If I were you, I would work on teaching your son coping mechanisms. There are many great books out there, but the one I find myself going back to time and time again is "Joint Custody With a Jerk". It is very helpful when your x is, well a jerk.
If your child tells you they don't want to visit you at the prescribed time.. reassure them and tell them that the details are grownup stuff but that you are very excited to see them and will have a great time together.
Buying into the blame game only reinforces in children that parents can be bought with emotion (ie: hurting one parent gains you the affection of the other) this is a dangerous road and one best to avoid.
JMHO of course.