I think your son's father is manipulating you because he is hurt. As a single father who does not have primary custody of my son, I can relate to why he may say those things to you. However from what I have been told by 2 different lawyers, even if he takes you back to court, your child will be interviewed and the court will know if he is being manipulated or honestly wanting his father. I would encourage you do be supportative of their relationship and do not be afraid to do guy things with your son. That will show him where he is better off.