The fact that he has dirt on him just helps his case.
What exactly are his allegations. And define taking away, Does he have minimal contact now or joint?
People need to remember that when you go to court you asking for the moon, so that you might be able to negotiate or more given some of what you really want.