Tell us what your spouse argued in his affidavits regarding your parenting skills. Quote him. Also, what did athe evaluators say?
This sounds like half a story, if that.
I am sure that it is very difficult being away from your children. I can also understand being angry when finding out about the affair. You do have my sympathy. Nonetheless, I remain a skeptic about the fullness of the information provided.