The fact you have your children in the presence of a former drug user/dealler, whatever makes me question you judgement. Wh would you risk that? I mean you justify it by saying "never had anything around them" and "I am not anything illegal"?
. You married the guy so you are stuck. Maybe these charges were from a long time ago...we don't know because of the limited info you provided.
I also agree in that the father has found an opportunity. What is your relationship like with dad? And be hones. Do you 2 dislike each other? Do you withold the kids from him? If he asks for more time do you say no?
More information would be greatly appreciated. And they are not your babies...lose that attitude and you will be better off.
Look, you are more worried about losing control than doing the right thing