But as you are aware, this is not a perfect world. In particular, you situation is not the perfect situation.
Sooooo, ignoring the financial issues for the moment, this dad probably doesn't win any prizes for world's best dad. OK, fine. But why are the kids so terrified, to use your terminology?
Is he physically abusive to them? Or on the other end of the spectrum, are you signalling in any way (verbally or body language) that the kids should be terrified, even though dad is completely non-threatening? Somewhere in that spectrum lies the truth, which we on this board, will never have any way of knowing.
Here's what I suggest, if you haven't done it already. First try to get dad to be more regular. Second, tell your kids there is nothing to be afraid of (unless there really is something, in which case you should be contacting the police or CPS or something). But your judge wouldn't have given the father any visitation if there was a real danger, most likely (understanding that not all judges are peferct either).
So, let dad do his thing with the kids, even though you may find it disruptive to your pattern. Undoubtedly, if the kids don't see their dad that often, and he isn't abusive to them, then they will remember these visits with him when they are older.
If they don't see him at all, what memories will they have?
I don't want to sound like your grandmother, but seems to me, the older we get, the fewer things we have left to cherish. And among the last things anyone ever has left is their memories, according to my aging father.
I say, give the kids a chance to have some memories, whether pleasant or not so pleasant, with their dad. It's all part of growing up.
Do those suggestions help your situation in any way? Hope so.
Good Luck.