However the bio-father parents, you can't deal with his PARENTING, and that is where people will tell you that there is nothing you can do about that. Your kid is too little to "have a voice" too. So, that means that if you are serious and really feel that your son is in DANGER, serious "gonna die" danger, then you need to stop the visitation through the courts. Hire a lawyer to see what your options are.
Otherwise, in and out, irresponsibility, no child support - those are forever going to be issues and YOU and your son need to learn to find a way to deal with it. The problems aren't going away. Just getting rid of the Dad for once and all isn't the answer. Your son will learn over time and appreciate you "allowing" the relationship to develop and yes, one day your son will have a choice.
