Being a bad partner does not make for being a bad parent. He did try to contact you so he could see the child you made together. You turned him down. How many times does he need to contact you? I imagine that he if kept calling than you would say that he is harrassing you.
Sorry that it didn't work out the way you expected. But he IS the child's Father and he does have rights. Why not contact the court to get a paternity test to declare him the father and than get a CO for visitation. If he does not want visitation than you get that in a CO. Do this legally otherwise there will be more problems down the road.