Living with his mother is also legal and the fact that he has no job could actually work against you, as stupid as that seems. Unemployed mothers nearly always get custody, unless deemed unfit. Unemployed fathers tend to be frowned upon, but the reality is that he and/or his mother are available to watch the child 24/7. What do you do with the child when you are at work?
File for some form of court-ordered custody immediately. Get a lawyer. Yes, mediation is a good step, but do something. Keep a journal of all that transpires. The journal should include dates and anything related to custody issues, but no opinion or emotion, just facts. If you do get a mediation you may be able to work out a way to share the child in a way that will benefit the child and you both agree on it.