Maybe the baby goes over to dad's house for a few hours every Saturday. This time would be "dad's" time. Mom would not need to be present and dad could decide who is or isn't present. So if he wanted to give the time to his mother, that would be his decision.
As the baby get older (and the parents more responsible) they could work into a more equal time share (and split of the responsibilities).
If they can't work something out informally, then it might be better to file for a custody plan. That way everyone knows the ground rules. Usually an attorney will give a free consult or at least a consult at a reduced fee. Might be worth looking into.