The judge acknowledges that going to a friends house is normal part of being a kid. I think that going to grandmas is as well, but if dad is working and sending them to grandma he is using them AS a sitter and not family. it just convienent that he doesn't have to pay them.
Im going to assume that time with his girlfriend was while he was working or maybe out with the guys, otherwise you would have hoped he would have been there too