are you and EX able to have civil communication at all? if so, maybe you could give him a call directly and just kindly tell him it is hard to carry on a conversation that way. if not, i would definitely either email or write a letter requesting time to visit with your son on the phone without being on speaker as it impairs your ability to carry on a conversation because you cannot understand what is being said. let's face it--keeping a 4 year old on the phone very long is a challenge in and of itself, let alone having to ask him to repeat things over and over so you can get part of what he's telling you. that gets frustrating for adults, let alone children. i'm sure the visits are shorter as a result of this. *sigh* (by doing this email or mail you have record of your request and also of the exact things that were said) --for future use if needed.
good luck--i know all too well about the aggravation that comes with constant distractions during phone visits. maybe others will have better suggestions here too...