For example, you mentioned neglect as a possible reason for custody change. Let's look at two possibilities: One, a child says that they want to live with Mom because Dad is never around and they feel neglected. In the second scenario a child says that they want to live with Dad because he is at work most of the time and they feel smothered by their mother being around so much. In the first scenario the child will be likely to get their wish, but not in the second.
Why? Because the parenting is more important than the preference. Or at least it SHOULD be. Parenting is not a popularity contest.