I've heard the same thing, but no explanation. I've used 4s and can't see what people are complaining about.
Here's what little I do know:
Using #4's to create a crossover in T-Trak keeps the proper spacing between tracks
Using #6's increases the spacing requiring
1) fudging around to get the spacing back and requires at least a double module
2) surgery on the turnout to shorten the curved section to keep the same spacing
#6 if I remember correctly are power routing -- kills the power to the non-switched track.
#4 has a jumper underneath to set the turnout to power routing or not.