Looking at the turnout afterward, I would say it's wear on the sides of the rails, allowing the wheels to wander off course, so falling into the flangeway is the probable result. The tracks get used for all sorts of different rolling stock scales. If it will run on code 100 HO, it does. It would be interesting to compare Code 100's gaps between stock rails and frogs etc, with the smaller codes. I suspect they would be narrower, in anticipation of narrower wheel treads being used. If wheels are correctly gauged and check/guard/frog rails are correctly placed to prevent excess side movement of the wheels, then the depth of the flangeway, if deep, shouldn't matter. Not deep enough, yes, if the flanges can make contact. If the wheels can't leave the inside edge of the rail it's traversing, then it shouldn't fall into the flangeway.
The main thing I think that caused my little excursions would be wear on the point of the frog, allowing the gap between it and the stock rail the wheels were aiming at to get to be too far away, combined with too wide a gap between the opposing guard and stock rails controlling sideways movement of the axle. Too an extent, it's the fault of using too wide of wheel treads on most of the production rolling stock. If we were to use the narrower tread profiles, we would be forced to make sure our wheel guaging was accurate. Then we could also make sure that the gaps between rails and guard rails was narrower, thereby eliminating side to side oscillation of the cars. We could still do that anyway, even though the treads may be wide, doesn't matter if they overhang the outside of the rails, does it?