There's no fallacy. That is, in fact, one way to cut a groove in a vinyl record that is a true, accurate square wave. Basically, you run the lathe like a circular plotter. It's sort of the limit case for the practice of half-speed mastering (which, done right, is a Very Good Thing to do).
Yup. Mathematics, now, can handle them just fine. But if you want to claim that a band-limited square wave still should be called a "square wave", I disagree. If the wave's fundamental is, say, 15 kHz, then what gets carved into the vinyl will be pretty close to a sine wave. And I think not a lot of folks would describe that as a "square wave".
Having compensated my share of "high bandwidth" 'scope probes (and built a few special-purpose probes along the way) I'd have to say, nope, those things with the horns on the edges (Gibbs' phenomenon) don't look very square to me.
Generally, that is true. There are situations, however, where the "functional bandwidth" of the DUT seems to predict one thing, while the "trouble-causing bandwidth" causes something rather different to happen. Ever see an audio amp that works (and measures) fine with the expected (and designed in) rolloff above the audible range, but then the gain pops back up again at a few megahertz (or more), which cause the thing to go all unstable under certain conditions?
One reason for testing with the fastest-edged waves you can get is to discover things like that.
Isaac