Why does streaming from my media server stutter when SD card playback is fine?

Buffer size is my first bet, but there's a second suspect worth checking before you touch code. Full signal bars don't mean much for streaming, because the ESP32's default WiFi power save mode puts the radio to sleep between packets, and that alone causes exactly the kind of random every-few-seconds hiccup you're describing. Try disabling power saving and see if the dropouts vanish. On ESP-IDF it's one call with WIFI_PS_NONE, on Arduino there's an equivalent setting. If that alone fixes it, you're done for the evening.

If it still stutters, then yes, it's almost certainly your buffer. SD card reads are fast and local, so even a small buffer hides any jitter. Streaming over WiFi is bursty, packets arrive in clumps with gaps, and if your audio task drains the buffer faster than the network task refills it, you get underruns at unpredictable intervals. Bump your ring buffer way up, think tens of kilobytes rather than a few, and run the network fetch in its own task with a decent stack so it never blocks on the decoder. MP3 decoding on the S3 is cheap, the dual core helps alot here.

On the display question: a redraw every second or two on a small TFT over SPI only steals a few milliseconds, so it shouldn't cause audio gaps on its own as long as the display runs on the same core as something that can wait. What does hurt is doing the redraw and the network read and the decode all in one loop, because one slow SPI flush delays your fill task. If you later notice dropouts getting worse with the screen busy, that's the sign to move the display updates onto the other core or drop the redraw rate.

Quick isolation trick: flash a barebones internet radio sketch with nothing else attached. If that plays clean for an hour, your network path is fine and the problem is task contention in your firmware, which points you straight at restructuring rather than rewiring anything.

Following this one because I ran into somethig similar with a little internet radio box I made, except mine streamed from a free online station instead of my own server. Turned out the stream itself was coming thru at a variable rate, so no amount of fiddling with the ESP32 side helped. Might be worth testing your NAS server from a laptop or phone first, just to check the stream is actually smooth before it even reaches the player. Also, is your NAS on a wired connection to the router or does it use wifi too? Mine was wireless and moving it onto a cable made a bigger difference than I expected.

Been down this road myself, back when we were doing this with an ESP8266 and praying the thing kept up. One thing that bit me: the server side was throttling or re-encoding on the fly, and the ESP just choked whenever the stream stalled. If your NAS is transcoding before sending, that alone will do it. Set the server to serve the raw file and see what happens.

Also, old trick from my radio days: log your buffer level for a minute while it plays. If you see it climbing and dropping in waves instead of holding steady, the problem is upstream of the ESP, not your code.

I'd push back on the claim that a screen redraw only costs a few milliseconds. In my experience that's optimistic. I had a similar player with a small TFT and the full frame push over SPI took long enough that my audio buffer ran dry whenever the display updated, and it looked exactly like network stutter. I chased WiFi settings for days before I paused the redraws and the gaps vanished instantly. So before bumping any buffer, try commenting out the display code entirely for one test run. If the dropouts stop, the screen is your problem and you can rate limit it or split the work across cores, no rewiring needed.

Before changing anything, I'd log the buffer fill level to serial every second with a timestamp, because in my own streaming project the pattern told me everythign once I coud actually see it. If the level drains in a smooth ramp and refills in jumps, the network side is bursty but coping; if it sits healthy and then plunges right after a redraw, the display is starving the fill loop and no WiFi setting will save you. Mine turned out to be the second case, and simply spacing the redraws further apart made the dropouts disappear without touching the audio code at all, which saved me a weekend of rewiring for nothing.