Been down this road with a weather station build, so here's what I learned. Yes, the concept works, but the weak spot isn't the panel size, it's the converter. A cheap buck module ussually needs a few volts above its output to regulate properly, and a bare panel drifting in and out of full sun can dip below that fast. When the converter drops out, the microcontroller browns out and reboots in a loop, which looks exactly like a broken board. Add a big capacitor across the input side, think supercapacitor territory, and it rides thru those dips like a tiny battery without the charging circuit headaches.
Second thing people skip: put a voltage divider on a spare analog pin and watch the panel voltage in your code. If it sags below your threshold, delay the risky stuff like writing to flash, and resume when the sun's back. Costs almost nothing and saves your filesystem. Last tip, oversize the panel way more than feels reasonable. My rule ended up being roughly triple what the average math said I needed, since morning and evening output is a fraction of the rating even on a clear day.