The clock drift is almost certainly happening because the device is relying too much on its internal oscillator rather than consistent network synchronization. Even if it maintains a solid WiFi connection, the internal Real-Time Clock chip on the ESP board itself isn't designed to keep perfect time when the power is fluctuating or when the chip is doing heavy processing elsewhere. Once it loses contact with the time server, it starts drifting, and if you don't force a re-sync, the error just piles up untill the next successful check. Try forcing the NTP component to update its time every hour instead of once a day, or if that fails, check if your specific ESP chip supports a battery backed RTC and use that method instead.
Regarding the drift itself, if the component is configured to check in, it should be doing so reliably. If you are using the standard NTP component, you might have the update interval set to something like ten hours, which just isn't precise enough for a clock. Increasing that update frequency is the easiest fix. Also, look through the logs during a time when the drift is apparent; if you see massive jumps in the reported time, that suggests a transient network issue is causing a bad time sync instead of a steady drift.
As for the jumpiness of the brightness sensor, that is extremely common. Sensors are noisy. They are measuring instantaneous light changes, and since you are in a kitchen, every little ripple of light from a passing car or a turning appliance will cause the raw reading to jump. Your automation is likely reacting to these rapid, momentary spikes instead of the overall light level.
The fix for the sensor jitter is to add some logic to filter the input. Instead of reacting to the raw sensor value, you should introduce an average filter in ESPHome, perhaps taking ten consecutive readings over a minute and only then passing that averaged value to Home Assistant. Alternatively, in your Home Assistant automation, you need to implement hysteresis or a deadband. For example, don't turn the light on until the sensor reading is above 500 lux, and don't turn it off until the reading drops below 400 lux, giving it a stable range to operate within.