The jumpiness you're seeing is usally related to two things: either the sampling rate is too low for how fast your loads are cycling, or there's electrical noise on the ADC line that the ESP32 is picking up. If you're just running the default loop timing, the sampling rate might not be fast enough to capture the quick spikes and dips, making the average look unstable. What I found when I was setting up monitoring for a small network closet circuit was that simply increasing the frequency at which the ESP32 reads the analog pin helped stabilize the output significantly. If you're using a custom loop or function to read the sensor, try increasing the frequency to something like every 50 milliseconds, or even faster if the processor can handle it. More importantly, you should be using a moving average filter on the readings within your code before feeding them to ESPHome. Instead of reporting the raw instantaneous value, average the last ten or twenty readings. This is the most effective way to smooth out the high-frequency electrical noise inherent in AC sensing.
Regarding calibration, yes, you absolutely need a proper calibration factor. The raw ADC values from the CT clamps are proportional to the current, but that proportionality constant is unique to your sensor unit and the specific wiring setup. You can't just assume the raw data maps perfectly to amperes. You need a known good test. Run the circuit with a measurable, steady load, say 10 amps, and record the average raw ADC value. Then calculate the calibration factor needed to translate that specific raw reading into 10 amps. This factor should be applied every time you read the current.
Finally, check your hardware connections. If you're running the ESP32 close to high-current paths or if your power supply ground isn't well shielded, you are introducing noise. Try to keep the analog input wiring as short and shielded as possable. Sometimes, moving the ESP32 unit away from the mains power wires completly resolves erratic behavior without needing massive code changes.