Is thin household-style wire okay for linking two 12v batteries in series?

Swap it now. That jumper carries the full load current, not part of it. In a series string every bit of current that leaves the batteries passes through every connection in the loop, so the link between them needs to handle the same amps as your main output leads. Doorbell wire is maybe 20 gauge at best, and a couple hundred watts at 12 volts is a lot of amps. It stayed cool beacuse your test load was tiny and short. Run the inverter hard for twenty minutes and that skinny wire becomes a fuse, except it fails wherever the insulation is thinnest, possibly inside a wall or under something flammable. One thing to double check while you're at it. You said you jumped negative to positive between the two batteries, which puts them in series for 24 volts. If that inverter is a standard little 12V unit, it won't live long on 24. Some people acutally take the output across just one battery and use the second as a spare, which works but drains them unevenly. If your inverter is genuinely a 24V model, you're fine, but confirm the label before the next outage. On sizing, match the jumper to the thickest cable you're using on the terminals, or at least the same gauge as the leads going to the inverter. Short runs like yours are cheap, so there's no reason to skimp. I'd also stick a blade fuse or breaker on the positive lead coming off the battery bank, sized a bit above what the inverter can actually draw. Cheap insurance, and it's the piece people always leave off the first build. Do the swap before you test it under real load. Low load tests tell you almost nothing about how wiring behaves when you actually pull serious amps.

One thing I'd add about the batteries themselves is that pairing two old sealed units in series only works well if they're the same age and same state of health, because the weaker one limits the whole string and can get dragged below its comfort zone while the healthy one still looks fine. I've had a gate backup setup where one battery sagged and the whole output dropped off long before the pair was actually empty. Also, how are you planning to top them up between outages? If there's no float charger on the pair they'll sit partially drained after every use and sulfation will finish them off within a season or two.

Been through this exact upgrade on an old trolling motor bank. What actually fixed my spaghetti mess of jumpers was a cheap ratcheting crimper and a pack of ring terminals. Soldered spades always worked loose on mine after a season of vibration, the crimped rings have never budged. Cut the cable to length, crimp both ends, done in ten minutes and it looks like it came from a shop. Also check your spade connectors the same way. If they're the push-on style, give them a firm tug test. If one slides off under light hand pressure it'll definately let go the day you actually need the thing.

I totally get why you're worried about that thin blue wire, but even if you swap it out for a heavy gauge one, you gotta make sure the connections themselves aren't the weak point. I've had folks install big setups where the wiring was thick, but the terminal connection was just a little loose, and they still developed hotspots. Make sure those battery terminals are cleaned down and the link is completley tight. You don're looking for rock solid contact, not just a wire that fits.

Quick update. I decided to switch that thin blue jumper out for a heavier gauge wire like you suggested. It makes total sense that the entire load current passes through that link. It's running smoothly now and I feel much better about the whole setup.