Disaster recovery, how to automate as far as possible?

I had to practice what I preached yesterday. I did an upgrade of all 14 of my active Raspberry Pi's, but had problems with 4 of the Zero W's, which wouldn't come back up immediately. I got two of them back up after a few reboots, but attaching the other two to a monitor revealed networking wouldn't come up. After trying lots of stuff I decided to rewrite the card with the previous night's image. This fixed one of the Pi's but the other fails to run avahi, wpa-suplicant and dhcpcd, I can get only get networking up by manually running dhcpcd.

I'd say this could be a hardware failure as Wifi is working, it doesn't report any Bluetooth devices. Although one of the other ZeroW's which didn't come up immediately is also lacking Bluetooth, and the Bluetooth driver was one of the things that was updated.

But anyhow the disaster recovery element worked, being able to successfully roll back a suspect update.

---druck