I woke up this morning to a text from my ISP, “There is an outage in your area, we are working to resolve the issue”
I laugh, this is what I live for! Almost all of my services are self hosted, I’m barely going to notice the difference!
Wrong.
When the internet went out, the power also went out for a few seconds. Four small computers host all of my services. Of those, one shutdown, and three rebooted. Of the three that ugly rebooted some services came back online, some didn’t.
30 minutes later, ISP sends out the text that service is back online.
2 hours later I’m still finding down services on my network.
Moral of the story: A UPS has moved to the top of the shopping list! Any suggestions??
Though I wonder if even besides adding an uninterruptible power supply (UPS) (writing acronym out for anyone else who would’ve had to Google it), this might be a useful exercise recovering from outages in general. This is coming from someone who hasn’t actually done any self hosting of my own, but you saying you’re still finding down services reminds me of when I learned the benefit of testing system backups as part of making them.
I was lucky in that I didn’t have any data loss, but restoring from my backup took a lot more manual work than I’d anticipated, and it came at an awkward time. Since then, my restoring from backup process is way more streamlined.