At 9:48 PM Central US time we received a notification that the database server has not sent events for the past 10 minutes and Kai confirmed the site was unreachable.
At 10:24 I logged in to Linode and saw we had an "Emergency Maintenance" notification for the database server - the Physical host that it runs on experienced a hardware failure. You cant choose when hardware fails, and today was the day something on it failed. Backups were verified as having happened today.
This happens on occasion as all hardware will eventually fail. This was already late for me and was Christmas eve so since there was nothing to do but wait for Linode to recover the system. I posted a notice in Discord's announcements channel at 10:32 pm and went to bed.
Linode's recovery took quite a while as well and we did not get the "all clear" till 7:15am the following morning. Linode's email said it should be "in the last state" it was when the incident happens (on) but it was not. After opening Santa's presents I clicked the "Power On' button at 7:18 am and the site was recovered at 7:19am.
I am posting this before heading out to Grandma's house for Christmas and I hope you are all enjoying your holidays. Any comments or questions will be addressed when time permits.
Total outage: 12/24 9:48pm - 12/25 7:19am central us.
At 10:24 I logged in to Linode and saw we had an "Emergency Maintenance" notification for the database server - the Physical host that it runs on experienced a hardware failure. You cant choose when hardware fails, and today was the day something on it failed. Backups were verified as having happened today.
This happens on occasion as all hardware will eventually fail. This was already late for me and was Christmas eve so since there was nothing to do but wait for Linode to recover the system. I posted a notice in Discord's announcements channel at 10:32 pm and went to bed.
Linode's recovery took quite a while as well and we did not get the "all clear" till 7:15am the following morning. Linode's email said it should be "in the last state" it was when the incident happens (on) but it was not. After opening Santa's presents I clicked the "Power On' button at 7:18 am and the site was recovered at 7:19am.
I am posting this before heading out to Grandma's house for Christmas and I hope you are all enjoying your holidays. Any comments or questions will be addressed when time permits.
Total outage: 12/24 9:48pm - 12/25 7:19am central us.
