Yesterday (October 24, 2013) six cabinets in the Backblaze datacenter lost all electricity at the same moment (10:58am). No customer data was lost, but during the repair of the electrical circuits the pods containing some customer data were offline for the next few hours until 9:42pm when the final pod was brought back up and online.
These six cabinets are made up of 54 Backblaze Storage Pods containing a total of 2,430 hard drives and approximately 9.7 Petabytes of storage, which is approximately 1/8th of the Backblaze fleet of pods.
New data from all customers continued to be safely backed up onto the other 7/8th of the Backblaze pods the whole time due to automatic software fallbacks built into the Backblaze system.
However, at 3am (October 25, 2013) a side effect of having so many pods offline triggered some customer backups to pause temporarily. We have made adjustments and are allowing more and more customer backups to continue. Our goal is to have 100 percent of Backblaze customers fully working and backing up by 3pm today.
We will continue to update this blog post with information as it develops. Thank you for your patience while we resolve this issue.
*Update – 10/25/13 12:53PM PST* – Maintenance is almost complete. 95% of customers should now be fully online and able to upload data.
*Update 2 – 10/25/13 1:11PM PST* – 100% of customers are now back online and can upload data. We will continue to investigate the cause of this outage.
*Update 3* – All systems are normal, and pods are online. The cause of the outage was 12 independent circuit breakers being tripped simultaneously, which affected a small percentage of our users. No customer backup data was lost. We are working with the data center to audit their power infrastructure.