As some of you noticed, Backblaze experienced an outage. I want to provide some detail on what happened and the current status.

The Cause
Backblaze stores your backed up data in a top-tier data center facility. Last night at 7:35 p.m., a security guard entered the facility. The door slammed, causing the protective covering to open on an “Emergency Power Off” switch and setting off alarms. While this had no impact, in a moment of confusion, the guard hoping to turn the alarms off, pressed the Big Red Button, and shut off all power to that zone. At 7:36 p.m., the duty engineer escalated the situation and a resolution plan was designed. By 8:03 the power was fully restored.

Backblaze Response
As soon as the power went out, Backblaze’s monitoring systems alerted us to the issue and we mobilized the company. Most of us went immediately to the data center, while others double-teamed in support to help instantly address any questions. We then started the phased procedure of bringing the service up again: Static web content (home page, help pages), dynamic web content (account pages, restore selection, billing), and finally all of the actual cloud storage.

We could have brought everything up very quickly, but we believe it’s critical to carefully check every system first. With over 5,000 spinning hard drives, this process takes a little while. Much of the team worked diligently through the night to bring the service back as quickly as possible.

Status
The static web pages were live within minutes of the power coming back online. We ran thorough tests throughout the system and fully brought the dynamic pages up this morning. This means you can browse the entire site, sign-in to your account, browse the files you have backed up, and even request (but not yet receive) a restore.

We expect to finish checking enough of the cloud storage systems later this afternoon to turn on the ability for backups to resume. At that point, most requests to restore data will also be fulfilled. However, some restores will be delayed a bit longer if they contain data on systems that we have not finished testing. As soon as we’re done, all restores will complete.

At this point, everything is progressing smoothly and we expect to have every piece of the service restored to complete operating procedure sometime this evening. While it is tempting to lock the Emergency Power Off switches, that would obviously defeat their purpose. However, we are looking at ways to speed the process in the future of performing all necessary tests in order to recover more quickly from any type of unplanned shutdown. Thank you for being patient with us as we work through this.

“Jason”, not his real name, is an electrical engineer contracting for the military. I wanted to share the comments and photos he sent us about his experience with Backblaze in Bagdad:

My being here is a once in a lifetime opportunity, and I
don’t want something like a hard drive failure to sour it. My computer
is my livelihood, and not having to worry about my irreplaceable data
allows me to focus on the mission in support of the Coalition
soldiers.

That being said, bandwidth is at a premium, and Backblaze has been
great to let me know EXACTLY what’s backed up, what needs to back up,
and for being able to adjust how much bandwidth it uses has also been
critical.

My family will be very happy to see all the photos that I
have taken over my travels and stay while in Baghdad, and I’m glad
knowing that even if my computer is damaged or destroyed, I will be
able to get that data back.

Here are a few of Jason’s photos of Baghdad:

Nearly 1 of 2 people (46%) lose data every year according to a study of Backblaze customers. Shocking? It surprised me at first glance, but then I dug in deeper.

Drive Failures
According to a Google study of hard drive failures, disk drives over a year old have about a 1 in 10 chance of failure each year. At this rate 1 of every 2 drives will fail every 5 years. A person that has a hard drive in their computer and an external drive for that period is nearly guaranteed to have one die.

Computer Theft and Computer Loss
15% of households annually experience burglary or theft according to the Bureau of Justice. While statistics are not available for what was stolen, when a home is burglarized, a computer is a likely target. According to the Ponemon Institute, 637,000 laptops are lost at airports across the country every year. How many more in taxi cabs, coffee shops, and at vacation destinations?

Viruses and Software Corruption
Various surveys across the web have shown that viruses cause 4% - 7% of all data loss. Add software corruption to the mix, boot sector issues, registry issues, etc. and this starts becoming significant.

Flooding, Fire, Earthquakes, and Other Disasters
According to FEMA, about 1 in 10 households that have flood insurance suffer a loss each year due to flood damage. Half a million buildings catch fire every year based on USFA statistics. Nearly 200 earthquakes with a magnitude of 6.0 or greater occur worldwide annually according to the USGS. Computers are sensitive devices and don’t like to be wet, hot, or shaken.

Human Error
“Oops” is the #2 most common cause of data loss (after hardware failure) according to data recovery specialist Ontrack. Is the delete key too big on the keyboard? Blame it on that…but all of us have done it and wished there there were an undo key that was just as big.

Losing data doesn’t always mean a hard drive crash. Sometimes it just means we deleted a folder (with our kids photos) or our dog knocked over an external drive (with our music library.) Whatever the cause, based on the actual needs of our customers, the various causes of data loss compile to require 1 of every 2 customers to restore data each year.