Backblaze logo

Data center outage

*UPDATE – 2/12/2014 – The post mortem on the data center outage is complete. The outage started when the Emergency Power Off (EPO) system was activated. The outage began on 02/05/2014 at 9:08 am PST and lasted 1 hour and 43 minutes until 10:51 am PST. At this point the Backblaze servers started recovering and were operational shortly thereafter. There was no loss of any data backed up to the Backblaze Storage Pods prior to the outage.

The outage began when the EPO system was triggered by a false report of a fire in an area of the data center. The EPO system is designed to activate when it detects a fire so that emergency workers and data center staff are not at risk of electrocution while responding to the emergency. There was no fire. The source of the failure occurred as workers were removing old wiring from under the data center floor and they inadvertently exposed and crossed two or more live wires from the current active system. This occurred in such a way as to indicate a fire to the EPO system and the system was automatically activated.

To minimize this from occurring in the future, the work of removing the old system wiring has been abandoned. In addition, a hardware upgrade is planned to include a 2nd trigger requirement to activate the EPO system.

*UPDATE – 2/5/2014 Noon (California time) – all account servers are now up, and backups can continue. This means you can now Install trials, Purchase the service, Update to the new version, Access the account, Browse your files, and Prepare Restores. A few servers are still being checked, so we’ll update this blog as any new information comes out. — Brian Wilson, CTO

original blog post below here

Ouch. It’s hard to say anything else than that.

On Monday we announced our new data center in Sacramento and today we launched version 2.5 of our service, with a number of highly-requested features. We sent an email announcing the new product release to half of our customers (with the other half scheduled to receive the email tomorrow) and offered them to upgrade today to get the new features.

Unfortunately, some combination of events brought power down in our new data center at 9am this morning. The timing couldn’t be worse.

We have people onsite at the data center and have been continuously in communication with the data center to ensure all steps were taken to bring power back on. At this point, 11:05 am, power has been turned back on.

Status as of 11:05am:
* Power has been restored to the data center
* Most services (new trials, purchases, backups, restores) are unavailable for most users
* Already backed up data is still backed up

Now starts an involved process where we will bring up all equipment, check that no systems have been damaged, and bring all services back online. This will take a while to be 100% complete – likely toward the end of the day.

However, within an hour we expect that users will be able to do almost everything:
* Install trials
* Purchase the service
* Update to the new version
* Access the account
* Restore files
(Backups may start as well, but will be rolling out slowly across all customers.)

We will continue to update this post with new information. Please bear with us.

Computers Need Electricity

blog-sad-cloudYesterday (October 24, 2013) six cabinets in the Backblaze datacenter lost all electricity at the same moment (10:58am).  No customer data was lost, but during the repair of the electrical circuits the pods containing some customer data were offline for the next few hours until 9:42pm when the final pod was brought back up and online.

These six cabinets are made up of 54 Backblaze Storage Pods containing a total of 2,430 hard drives and approximately 9.7 Petabytes of storage, which is approximately 1/8th of the Backblaze fleet of pods.

New data from all customers continued to be safely backed up onto the other 7/8th of the Backblaze pods the whole time due to automatic software fallbacks built into the Backblaze system.

However, at 3am (October 25, 2013) a side effect of having so many pods offline triggered some customer backups to pause temporarily.  We have made adjustments and are allowing more and more customer backups to continue.  Our goal is to have 100 percent of Backblaze customers fully working and backing up by 3pm today.

We will continue to update this blog post with information as it develops. Thank you for your patience while we resolve this issue.

*Update – 10/25/13 12:53PM PST* – Maintenance is almost complete. 95% of customers should now be fully online and able to upload data.

*Update 2 – 10/25/13 1:11PM PST* – 100% of customers are now back online and can upload data. We will continue to investigate the cause of this outage.

*Update 3* – All systems are normal, and pods are online. The cause of the outage was 12 independent circuit breakers being tripped simultaneously, which affected a small percentage of our users. No customer backup data was lost. We are working with the data center to audit their power infrastructure.

Backblaze Halloween Contest


Want to win an iPad Mini or a Nexus 7?
We’re having a scary video contest! Want the rules? Read below, and don’t forget to check out the example video at the bottom! Oh, and hurry! The contest ends October 29th!

How to Enter the Contest?

1. Prepare a short, 30 seconds or less, video recreating your favorite horror movie scenes using your computer or hard drive as the victim.
2. Insert the following image at the end of the video (right-click and save as):


3. Upload your video online, to YouTube for example.
4. Post a link to your video on the Backblaze Facebook wall or on Twitter with the hashtag #Backblaze so we can enter it in to the contest. Or, link to it in the comments below!
5. Share your video with friends.

General Questions:

1.  How many people can be in the video?
Answer:  However many you need in order to recreate the scene!

2. Can I make it longer than 30 seconds?
Answer: Maybe 32 seconds, but that’s it.  If you want to make a longer “director’s cut” we’d love to see it, but the contest video should be close to 30 seconds. Please keep it short and scary.

3. Can I record it on an iPhone, Android, iPad, Camera, etc?
Answer:  You can use whatever device you wish to record your video.

4. Can I submit multiple videos?
Answer: If you have multiple favorite scenes, make a vignette! But only please submit one video.


1. You must use a valid Facebook account and comply with all Facebook rules for age, content, copyrights, etc to post a link to your video on the Backblaze Wall.
2. We reserve the right to remove and not consider as a valid entry any videos which we deem inappropriate.  We reserve the exclusive right to determine what is inappropriate.
3. Backblaze reserves the right to use your video for other purposes.
4. The contest will end on October 29, 2013 at 11:59:59 PM Pacific Standard Time.  The winner will be announced on October 31, 2013.  The winner will be determined by Backblaze.
5. One 16GB iPad Mini or one 16GB Nexus7 will be awarded to the winner.  The winner will determine which they would like (either an iPad Mini or Nexus7). The prize will be mailed to the winner in a timely manner once it becomes generally available to the public.
6. Please keep the content of the post PG rated – no cursing or extreme gore/violence.
7. By submitting a video you agree to all of these rules.

Want an example?

That’s it! Have fun!

What Would You Like To Read On The Backblaze Blog?

The Backblaze Blog has been around for quite a while and we’d like to know what you think of it, and what you’d change about it, if anything. Thank you for your feedback, and we hope we can keep bringing you quality, entertaining, and informative content!

*Update* Thanks for all the feedback! We’ll be posting our results soon!

*Update 2* The results are below! More datacenter and Backblaze Storage Pod related information won by a wide margin…we’ll see what we can do!

Unboxing The Japanese Retail Box!

Last year Backblaze announced a partnership with Sourcenext to help spread online backup throughout Japan.  After all, backing up your computer is a good idea, no matter where you live!  This year Sourcenext is releasing a retail box for us!  Let’s take a look inside shall we?

Incredible Invoices!

Backblaze invoice

Backblaze has a great support team and one of the most frequent requests they receive is, “could you please provide an invoice?” From the beginning, we’ve been handcrafting every invoice that’s been sent out, and while we love making our customers happy, as Backblaze has grown it’s become increasingly difficult to keep up with all the requests. Over the last few years, we’ve been hard at work making the best online backup solution possible and that meant that our engineers were working on iterating and expanding the functionality of our backup client. Unfortunately this meant that some of the non-mission critical improvements we’ve wanted to make had to take a backseat. Well, the day of automated invoicing is finally upon us! Thanks to Monika, the latest addition to our engineering team. After joining Backblaze she was tasked with something of the utmost importance, creating an invoicing system so that you can get your own invoices!

That’s right! Starting today you will be able to go to the “Transactions” section of your Backblaze account and select a year or billing period for which you would like to receive an invoice. Once created, you will be able to print or download the invoice which will have all of your Backblaze transactions including your dates of purchase, the number of licenses purchased, and the total cost. Even though this may sound like a minor improvement it is something we are very excited about!

10% now back up daily, 90% to go!

Backup Awareness 2012
Five years worth of data
10% percent may not seem like a lot, but in the world of online backup it’s a milestone. For the last 5 years Backblaze has had the folks at Harris Interactive ask “How often do you backup all the data on your computer?” To our dismay, daily backup of the photos, videos, music, data and more on people’s computers has languished at 6-8%. But our 2012 survey said “10%” now do daily backups, a 47% increase over 2011 – Woo Hoo. Even better, the survey also revealed that weekly backups increased 38% over 2011 and monthly backups increased 29% over 2011. In other words, people are backing up their data more often and that is GREAT news.

There’s still work to do
But all is not rosy in backup land as 29% of computer users have never backed up all the data on their computer (cue scary music here). While 29% is a decrease of 15% over 2011, it is still too high. So we wanted to know why would someone not do daily backups? Amazingly, 59% thought that daily data backups were either too complicated or too expensive, or they just plain forget to do it. Hmmm, so is there a way to backup a computer that is easy to use, affordable, automatic and continuous?

Our challenge for 2013
Here’s our audacious goal for 2013. Let’s get the percentage of people who backup daily (10% in 2012) to be higher than the percentage of people who don’t backup at all (29% in 2012). Impossible you say. Let’s try – “no try, do”. How can you help? Give the gift of Backblaze, to someone you know from the 90% who don’t back up daily. Of course we’ll keep plugging away here and between us we can reach the 2013 goal. Thanks for your help.

“How often do you backup all the data on your computer?”

Other interesting statistics from the 2012 survey:
• 43% of computer owners backup less than once a year (or never.)
• Only 3% backup more frequently than once-per-day
     (Backblaze recommends backing up continuously!)

How backup corresponds with education; percent of people who have ever backed up their data:
• 62% High-school or less
• 74% Some college
• 79% College or more

How backup corresponds with income; percent of people who have ever backed up their data:
• 71% Less than $35k
• 68% $35k – $49k
• 69% $50k – $74k
• 74% More than $75k

How backup corresponds to employment status; percent of people who have ever backed up their data:
• 75% Full-time employed or self-employed
• 70% Part-time employed
• 68% Unemployed
• 71% Student
• 65% Retired

Survey Methodology
The surveys were conducted online within the United States by Harris Interactive on behalf of Backblaze as follows: May 31–June 4, 2012 among 2,209 respondents, June 28–30, 2011 among 2,257 respondents, June 3–7, 2010 among 2,071 respondents, May 13–14, 2009 among 2,185 respondents, and May 27–29, 2008 among 2,761 respondents. In all surveys, respondents consisted of U.S. adult computer users (aged 18+), weighted to the U.S. adult population of computer users. No estimates of theoretical sampling error can be calculated.

Quick maintenance at 2pm PDT

Our online backup service will have a 30 minute maintenance at 2pm PDT today (Tuesday 3/13/12.) This is a slight finish up after last week’s maintenance.

During this maintenance, some parts of our service will be affected:
* Restores and access to My Account will be unavailable.
* Creating new accounts will be unavailable. We will provide a form to enter your email address to be notified when maintenance is complete.
* Backups that are in process will continue unless interrupted by your computer shutting down or your Internet connection losing connectivity.
* Buying will be unavailable on our website. However, it will be possible to purchase annual licenses on Amazon.

After the maintenance completes, there will be a ramp up period during which backups will start that may last up to an hour for some users. Thank you for your patience!

Update: Maintenance is done! 22 minutes total – just slightly ahead of schedule. As mentioned above, backups will take up to one hour for some users to restart.