Backblaze logo

Scheduled Maintenance (7/22/2014)


blog_maintenance3
Tomorrow (7/22/2014) from 7-5PM California Time, Backblaze will be undergoing scheduled maintenance. During this time, we will be updating a cluster, resulting in more stability and speed for some of our earliest customers.

During this maintenance, the following parts of our service will be affected for some of our users:

  • Access to the My Account section of the website will be unavailable, this includes access to restores and the ability to purchase additional licenses for your existing account
  • Backups will not be running, and you will see the following offline message in the Backblaze control panel:

Offline
After the maintenance completes, those users will immediately be able to access their account, download restores, and add licenses. All backups will restart from where they left off, however, there may be some ramp-up time before clients start uploading backup data. Not all users will be affected.

We will be updating this page throughout the day, and when the system is fully back online.

Update 7:20AM -> Maintenance has started and should last until 3PM.

Update 9:22AM -> As a result of maintenance, some users will be unable to log in to our help system. If you need to open a new support ticket, please write an email to: helpme@backblaze.com and our support team will be able to respond to you, or go to http://www.backblaze.com/help.html and you’ll be able to chat with our support team.

Update 3:32PM -> The majority of work has been completed and the final maintenance stages have begun. We estimate another hour or so of downtime.

Update 5:17PM -> Systems are now coming back online and users should start being able to fully use the site soon. We’ll update once we’re fully back online.

Update 9:26PM -> As of a few hours ago, all systems are back online, and backups are resuming. Maintenance is complete.

Author = Yev / @YevP



Wanted: Sr. Systems Admin, Web Services


blog-job-sys-adminResponsibilities

  • Own Apache, Tomcat, WordPress and Java services used to host static and dynamic web sites
  • Maintain and expand platforms used to host web services including Dell, Apple, and Supermicro servers plus Dell & EMC storage hardware
  • Oversee daily and weekly backups of critical metadata
    Research new applications as needed and recommend specific packages and configurations (DNS, NTP, DHCP, etc)
  • Help maintain multi-petabyte storage farm including low level RAID rebuilds, file system recovery and hardware repairs
  • Help deploy, configure and maintain 100s of Linux systems and services using Ansible
  • Perform routine drive replacements as needed
  • Help create, maintain and improve operational bash shell scripts and manual procedures
  • Help hire and train IT staff as needed, including those in remote (global) locations
  • Participate in the pager rotation, respond to alerts as needed, review system logs and monitor security forums
Requirements
  • Expert knowledge of Linux system administration, Debian experience preferred
  • Experience running Apache, Tomcat, WordPress and Java services
  • 8+ years of experience
  • Bash scripting skills desired
  • Ability to lift/move 50-75 lbs and work down near the floor as needed
  • Position based in the San Francisco Bay Area, California requiring 3+ days/week in San Mateo
All Backblaze Positions Require
  • Good attitude and willingness to do whatever it takes to get the job done
  • Strong desire to work for a small fast paced company
  • Desire to learn and adapt to rapidly changing technologies and work environment
  • Rigorous adherence to best practices
  • Relentless attention to detail
  • Excellent interpersonal skills and good oral/written communication
  • Excellent troubleshooting and problem solving skills

Want to join our team? Follow these three steps:

  • Send an Email to jobscontact@backblaze.com with the position title listed above in the subject line
  • Include your resume
  • Include your answers to 2 of the following 3 questions
    • What about working at Backblaze excites you the most?
    • Provide 3 adjectives that best describe your personal workspace.
    • How would you manage boot images and system configurations on 1,000+ servers (i.e. Backblaze Storage Pods)?

We’ll be interviewing candidates over the next couple of weeks. Join us and help us continue to build a great online backup company.

Backblaze is an Equal Opportunity Employer.

Comments Closed


Data center outage


*UPDATE — 2/12/2014 — The post mortem on the data center outage is complete. The outage started when the Emergency Power Off (EPO) system was activated. The outage began on 02/05/2014 at 9:08 am PST and lasted 1 hour and 43 minutes until 10:51 am PST. At this point the Backblaze servers started recovering and were operational shortly thereafter. There was no loss of any data backed up to the Backblaze Storage Pods prior to the outage.

The outage began when the EPO system was triggered by a false report of a fire in an area of the data center. The EPO system is designed to activate when it detects a fire so that emergency workers and data center staff are not at risk of electrocution while responding to the emergency. There was no fire. The source of the failure occurred as workers were removing old wiring from under the data center floor and they inadvertently exposed and crossed two or more live wires from the current active system. This occurred in such a way as to indicate a fire to the EPO system and the system was automatically activated.

To minimize this from occurring in the future, the work of removing the old system wiring has been abandoned. In addition, a hardware upgrade is planned to include a 2nd trigger requirement to activate the EPO system.

========================
*UPDATE – 2/5/2014 Noon (California time) – all account servers are now up, and backups can continue. This means you can now Install trials, Purchase the service, Update to the new version, Access the account, Browse your files, and Prepare Restores. A few servers are still being checked, so we’ll update this blog as any new information comes out. — Brian Wilson, CTO

=========================
original blog post below here
=========================

Ouch. It’s hard to say anything else than that.

On Monday we announced our new data center in Sacramento and today we launched version 2.5 of our service, with a number of highly-requested features. We sent an email announcing the new product release to half of our customers (with the other half scheduled to receive the email tomorrow) and offered them to upgrade today to get the new features.

Unfortunately, some combination of events brought power down in our new data center at 9am this morning. The timing couldn’t be worse.

We have people onsite at the data center and have been continuously in communication with the data center to ensure all steps were taken to bring power back on. At this point, 11:05 am, power has been turned back on.

Status as of 11:05am:
* Power has been restored to the data center
* Most services (new trials, purchases, backups, restores) are unavailable for most users
* Already backed up data is still backed up

Now starts an involved process where we will bring up all equipment, check that no systems have been damaged, and bring all services back online. This will take a while to be 100% complete – likely toward the end of the day.

However, within an hour we expect that users will be able to do almost everything:
* Install trials
* Purchase the service
* Update to the new version
* Access the account
* Restore files
(Backups may start as well, but will be rolling out slowly across all customers.)

We will continue to update this post with new information. Please bear with us.

Comments Closed


Computers Need Electricity


blog-sad-cloudYesterday (October 24, 2013) six cabinets in the Backblaze datacenter lost all electricity at the same moment (10:58am).  No customer data was lost, but during the repair of the electrical circuits the pods containing some customer data were offline for the next few hours until 9:42pm when the final pod was brought back up and online.

These six cabinets are made up of 54 Backblaze Storage Pods containing a total of 2,430 hard drives and approximately 9.7 Petabytes of storage, which is approximately 1/8th of the Backblaze fleet of pods.

New data from all customers continued to be safely backed up onto the other 7/8th of the Backblaze pods the whole time due to automatic software fallbacks built into the Backblaze system.

However, at 3am (October 25, 2013) a side effect of having so many pods offline triggered some customer backups to pause temporarily.  We have made adjustments and are allowing more and more customer backups to continue.  Our goal is to have 100 percent of Backblaze customers fully working and backing up by 3pm today.

We will continue to update this blog post with information as it develops. Thank you for your patience while we resolve this issue.

*Update – 10/25/13 12:53PM PST* – Maintenance is almost complete. 95% of customers should now be fully online and able to upload data.

*Update 2 – 10/25/13 1:11PM PST* – 100% of customers are now back online and can upload data. We will continue to investigate the cause of this outage.

*Update 3* – All systems are normal, and pods are online. The cause of the outage was 12 independent circuit breakers being tripped simultaneously, which affected a small percentage of our users. No customer backup data was lost. We are working with the data center to audit their power infrastructure.

Comments Closed


Backblaze Halloween Contest


email-scary-contest

Want to win an iPad Mini or a Nexus 7?
We’re having a scary video contest! Want the rules? Read below, and don’t forget to check out the example video at the bottom! Oh, and hurry! The contest ends October 29th!

How to Enter the Contest?

1. Prepare a short, 30 seconds or less, video recreating your favorite horror movie scenes using your computer or hard drive as the victim.
2. Insert the following image at the end of the video (right-click and save as):

logo_online_backup_transparent

3. Upload your video online, to YouTube for example.
4. Post a link to your video on the Backblaze Facebook wall or on Twitter with the hashtag #Backblaze so we can enter it in to the contest. Or, link to it in the comments below!
5. Share your video with friends.

General Questions:

1.  How many people can be in the video?
Answer:  However many you need in order to recreate the scene!

2. Can I make it longer than 30 seconds?
Answer: Maybe 32 seconds, but that’s it.  If you want to make a longer “director’s cut” we’d love to see it, but the contest video should be close to 30 seconds. Please keep it short and scary.

3. Can I record it on an iPhone, Android, iPad, Camera, etc?
Answer:  You can use whatever device you wish to record your video.

4. Can I submit multiple videos?
Answer: If you have multiple favorite scenes, make a vignette! But only please submit one video.

Rules:

1. You must use a valid Facebook account and comply with all Facebook rules for age, content, copyrights, etc to post a link to your video on the Backblaze Wall.
2. We reserve the right to remove and not consider as a valid entry any videos which we deem inappropriate.  We reserve the exclusive right to determine what is inappropriate.
3. Backblaze reserves the right to use your video for other purposes.
4. The contest will end on October 29, 2013 at 11:59:59 PM Pacific Standard Time.  The winner will be announced on October 31, 2013.  The winner will be determined by Backblaze.
5. One 16GB iPad Mini or one 16GB Nexus7 will be awarded to the winner.  The winner will determine which they would like (either an iPad Mini or Nexus7). The prize will be mailed to the winner in a timely manner once it becomes generally available to the public.
6. Please keep the content of the post PG rated — no cursing or extreme gore/violence.
7. By submitting a video you agree to all of these rules.

Want an example?

That’s it! Have fun!

Comments Closed


What Would You Like To Read On The Backblaze Blog?


The Backblaze Blog has been around for quite a while and we’d like to know what you think of it, and what you’d change about it, if anything. Thank you for your feedback, and we hope we can keep bringing you quality, entertaining, and informative content!

*Update* Thanks for all the feedback! We’ll be posting our results soon!

*Update 2* The results are below! More datacenter and Backblaze Storage Pod related information won by a wide margin…we’ll see what we can do!

Comments Closed


Unboxing The Japanese Retail Box!


Last year Backblaze announced a partnership with Sourcenext to help spread online backup throughout Japan.  After all, backing up your computer is a good idea, no matter where you live!  This year Sourcenext is releasing a retail box for us!  Let’s take a look inside shall we?

Comments Closed


Incredible Invoices!


Backblaze invoice

Backblaze has a great support team and one of the most frequent requests they receive is, “could you please provide an invoice?” From the beginning, we’ve been handcrafting every invoice that’s been sent out, and while we love making our customers happy, as Backblaze has grown it’s become increasingly difficult to keep up with all the requests. Over the last few years, we’ve been hard at work making the best online backup solution possible and that meant that our engineers were working on iterating and expanding the functionality of our backup client. Unfortunately this meant that some of the non-mission critical improvements we’ve wanted to make had to take a backseat. Well, the day of automated invoicing is finally upon us! Thanks to Monika, the latest addition to our engineering team. After joining Backblaze she was tasked with something of the utmost importance, creating an invoicing system so that you can get your own invoices!

That’s right! Starting today you will be able to go to the “Transactions” section of your Backblaze account and select a year or billing period for which you would like to receive an invoice. Once created, you will be able to print or download the invoice which will have all of your Backblaze transactions including your dates of purchase, the number of licenses purchased, and the total cost. Even though this may sound like a minor improvement it is something we are very excited about!

Comments Closed