Sometimes when you first hear an idea you think, “OK, that’s crazy.” That’s what happened when our PR folks came to us in marketing with the idea of having Backblaze be on The Ellen DeGeneres Show. It was a crazy idea.
For those of you who know Backblaze, we have grown from a gleam in our founders’ eye to a profitable, reasonably well known company in the Online Backup business. We didn’t spend our money on TV commercials, radio ads and the like, we relied on making a great product, and having our customers tell their family and friends about us. So when someone said, “you can be on Ellen,” you can understand why we thought they might be a bit of a loon.
It was just before Christmas when we were first approached with the idea of having Backblaze be on Ellen. We weren’t ready. We were approached again just before Valentine’s Day, then again before Mother’s Day. Each time we’d talk through it and each time we’d get closer to saying yes. But we still said no. This was a really big deal. Was it the right thing for Backblaze to do? Could we handle the new demand without compromising service to our existing customers? What would happen if nothing happened?
We talked around this for months. Our PR folks continued to nudge and slowly the rest of us in marketing got comfortable with the idea. It could work, we could do it, and it would be amazing.
So we said yes.
Contracts were signed, emails were sent, and we were on our way. A couple of Backblaze folks went down to Burbank to watch the filming (more on that in another post).
We are very proud to announce that Backblaze was selected as a finalist for the 2013 SIIA Software CODiE Awards in the Best Cloud Storage and Back Up Solution category. The SIIA CODiE Awards are the premier award for the software and information industries, and have been recognizing product excellence for 27 years.
To become a finalist, Backblaze had to be evaluated by a panel of industry judges and be compared to the other category entrants. There were a myriad of questions to answer and the judges ran Backblaze through it’s paces to make sure it did everything we said. We also had to address how Backblaze delivered in the areas of agility, flexibility, redundancy, reliability, scalability and most importantly security. The thoroughness of the evaluation not only included the Backblaze client like the one on your PC or Mac, but the entire Backblaze infrastructure. This makes sense, given the breadth of the category “Best Cloud Storage and Back up Solution”.
While many people find our service to be the easiest online backup solution around, the real magic is masking the complexity that is inherent in backing up, storing and restoring our customer’s data. Over the past 5 plus years Backblaze has safely backed up and now stores nearly 50 Petabytes of user data and so far we’ve restored over 3 Billion files for our customers.
We’d like to thank the SIIA for naming us as an SIIA CODiE Awards finalist in the Best Cloud Storage and Back Up Solution category. And we’d like to thank you our customers for believing in us as well.
We thought ten people would care; instead a million people read our Storage Pod 1.0 blog post where we open sourced the Backblaze Storage Pod design and introduced the world’s most cost-efficient way to store big data. The interest grew when we published our Petabytes on a Budget: Revealing More Secrets blog post that announced Storage Pod 2.0, which doubled the amount of storage and reduced the price. Since then several companies have built businesses selling Storage Pods inspired by Backblaze to hundreds of organizations around the world who are storing hundreds of petabytes of data on their own Storage Pods. Today we introduce Backblaze Storage Pod 3.0 which stores more data, costs less, is more reliable, and is easier to service.
What’s New?
For Storage Pod 3.0 we redesigned the chassis and upgraded many of the components. Most of the changes are aimed at improving the reliability and flexibility of the Storage Pod. The full parts list is in Appendix A at the end of this post.
Here are the highlights:
180 terabytes of storage. With the availability of 4 TB hard drives a Storage Pod can now be configured to store 180 TB (45 x 4 TB). As a bonus the same chassis and components can be used with any capacity of 3.5” hard drives.
Anti-vibration drive bay assemblies. There are now 3 assemblies, one for each row of 15 drives. Each assembly is designed to lock down a row of drives in place. These assemblies replace the “drive bands” around each drive. This saves nearly an hour during Pod assembly and makes drive replacement easier as well.
The key advantage of the drive bay assemblies is to reduce vibration. These assemblies not only keep the drives still, they also keep them firmly seated in the backplanes. Over the past several months we have tested different models of drives in the new drive bay assemblies and we have seen a dramatic improvement in overall system performance along with lower drive failure rates.
Upgraded motherboard. We now use the Supermicro MBD-X9SCL-F motherboard which replaces the previous model. The new motherboard adds a host of advanced processing features from Intel, as well as upgrades the PCIe slots to double throughput. Note that while our currently specified SATA cards do not take advantage of this increased throughput, it’s nice to know we can use it in the future.
More motherboard choices. We added standoffs to the chassis to provide better support for Micro ATX motherboards while still supporting the Standard ATX form factor. Specifically the new standoffs support the outer edge of the Micro ATX boards.
CPU. We upgraded the CPU to a 2nd generation Intel Core i3-2100 processor to replace the end of lifed (EOL) i3-540 model. This also gets us a little bump in clock speed (3.06 to 3.1 GHz), lower power usage (65 versus 73 watts), and more supported RAM (32GB up from 16GB).
Memory. We changed memory suppliers, so now the memory is certified by Supermicro.
Boot drive options. With Storage Pod 3.0, boot drives can now be 2.5” or 3.5” and we allow a second 2.5” drive to be attached for use in a redundant RAID1 boot volume. Boot-up drives can also be traditional HD or SSD drives. Backblaze switched to 2.5” boot drives because they are less expensive and are more reliable, but we didn’t want to eliminate support for 3.5” drives in case anyone needs a higher capacity 3.5” drive.
Backplanes. While we continue to use the same backplanes (Sil3726 chipset) as with Pod 2.0, there is another backplane based on the newer Silicon Image Sil3826 chipset that can be used. If you do use the Sil3826 based backplanes, you’ll notice that boot up can take a long time due to a large number of time-out/retry errors during the boot process. Eventually the boot up process will succeed. To fix this, you can use these instructions to update the backplane driver in your Linux kernel so you can use the Sil3826 based backplanes.
SATA cables. We replaced our SATA cable vendor with Nippon Labs. If you decide to use another vendor, look for cables that are SATA II or SATA III compliant and test them extensively. While any SATA II or SATA III cable should work we have found quality control problems with a number of vendors. In all cases it was the connectors, not the cables, which were defective.
Metal standoffs in the chassis. We replaced the plastic standoffs with metal components that can be manufactured as part of the chassis to reduce the cost and assembly time.
Improved airflow. The vent design was improved to increase airflow through the pod. We’ve never really had a problem with heat in the pods, and we’d like to keep it that way.
Chassis rivets. We replaced many of the screws with rivets. This simplifies the manufacturing process and saves times during the assembly process. Note, if you end up buying a case from Protocase (more on them later), they continue to use screws versus rivets. The two manufacturing processes produce the same basic chassis regardless.
Costs Less. For Storage Pod 2.0, the price for the components without drives was $1,984.00. For Pod 3.0 the price is $1,942.59, or $37.41 less – or about a 1.9% decrease in the cost (see Appendix A for a parts breakdown). The main component in the total cost will be the hard drives. The lingering effects from the Thailand drive crisis and consolidation in the drive industry have meant that even today hard drive prices are higher than they were when Pod 2.0 was introduced back in July of 2011 but here’s one way you can save a little on the price of hard drives.
Lessons Learned
Backblaze currently has over 450 Storage Pods deployed and manages nearly 50 petabytes of data. There are also many Storage Pods being used by organizations around the globe. Along the way we’ve learned a few things:
Firmware revisions matter. Watch out, manufacturers update hardware and upgrade firmware without changing their model number. These updates are intended to fix bugs but in the process new bugs can be introduced. Read the release notes carefully and downgrade the firmware to the version you’ve tested whenever possible. When updates are unavoidable, test them thoroughly before deploying them.
Let your vendors do the testing. Components like memory, PCIe cards and hard drives are modular and should all play well together. Unfortunately, the specs that allow for this interoperability (PCIe, SATA, DDR3, etc) are as complicated as are the components themselves. Because of this, use ‘certified’ components whenever possible. This will minimize problems and avoid finger pointing between vendors if problems do arise.
Don’t make random changes to our design. It might be tempting to try a build a pod out of the spare parts lying around your office but don’t do it. The components specified in the parts list in Appendix A are known to work well together. We believe in iteration and experimentation but don’t reinvent the wheel unless you have to.
There is more to power than just Watts. ATX power supplies deliver power at several voltages or ‘rails’ (12V, 5V, 3.3V, etc). Each vendor imposes unique limits on the amount of power you can draw off of each rail and unused power on one rail cannot be used on another. In particular, most high end power supplies are designed to deliver most of their power on the 12V rail because that is what high end gamer PCs use. Unfortunately, hard drives draw a lot of power off the 5V rail and can easily overwhelm a high wattage power supply. You will hit serious problems if power requirements for each component are not met so be careful if you don’t use the power supplies we recommend.
Keep it simple. For hard drives, SATA cards, SATA cables, backplanes, etc. you should use the same vendor, part number and/or model number as much as possible. By keeping things simple you reduce the number of variables that need to be considered if things go wrong. If you do mix components do it intentionally and for a good reason like comparing the performance of 3 different hard drives. And when you do this, make sure you’ve set things up in a way that allows you to draw clear conclusions. For example: with hard drives, if you want to compare the performance of 3 different models you should arrange them so that each RAID array is homogeneous. You should also take care to make sure each array is spread across all backplanes and SATA cards so that you don’t have IO hotspots which could taint your results.
Things change. Just when you get comfortable with a part, it will be discontinued or upgraded. We buy in quantity, we buy spares, and we have substitutes ready to go at any time. We also look for parts that have long-term support policies. We realize you may not be able to do these things, so be prepared when you upgrade one component that something may break.
The Uses and Users of a Storage Pod
In general there are three types of storage:
Transactional storage that provides real time or near real time access to data.
Bulk storage which stores a large amount of data yet provides access to that data within seconds to minutes.
Archive storage which keeps all of your data and often requires hours to days to access the data you need.
All are important. Our Backblaze Storage Pods were designed, built and implemented to economically hold large amounts of data yet provide access to that data in seconds to minutes, thus they are purpose-built for bulk storage. The economics of using a Backblaze Storage Pod for storage have driven many organizations and people to build and utilize Storage Pods inspired by Backblaze. This community has also built upon the Backblaze design and specifications to build Storage Pods for all types of storage needs. Here are a few examples:
Rensselaer Polytechnic Institute, the NASA Jet Propulsion Laboratory, Heritage Auctions, the University of New Mexico and many more.
Storage Pod Economics
When we announced Storage Pod 2.0 in July of 2011, the price dropped by 15% and we doubled the amount of storage versus Storage Pod 1.0. So with Storage Pod 3.0, this should be the section where we would say something like “as hard drive prices have dropped, the economics have continued to improve.” We would say that, except that the Thailand drive crisis which began in October of 2011 raised hard drive prices dramatically. That event, and perhaps the continuing consolidation of hard drive manufacturers, has meant that hard drives prices have yet to return to their July 2011 levels. For example, the hard drive we specified in the Storage Pod 2.0 blog post was listed at $120 (Hitachi 3TB 5400 HDSS5C3030ALA630). To purchase that same drive today you would pay $199 with very limited availability. To be fair there are other drives available that we use, but even the least expensive drive is $125 and that’s an external USB drive that must be ‘shucked’ before it can be used. With the increase in drive prices and decrease in availability, we have validated a variety of drives that work in Backblaze Storage Pods. Here is a list of the drives we have tested and currently use:
4 TB drives
Hitachi – HDS5C4040ALE630 (Just starting to use these, but they look good.)
Another thing we expected to happen was wide availability of 4 TB drives with the subsequent price drop. Only recently have we been able to get a reliable supply of 4 TB drives and prices are starting to drop. While we have built and deployed 180 TB Storage Pods (forty-five 4 TB drives) and we expect to move to exclusively building 180 TB Storage Pods in the near future, the cost per TB for 3 TB systems versus is 4 TB systems is nearly the same.
Storage Costs
The table below compares the total cost of different Backblaze Storage Pod configurations.
Description
Pod 2.0
Pod 3.0
Pod 3.0
Drive size
3 TB
3 TB
4 TB
Our cost per drive
$120
$125
$195
Number of drives
45
45
45
Chassis cost
$1,984.00
$1,942.59
$1,942.59
Drive cost
$5,400.00
$5,625.00
$8,775.00
Total cost
$7,384.00
$7,567.59
$10,717.59
Total size in TB
135
135
180
Cost per TB
$54.70
$56.06
$59.54
Cost per GB
$0.0547
$0.0561
$0.0595
While it looks like the cost of a 4TB drive system is more expensive, when you factor in rack space, electricity, installation labor, etc. the long term cost for Backblaze leans towards using 4 TB drives. Our monthly cost for a full rack of Storage Pods with 3 TB drives is $0.63 per TB, while a full rack of Storage Pods with 4 TB drives is $0.47 per TB. When you factor all the costs together, it takes about 5 months for us to recover the extra cost encountered when building 4 TB based Storage Pods.
Building Your Own Storage Pod
Since Backblaze has given away the design, you can build your own Storage Pod. You’ll want to start with the parts list in Appendix A. From there feel free to change out some of the components, redesign the case, or completely change out the design. It’s up to you, but pay attention to the “lessons learned” noted earlier. To help you out, here’s a nice screen shot assembly walk-through from Protocase you can review and here’s a Storage Pod assembly overview (PDF-1.5MB) that you can download. You may also find some fellow Pod Builders by visiting the openstoragepod.org web site for information on their efforts to develop “affordable & energy-efficient high-capacity storage servers, built from commodity components”.
One thing to know is since we don’t sell the Storage Pods and don’t make money off any ancillary services, we don’t provide support or a warranty for people who build their own. To all of those builders who take up the challenge, we’d love to hear from you and welcome any insights you provide about the experience.
Buying a Storage Pod
If you are interested in buying a Storage Pod or having one built for you we recommend you talk to the folks at Protocase. They have built hundreds of Storage Pods. They can also help you with design changes for a reasonable fee. Also, their 45 Drives wiki site contains useful technical information about their Backblaze inspired Storage Pods.
As noted earlier Protocase continues to use screws versus rivets in the assembly process of their Storage Pods. From a functional point-of-view you will notice no difference in the finished product.
Another thing to know about Protocase is that they do not include hard drives with their Storage Pods. Protocase does test each Storage Pod they build with drives before they ship the unit to you, but it’s up to you to buy and install the hard drives. This is pretty easy to do, but be forewarned; sometimes buying 45 hard drives at Costco can be an issue.
Thanks for All the Fish
Thank you to everyone that has helped with ideas for how to improve the Storage Pods and to those who have helped spread the word on the reality of inexpensive big data storage. We welcome your continued feedback and support. Enjoy the design!
Appendix A – Price List:
The prices listed below are what we pay for the parts. To obtain these prices we do purchase then in quantity.
Item
Qty
Price
Total
4U Custom Case Includes case, anti-vibration assemblies, power supply bracket, etc. (Available in quantities of 1 from Protocase for more money)
1
$382.35
$382.35
760 Watt Power Supply Zippy PSM-5760V Power Supply
2
$269.00
$538.00
Dampener Kits Vantec VDK-PSU Power Supply Vibration Dampener
2
$3.99
$7.98
Motherboard Supermicro MBD-X9SCL-F (MicroATX)
1
$146.00
$146.00
Port Multiplier Backplanes CFI-B53PM 5 Port Backplane (SiI3726 Chipset)
9
$40.00
$360.00
8GB DDR3 RAM 4GB Samsung BDDR3-1333 PC3-10600 (Certified by Supermicro)
2
$38.00
$76.00
CPU Intel Core i3 processor i3-2100
1
$114.00
$114.00
Case Fan Mechatronics G1238M(OR E)12B1-FSR 12V 3-Wire Fan
6
$12.62
$75.72
Port PCIe SATA II Card Syba PCI Express SATA II 4-Port RAID Controller Card SY-PEX40008
3
$48.00
$144.00
SATA Cable SATA cables RA-to-STR 3 ft locking from Nippon Labs
9
$1.50
$13.50
On/Off Switch FrozenCPU ele-302 Bulgin Vandal Momentary LED Power Switch 12″ 2-pin
1
$24.95
$24.95
Boot Drive WD SCORPIO BLUE WD1600BPVT 160GB 2.5″ Internal
1
$48.99
$48.99
Screw: 4-40 X 3/16 Phillips: Secure Boot drive
4
$0.02
$0.08
Standoff: Round 6-32 X 1/4 Dia X 5/16 Lng: Motherboard mount
1
$0.90
$0.90
Screw: 6-32 X 3/16 Phillips PAN SST: Motherboard standoff
1
$0.02
$0.02
Screw: 6-32 X 1/4 Phillips PAN ZPS: Motherboard standoff
9
$0.004
$0.04
Screw: 4-40 X 5/16 Phillips PAN ZPS: Backplanes
72
$0.02
$1.44
Crimp Terminal, 22-30 AWG Power (Tin): Fan Assembly
18
$0.02
$0.36
Fan Connection Housing, 1×3 with ramp and rib: Fan Assembly
6
$0.04
$0.24
Soft Fan Mount, AFM02B (1 flat end): Fan Assembly
12
$0.175
$2.10
Soft Fan Mount, AFM03B (2 tab ends): Fan Assembly
12
$0.175
$2.10
Foam Tape, 1″ x 50′ x 1/16 in black: Vibration lids
The rain started in August and by mid-October 2011, violent floods in Thailand had crippled the factories that helped produce nearly half of the world’s hard drives. As an online backup company, Backblaze fills more than 50 TB of new drives every day. To survive this crisis without raising prices or compromising service, Backblaze deployed every last employee, as well as friends and family, to acquire drives in what became known internally as “drive farming”. What follows is how we did it.
Good morning, your costs just tripled!
Brian, our Buyer, gasped, “$349 for a 3TB internal Hitachi hard drive!” How could that be? The last drive order he placed just two weeks ago on NewEgg was at $129 per drive. He rechecked the SKU three times. He found Seagate and Western Digital prices were similar. What was going on? Over the next few days prices spiked in all of our traditional purchasing channels, everyone was quoting ridiculously high numbers. One “friend” offered us 100 drives, non-returnable, for a mere $568 each – we said no. In a matter of days the cost of the hard drives that are core to our business had jumped over 200% meaning the cost of providing our customers unlimited storage was going to skyrocket as well. We needed a plan.
We didn’t plan for rain in Thailand
We started drive farming in November 2011. The reason was simple, the supply of the 3TB hard drives used in our Storage Pods had dried up – or more correctly was under water. The tragic flooding in Thailand began in August 2011 and by early-October had submerged houses, schools and factories. Over 800 people died and many more were homeless and hungry, with over 1 million people thrown out of work. As the water receded, the human cost of the flooding was obvious and the economic impact was slowly coming into focus. In late-October, it was estimated that up to 50% of the worldwide hard drive manufacturing capacity was lost or damaged. The impact of the lost capacity was immediate as hard drive prices nearly tripled overnight.
Wanted: a really good idea
We huddled in the conference room. CTO Brian stood at the whiteboard listing retailers, store locations and employee names. While we had a storage pod buffer in our datacenter that could last a few months, some industry analysts were already predicting that the drive shortage could last well over a year. Backblaze is committed to providing unlimited data backup, but with our normal channels charging usury prices for the hard drives core to our business, we needed a miracle. We got two: Costco and Best Buy. On Brian’s whiteboard he listed every Costco and Best Buy in the San Francisco Bay Area and then some. We would go to each location and buy as many 3 TB drives as possible.
Shucking is a good thing
The going rate for a 3TB external drive at Costco or Best Buy was $169. Internal drives, like those inside your computer and the ones we typically use in our storage pods, were at least $100 more and were becoming nearly impossible to find, having been bought by HP, Apple and Dell for their systems. Our “choice” to use external drives meant they had to be “shucked” – a process where the internal drive is removed from its case – before being deployed into a Backblaze storage pod. In addition to the price of the drive there were now extra costs to shuck the drives, extra costs to recycle the shucked parts, and once shucked we couldn’t return a bad drive, but it was still less expensive than buying internal drives even if we could find them. Problem solved, Costco here we come.
“Aaargh, there the treasure be”
Backblaze employees Yev and Ken stared exhausted into the car trunk at their treasure – fifty-two 3 TB disk drives. That would be enough for 2 maybe 3 days at the current burn rate they thought as they drove back to the office to unload their bounty. They hoped Damon, KC and the other Backblaze employees would have similar success and we’d add a few more weeks’ worth of hard drives to our stockpile.
The economics of storage pods
Backblaze provides online backup for unlimited data at $5/month. We can do this because we have implemented our own backup and data storage management software on our own Storage Pods. These pods are built from commercially available parts such as 3TB internal hard drives from Seagate or Western Digital, motherboards from Supermicro, etc. Each Storage Pod holds 45 drives or 135 TB of data. We can reliably store and retrieve data at up to 25 times lower than the cost of other services such as Amazon S3 by using our own purposed-designed cloud storage.
Did you really, really, really mean 2?
The “Two Drive Limit” signs started appearing in retail stores in mid-November. At first we didn’t believe them, but we quickly learned otherwise. Sometimes, we talked our way into more, but we heard “2 is the limit” a lot. We started doing “drive math”: 2 drives a day per store, times 3 stores per day, times 5 “farmers”, times 7 days a week is 210 drives. That would be sufficient, but in reality it didn’t work out that way. Stores were stocking out of drives on a regular basis and we really couldn’t farm every day, but we kept at it. One Wednesday afternoon, after working all day at Backblaze, Yev circled the San Francisco Bay hitting local Costco and Best Buy stores – 10 stores, 46 drives, 212 miles on his Nissan.
Family and friends to the rescue
In late November, Brian was banned from buying drives on Costco’s online store, and Billy was banned from purchasing them at any Costco in the Bay Area. Other Backblaze employees were also asked to leave stores empty handed. Other vendors, along with the seasonal demand from Christmas were rapidly draining supplies. It was time to get creative, again. At an impromptu drive crisis meeting, fueled by stale pizza and good beer, we decided we needed to scale drive farming and the idea of friends and family drive farming was born. Emails and text messages were sent, phones calls were made, Facebook posts were posted, tweets were tweeted, you name it – the call went out to friends and family – buy hard drives and send them to Backblaze, NOW.
Simon says, “Raise prices” – Backblaze doesn’t play Simon says.
While we were drive farming other vendors reacted more traditionally. Intel warned that the drive crisis could lower its quarterly revenue by about $1 billion, give or take $300 million. EMC notified its Velocity channel partners, “…our Q1 2012 HDD list prices will rise between 5-15% over Q4 2011 levels.” HP announced that, “…HP will be forced to increase the prices that we charge for certain disk drives”. Dell sent us a nicely worded email explaining that “HDD costs have increased…and Dell’s list pricing has been adjusted as a result.” So while we pondered the economics of renting a truck and driving it across the country buying drives along the way, others raised prices – we would not do that.
Drive farming at work
Stores throughout the rest of the country were also limiting hard drive availability, but now we had more farmers. Drives started to trickle in, 2 at a time. It was cheaper to buy external drives at a store in Iowa and have Yev’s dad, Boris, ship them to California than it was to buy internal drives through our normal channels. Nearly every day the postman would deliver packages of drives from friends and family around the country. Tim would pile the boxes up on a table in the office and every week the van from our pod assembly company would pick them up. A few weeks later the Storage Pods, filled with 45 drives in each, would arrive in the datacenter. There, Sean and Guido would deploy the pods to store the petabytes of data backed up by our customers.
Holiday drive(s)
On Christmas Eve, Gleb, our CEO, stopped by a friend’s house to pick up eighty 3TB drives his friend had acquired when an online site forgot to limit the quantity he could order. It had taken the FedEx guy nearly 30 minutes to unload them and carry them up to his friend’s apartment. As Gleb was lugging boxes of hard drives to his car to take back to Backblaze, he realized that the hard drives he was loading were worth more than the car he was loading them into.
More stories
There are many great adventures from our days as drive farmers, here are just a few…
Getting back to normal
Drive farming continued throughout January and into mid-February of 2012. By then we had farmed 5.5 Petabytes of data storage. Eventually smaller distributors and resellers appeared offering decent prices on internal drives – no more shucking. Throughout the crisis, we didn’t raise our prices, we continued our policy of unlimited storage, and we didn’t throttle our customers’ backup speeds. Prices have slowly decreased over the past several months as Thailand has gotten back on its feet and production shifted to other factories. We recently purchased another 2,000 internal drives to add to our stockpile. Lower prices are nice.
Thank you
We want to say thank you to our friends and family who helped us through the drive crisis; Boris, Cara, Jim, Susan, Dave, Meng, Mike, Ben, Evelyn, Katherine, Randy, Ramey, Lise, Pete, Paul, Vladik, Tasha, Hakan, Alec, Alla, Katia, Leya, Yan, Jessica, Sergey, Mary, John, Eugenia, Penny, Sergey, Rebecca, Tim and many others. We’d also like to thank the retailers who kept the price of external drives reasonable – even Costco.
Yet, even though we at Backblaze weathered the drive challenge it pales in comparison to the challenges faced by the people of Thailand as they continue to recover from the devastating floods that have ravaged their country. Our hearts go out to them. If you wish to help, please consider a donation through Give2Asia or the relief organization of your choice.
Epilogue
On July 25th of this year, Backblaze took $5M in venture funding. At the same time, Costco was offering 3TB external drives for $129 about $30 less than we could get for internal drives. The limit was five drives per person. Needless to say, it was a deal we couldn’t refuse. Old habits die hard.
References:
EMC and HP quotes: http://www.storagenewsletter.com/news/miscellaneous/emc-and-hp-increasing-hdd-prices.
Charges against Armando Angulo, a Miami doctor who was pursued by the federal government, have been dropped because the evidence was taking too much space on the DEA servers. Now, to give them their due, the government has more than 400,000 pages of evidence. That must have taken a tremendous amount of resources to assemble. Lawyers, courts, investigations, etc.
But those 400,000 pages of evidence take just 2 TB of data. You can buy a 2 TB drive for $100 at Amazon. But the DEA claims this will be “an economic and political hardship”. $100 versus the $6.5 million he allegedly cost Medicaid through his fraudulent activities.
What may be even crazier is that the DEA claims this 2 TB took up 5% of their entire worldwide global storage network. In other words, the DEA has a total of 40 TB of storage. Worldwide.
If $6.5 million isn’t enough to keep 2 TB of data, we would be happy to provide an entire Backblaze Storage Pod to them for that much. And we’d be happy to provide online backup for all 2 TB for just $5/month.
40 petabytes of data takes up a lot of room. Those little ones and zeroes that make up your photos, videos, spreadsheets – your digital life – that we backup online for you are adding up. We started out nearly five years ago with 20 terabytes of data storage. Shortly after that, we signed the deal for our first quarter-cabinet in our current datacenter and we have steadily added more and more cabinets. Each full cabinet has 8-10 storage pods and when you do the math, we now have about 40 petabytes of storage. Growth is good.
So we are looking for another datacenter. We’ve created and issued an RFP (Request for Proposal) to get the process going. It’s a fairly short document, 8 pages, which outlines our requirements. And what do we need in a datacenter, here are a few things…
Space – Air conditioned, clean, raised floor for up to 225 cabinets – Hey, we provide unlimited data storage, so we need lots of space.
Network/Bandwidth – Multi-vendor 10Gb redundant symmetric connections – we don’t throttle backup speeds, so big pipes are mandatory.
Power – Each cabinet requires a 30 AMP 208Volt circuit, which powers 8-10 storage pods in each cabinet.
Physical security – Access controls, CCTV, good location, fences and barriers, manned patrols, etc.
Amenities – Space for Guido, parking, storage, loading dock and free back massages (just kidding, but it doesn’t hurt to ask).
Providers must provide documentation on their SSAE 16 or “SAS 70 type II” status. A LEED certified facility is desired. Of course we want a reputable provider with a proven history of providing stellar service.
Backblaze has grown from a commitment of just 1/4 cabinet to committing to up to 225 cabinets, a 1000x increase, and we are looking for a great partner to work with us.
We want to start using our new datacenter in December. It’s aggressive, but we believe it can be done. If you are interested in being our next datacenter provider you can download and view the RFP to learn how to submit your proposal, but act now as initial responses are due on Aug 20th.
Netflix designed a 36-hard drive custom chassis filled with the most popular movies and deployed at every datacenter near their customers. Not only will this this reduce the load on Internet infrastructure and lower the cost for ISPs to provide the service, but it will also increase the speed and improve the user experience for end users. It’s a step that shows why Netflix is a leader in online video: it’s due not only to content, but also to continuous innovation.
When we open sourced our Storage Pod design in 2009, one of the most common reactions was, “Are you crazy?! Why would you give that away?” We are incredibly excited to see companies that need inexpensive storage not only building their own, but customizing it for their specific needs, and contributing their designs and learnings back to the community.
Derrick Harris penned a great analysis at GigaOm today, Why Netflix’s CDN should scare the storage industry. He’s right, of course. The open source software movement started from a desperate need by developers to have access to base components upon which they could build and innovate. That demand created a treasure trove of invaluable applications.
Open source hardware comes from the same goals and is starting to have the same results. Companies operating at “cloud scale” have a critical need for physical systems that can support the needs at efficient costs. With the Backblaze Storage Pods, Facebook Open Compute, and Netflix Open Connect, we may have reached a tipping point.
I love how Netflix puts their goal on their site:
We welcome commentary and improvements, which will be shared with the community with the goal of a faster, less expensive Internet for all.
Next time you stream some Top Gear on Netflix, it could be from one of these Flixapods.
Many of you enjoyed the Part 1, Behind Backblaze video. Here’s Part 2, Behind Backblaze: Scaling Big Storage, which takes a look at our datacenter and the people behind it that make sure your data is safe and secure with Backblaze.
For those of you who think this kind of stuff is really cool, Backblaze has job openings in our Operations Group for a Director of Web Services, a Systems Administrator and a Datacenter Technician. All positions are in our San Mateo office.