Backblaze logo

What Hard Drive Should I Buy?


blog-which-drive-to-buy

My last two blog posts were about expected drive lifetimes and drive reliability. These posts were an outgrowth of the careful work that we’ve done at Backblaze to find the most cost-effective disk drives. Running a truly unlimited online backup service for only $5 per month means our cloud storage needs to be very efficient and we need to quickly figure out which drives work.

Because Backblaze has a history of openness, many readers expected more details in my previous posts. They asked what drive models work best and which last the longest. Given our experience with over 25,000 drives, they asked which ones are good enough that we would buy them again. In this post, I’ll answer those questions.

Drive Population

At the end of 2013, we had 27,134 consumer-grade drives spinning in Backblaze Storage Pods. The breakdown by brand looks like this:

Hard Drives by Manufacturer Used by Backblaze
Brand Number
of Drives
Terabytes Average
Age in Years
Seagate 12,765 39,576 1.4
Hitachi 12,956 36,078 2.0
Western Digital 2,838 2,581 2.5
Toshiba 58 174 0.7
Samsung 18 18 3.7

As you can see, they are mostly Seagate and Hitachi drives, with a good number of Western Digital thrown in. We don’t have enough Toshiba or Samsung drives for good statistical results.

Why do we have the drives we have? Basically, we buy the least expensive drives that will work. When a new drive comes on the market that looks like it would work, and the price is good, we test a pod full and see how they perform. The new drives go through initial setup tests, a stress test, and then a couple weeks in production. (A couple of weeks is enough to fill the pod with data.) If things still look good, that drive goes on the buy list. When the price is right, we buy it.

We are willing to spend a little bit more on drives that are reliable, because it costs money to replace a drive. We are not willing to spend a lot more, though.

Excluded Drives

Some drives just don’t work in the Backblaze environment. We have not included them in this study. It wouldn’t be fair to call a drive “bad” if it’s just not suited for the environment it’s put into.

We have some of these drives running in storage pods, but are in the process of replacing them because they aren’t reliable enough. When one drive goes bad, it takes a lot of work to get the RAID back on-line if the whole RAID is made up of unreliable drives. It’s just not worth the trouble.

The drives that just don’t work in our environment are Western Digital Green 3TB drives and Seagate LP (low power) 2TB drives. Both of these drives start accumulating errors as soon as they are put into production. We think this is related to vibration. The drives do somewhat better in the new low-vibration Backblaze Storage Pod, but still not well enough.

These drives are designed to be energy-efficient, and spin down aggressively when not in use. In the Backblaze environment, they spin down frequently, and then spin right back up. We think that this causes a lot of wear on the drive.

Failure Rates

We measure drive reliability by looking at the annual failure rate, which is the average number of failures you can expect running one drive for a year. A failure is when we have to replace a drive in a pod.

blog-fail-drives-manufacture

This chart has some more details that don’t show up in the pretty chart, including the number of drives of each model that we have, and how old the drives are:

Number of Hard Drives by Model at Backblaze
Model Size Number
of Drives
Average
Age in
Years
Annual
Failure
Rate
Seagate Desktop HDD.15
(ST4000DM000)
4.0TB 5199 0.3 3.8%
Hitachi GST Deskstar 7K2000
(HDS722020ALA330)
2.0TB 4716 2.9 1.1%
Hitachi GST Deskstar 5K3000
(HDS5C3030ALA630)
3.0TB 4592 1.7 0.9%
Seagate Barracuda
(ST3000DM001)
3.0TB 4252 1.4 9.8%
Hitachi Deskstar 5K4000
(HDS5C4040ALE630)
4.0TB 2587 0.8 1.5%
Seagate Barracuda LP
(ST31500541AS)
1.5TB 1929 3.8 9.9%
Hitachi Deskstar 7K3000
(HDS723030ALA640)
3.0TB 1027 2.1 0.9%
Seagate Barracuda 7200
(ST31500341AS)
1.5TB 539 3.8 25.4%
Western Digital Green
(WD10EADS)
1.0TB 474 4.4 3.6%
Western Digital Red
(WD30EFRX)
3.0TB 346 0.5 3.2%
Seagate Barracuda XT
(ST33000651AS)
3.0TB 293 2.0 7.3%
Seagate Barracuda LP
(ST32000542AS)
2.0TB 288 2.0 7.2%
Seagate Barracuda XT
(ST4000DX000)
4.0TB 179 0.7 n/a
Western Digital Green
(WD10EACS)
1.0TB 84 5.0 n/a
Seagate Barracuda Green
(ST1500DL003)
1.5TB 51 0.8 120.0%

The following sections focus on different aspects of these results.

1.5TB Seagate Drives

The Backblaze team has been happy with Seagate Barracuda LP 1.5TB drives. We’ve been running them for a long time – their average age is pushing 4 years. Their overall failure rate isn’t great, but it’s not terrible either.

The non-LP 7200 RPM drives have been consistently unreliable. Their failure rate is high, especially as they’re getting older.

1.5 TB Seagate Drives Used by Backblaze
Model Size Number
of Drives
Average
Age in
Years
Annual
Failure
Rate
Seagate Barracuda LP
(ST31500541AS)
1.5TB 1929 3.8 9.9%
Seagate Barracuda 7200
(ST31500341AS)
1.5TB 539 3.8 25.4%
Seagate Barracuda Green
(ST1500DL003)
1.5TB 51 0.8 120.0%

The Seagate Barracuda Green 1.5TB drive, though, has not been doing well. We got them from Seagate as warranty replacements for the older drives, and these new drives are dropping like flies. Their average age shows 0.8 years, but since these are warranty replacements, we believe that they are refurbished drives that were returned by other customers and erased, so they already had some usage when we got them.

Bigger Seagate Drives

The bigger Seagate drives have continued the tradition of the 1.5Tb drives: they’re solid workhorses, but there is a constant attrition as they wear out.

2.0 to 4.0 TB Seagate Drives Used by Backblaze
Model Size Number
of Drives
Average
Age in
Years
Annual
Failure
Rate
Seagate Desktop HDD.15
(ST4000DM000)
4.0TB 5199 0.3 3.8%
Seagate Barracuda
(ST3000DM001)
3.0TB 4252 1.4 9.8%
Seagate Barracuda XT
(ST33000651AS)
3.0TB 293 2.0 7.3%
Seagate Barracuda LP
(ST32000542AS)
2.0TB 288 2.0 7.2%
Seagate Barracuda XT
(ST4000DX000)
4.0TB 179 0.7 n/a

The good pricing on Seagate drives along with the consistent, but not great, performance is why we have a lot of them.

Hitachi Drives

If the price were right, we would be buying nothing but Hitachi drives. They have been rock solid, and have had a remarkably low failure rate.

Hitachi Drives Used by Backblaze
Model Size Number
of Drives
Average
Age in
Years
Annual
Failure
Rate
Hitachi GST Deskstar 7K2000
(HDS722020ALA330)
2.0TB 4716 2.9 1.1%
Hitachi GST Deskstar 5K3000
(HDS5C3030ALA630)
3.0TB 4592 1.7 0.9%
Hitachi Deskstar 5K4000
(HDS5C4040ALE630)
4.0TB 2587 0.8 1.5%
Hitachi Deskstar 7K3000
(HDS723030ALA640)
3.0TB 1027 2.1 0.9%

Western Digital Drives

Back at the beginning of Backblaze, we bought Western Digital 1.0TB drives, and that was a really good choice. Even after over 4 years of use, the ones we still have are going strong.

We wish we had more of the Western Digital Red 3TB drives (WD30EFRX). They’ve also been really good, but they came after we already had a bunch of the Seagate 3TB drives, and when they came out their price was higher.

Western Digital Drives Used by Backblaze
Model Size Number
of Drives
Average
Age in
Years
Annual
Failure
Rate
Western Digital Green
(WD10EADS)
1.0TB 474 4.4 3.6%
Western Digital Red
(WD30EFRX)
3.0TB 346 0.5 3.2%
Western Digital Green
(WD10EACS)
1.0TB 84 5.0 n/a

What About Drives That Don’t Fail Completely?

Another issue when running a big data center is how much personal attention each drive needs. When a drive has a problem, but doesn’t fail completely, it still creates work. Sometimes automated recovery can fix this, but sometimes a RAID array needs that personal touch to get it running again.

Each storage pod runs a number of RAID arrays. Each array stores data reliably by spreading data across many drives. If one drive fails, the data can still be obtained from the others. Sometimes, a drive may “pop out” of a RAID array but still seem good, so after checking that its data is intact and it’s working, it gets put back in the RAID to continue operation. Other times a drive may stop responding completely and look like it’s gone, but it can be reset and continue running.

Measuring the time spent in a “trouble” state like this is a measure of how much work a drive creates. Once again, Hitachi wins. Hitachi drives get “four nines” of untroubled operation time, while the other brands just get “two nines”.

Untroubled Operation of Drives by Manufacturer used at Backblaze
Brand Active Trouble Number of Drives
Seagate 99.72 0.28% 12459
Western Digital 99.83 0.17% 933
Hitachi 99.99 0.01% 12956

Drive Lifetime by Brand

The chart below shows the cumulative survival rate for each brand. Month by month, how many of the drives are still alive?

blog-36-month-drive-survival-rate

Hitachi does really well. There is an initial die-off of Western Digital drives, and then they are nice and stable. The Seagate drives start strong, but die off at a consistently higher rate, with a burst of deaths near the 20-month mark.

Having said that, you’ll notice that even after 3 years, by far most of the drives are still operating.

What Drives Is Backblaze Buying Now?

We are focusing on 4TB drives for new pods. For these, our current favorite is the Seagate Desktop HDD.15 (ST4000DM000). We’ll have to keep an eye on them, though. Historically, Seagate drives have performed well at first, and then had higher failure rates later.

Our other favorite is the Western Digital 3TB Red (WD30EFRX).

We still have to buy smaller drives as replacements for older pods where drives fail. The drives we absolutely won’t buy are Western Digital 3TB Green drives and Seagate 2TB LP drives.

A year and a half ago, Western Digital acquired the Hitachi disk drive business. Will Hitachi drives continue their excellent performance? Will Western Digital bring some of the Hitachi reliability into their consumer-grade drives?


Correction: Hitachi’s 2.5″ hard drive business went to Western Digital, while the 3.5″ hard drive business went to Toshiba.

At Backblaze, we will continue to monitor and share the performance of a wide variety of disk drive models. What has your experience been?



Enterprise Drives: Fact or Fiction?


blog-enterprise-vs-consumer

Last month I dug into drive failure rates based on the 25,000+ consumer drives we have and found that consumer drives actually performed quite well. Over 100,000 people read that blog post and one of the most common questions asked was:

“Ok, so the consumer drives don’t fail that often. But aren’t enterprise drives so much more reliable that they would be worth the extra cost?”

Well, I decided to try to find out.

In the Beginning
As many of you know, when Backblaze first started the unlimited online backup service, our founders bootstrapped the company without funding. In this environment one of our first and most critical design decisions was to build our backup software on the premise of data redundancy. That design decision allowed us to use consumer drives instead of enterprise drives in our early Storage Pods as we used the software, not the hardware, to manage redundancy. Given that enterprise drives were often twice the cost of consumer drives, the choice of consumer drives was also a relief for our founders’ thin wallets.

There were warnings back then that using consumer drives would be dangerous with, people saying:

    “Consumer drives won’t survive in the hostile environment of the data center.”
    “Backblaze Storage Pods allow too much vibration – consumer drives won’t survive.”
    “Consumer drives will drop dead in a year. Or two years. Or …”

As we have seen, consumer drives didn’t die in droves, but what about enterprise ones?

Failure Rates
In my post last month on disk drive life expectancy, I went over what an annual failure rate means. It’s the average number of failures you can expect when you run one disk drive for a year. The computation is simple:

Annual Failure Rate = (Number of Drives that Failed / Number of Drive-Years)

Drive-years a measure of how many drives have been running for how long. This computation is also simple:

Drive-Years = (Number of Drives x Number of Years)

For example, one drive for one year is one drive-year. Twelve drives for one month is also one drive-year.

Backblaze Storage Pods: Consumer-Class Drives
We have detailed day-by-day data about the drives in the Backblaze Storage Pods since mid-April of 2013. With 25,000 drives ranging in age from brand-new to over 4 years old, that’s enough data to slice the data in different ways and still get accurate failure rates. Next month, I’ll be going into some of those details, but for the comparison with enterprise drives, we’ll just look at the overall failure rates.

We have data that tracks every drive by serial number, which days it was running, and if/when it was replaced because it failed. We have logged:

    14719 drive-years on the consumer-grade drives in our Storage Pods.
    613 drives that failed and were replaced.

Commercially Available Servers: Enterprise-Class Drives
We store customer data on Backblaze Storage Pods which are purpose-built to store data very densely and cost-efficiently. However, we use commercially available servers for our central servers that store transactional data such as sales records and administrative activities. These servers provide the flexibility and throughput needed for such tasks. These commercially available servers come from Dell and from EMC.

All of these systems were delivered to us with enterprise-class hard drives. These drives were touted as solid long-lasting drives with extended warranties.

The specific systems we have are:

  • Six shelves of enterprise-class drives in Dell PowerVault storage systems.
  • One EMC storage system with 124 enterprise drives that we just brought up this summer. One of the drives has already failed and been replaced.
  • We have also been running one Backblaze Storage Pod full of enterprise drives storing users’ backed-up files as an experiment to see how they do. So far, their failure rate, has been statistically consistent with drives in the commercial storage systems.

    In the two years since we started using these enterprise-grade storage systems, they have logged:

      368 drive-years on the enterprise-grade drives.
      17 drives that failed and were replaced.

    Enterprise vs. Consumer Drives
    At first glance, it seems the enterprise drives don’t have that many failures. While true, the failure rate of enterprise drives is actually higher than that of the consumer drives!

    Enterprise Drives Consumer Drives
    Drive-Years of Service 368 14719
    Number of Failures 17 613
    Annual Failure Rate 4.6% 4.2%

    It turns out that the consumer drive failure rate does go up after three years, but all three of the first three years are pretty good. We have no data on enterprise drives older than two years, so we don’t know if they will also have an increase in failure rate. It could be that the vaunted reliability of enterprise drives kicks in after two years, but because we haven’t seen any of that reliability in the first two years, I’m skeptical.

    You might object to these numbers because the usage of the drives is different. The enterprise drives are used heavily. The consumer drives are in continual use storing users’ updated files and they are up and running all the time, but the usage is lighter. On the other hand, the enterprise drives we have are coddled in well-ventilated low-vibration enclosures, while the consumer drives are in Backblaze Storage Pods, which do have a fair amount of vibration. In fact, the most recent design change to the pod was to reduce vibration.

    Overall, I argue that the enterprise drives we have are treated as well as the consumer drives. And the enterprise drives are failing more.

    So, Are Enterprise Drives Worth The Cost?
    From a pure reliability perspective, the data we have says the answer is clear: No.

    Enterprise drives do have one advantage: longer warranties. That’s a benefit only if the higher price you pay for the longer warranty is less than what you expect to spend on replacing the drive.

    This leads to an obvious conclusion: If you’re OK with buying the replacements yourself after the warranty is up, then buy the cheaper consumer drives.



    How long do disk drives last?


    blog-drive-study

    How long do disk drives last? The short answer is: we don’t know yet, but it’s longer than you might guess.

    Why does a company that keeps more than 25,000 disk drives spinning all the time not know how long they last? Backblaze has been providing reliable and unlimited online backup for over five years. For the past four years, we’ve had enough drives to provide good statistics, but 74% 78% of the drives we buy are living longer than four years. So while 26% 22% of drives fail in their first four years, and we have detailed information about the failure rates of drives in their first four years, we don’t yet know what will happen beyond that. So how long do drives last? Keep reading.

    How Drives Are Used At Backblaze

    Backblaze uses lots of hard drives for storing data. 45 drives are mounted in each Backblaze Storage Pod, and the Storage Pods are mounted in racks in our data centers. As new customers sign up, we buy more disk drives, test them, and deploy them. We are up to 75 petabytes of cloud storage now.

    Before being deployed, each Backblaze Storage Pod is tested, including tests on all of the drives in it. Recently, Andy posted about Poor Stephen, a disk drive that failed this testing. His post describes the process Backblaze uses to set up, load test, and deploy a Storage Pod.

    Types Of Hard Drives In The Analysis

    Backblaze has standardized on “consumer-grade” hard drives. While hard drive companies say these drives are not designed to work in RAID arrays or the 24×7 workload of a data center environment, Backblaze uses software redundancy to protect data. In a future blog post we will delve into the statistics comparing “consumer” and “enterprise” hard drives.

    By far the majority of these hard drives are “raw” or “internal” hard drives. However, because the Thailand Drive Crisis made it nearly impossible to find internal hard drives for sale at reasonable prices, Backblaze started to farm hard drives. Thus, approximately 6 petabytes of the drives in this analysis were originally “external” hard drives that were “shucked” out of their enclosures.

    Number of Hard Drives

    The chart below shows the age distribution of the drives in the Backblaze data centers. The shape of the chart is mostly a reflection of the growth of the company, and the addition of drives as the customer base grew. Overall, not that many drives fail.

    blog-drivestats-total-drives-cuml

    Failure Rates

    Before diving into the data on failure rates, it’s worth spending a little time clarifying what exactly a failure rate means. At first glance, you might think that a failure rate of 100% is the worst possible. Every drive is failing! That’s not the whole story, though.

    Continue reading…



    Data survivability with ultrabook SSDs:
    A dropped Zenbook and teardown findings



    New ultrabook design decisions limit data recovery options
    When my girlfriend recently dropped (and killed) her new ultrabook, I found out the Solid State Drive (SSD) inside of it was not easily removable like the old hard drives always were. This means a dead laptop = completely lost data if you aren’t fully backed up. Now don’t get me wrong, storing my valuable data on precariously spinning platters inside laptop hard drives has always worried me. Even before I co-founded an online backup company, I’d wince when a friend dropped a laptop roughly on a desk or spun the laptop around with a fast nudge so others could see the screen. So as soon as I could, I moved over to using SSDs that contain no moving parts. The new SSDs are more reliable than the old spinning platters overall, but the current designs cause catastrophic failure modes that will result in MORE data loss in many situations.

    Continue reading…



    VW takes Backblaze Storage Pod for a ride


    VW on Backblaze Storage Pod
    You may not be familiar with the name Crispin Porter + Bogusky, but you’re probably familiar with their work. The firm, which was named U.S. Agency of the Year by Adweek last year, created “The King” and “Whopper Freakout” campaigns for Burger King; the Windows Mojave, Jerry Seinfeld/Bill Gates and I’m a PC campaigns for Microsoft; as well as ads for Guitar Hero, Old Navy, Best Buy, Coke Zero, and others.

    For the previous four years they have also been the official U.S. agency for Volkswagen and have created a lot of media during that time. So, when it came time to archive all of that media somewhere…they decided to build their own Backblaze Storage Pod.

    Ryan Banham, Windows Evangelist at Crispin Porter + Bogusky took on the task:

    Just as everyone is settling down for a big turkey dinner our first
    Backblaze storage pod will be preparing to feast on terabytes of data.

    He customized the Backblaze storage pod reference design with a different motherboard, more memory, Samsung instead of Seagate drives, a single power supply, and used Windows Server 2008 as the operating system. It’s great to see people making the design suit their particular purpose. Once he hones in on the final design for their purpose, he plans to deploy several racks of mirrored archive servers to support their storage needs.

    Some of the feedback Ryan provided to us on his customized version included:
    * The pod is uber-cool: Even under full load the drives stay under 72 F, so he also swapped our fans for quieter and lower power intake fans.
    * No trampolines for the pod: Moving the pod around requires the RAID cards to be reseated (possibly because the bottoms of the RAID cards stick out of the case.)
    * $20 gets you far. A pod running 50% – 75% of the month costs just $20 in electricity.

    Ryan says:

    Thanks for sharing the build and giving me something fun and
    interesting to do over the last few weeks! I learned a lot.

    Glad it was interesting and useful and thank you for sharing your learnings!

    Photos Ryan sent us of his pod:

    VW Backblaze Storage Pod



    The future of the data center is green:
    Takeaways from WiredRE data center event


    Green Datacenter

    What do Google providing search, Coca-Cola operating its systems to track inventory, and Backblaze backing up your data have in common? The computers that handle all of this live in data centers. And those data centers use power – lots of it.

    In the U.S. alone there are over 20,000 data centers – each of which houses thousands or tens of thousands of servers. Combined, these data centers make up 3% of all U.S. energy consumption (not just electricity) – more than the entire domestic air fleet.

    So when I went to an event on Wednesday called:
    THE TRUTH ABOUT THE FUTURE OF THE DATA CENTER:
    CLOUD, COLOCATION, & DATA CENTER REAL ESTATE

    it should be no surprise that the focus was on power, power, power.

    And lest you think this is people getting wrapped up in the green movement or just jumping on a marketing trend – let me dissuade you. Datacenters in the U.S. spend $23 billion a year on electricity according to KC Mares of MegaWatt Consulting. In fact, electricity can often cost over 50% of the purchase price of a server over it’s lifetime. Minor improvements can have massive implications not only on global warming but also company bottom lines.

    KC provided a fascinating overview of innovations and experiments that operators of data centers and the companies building out large server deployments are pursuing. Some examples:

    * VFDs – variable frequency drives to adjust the speed of blower fans that adjust to need rather than spinning at a constant rate.
    * Natural cooling – using outside air and fans rather than air-conditioning to keep data centers cool; it turns out most servers are perfectly happy running at temperatures much higher than what data centers attempt to keep them at.
    * Shorter cooling regions – having air flow almost directly around a server in the process of cooling it rather than through the entire building; shorter distances mean less air friction and less energy spent moving it around.
    * Eliminating UPS systems – getting rid of the backup power systems and assuming servers will go down…and having backup servers or data centers instead.
    * Using 480 volts – higher voltage means lower amperage and thus less heat loss and higher efficiency. More of today’s server systems are capable of handling this voltage.
    * Higher efficiency power supplies – switching to 90% efficient power supplies on servers rather than using 70% or 80% ones; these are more expensive upfront but can still pay off fairly quickly.

    A number of these items pay for themselves in a couple months and then generate savings ongoing from then on. KC has a variety of information on his site and blog.



    User builds “Extreme Media Server” based on a Backblaze storage pod


    Extreme Media Server
    Don Honabach has the honor of being the first person to successfully build his own Backblaze storage pod. (At least the first we know about.)

    With four servers running at home for media storage, Don, was using a fair bit of power (and probably generating a lot of heat and noise and taking up space.) For five years he was working to come up with an “Extreme Media Server” and after reading about the Backblaze storage pod, he decided this may be the way to go.

    Having expertise in the space, Don customized a variety of items in the pod including:
    * The operating system (switching to Microsoft Windows Server 2008 R2)
    * Power supplies
    * Motherboard
    * and more…

    In just a couple weeks Don had completed his “Extreme Media Server”. Combining all four servers into one, Don is saving 500 watts of power, and can run 16 independent movie streams across two monitors from a single storage pod.

    Don created a blog that describes his experiences building his Extreme Media Server.

    Congratulations Don and good luck watching all those movies at the same time!
    Extreme Media Server from Backblaze Storage Pod



    Backblaze storage pod:
    Vendors, Tips, and Tricks


    Storage_Pod_Tips
    Last month’s blog post about building our Backblaze storage pods generated a ton of interest and many people are building their own pods! Our post also generated a ton of questions so below we answer the common ones and provides more detail about where to get components.
    Continue reading…