Updated on 19/08/2025
Due, in part, to the different views and opinions regarding the usage of hot spare disks in our previous post, we’ve decided to add an update for clarification.
The Problematic Aspects of Using a Hot Spare Disk
As stated in almost every theory, using a hot spare disk with ZFS, or in any other data storage environment is a good solution, as it automatically responds to malfunctions in a RAID and helps minimize the duration of a degraded array state.
That being said, the primary goal of creating a RAID is to ensure continuous operation and prevent data loss in the event of a disk failure. Therefore, anything that increases the risk of data loss could be considered a bad idea. Let’s take a closer look at some of the problematic aspects of using hot spare disks.
Hot Spare Disks Add Stress to Vulnerable Systems
The primary issue with hot spare disks is that they enable the rebuilding (resilvering) of a system that is still in active use as a production server. While the resilvering process is taking place, the system will also continue to process the usual production data reads and writes.
Resilvering is a process that consumes significant server resources. When executed while the server is still in use, it must compete with production workloads. Because it is treated as a low-priority task, the resilvering process can take an extended amount of time – sometimes even several days. This prolonged operation at maximum throughput can put considerable strain on the disks, especially HDDs, and may lead to serious wear or potential failures.
Having decades worth of experience, we’ve realized that the use of hot spare disks in complex enterprise systems increases the probability of additional disks failing as the resilvering process starts to put more and more stress on the existing disks and the system itself.
Problems in Overall Hot Spare Disk Design
The next flaw of a hot spare disk is that it degrades over time. From the moment it is connected to the system, it keeps on working. And when, eventually, it’s time for it to be used as a damaged disk’s replacement; the hot spare disk itself could simply not be in a good enough state to actually replace the damaged disk.
Another issue with hot spare disks is that they are activated automatically when a disk failure is detected, even if the failed disk is still connected to the system. The faulty disk might attempt to reconnect and operate again while the hot spare is taking over its role, creating additional stress on the system. This can impact overall performance and, in some cases, increase the risk of data loss.
Hot Spare Disks Create a Single Point of Failure
If your goal is to build a system with no single point of failure, relying on a hot spare disk won’t provide much confidence. The process of automatically replacing a failed disk can sometimes fail (partially or completely), which may lead to data loss.
From our decades of experience providing data storage solutions with Open-E, we’ve seen many cases where a hot spare disk actually caused a full server failure or even permanent data loss. The risk comes from automation: once triggered, it can set off a domino effect, especially in older infrastructures where hardware has already experienced years of wear.
Recommended Procedure in Case of a Disk Failure
These problematic aspects of hot spare disks are why our advice would be to not rely on hot spare disks in complex data storage architectures and to use other business continuity solutions instead. This is On- & Off-site Data Protection with user defined backup retention-interval plans reducing RPO & RTO to minutes.
Using the ZFS file system, it’s much easier to monitor the system and create a proper backup. With that, you have the ability to retrieve data from a damaged disk and write it onto a new one. Additionally, when using a HA cluster, there is an option to manually switch the production from the affected node to a secondary one, allowing for maintenance on the affected node.
Recommended Procedure in Case of a Disk Failure:
We’d advise following this procedure once the array shows that a degraded state has occurred as a result of a disk failure:
- Run a full data backup.
- Verify the backed-up data for consistency and confirm that the data restore mechanism is functioning properly.
- Identify the problem source, i.e., find the erroneous hard disk. If possible, shut down the server and ensure the serial number of the hard disk matches the one reported by the event viewer or system logs.
- Replace the faulty hard disk with a new, unused one. If the replacement hard disk had already been used within another RAID array; make sure that any residual RAID metadata on it has been deleted via the original RAID controller.
- Start a rebuild of the system.
With this manual process, the rebuild involves 5 steps. By contrast, using a hot spare disk skips the first four critical steps and automatically moves to steps 4 and 5. This means the rebuild completes before you’ve had the chance to run backups, verify data, or confirm the faulty hardware – steps that often make the difference between safe data and lost data.
Anyway, it’s still completely up to you as to how to build a proper system. Anyway, it’s still entirely up to you to determine how to build a proper system. However, we suggest avoiding dependency on hot spare disks in a ZFS RAID array due to the potential data loss they can cause.
45 Comments
MJI /
20, 02 2016 10:52:45I have some sympathy for this argument, but wouldn’t taking a full system backup and then a full verification of that backup stress the HDDs that remain in the array even more than rebuilding the array with the hot spare?
As other posters have pointed out, RAID is not a proper backup in the first place, but rather a way to ensure continuity of service – and a hot spare is the quickest way to ensure that the array is reconstructed and brought back into full operation as quickly as possible.
Frank /
24, 03 2016 02:24:07Hi, I’m getting this message even before the system start normally:
1785- Slot 0 Drive Array Not Configured. Run hpssa.
When I run HPSSA it asks for configuring a new array. i wasn’t the first person who configured it so i don’t know which raid system they were using. does it affects stored data?
Is it really bad? What’s the best solution?
matthew /
10, 08 2016 10:58:59Let me posit some reasons why a hot spare is a *really* good idea.
RAID array fails while no-one is at work (does your site remain manned 24×365, even in the event of a fire alarm or other security alert?), you’re running at risk of data loss until the situation is addressed.
SMART monitoring on the controller spots the disk is failing *before* it goes offline and fails to the hot spare with no risk of data loss and no downtime. I know it won’t catch all failures, but on the HP kit I’ve worked on the majority of failures (media failures as opposed to electronics failures) have been spotted and fixed on the fly while the disk was still usable.
In over 20 years I have *once* had an array fail a second disk while recovering onto a hot spare. I have lost count of the number of times that a hot spare has saved the day…
I have worked on one customer’s system where the transactional traffic was so high that they would not restore from backup… the lost income from the restore time outweighed the costs of abandoning the data and starting again with an empty database… for them, hot spares provided a better financial risk than an outage to perform steps (1) and (2)
If your data is that critical, you should use some RAID that allows multiple devices to fail… and probably have more than one hot spare available.
So I’m afraid my real-world experience teaches me that your article is really not a good way to go for every business. There may be edge cases where a hot spare proves bad, but there are edge cases where not wearing a seatbelt in a car proved beneficial.
Luke /
11, 10 2016 04:21:30I also think hot-spare is a bad idea with RAID5. Here is my reasoning….
OptionA: RAID 5 without hot-spare (3 drives in total)
OptionB: RAID 5 with a hot-spare (4 drives in total)
In OptionA: if one drive fails, you simply replace that drive ASAP and then system will rebuild failed drive. Let’s say whole rebuild will take 24hrs.
In OptionB: if one drive fails, immediately rebuild begins on a hot-spare. It will also take 24hrs. Once rebuild is complete you still need to swap the broken drive. Which will trigger another rebuild this time from hot-spare assigned drive to the freshly inserted drive – this process will also take 24hrs.
So not only you did the rebuild twice… it also took twice as much time. Effectively doubling amount of time where your Raid Array is danger of having another faulty drive. With all the extra stress caused by doing rebuild twice it’s walking on a thing ice really.
It almost feels it’s better to have a spare drive ready, kept on shelf, not as part of system (and not assigned as a hot-spare). As soon as faulty drive is detected start rebuild onto that drive. Obviously, if you cannot be next to your system everyday then maybe a hot-spare is a better option.
If you really insist on RAID 5 then maybe not having hot-spare is a safer option in this case. Unless I am missing something really obvious here.
Would love more feedback on this case.
** I know in some cases you can mark freshly rebuild hot-spare as your new drive. And then simply add another drive as a hot spare. But I am not sure if this is a default behavior for Raid Controllers. I think most raid controllers usually just rebuild back from hot-spare into a freshly slotted drive.
MJI /
01, 11 2016 01:09:37I don’t understand why option B would trigger two rebuilds. If you have a hot spare, the NAS will start an automatic rebuild, at the end of which you have a restored 3 drive setup. When you replace the broken drive, you can add the new drive as a hot spare if you want it, there is nothing that forces you to turn it into a 4 drive NAS.
Scott in Texas /
28, 12 2016 01:45:46If you use real RAID, not motherboard RAID or software RAID, you need not move the drive, and if you DID move the drive from the hot swap port to the primary port it would not result in a rebuild, it is called “Disk Roaming”.