Introduction
The RAID concept—Redundant Array of Inexpensive Disks—dates back to around 1980. At the time, the disks now known as Enterprise drives, with higher performance and greater resistance to data loss, were extremely expensive.
Researchers gradually began pursuing the idea of distributing data across a larger number of less expensive disks to achieve higher performance and reliability. This became the foundation of RAID technology and, for many years, the foundation of the global storage industry.
Although RAID remains a fundamental component of many storage systems, the rapid evolution of the IT industry and the explosive growth of data have exposed the limitations of this aging technology. These limitations have become a major concern for IT administrators, a source of unexpected infrastructure costs, and, in some cases, the cause of catastrophic data loss.
What are RAID’s design weaknesses in the Big Data era?
Designed for a Small Number of Drives
RAID was designed at a time when five 10 GB hard drives seemed extraordinary. Its protection model was therefore built around relatively small numbers of disks and the loss of only one or two drives.
With RAID 5, the loss of one disk in an array can be tolerated; with RAID 6, the loss of two disks can be tolerated while the data remains available. In practice, an array is often kept below about 10 disks because increasing the number of disks increases the probability of losing more than the one or two failures the protection scheme can tolerate, potentially resulting in total data loss. Even in smaller arrays, however, there is no guarantee that two or three disks will not fail around the same time. Drives in an array often begin operating at the same time, which can also increase the possibility of correlated failures.
This approach may be acceptable for a small environment with 30 or 40 drives, but it becomes impractical as the number of drives grows. For example, with 1,000 disks, it is no longer practical to divide them into groups of ten and dedicate two disks in every group to redundancy. The probability of drive failure also increases with the number of drives. When we consider environments containing millions of disks, such as Google’s infrastructure, it becomes clear that conventional RAID cannot be the foundation for such a scale.
Designed for Small Capacities
RAID was designed when 10 GB was considered a large hard-drive capacity. At that scale, rebuilding an array after a drive failure did not take an excessive amount of time.
As drive capacities have grown rapidly, with 6 TB and 8 TB drives becoming commonplace, repair and rebuild times have become much longer. In many parity-based RAID systems, such as RAID 5, configuring rebuild for high performance can take 48 hours or more while reducing storage-system performance by around 50 percent. RAID 6 can be even more challenging.
Because such a performance reduction is often unacceptable, rebuild operations are typically run in the background to minimize the impact on production workloads. The trade-off is that rebuilds can take several times longer. If another drive fails during that period, the entire array can become unrecoverable.
Google was estimated to use between 10 and 15 exabytes of data in 2026 (one exabyte is one million terabytes), a scale that is fundamentally outside the design assumptions of traditional RAID.
Designed Only for Block-Level Access
A RAID system combines all disks in an array and presents them to the connected operating system as a Block-Level Access Device—or, more simply, as one large disk. LUNs (Logical Units) are then created on this disk, with each LUN appearing to the operating system as a separate disk.
Many RAID controllers cannot create LUNs larger than 32 or 64 TB, so the operating system may never see a disk larger than that. This is usually not a problem when working with databases or mail servers, but it becomes a major issue when dealing with hundreds of terabytes of video archives or backup repositories. The result is a large number of separate volumes that administrators must somehow combine again to obtain a unified view of the data.
Dependent on Specialized Hardware
In a RAID array, synchronized writing and reading of data stripes—the fragments into which the original data is divided—is critical. If, for example, one of ten disks fails to respond within the timeout defined by the RAID controller, read and write operations can be disrupted, and the controller may mark the entire array, or even a healthy disk, as failed.
This is one reason RAID systems commonly require at least NearLine SAS or Enterprise SATA drives. A desktop hard drive may not provide sufficiently predictable response times and can cause incorrect behavior by the RAID controller. For RAID-oriented drives, TLER is typically expected to be below several seconds, while desktop drives may take much longer to respond to an error.
NL-SAS and Enterprise SATA drives can cost several times more than comparable desktop drives, significantly increasing storage costs. In addition, because of issues such as the Write Hole problem, RAID algorithms are often implemented in specialized hardware. That hardware can represent a substantial portion of the total storage-system cost and, when it fails, may need to be replaced with a vendor-specific component at a price determined by the manufacturer.
Not Optimized for Maximum Usable Capacity and Protection
In a typical RAID 5 array, roughly 20 to 25 percent of the raw capacity is consumed by parity. If RAID 6 is used for stronger protection, parity overhead can rise to around 25 to 35 percent.
The fundamental issue is that, despite sacrificing this amount of capacity, the protection is still limited to one or two disk failures within each array. For example, consider a 1,000-disk environment made up of one hundred 10-disk RAID 6 arrays, each containing eight data disks and two parity disks. Even though 200 disks are dedicated to parity, each individual array can tolerate only two disk failures. A third failure in any one array can make the data in that array unavailable.
Designed Without Operational Maintenance in Mind
When a disk fails in a RAID array, the administrator must replace it quickly so that rebuilding can begin immediately. If the administrator is unaware of the failure or replacement is delayed, another disk can fail and the entire array may be lost.
A Hot Spare can be assigned to reduce this risk, but doing so consumes additional drives and controller bays. More expensive controllers can provide a Global Hot Spare and reduce the problem to some extent. Nevertheless, as a general rule, RAID environments require failed drives to be replaced as quickly as possible to reduce the risk of total data loss.
Designed Without Flexible Data Growth in Mind
Changing the configuration of a RAID array is generally not possible without rebuilding or recreating the array. In other words, when an administrator wants to change the array configuration, the existing data must first be backed up, the array must be destroyed, and a new configuration must be created.
At tens or hundreds of terabytes, the temporary storage required for that backup—and the backup operation itself—can become a major operational challenge.
Designed Without Protection Against Silent Data Corruption
It has long been known that data stored on hard drives can suffer corruption and errors for many reasons, including cosmic radiation and firmware bugs. Although drive manufacturers may specify an error rate of approximately one error per 10^16 bits, research such as CERN studies has shown that real-world error rates can be higher.
As the amount of data stored worldwide grows exponentially, even a very small error rate becomes significant. CERN has estimated that in a 97-petabyte array, the corresponding amount of corrupted data could be on the order of 120 GB.
This phenomenon, known as Silent Data Corruption, is especially serious for applications such as financial systems, where even a single corrupted bit can potentially affect a large monetary value.
Traditional RAID systems generally do not continuously validate stored data against this class of corruption. The RAID controller effectively assumes that data written to disk is correct—an assumption that becomes increasingly unsafe at very large storage scales.
Controller Bottlenecks
Another weakness in RAID design is the controller bottleneck. In a RAID system, the controller is responsible for distributing data across array disks during writes and reconstructing it during reads.
As a result, the RAID controller can become a bottleneck for both performance and redundancy. Read and write throughput depends on the controller’s interface and internal processing capabilities. If the controller fails, the entire RAID system and the data stored on it can become inaccessible.
More expensive RAID systems may use dual controllers and multipath connectivity, but the fundamental limitations remain. Even if a controller can connect to thousands of disks through modern JBOD systems, aggregate performance cannot exceed the controller’s internal processing and interface capabilities.
Controllers also impose other limits. For example, many mid-range controllers cannot support more than a specified number of drives—such as 192—and require an additional controller beyond that limit.
Deliberate Vendor Dependence
This is not strictly a RAID design weakness, but storage vendors sometimes introduce product limitations that encourage customers to replace equipment and purchase newer hardware on a fixed cycle.
For example, some controllers support hard drives only up to a specified capacity. When higher-capacity drives become available, compatibility or firmware limitations may prevent customers from using them. Eventually, maintaining older equipment becomes economically unattractive and customers are pushed toward newer vendor hardware. Storage vendors commonly plan such hardware refresh cycles around five years.
So What Should Be Done?
The limitations of RAID have encouraged storage architects to develop systems that avoid these fundamental design constraints. Many of these newer architectures—including Xendure—are based on distributed systems designed to provide storage at petabyte scale and beyond.
Xendure addresses the limitations described above through its distributed architecture and purpose-built capabilities:
Designed for virtually unlimited capacity and drive count: Xendure is built around a distributed architecture of network-connected nodes. It is designed for very large numbers of drives and extremely large capacities. Unlike RAID, its advantages become more pronounced as drive count and capacity increase. Thousands of drives and tens of petabytes in a single Xendure cluster are considered a normal operating scale.
Designed without a centralized bottleneck: Xendure is distributed by design. When additional capacity or performance is required, new nodes can be added without redesigning the existing system. Unlike RAID, where adding drives can introduce additional operational complexity, adding nodes and drives to Xendure is a straightforward scaling operation. Because the Xendure client accesses all nodes in parallel, increasing the number of nodes increases overall performance as well as capacity.
Designed for unified file access: Xendure is designed for file-based access, including access to very large files. Multiple access methods—including REST API, file-system mounting, and SDKs—are provided so applications and users can access their data easily. Xendure does not use the traditional LUN-or-partition model; instead, it uses namespaces, allowing the entire multi-petabyte capacity of a cluster to be presented as a unified space.
Designed around available, non-proprietary hardware: Xendure uses hardware that is readily available on the market. No component is tied to a proprietary hardware platform, so a failed component can be replaced with a standard market-available part. There is no dependency on a single manufacturer, and customers are not forced to pay a premium for proprietary components. This use of standard hardware also reduces the learning curve. Administrators familiar with IP networking, conventional hardware, and operating systems can work with the platform without having to master a vendor-specific hardware ecosystem.
Designed for efficient capacity utilization and strong data protection: Unlike conventional RAID, Xendure uses its BitExpander algorithm for data protection. BitExpander enables protection levels that would be difficult to achieve with traditional RAID. For example, with a 16/20 configuration, data is divided into 16 parts and expanded to 20 parts, allowing the original data to be recovered even if four parts are lost. In a 20-disk set, this means the data remains recoverable even if four disks fail simultaneously, using less than 20 percent of the raw capacity for protection.
In addition, the Xendure data path uses a 160-bit checksum to verify stored data and protect against Silent Data Corruption. Xendure can also apply different protection levels to different files—for example, a 16/20 configuration for one file and a stronger 10/20 configuration for a critical file, where the data remains available even if ten of the twenty disks fail simultaneously.
Designed for straightforward administration: The entire system can be managed through a Web GUI or CLI. Adding or removing disks and nodes is straightforward. When a node or disk is added, its capacity becomes visible and usable without requiring changes to the existing configuration. If a disk or node fails, the system automatically begins rebuilding into available capacity without requiring administrator intervention. The platform uses conventional hardware familiar to IT administrators, simplifying both learning and ongoing maintenance.
Overall, Xendure is a storage system designed for Big Data that addresses the fundamental limitations of conventional RAID architecture.