{"id":52328,"date":"2022-12-09T10:27:03","date_gmt":"2022-12-09T09:27:03","guid":{"rendered":"https:\/\/www.open-e.com\/blog\/?p=52328"},"modified":"2025-06-18T08:21:49","modified_gmt":"2025-06-18T08:21:49","slug":"data-deduplication-in-zfs-yes-or-not","status":"publish","type":"post","link":"https:\/\/www.open-e.com\/blog\/data-deduplication-in-zfs-yes-or-not\/","title":{"rendered":"Data Deduplication in ZFS \u2013 Yes or Not?"},"content":{"rendered":"

\t\t\t\tWhen you scroll through the different data storage articles and read about the file systems\u2019 specifications, you might come across such a catchy term as \u201c<\/span>data deduplication<\/b>\u201d. In case you haven\u2019t checked it so far – it is a file system<\/a>\u2019s feature that <\/span>allows for reducing <\/b>data. It can work on two levels: <\/span>file- and block-level <\/b>(in some specific cases, it can be a <\/span>bit-level<\/span><\/a> as well), which, as you can guess, depends on the types of deduplication. File-level deduplication works with single files, while the block-level – with the whole blocks. Both have their advantages: the file level takes fewer resources and thus may be deployed over larger amounts of physical storage, and the block level can eliminate chunks of data smaller than a file.<\/span><\/p>\n

Sounds good so far, right?<\/span><\/p>\n

However, before happily implementing the feature just because it sounds desirable, you should first analyze the data you store. Let us explain how it works and what it actually offers you, on the example of Open-E JovianDSS.\u00a0<\/span><\/p>\n

Pros of Deduplication<\/span><\/h2>\n

Used Space Reduction:<\/h4>\n

Well, as it has been stated already in the description of the term: data deduplication is a technique for eliminating repeated data. So, the main goal of deduplication is to reduce the occupation of space in data storage. How does it work? During this process, the system compares the data byte patterns (chunks), which are contiguous blocks of data. These chunks are compared to other chunks within existing data to find matching patterns. When the system claims \u201cIt\u2019s a match!\u201d, the redundant chunk is replaced with a small-size reference that points to the \u201ctwin\u201d. Also,<\/span> it can effectively reduce the used capacity, even up to the disk number<\/b>, as the <\/span>deduplication ratio is higher than 1:1<\/span><\/a>.<\/span><\/p>\n

Improved Performance:<\/h4>\n

Due to the duplicates\u2019 elimination, there is a much greater chance to improve the I\/O performance in terms of write operations.<\/span> The more data is deduplicated, the fewer data there is left to be written<\/b>, as the system records just the references to the previous files.<\/span><\/p>\n

Cons of Deduplication<\/span><\/h2>\n

As we\u2019ve mentioned already, you should analyze your business requirements and the data you\u2019d like to store on your data storage solution.<\/span> If the data is not duplicable (which means that the data chunks are unique and cannot be reduced), this feature can turn out to be useless<\/b>. It will affect neither the capacity nor the performance on a noticeable level.\u00a0<\/span><\/p>\n

In addition, in some cases deduplication influences the read performance of your system. For example, <\/span>during a restore operation, performance of a deduplicated storage pool can be slower than a restore from a non-deduplicated pool<\/b>. When used, data deduplication can spread the extent of a given file across multiple volumes on the server.\u00a0<\/span><\/p>\n

When it comes to an automatic data comparison on all deduplication levels, the system counts the binary codes to state if the bites\/files\/blocks are the same. <\/span>This way, if two random pieces of data are stated as identical, it can not always be true and it may lead to data loss after deduplication<\/b>. The smaller the deduplication level, the smaller the risk of data loss.\u00a0\u00a0<\/span><\/p>\n

Deduplication disabling<\/span><\/h2>\n

IMPORTANT: If you decide to turn the deduplication off, for instance in order to restore the performance level, those settings will concern only the new data<\/b>. The old files will remain unchanged since disabling the deduplication process doesn\u2019t reduplicate the data. The deduplicated files remain deduplicated so it won\u2019t improve the storage performance. To do so you can use the <\/span>ZFS send<\/span><\/i> and <\/span>ZFS receive<\/span><\/i> features, and send the data to a dataset that doesn\u2019t use deduplication.<\/span><\/p>\n

Types of data suitable for deduplication:<\/span><\/p>\n