urn:uuid:dac04788-378c-5224-b043-e707a6377cc7 Daniel Aleksandersen https://www.daniel.priv.no/ Copyright © 2022 Daniel Aleksandersen. https://www.ctrl.blog/assets/logo/logo-square.svg 2022-05-05T20:46:00Z weekly 10 urn:uuid:55d51586-281a-47b7-a134-977e8eed78d2 2022-05-05T20:46:00Z 2022-05-05T20:46:00Z

Syncthing can deduplicate data before network transfers and even utilize on-disk storage deduplication on modern file systems on Linux, Solaris, and Windows.

<p>Syncthing is an open-source encrypted peer-to-peer folder synchronization program. It uses deduplication techniques to reduce the amount of data it needs to transfer over the network; saving you bandwidth costs, energy, and time. You can now optionally also let it deduplicate data storage to reduce your storage costs.</p> <p><a href="https://www.ctrl.blog/entry/syncthing-deduplication-master.html">Read more …</a></p> urn:uuid:37e2f2cf-f3c5-4222-abef-a22378910387 2022-02-07T15:26:00Z 2022-02-07T15:26:00Z

An overview of which operating and file systems support sparse files, how to identify such files, program support, and sparse files versus file system compression.

<p>Sparse files are files stored in a file system where consecutive data blocks consisting of all zero-bytes (null-bytes) are compressed to nothing. There is often no reason to store lots of empty data, so the file system just records how long the sequence of empty data is instead of writing it out on the storage media. This optimization can save significant amounts of storage space for other purposes.</p> <p><a href="https://www.ctrl.blog/entry/sparse-files.html">Read more …</a></p> urn:uuid:16597654-c539-4070-9e0e-4492d3665cfd 2021-01-23T14:10:00Z 2021-01-23T14:10:00Z

What would case a file-cloning attempt using the copy command to return “invalid argument”? It can be hard to identify the underlying issue.

<p>Most modern copy-on-write file systems, such as Btrfs and XFS, support file cloning. (OpenZFS being the notable exception.) However, the tools that support this space-saving innovation can be difficult to use. Here’s an example situation detailing how the simple copy (<code>cp</code>) command on Linux can make it hard to understand what’s going on.</p> <p><a href="https://www.ctrl.blog/entry/cp-reflink-einval.html">Read more …</a></p> urn:uuid:6b6aa72f-7588-4f1c-a2e3-9edf6d6b431b 2020-10-14T00:30:00Z 2020-10-14T00:30:00Z

Six percentage of each music files you’ve bought from Apple Music (iTunes Store) contain, literally, nothing. Why is Apple padding their music with zero-bytes?

<p>Have you ever bought a storage upgrade for your iPod, iPhone, or Mac to have more room for your Apple Music (formerly iTunes) digital music collection? You may have been filling it with gigabytes worth of empty space.</p> <p><a href="https://www.ctrl.blog/entry/apple-music-nullbytes.html">Read more …</a></p> urn:uuid:3afd4e8a-35a9-49dd-a2b1-36d96a03538a 2020-10-07T12:38:00Z 2020-10-07T12:38:00Z

Exploring whether deduplicating file systems or other deduplication alternatives can be used to reduce the storage cost of duplicated media content.

<p>Modern file systems like APFS, Btrfs, XFS, and ZFS support deduplicating whole files and chunks of files. The same applies to network sharing protocols like BitTorrent and IPFS. Can storage deduplication be used to reduce the storage requirement for something like the intro sequences of TV shows? What other options are there?</p> <p><a href="https://www.ctrl.blog/entry/deduplicate-tv-intros.html">Read more …</a></p>