home :: FreeBSD :: stow01

Content Addressable Storage

Because I am backing up a whole bunch of machines that may have identical files on them (many copies of Windows, old backups, etc), I am interested in finding ways of having n copies of a file not occupy n times the amount of storage.

I see three ways of doing this:

1. Using ZFS file depuplication on my backup server. This sounds good, but all of the ZFS configuration guides I've seen recommend having lots of RAM when you use dedup, and my little backup servers have only 2 to 4 GB of RAM, so this seems like a non-starter.
2. Using a manual dedup script that uses hardlinks to free up diskspace on a Unix volume... Something like dedup from Roderick Schertler (documented here), trimstrees at CPAN, harlink from Julian Andres Klode, opendedup, or fileuniq on sourceforge (there is a list here) actually. That sounds attractive, but I am not sure how long it would take on 1 TB of data, and I am not sure hardlinks will work well on files owned by multiple users.
3. Then, there is the possibility of Content Addressable Storage. EMC does this on its Centera product line, and it is quite useful in the Corporate space... in the Open Source world, I have been interested by Poul-Henning Kamp's stow package for a while (see here for the old version or here for the new version that phk is working on this year).

Stow looks interesting... I'll try to find some time to dig further into it.

/FreeBSD | Posted at 11:48 | permanent link