I'm working on a project to back up my family photos from TrueNas to Blu-Ray disks. I have other, more traditional backups based on restic and zfs send/receive, but I don't like the fact that I could delete every copy using only the mouse and keyboard from my main PC. I want something that can't be ransomwared and that I can't screw up once created.
The dataset is currently about 2TB, and we're adding about 200GB per year. It's a lot of disks, but manageably so. I've purchased good quality 50GB blank disks and a burner, as well as a nice box and some silica gel packs to keep them cool, dark, dry, and generally protected. I'll be making one big initial backup, and then I'll run incremental backups ~monthly to capture new photos and edits to existing ones, at which time I'll also spot-check a disk or two for read errors using DVDisaster. I'm hoping to get 10 years out of this arrangement, though longer is of course better.
I've got most of the pieces worked out, but the last big question I need to answer is which software I will actually use to create the archive files. I've narrowed it down to two options: dar and bog-standard gnu tar. Both can create multipart, incremental backups, which is the core capability I need.
Dar Advantages (that I care about):
- This is exactly what it's designed to do.
- It can detect and tolerate data corruption. (I'll be adding ECC data to the disks using DVDisaster, but defense in depth is nice.)
- More robust file change detection, it appears to be hash based?
- It allows me to create a database I can use to locate and restore individual files without searching through many disks.
Dar disadvantages:
- It appears to be a pretty obscure, generally inactive project. The documentation looks straight out of the early 2000s and it doesn't have https. I worry it will go offline, or I'll run into some weird bug that ruins the show.
- Doesn't detect renames. Will back up a whole new copy. (Problematic if I get to reorganizing)
- I can't find a maintained GUI project for it, and my wife ain't about to learn a CLI. Would be nice if I'm not the only person in the world who could get photos off of these disks.
Tar Advantages (that I care about):
- battle-tested, reliable, not going anywhere
- It's already installed on every single linux & mac PC , and it's trivial to put on a windows pc.
- Correctly detects renames, does not create new copies.
- There are maintained GUIs available; non-nerds may be able to access
Tar disadvantages:
- I don't see an easy way to locate individual files, beyond grepping through
snarmetadata files (that aren't really meant for that). - The file change detection logic makes me nervous - it appears to be based on modification time and inode numbers. The photos are in a ZFS dataset on truenas, mounted on my local machine via SMB. I don't even know what an inode number is, how can I be sure that they won't change somehow? Am I stuck with this exact NAS setup until I'm ready to make a whole new base backup? This many blu-rays aren't cheap and burning them will take awhile, I don't want to do it unnecessarily.
I'm genuinely conflicted, but I'm leaning towards dar. Does anyone else have any experience with this sort of thing? Is there another option I'm missing? Any input is greatly appreciated!
I'm using standard BD-DLs. M-Disks are almost triple the price, and this project is already too costly. I'm not looking for centuries of longevity, I'm using optical media because it's read-only once written. I read that properly stored Blu-Rays should be good for 10 or 20 years, which is good enough for me. I'll make another copy when the read errors start getting bad.
Copying files directly would work, but my library is real big and that sounds tedious. I have photos going back to the 80s and curating, tagging, and editing them is an ongoing job. (This data is saved in XMP sidecars alongside the original photos). I also won't be encrypting or compressing them for the same reasons you mentioned.
For me, the benefit of the archive tool is to automatically split it up into disk-sized chunks. That and to automatically detect changes and save a new version; your first key doesn't hold true for this dataset. You're right though, I'm sacrificing accessibility for the rest of the family. I'm hoping to address this with thorough documentation and static binaries on every disk.
The densities I'm seeing on M-Discs - 100GB, $5 per, a couple years ago - seemed acceptable to me. $50 for a TB? How big is your archive? Mine still fits in a 2TB disk.
I mean, putting it in an archive isn't going to make it any smaller. Compression on even lossless compressed images doesn't often help.
And we're talking about 100GB discs. Is squeezing that last 10MB out of the disk by splitting an image across two disks worth it?
The metadata is a different matter. I'd have to think about how to handle the sidecar data... but that you could almost keep on a DVD-RW, because there's no way that's going to be anywhere near as large as the photos themselves. Is your photo editor DB bigger than 4GB?
I never change the originals. When I tag and edit, that information is kept separate from the source images - so I never have multiple versions of pictures, unless I export them for printing, or something, and those are ephemeral and can be re-exported by the editor with the original and the sidecar. Music, and photos, I always keep the originals isolated from the application.
This is good, though; it's helping me clarify how I want to archive this stuff. Right now mine is just backed up on multiple disks and once in B2, but I've been thinking about how to archive for long term storage.
I think in going to go the M-Disc route, with sidecar data on SSD and backed up to BluRay RW. The trick will be letting DarkTable know that the source images are on different media, but I'm pretty sure I saw an option for that. For sure, we're not the first people to approach this problem.
The whole static binary thing - I'm going that route with an encrypted share for financial and account info, in case I die, but that's another topic.
Where I live (not the US) I’m seeing closer to $240 per TB for M-disc. My whole archive is just a bit over 2TB, though I’m also including exported jpgs in case I can’t get a working copy of darktable that can render my edits. It’s set to save xmp sidecars on edit so I don’t bother with backing up the database.
I mostly wanted a tool to divide up the images into disk-sized chunks, and to automatically track changes to existing files, such as sidecar edits or new photos. I’m now seeing I can do both of those and still get files directly on the disk, so that’s what I’ll be doing.
I’d be careful with using SSDs for long term, offline storage. I hear they lose data if not powered for a long time. IMO metadata is small enough to just save a new copy when it changes
This is more expensive in your country?
https://a.co/d/9DiKeie
That's a little over $11 USD per 100 GB disk. Is it just more expensive where you live, or is it shipping?
I'd be really surprised if these weren't manufactured in Asia somewhere.
My options look like this:
https://allegro.pl/kategoria/nosniki-blu-ray-257291?m-disc=tak
Exchange rate is 3.76 PLN to 1 USD, which is actually the best I’ve seen in years
Just out of curiosity, is the product on Amazon, and is it that same price?
Broadly similar from a quick glance: https://www.amazon.pl/s?k=m-disc+blu+ray
Shit, that's way more expensive. If only you knew someone in the US who would buy a few boxes and ship them to you...
But, seriously, yeah, that basically eliminates it as an option.