Asklemmy

43905 readers

1141 users here now

A loosely moderated place to ask open-ended questions

Search asklemmy 🔍

If your post meets the following criteria, it's welcome here!

Open-ended question
Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
Not ad nauseam inducing: please make sure it is a question that would be new to most members
An actual topic of discussion

Looking for support?

Looking for a community?

Lemmyverse: community search
sub.rehab: maps old subreddits to fediverse options, marks official as such
!lemmy411@lemmy.ca: a community for finding communities

~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~

founded 5 years ago

MODERATORS

Best way to backup files (sh.itjust.works)

submitted 5 months ago by emergencyfood@sh.itjust.works to c/asklemmy@lemmy.ml

28 comments fedilink hide all child comments

I have about 500GB of data (photos, documents, videos etc.) that I have accumulated over the years. Currently, I keep them on my computer and rsync all additions / changes once a month or so to an external hard drive. Do I need to be worried about data loss (sectors going bad, bit rot, bit flip, whatever it is called)?

To clarify,

None of this is commercially important; I just don't want to get into a situation where I look up an old family photo or video twenty years down the line and it has got corrupted.
Both my computer and the external HD are HDDs. They are fairly cheap here (and very cheap if second hand). Buying SSDs or dedicated hardware would be expensive.

you are viewing a single comment's thread
view the rest of the comments

[–] Dark_Arc@social.packetloss.gg 3 points 5 months ago (1 children)

I recommend kopia. It lets you backup automatically to a primary location, copy that data periodically to a secondary location, and it has a command that you can use to verify all the data is actually what it was when the backup was created.

[–] emergencyfood@sh.itjust.works 2 points 5 months ago (1 children)

Thank you. On that note, when backing up, is there a way to compare the two versions, see if one has become corrupted, and copy the good version to both? It would be sad if your primary copy got corrupted, and you overwrote all other copies with it.

[–] Dark_Arc@social.packetloss.gg 1 points 5 months ago* (last edited 5 months ago)

Kopia uses content addressable storage. So basically when it copies things, it only copies what data is new. Files that haven't changed will not be overwritten.

You kind of need to run the verification command on both the source and the "backup copy" for maximum paranoia. If you're running it on a local copy, that should be a relatively fast process as you don't need to download stuff.

You'd basically connect on the command line to the copy you just updated via sync-to and then ask kopia to verify 100% of the file integrity ... it should then run through everything and make sure it matches what's supposed to be there. I'm not sure how you fix it if it detects something wrong, I've yet to run into that ... I'm sure there's a way 🙂

You could also use two backup drives and sync to both, then if you get an error restoring a particular file from one, you could in theory restore it from the other. A ZFS cluster with redundant copies and/or a RAID-1, RAID-5 or RAID-6 style setup could also help ... but most people aren't going to run an entire NAS just to turn it on periodically and backup their data "offline". Most people are going to be better served (IMO) by using cloud storage like B2 (where bitflips aren't really a concern) or a NAS (where bitflips similarly are a minimal concern, ideally in another location) with a periodically updated offline copy (on say an external hard drive) should be enough to protect most people's data well.

Also going to like to what I'm talking about: