Any good tools to deduplicate files? Got a bunch of images with different names, on different folders, sometimes with different resolutions, spread out across 10 external hard drives that I need to go through...
this post was submitted on 29 Feb 2024
1 points (100.0% liked)
datahoarder
6616 readers
2 users here now
Who are we?
We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.
We are one. We are legion. And we're trying really hard not to forget.
-- 5-4-3-2-1-bang from this thread
founded 4 years ago
MODERATORS
Dupe Guru has been a useful GUI on windows. It is also cross platform.
This was done by hand on my end. If I had needed to do it procedurally, I’m not sure what my approach would have been, but the direction I’d probably head in would be to look into finding duplicate MD5 hashes.
md5sum to get the low hanging fruit
Storage is dirt cheap. Just add more. IT at work bugs out their eyes when I talk about adding more storage space. I have more at home than they do in the office. Lol.
I’ve been buying used 8TB HGST drives from eBay. Dirt cheap and haven’t had one fail yet.
I’ve always been one for moderation. Plus, the big issue was just being able to find all of it. Everything was so scattered and in some cases I had half my Steam library downloaded onto two or three of the hard drives, all outdated.
Now I know where it all is and can easily back it up, where before I only had one copy of a lot of my most important files.
On what types of junk files did you save most of the space?
I had a bad habit whenever I would go to reinstall an OS to just copy the entire user folder into one of many places on my largest hard drive. I had at least 4 or 5 of these. So tons of it was cache files.
So it's not just me 😞
New Folder (5)(copy) in a directory called TO SORT.
Hah i have multiple to sort folders. Sigh
Never gets sorted
I've still got "To Sort" folders on spindles on DVDs lol
Oh yeah I can definitely relate! Though I do delete the more useless parts of them gradually.
WHAT IF I NEED IT YO
Now Jesse, we've talked about this...
There were some folders that were 50+GB that had like 3 txt files and a song worth saving in them. In some ways, I was impressed.