Linux

Multiple TB when setting up a new server to mirror an existing one. (Did an initial copy with both together in the same room, before moving the clone to a physically separate location. Doing that initial copy would saturate the network connection for a week or more otherwise)

I once moved ~5TB of research data over the internet. It took days and unfortunately it also turned out that the data was junk :/

In the middle of something 200tb for my Plex server going from a 12 bay system to a 36 LFF system. But I've also literally driven servers across the desert because it was faster than trying to move data from one datacenter to another.

Upgraded a NAS for the office. It was reaching capacity, so we replaced it. Transfer was maybe 30 TB. Just used rsync. That local transfer was relatively fast. What took longer was for the NAS to replicate itself with its mirror located in a DC on the other side of the country.

Probably ~15TB through file-level syncing tools (rsync or similar; I forget exactly what I used), just copying up my internal RAID array to an external HDD. I've done this a few times, either for backup purposes or to prepare to reformat my array. I originally used ZFS on the array, but converted it to something with built-in kernel support a while back because it got troublesome when switching distros. Might switch it to bcachefs at some point.

With dd specifically, maybe 1TB? I've used it to temporarily back up my boot drive on occasion, on the assumption that restoring my entire system that way would be simpler in case whatever I was planning blew up in my face. Fortunately never needed to restore it that way.

~340GB, more than a million small files (~10KB or less each one). It took like one week to move because the files were stored in a hard drive and it was struggling to read that many files.

My cousin once stuffed an ISO through my mail server in '98. His connection up in Bella Bella restricted non-batched comms back then, so he jammed it through the server as email to get on the batched quota.

It took the data and passed it along without error, albeit with some constipation!

Probably some vigeo game on that is ~150-200 GiB. Does that count?

While I haven't personally had to move a data center I imagine that would be a pretty big transfer. Probably not dd though.

I've imaged an entire 128GB SSD to my NAS...

80GB, it was 8 hours of (supposedly) 4k content in the MP4 format. https://www.youtube.com/watch?v=VF5JWdaJlvc Here's the link (hoping for the piped bot to appear).

When I was in highschool we toured the local EPA office. They had the most data I've ever seen accessible in person. Im going to guess how much.

It was a dome with a robot arm that spun around and grabbed tapes. It was 2000 so I'm guessing 100gb per tape. But my memory on the shape of the tapes isn't good.

Looks like tapes were four inches tall. Let's found up to six inches for housing and easier math. The dome was taller than me. Let's go with 14 shelves.

Let's guess a six foot shelf diameter. So, like 20 feet circumference. Tapes were maybe .8 inches a pop. With space between for robot fingers and stuff, let's guess 240 tapes per shelf.

That comes out to about 300 terabytes. Oh. That isn't that much these days. I mean, it's a lot. But these days you could easily get that in spinning disks. No robot arm seek time. But with modern hardware it'd be 60 petabytes.

I'm not sure how you'd transfer it these days. A truck, presumably. But you'd probably want to transfer a copy rather than disassemble it. That sounds slow too.

Why would dd have a limit on the amount of data it can copy, afaik dd doesn't check not does anything fancy, if it can copy one bit it can copy infinite.

Even if it did any sort of validation, if it can do anything larger than RAM it needs to be able to do it in chunks.

I work in cinema content so hysterical laughter

When I was moving from a Windows NAS (God, fuck windows and its permissions management) on an old laptop to a Linux NAS I had to copy about 10TB from some drives to some other drives so I could re-format the drives as a Linux friendly format, then copy the data back to the original drives.

I was also doing all of this via terminal, so I had to learn how to copy in the background, then write a script to check and display the progress every few seconds. I'm shocked I didn't loose any data to be completely honest. Doing shit like that makes me marvel at modern GUIs.

Took about 3 days in copying files alone. When combined with all the other NAS setup stuff, ended up taking me about a week just in waiting for stuff to happen.

I cannot reiterate enough how fucking difficult it was to set up the Windows NAS vs the Ubuntu Server NAS. I had constant issues with permissions on the Windows NAS. I've had about 1 issue in 4 months on the Linux NAS, and it was much more easily solved.

The reason the laptop wasn't a Linux NAS is due to my existing Plex server instance. It's always been on Windows and I haven't yet had a chance to try to migrate it to Linux. Some day I'll get around to it, but if it ain't broke... Now the laptop is just a dedicated Plex server and serves files from the NAS instead of local. It has much better hardware than my NAS, otherwise the NAS would be the Plex server.

I'm currently backing up my /dev folder to my unlimited cloud storage. The backup of the file /dev/random is running since two weeks.

Do cloud platform storage operations count? If so, in the hundreds of terabytes (work)

I synced to the BSV shitcoin which is 11+ terabytes. So large I had to turn on throwing away the rest of what I downloaded because it wouldn't fit on all of the storage media I own. I feel sorry for the people running an archive node.

Manually transferred about 7TBs to my new Rpi4 powered NAS. It took a couple of days because I was lazy and transferred 15 GBs at a time which slowed down the speed for some reason. It could handle small sub 1 GB files in half a minute otherwise.

I once abused an SMTP relay (my own) by emailing Novell a 400+ MB memory dump. Their FTP site kept timing out.

After all that, and them swearing they had to have it, the OS team said "Nope, we're not going to look at it". Guess how I feel about Novell after that?

This was in the mid-90's.