this post was submitted on 21 Jul 2024
208 points (94.4% liked)

Selfhosted

40717 readers
297 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

Sorry but I can't think of another word for it right now. This is mostly just venting but also if anyone has a better way to do it I wouldn't hate to hear it.

I'm trying to set up a home server for all of our family photos. We're on our way to de-googling, and part of the impetus for the change is that our Google Drive is almost full.We have a few hundred gigs of photos between us. The problem with trying to download your data from Google is that it will only allow you to do so in a reasonable way through Google takeout. First you have to order it. Then you have to wait anywhere from a few hours to a day or two for Google to "prepare" the download. Then you have one week before the takeout "expires." That's one week to the minute from the time of the initial request.

I don't have some kind of fancy California internet, I just have normal home internet and there is just no way to download a 50gig (or 2 gig) file in one go - there are always intrruptions that require restarting the download. But if you try to download the files too many times, Google will give you another error and you have to start over and request a new takeout. Google doesn't let you download the entire archive either, you have to select each file part individually.

I can't tell you how many weeks it's been that I've tried to download all of the files before they expire, or google gives me another error.

(page 2) 41 comments
sorted by: hot top controversial new old
[–] [email protected] 8 points 5 months ago

I have fancy California Internet and the downloads are surprisingly slow and kept slowing down and turning off. It was such a pain to get my data out of takeout.

[–] [email protected] 53 points 5 months ago* (last edited 5 months ago) (8 children)

Honestly I thought you were going to bitch about them separating your metadata from the photos and you then having to remerge them with a special tool to get them to work with any other program.

[–] [email protected] 28 points 5 months ago

omg they WHAT

[–] [email protected] 25 points 5 months ago

I'm not really looking forward to that step either

load more comments (6 replies)
[–] [email protected] 16 points 5 months ago* (last edited 5 months ago) (1 children)

I think this is a bit unfair. Most Google Takeout requests are fulfilled in seconds or minutes. Obviously collating 100GB of photos into a zip takes time.

And it's not googles fault you have internet issues: even a fairly modest 20Mbps internet connection can do 50GB in 6h. If you have outages that's on your ISP not Google. As others have said, have it download to a VPS or Dropbox etc then sync it from there. Or call your ISP and tell them to sort your line out, I've had 100℅ uptime on my VDSL copper line for over 2 years.

I was able to use Google Takeout and my relatively modest 50Mbps connection to successfully Takeout 200GB of data in a couple of days.

[–] [email protected] 9 points 5 months ago (2 children)

What download manager did you use? I've tried with whatever's built into Firefox on two different networks and similar results. The downloads freeze every so often and I have to restart them (it picks up where it left off). Sometimes it just won't reconnect, which I'm guessing is a timeout issue with Google, although I really have no idea.

I don't ever have to manage downloads of this size, so sorry if it's an obvious question

[–] [email protected] 4 points 5 months ago

A download manager I found to work well generally was aria2c. Only really worth it if you are on linux but it is simple yet powerful.

[–] [email protected] 6 points 5 months ago (1 children)

Not OP, but I use this download manager. It has been good.

https://www.downthemall.org/

load more comments (1 replies)
[–] [email protected] 1 points 5 months ago

Find a way to borrow more bandwidth.

[–] [email protected] 7 points 5 months ago

It's bad because they don't want you to use it, but they made it exist so that they don't get sued by the European Union.

[–] [email protected] 8 points 5 months ago (4 children)

Just gone through this whole process myself. My god does it suck. Another thing you'll want to be aware of around Takeout with Google Photos is that the photo metadata isn't attached as EXIF like with a normal service, but rather it's given as an accompanying JSON file for each image file. I'm using Memories for Nextcloud, and it has a tool that can restore the EXIF metadata using those files, but it's not exact and now I have about 1.5k images tagged as being from this year when they're really from 2018 or before. I'm looking at writing my own tool to restore some of this metadata but it's going to be a right pain in the ass.

[–] [email protected] 7 points 5 months ago (1 children)

Wow thanks for that. I was looking into https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper but I haven't gotten to that step yet

[–] [email protected] 2 points 5 months ago

Ooh, might look into that instead, actually. I always love a reason to write myself a little tool, but dealing with Google's bull makes it much less appealing to me when existing tools can do it for me.

[–] [email protected] 2 points 5 months ago

There exist tools for this from takeout archives already, its how I migrated to immich.

load more comments (2 replies)
[–] [email protected] 3 points 5 months ago

It's not an ideal solution but you can save your Google takeouts to Dropbox. It might be worth signing up for Dropbox for one month and use Dropbox's sync features. I haven't used Dropbox in years but they used to have quite solid syncing.

[–] [email protected] 2 points 5 months ago

You need a solid wired connection. Maybe phone a friend for help.

Alternatively you could use curl. I think it as a resume option.

[–] [email protected] 28 points 5 months ago* (last edited 5 months ago)

Google takeout is there so they are technically compliant with rules that say you must be able to download your personal data, but they make it so inconvenient to use that practically it's almost impossible to download it. Google photos isn't a backup service so much as a way for Google to hold your photos hostage until you start paying for higher amounts of storage. And by the time you need that storage, Google takeout download has become impractical.

[–] [email protected] 8 points 5 months ago (1 children)

Can you do one album at a time? Select the albums you want to download, then do that file. Then do the next few albums. That way you have direct control over the data you're getting in each batch, and so you'll have a week to get that batch instead of having to start again if the whole thing didn't finish in a week.

[–] [email protected] 6 points 5 months ago

That may be a thought. I could organize the photos first and then do multiple takeouts. Thanks

[–] [email protected] 37 points 5 months ago (3 children)

I know it’s not ideal, but if you can afford it, you could rent a VPS in a cloud provider for a week or two, and do the download from Google Takeout on that, and then use sync or similar to copy the files to your own server.

[–] [email protected] 10 points 5 months ago (2 children)

I don't know how to do any of that but I know it will help to know anyway. I'll look into it. Thanks

[–] [email protected] 11 points 5 months ago

Be completely dumb and install a desktop OS like Ubuntu Desktop. Then remote into it, and use the browser just as normal to download the stuff on it. We'll help you with moving the data off it to your local afterwards. Critically the machine has to have as much storage as needed to store all of your download.

load more comments (1 replies)
[–] [email protected] 3 points 5 months ago

I was gonna suggest the same.

load more comments (1 replies)
[–] [email protected] 4 points 5 months ago (1 children)

Have you tried mounting the google drive on your computer and copying the files with your file manager?

[–] [email protected] 12 points 5 months ago (2 children)

From a search, it seems photos are no longer accessible via Google Drive and photos downloaded through the API (such as with Rclone) are not in full resolution and have the EXIF data stripped.

Google really fuck over anyone using Google Photos as a backup.

[–] [email protected] 5 points 5 months ago (1 children)

Yeah, with takeout, there are tools that can reconstruct the metadata. I think Google includes some JSONs or something like that. It's critical to maintain the dates of the photos.

Also I think if I did that I would need double the storage, right? To sync the drive and to copy the files?

[–] [email protected] 1 points 5 months ago

From what I've read, I would not trust any process other than the takeout process. Do the album thing to split it up.

[–] [email protected] 2 points 5 months ago

Yeah, they really want to keep your data.

[–] [email protected] 20 points 5 months ago (1 children)

It doesn't have an option to split it?

When I did my Google takeout to delete all my pics from Google photos there was an option to split in like "one zip every 2gb"

[–] [email protected] 10 points 5 months ago (2 children)

The first time I tried it in the two gigabyte blocks. The problem with that is I have to download them one or two at a time. It's not very easy to do over the course of a week on a normal internet connection. Keep in mind, I also have a job.

I got about 50 out of 60 files before the one week timer reset and I had to start all over.

[–] [email protected] 13 points 5 months ago

Apparently you can save it to Google drive then download the Google drive program and make that folder available offline so it downloads it to the computer.

  1. When you setup the Google Takeout export choose Save in a Google Drive folder

  2. Install the Google Drive PC client (Drive for desktop)

  3. It will create a new drive (i.e. G:) in your explorer. Right click on the takeout folder and select "Make available offline". All files in that folder will be downloaded by the Google Drive Desktop in the background, and you will be able to copy to another location, as they will be local files.

[–] [email protected] 10 points 5 months ago (1 children)

You could look into using a download manager. No reason for you to manually start each download in sequence if there's a way to get your computer to automatically start the next as soon as one finishes.

[–] [email protected] 2 points 5 months ago (3 children)

Any recommendations? Windows or Linux?

[–] [email protected] 3 points 5 months ago

Definitely recommend Motrix:

https://motrix.app/

If the Google download link supports it, it should be fairly resistant to interruptions. If it doesn't, this might not help much, but you should still use this instead of just a browser.

I haven't tried to download a Google takeout, so you might need to get clever with how you add the download link to it.

If you just can't get it to work, you can try getting the browser extension to automatically send all downloads to Motrix. There is some setup required, though:

https://github.com/gautamkrishnar/motrix-webextension

Good luck!

[–] [email protected] 7 points 5 months ago

jDownloader

It is the most patient downloader I know.

[–] [email protected] 1 points 5 months ago

I used uGet on windows, and it was fairly smooth. Not google, but an equally annoying large download. I believe it's on Linux as well.

[–] [email protected] 112 points 5 months ago (1 children)

There's no financial incentive for them to make is easy to leave Google. Takeout only exists to comply with regulations (e.g. digital markets act), and as usual, they're doing the bare minimum to not get sued.

[–] [email protected] 11 points 5 months ago* (last edited 5 months ago) (1 children)

Or why is Google Takeout as good as it is? It's got no business being as useful as it is in a profit-maximizing corpo. 😂 It can be way worse while still technically compliant. Or expect Takeout to get worse over time as Google looks into undermaximized profit streams.

[–] [email protected] 25 points 5 months ago* (last edited 5 months ago) (1 children)

Probably because the individual engineers working on Takeout care about doing a good job, even though the higher-ups would prefer something half-assed. I work for a major tech company and I've been in that same situation before, e.g. when I was working on GDPR compliance. I read the GDPR and tried hard to comply with the spirit of the law, but it was abundantly clear everyone above me hadn't read it and only cared about doing the bare minimum.

[–] [email protected] 7 points 5 months ago* (last edited 5 months ago)

Most likely. Plus Takeout appeared way before Google was showing any profit maximization signs and didn't even hold the monopoly position it does hold today.

load more comments
view more: ‹ prev next ›