this post was submitted on 09 Aug 2024
30 points (96.9% liked)

Selfhosted

40173 readers
652 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
30
Trying to get advice for photo/files storing/syncing (lemmy.sdf.org)
submitted 3 months ago* (last edited 3 months ago) by [email protected] to c/[email protected]
33 comments fedilink hide all child comments
 

Hi. Sorry for the vague title. Nowadays I'm using multiple computers and get to need files and such from other machines pretty often. My music and photos library has also increased and it's getting much harder to maintain with it being scattered across many machines. Basically I'm trying to have a photo library and plain files(documents, music, etc) shared across computers.

For plain files I'm thinking nfs+samba would be the best approach, but there are problems. They have speed issues, but as I can't afford large space for all my machines I can't keep a full rsync'd local copies everywhere too.

The photos are my bigger concern, as I'm looking for a tagging feature. A plain directory structure would be easy to sync but those tags would differ by programs.. desktop programs like digikam or xnview(sadly proprietary) would work well if I didn't need syncing, but I'm not sure if they'd work reliably with all their configs/files stored over nfs. Plus, these programs would have incompatibilities by platform and not work at all on android.

Web based solutions like Immich or NextCloud Photos appear to be pretty famous nowadays, but I'm not sure about them as well. They seem to be overkill for my purpose, and those mostly tend to be very new & i'm not too sure about their future, as they store tags and such on their own formats.

Edit: Oops, forgot to say. I have multiple servers right now, one offsite running FreeBSD, another running Devuan, and one at home running FreeBSD.

I'd love to hear how others are maintaining their system. Thanks for reading.

top 33 comments
sorted by: hot top controversial new old
[–] [email protected] 3 points 3 months ago

Maybe I have just had back luck but syncing my files across all of my devices has always burned me at some point. No matter what software I have used I have overwritten something by accident or I try and delete something and it doesnt get deleted on all device. Or get a bunch of conflicting files and now i need to figure out what file I want etc...

I do use some syncing but it is mostly between only two devices. Often times it is only a 1 way sync. For example photos on my phone get synced to my NAS automatically.

For me keeping all of my files in one place is the way to go. I just have everything on a NAS. (TrueNAS) All of my devices connect to that and i just edit them directly over the network.

I have a VPN for remote access.

[–] [email protected] 7 points 3 months ago (1 children)

As far as photo librabry, you can compare the different solutions at https://github.com/meichthys/foss_photo_libraries

[–] [email protected] 2 points 3 months ago (1 children)

Thanks. I've actually already seen it, but it wasn't enough as it only listed web hosted solutions and I found most of them lacking for my needs. Thankfully others' comments below are pretty inspiring :)

[–] [email protected] 1 points 3 months ago (1 children)

I don't see why you think those are web based, usually people run this at home but glad you find what you are looking for

[–] [email protected] 1 points 3 months ago

By web based I mean they're not a native desktop program and such.

[–] [email protected] 2 points 3 months ago* (last edited 3 months ago) (1 children)

Lots of good advice here.

The best place to start would be to ignore what you currently have, for the moment, and think about your requirements, at a high level.

In the corporate world, we start with Business/User Requirements - think "what does a user need to be able to do", these are pretty abstract things like:

Have all photos accessible on mobile

Have all photos accesible by App A

Have all photos accesible by App B

Etc

Then take all those User requirements as a guide to the Functional/System/Technical requirements (what solution meets which requirement?)

I kind of just focus on data stability myself (3 local copies, one cloud backup, with local copies being sync'd manually, to act as a sort of buffer from my own fuckups), and implement different solutions for each requirement/system.

Like Syncthing on Windows/Linux/Android, because it just works for regular sync, Resilio on my Media server and Mobile devices, because it has Selective Sync, Tailscale on mobile devices and a single server at home for remote access and remote control.

[–] [email protected] 1 points 3 months ago* (last edited 3 months ago) (1 children)

Thanks. I should try making my requirements clearer. I'm curious, is there a reason you use Resilio despite it being proprietary over other solutions(like rsync scripts)?

[–] [email protected] 1 points 3 months ago

Selective Sync is the one feature that Resilio provides that I use.

It enables me to grab any file, using any device, at any time, from anywhere, over any network, simply and quickly. I really wish Syncthing had this capability. Oh well.

So if I'm traveling, I can download a movie from my library with my phone or iPad while connected to hotel wifi. The Resilio UI is simpler than turning on Tailscale, launching a file explorer connecting to my server, then copying. Plus it's a robust sync job - I don't have to think about it, if the network goes away, Resilio will pick up the sync again when it can. On my mobile devices, Resilio is only run if started by the user, but Syncthing runs all the time to ensure stuff like photos, downloaded files, Backups, etc, are sync'd to my server.

I switched from Resilio to Syncthing for everything else (mobile devices mostly, since I can use other tools on laptops), because it's much lighter to run. Resilio is hell on mobile devices if you have a large library, as it keeps the index in memory, while Syncthing uses a file-based approach for indexes. Resilio is also resource intensive on my server - again because of the large media library.

[–] [email protected] 7 points 3 months ago* (last edited 3 months ago) (1 children)

I use Syncthing and a bunch of rsync scripts to keep my machines in sync. The stuff I want synced continuously is handled by Syncthing. Other stuff is synced on a daily basis using the rsync scripts and anachron. For Photos, I use PhotoPrism. I simply sync the Photos from my smartphone to a folder and make PhotoPrism scan it on a regular basis using ofelia. For cameras, I need to copy the photos manually, but I don't think there's a way around that.

[–] [email protected] 3 points 3 months ago

Seems like I'll be going on a setup pretty similar to this. Thanks!

[–] [email protected] 3 points 3 months ago (1 children)

I use digikam with sidecar files on my main photo editing PC.

We sync this directory with a media server on the local network that enables all of the local devices to access the photos and tags. In theory, it means we could run digikam on another device as well, and sync data between them via updates to the sidecar files, but in practice, we don't do this and the media server is effectively read only.

Then, we sync the media server images with a photoprism instance that we have running on external hosting. Photoprism recognises the keywords and sidecar data from digikam, which lets me search and access the images from anywhere.

[–] [email protected] 2 points 3 months ago

That's great, I never knew Photoprism can detect and work with digikam's sidecar files. Thanks for sharing.

[–] [email protected] 4 points 3 months ago (1 children)

I use nextcloud for syncing between different computers, because I tend to have different machines that are far separated geographically, and it works well. I put all my home folders on each computer into the nextcloud directory so I have all the same files everywhere I go and if I don't have one of my computers I can still log in and access those files.

I used to use nextcloud as my solution for everything, but a big problem with photos is it isn't really very navigable, and a problem with nextcloud as a general platform is everything is a plugin so if the plugin doesn't get updated you can be stuck on an older version of the software which carries its own risks. As well, given the interface, You have your media but you can't really go back and look at it. What I did instead is I set up a library in jellyfin with all my photos sorted into directories, and you can scroll and navigate through them fairly intuitively. I pulled my data out of google and facebook before deleting the accounts and so had many many photos but no way to really enjoy them, but that solution worked really well for me and I've been able to look at my old photos easily.

[–] [email protected] 1 points 3 months ago (1 children)

Nice. So you have home folders of each machines in NextCloud? Are you using their official client to sync automatically or on a certain schedule? I never knew Jellyfin can be used as a photo library. Thanks :)

[–] [email protected] 2 points 3 months ago (2 children)

The way I've got it set up is I have a Nextcloud\Desktop, Nextcloud\Downloads, Nextcloud\Documents, Nextcloud\Pictures, and Nextcloud\Videos folder, and on each machine I use I point each of those points in windows to use the folder in the nextcloud folder instead of my users folder, then I run the official client to sync the entire nextcloud folder. By doing that, whichever computer I'm on I've got the same stuff in my main folders and anything else I have I can keep in the nextcloud folder. I've also got it on my mobile device just to automatically upload new pictures to the InstantUploads folder, but the app is a bit limited.

I live equally on the road working as at home, and I've got completely different computers for home and travel, so in this way I've always got all my files available since once I start up the computer it automatically starts pulling the local files. If you don't want a full copy of everything on both machines, I think you can tell it to just create links of the files and the client will download the files from the server as they're required, but I prefer having a local copy of the files myself.

[–] [email protected] 2 points 3 months ago

I see. Thanks for sharing your setup :)

[–] [email protected] 2 points 3 months ago

When I'm using linux, I do something similar, I just sync'd my home folder as my nextcloud directory and that similarly made all my files available.

[–] [email protected] 9 points 3 months ago* (last edited 3 months ago) (1 children)

I would honestly consolidate your media and start storing it on a NAS, then have everything sync to the main shares on the NAS.

[–] [email protected] 1 points 3 months ago (2 children)

I wish lol. I don't have a dedicated(?) hardware, just multiple laptops in various locations. They've ran pretty stable for the last few years of 24/7 use though. I'm thinking having some files synced to local would be enough backup, compared to none i have now (aside from few manual work).

[–] [email protected] 1 points 3 months ago (1 children)

How much data in total do you have? Perhaps 2 or 3 external drives (each large enough to hold all your data) on those laptops could bridge the gap for now. Externals are relatively inexpensive, and using something like Syncthing, they could stay in sync.

I took this approach to consolidate my data to free up drives from machines so I could build a NAS running Proxmox. Then copied that data to the NAS, which is the authoritative data store, the other drives now act as local duplicates.

Alternatively, upgrade the drives in the laptops (depending on how much data you have).

Also, keep in mind growth - once you have your data sorted, watch it grow and use that to predict your need for new storage.

[–] [email protected] 1 points 3 months ago (1 children)

I don't have that much data now, I usually lack storage (or computer equipments in general) so I tend to not store much. Those laptops have about 1TB storage each so that would hopefully be enough for me now. As they're old and have ODD I can also have two drives on one, but I'm not sure if that's needed for me now. I'd avoid external drives as they aren't really reliable.

[–] [email protected] 1 points 3 months ago* (last edited 3 months ago)

Externals are perfectly reliable, it's how you manage things that matters.

I've had (over 30+ years of machines now) as many internal drives fail as externals - which is to say not many.

The key with externals is they lack cooling, so don't hammer them like an internal. I have some externals that are 10+ years old. One is currently my authoritative drive as I reconfigure my setup. It sits by itself and I tacked on an old case fan to keep it cool if needed. (That said, I do prefer to not use externals, the enclosure is another point of failure, and USB connectivity can be a bit unreliable).

I'd look at your total data, consolidate it into a single drive and folder structure, and duplicate that across the drives you have, using the first one as the authoritative drive.

Then get a cloud backup like storj.io on that authoritative drive, so you get local duplication and cloud backup.

[–] [email protected] 5 points 3 months ago (1 children)

So use syncthing on all your devices. It will let you selectively sync folders.

[–] [email protected] 4 points 3 months ago (1 children)

syncthing is great.

[–] [email protected] 1 points 3 months ago

Seems like Syncthing is loved by many :) Not too sure how much it'd differ from rsync scripts, but I'll give it a try. Thanks!

[–] [email protected] 6 points 3 months ago (2 children)

I use Nextcloud and Immich and would recommend both. Immich might be a bit overkill, but it's also well maintained, feature-rich and has a large community. It's super easy to set up and works great.

[–] [email protected] 1 points 3 months ago (1 children)

What is the cost associated with Immich? I keep hearing about it, but I still don't know how that really works I guess

[–] [email protected] 1 points 3 months ago

What's costs do you mean? It's free and open source.

[–] [email protected] 2 points 3 months ago (2 children)

Thanks for replying. Is there a reason you use Immich over NextCloud's NextCloud Photos? Also I've occasionally heard NextCloud is pretty slow, is it okay for you? You're using their official client program to sync files?

[–] [email protected] 2 points 3 months ago (1 children)

Immich has image and facial recognition by default and a very neat Android app. Also it's running in my home server, which has more power if Immich needs it. In that case I'd say software should serve one purpose and serve that good. Immich is just for picture management and does that very good. Nextcloud is a cloud and the Photos app is just a small extra that can't compete with a full-fledged software. Nextcloud runs fine on my Raspberry Pi 4, but it's only used by me and three friends. It's mainly limited by your network speed and disk speed I'd say. And I'm using an external hard drive without issues.

[–] [email protected] 2 points 3 months ago

I see. The lack of a desktop client seems to be Immich's biggest con. I guess NextCloud wouldn't be the best choice here. Thanks for replying.

[–] [email protected] 4 points 3 months ago (1 children)

Nextcloud Photos performs okay, but the interface is very ‘meh’. Plus, the mobile client’s sync is a little unstable. On iOS, there’s no background sync at all.

[–] [email protected] 1 points 3 months ago

Just curious, but is there anything that provides background sync on iOS except iCloud?