this post was submitted on 19 May 2024

72 points (98.6% liked)

Linux

48157 readers

619 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
No misinformation
No NSFW content
No hate speech, bigotry, etc

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago

MODERATORS

[email protected]

How to speed up accessing lots of files on another computer? Some kind of local cache? (discuss.tchncs.de)

submitted 5 months ago by [email protected] to c/[email protected]

63 comments fedilink hide all child comments

Title is TLDR. More info about what I'm trying to do below.

My daily driver computer is Laptop with an SSD. No possibility to expand.

So for storage of lots n lots of files, I have an old, low resource Desktop with a bunch of HDDs plugged in (mostly via USB).

I can access Desktop files via SSH/SFTP on the LAN. But it can be quite slow.

And sometimes (not too often; this isn't a main requirement) I take Laptop to use elsewhere. I do not plan to make Desktop available outside the network so I need to have a copy of required files on Laptop.

Therefor, sometimes I like to move the remote files from Desktop to Laptop to work on them. To make a sort of local cache. This could be individual files or directory trees.

But then I have a mess of duplication. Sometimes I forget to put the files back.

Seems like Laptop could be a lot more clever than I am and help with this. Like could it always fetch a remote file which is being edited and save it locally?

Is there any way to have Laptop fetch files, information about file trees, etc, located on Desktop when needed and smartly put them back after editing?

Or even keep some stuff around. Like lists of files, attributes, thumbnails etc. Even browsing the directory tree on Desktop can be slow sometimes.

I am not sure what this would be called.

Ideas and tools I am already comfortable with:

rsync is the most obvious foundation to work from but I am not sure exactly what would be the best configuration and how to manage it.
luckybackup is my favorite rsync GUI front end; it lets you save profiles, jobs etc which is sweet
freeFileSync is another GUI front end I've used but I am preferring lucky/rsync these days
I don't think git is a viable solution here because there are already git directories included, there are many non-text files, and some of the directory trees are so large that they would cause git to choke looking at all the files.
syncthing might work. I've been having issues with it lately but I may have gotten these ironed out.

Something a little more transparent than the above would be cool but I am not sure if that exists?

Any help appreciated even just idea on what to web search for because I am stumped even on that.

(page 2) 14 comments

sorted by: hot top controversial new old

[–] [email protected] 4 points 5 months ago* (last edited 5 months ago) (3 children)

You have two problems.

Transferring between your laptop and desktop is slow. There’s a bunch of reasons that this could be. My first thought is that the desktops got a slow 100mbps nic or not enough memory. You could also be using something that’s resource intensive and slow like zfs/zpools or whatever. It’s also possible your laptops old g WiFi is the bottleneck or that with everything else running at the same time it doesn’t have the memory to hold 40tb worth of directory tree.

Plug the laptop into the Ethernet and see if that straightens up your problems.

You want to work with the contents of desktop while away from its physical location. Use a vpn or overlay network for this. I have a complex system so I use nebula. If you just want to get to one machine, you could get away with just regular old openvpn or wireguard.

E: I just reread your post and the usb is likely the problem. Even over 2.0 it’s godawful. See if you can migrate some of those disks onto the sata connectors inside your desktop.

load more comments (3 replies)

[–] [email protected] 3 points 5 months ago (2 children)

zerotier + rclone sftp/scp mount w/ vfs cache? I haven't tried using vfs cache with anything other than a cloud mount but it may be worth looking at. rclone mounts work just as well as sshfs; zerotier eliminates network issues

load more comments (2 replies)

[–] [email protected] 2 points 5 months ago (1 children)

Why is VPN to your main device not an option?

[–] [email protected] 1 points 5 months ago (2 children)

I don't know what that means

load more comments (2 replies)

[–] [email protected] 7 points 5 months ago (1 children)

Easiest for this might be NextCloud. Import all the files into it, then you can get the NextCloud client to download or cache the files you plan on needing with you.

[–] [email protected] 3 points 5 months ago (3 children)

hmm interesting idea. I do not get the idea that nextcloud is reliably "easy" as it's kind of a joke how complex it can be.

Someone else suggested WebDAV which I believe is the filesharing Nextcloud uses. Does Nextcloud add anything relevant above what's available from just WebDAV?

load more comments (3 replies)

[–] [email protected] 3 points 5 months ago

A few ideas/hints: If you are up for some upgrading/restructuring of storage, you could consider a distributed filesystem: https://wikiless.org/wiki/Comparison_of_distributed_file_systems?lang=en.

Also check fuse filesystems for weird solutions: https://wikiless.org/wiki/Filesystem_in_Userspace?lang=en

Alternatively perhaps share usb drives from 'desktop' over ip (https://www.linux.org/threads/usb-over-ip-on-linux-setup-installation-and-usage.47701/), and then use bcachefs with local disk as cache and usb-over-ip as source. https://bcachefs.org/

If you decide to expose your 'desktop', then you could also log in remote and just work with the files directly on 'desktop'. This oc depends on usage pattern of the files.

[–] [email protected] 4 points 5 months ago (6 children)

NFS and ZeroTier would likely work.

When at home NFS will be similar to a local drive, though a but slower. Faster than SSHFS. NFS is often used to expand limited local space.

I expect a cache layer on NFS is simple enough, but that is outside my experience.

The issue with syncing, is usually needing to sync everything.

load more comments (6 replies)

[–] [email protected] 2 points 5 months ago (1 children)

Have you tried setting up WebDAV? From what I know it has local cache support. I use it to access the files on my NAS remotely. Of course, I could be wrong, and my NAS came with it preinstalled so I'm not actually sure how to set it up manually.

[–] [email protected] 2 points 5 months ago

I've used WebDAV here and there. I found some aspects of set up frustrating so I tend to keep away from it except for smaller, short term use cases.

Does it do the caching thing or is it more of an alternative to SSH/SFTP?

If it's an alternative, what is the benefit?

IIRC WebDAV can be set up from inside certain filemanagers (like nautilus with an extension installed) or by using a web server like apache, or by using smaller stand alone services.

[–] [email protected] 5 points 5 months ago (1 children)

What you suggest sounds a lot like the “Briefcase” that was in Windows 9x. I don’t know of something similar, especially not something integrated into Linux.

The easiest way might be to setup SyncThing to share all of your different folders and then subscribe to those you need on your laptop. Just be aware that if you delete a file on your laptop it will also be deleted on your desktop on the next sync. Unsubscribe from the folder first before freeing up the disk space.

[–] [email protected] 2 points 5 months ago (1 children)

if you delete a file on your laptop it will also be deleted on your desktop on the next sync

This is my fear! I have done it before.... Forgetting something is synced and deleting what I thought was "an extra copy" only to realize later that it propagated to the original.

load more comments (1 replies)

[–] [email protected] 2 points 5 months ago (1 children)

Can you upgrade the desktop? What speed is your laptops WiFi?

I just use rsync manually until I have syncthig working but these don't really solve slowdown issues and aren't mounted. I would look into a better NIC and/or storage for the desktop or possibly your router.

Try using something like iperf to measure raw speed of the connection between your 2 systems, see if its what it should be (around 300-600mbps for wireless to wired locally) and try to narrow down where the bottleneck is.

[–] [email protected] 3 points 5 months ago (1 children)

A few weeks ago I put some serious time/brainpower into the network and got it waaaay smoother and faster than before. Finally implemented some upgraded hardware that has been sitting on a shelf for too long.

I tried iperf. Actually iperf3 because that's the first tutorial I found. Do you have any opinion on iPerf vs iperf3? On Desktop I ran:

iperf3 -s -p 7673

On Laptop I am currently doing some stuff I didn't want to quit so this may not be a totally fair test. I'll try re running it later. That said I ran:

 iperf3 -c desktop.lan -p 7673 -bidir

And what looks like a summary at the bottom:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   102 MBytes  86.0 Mbits/sec  152             sender
[  5]   0.00-10.00  sec   102 MBytes  85.6 Mbits/sec                  receiver

I actually have AnotherDesktop on the LAN also connected via ethernet. Going from Laptop ---> AnotherDesktop gets similar to the above.

However going AnotherDesktop ---> Desktop gets 10x better results:

[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   933 Mbits/sec                  receiver

Laptop has Intel Dual Band Wireless-AC 8260 who's Max Speed = 867 Mbps. It probably isn't the bottleneck. Although with the distro running at the moment (Fedora) I have a LOT of problems with everything so possibly things aren't set up ideally here.

I still didn't upgrade the actual wireless access point for the network; don't recall what the max speed is for current WAP but could be around 100Mbps.

So this is an interesting path to optimize. However I am still interested in solving the original problem because even when I am directly using Desktop, things are slow. I do not really want to upgrade it is I can get away with a software solution. There are many items on my list of projects and purchases that I'd rather concentrate on.

load more comments (1 replies)

load more comments