Y'all, this is gonna be super broad, and I apologize for that, but I'm pretty new to all this and am looking for advice and guidance because I'm pretty overwhelmed at the moment. Any help is very, very appreciated.
For the last ~3 years, I've been running a basic home server on an old computer. Right now, it is hosting HomeAssistant, Frigate NVR, their various dependencies, and other things I use (such as zigbee2mqtt, zwave-js-ui, node-red, mosquitto, vscode, etc).
This old server has been my "learning playground" for the last few years, as it was my very first home server and my first foray into linux. That said, it's obviously got some shortcomings in terms of basic setup (it's probably not secure, it's definitely messy, some things don't work as I'd like, etc). It's currently on its way out (the motherboard is slowly kicking the bucket on me), so it's time to replace it, and I kind of what to start over (not completely - I've hundreds of automations in home assistant and node-red, for instance, that I don't want to have to completely re-write, so I intend to export/import those as needed) and do it "right" this time - at this point, I think this is where I'm hung up, paralyzed by a fear of doing it "wrong" and winding up with an inefficient, insecure mess.
The new server, I want to be much more robust in terms of capability, and I have a handful of things I'd really love to do: pi-hole (though I need to buy a new router for this, so that has to come later on unless it'd save a bunch of headache doing it from the get-go), NAS, media server (plex/jellyfin), *arr stuff, as well as plenty of new things I'd love to self-host like Trilium notes, Tandoor or Mealie, Grocy, backups of local PCs/phones/etc (nextcloud?)... obviously this part is impossible to completely cover, but I suspect the hardware (list below) should be capable?
I would love to put all my security cameras on their own subnet or vlan or something to keep them more secure.
I need everything to be fully but securely accessible from outside the network. I've recently set up nginx for this on my current server and it works well, though I probably didn't do it 100% "right." Is something like Tailscale something I should look to use in conjuction with that? In place of? Not at all?
I've also looked at something like Authelia for SSO, which would probably be convenient but also probably isn't entirely necessary.
Currently considering Proxmox, but then again, TrueNAS would be helpful for the storage aspect of all this. Can/should you run TrueNAS inside Proxmox? Should I be looking elsewhere entirely?
Here's the hardware for the recently-retired gaming PC I'll be using:
https://pcpartpicker.com/list/chV3jH
Also various SSDs and HDDs.
I'm in this weird place where I don't have too much room to play around because I want to get all my home automation and security stuff back up as quickly as possible, but I don't want to screw this all up.
Again, any help/advice/input at all is super, super appreciated.
My best advice is use that your old setup hasn't died yet while you can. I.e. start now and setup Proxmox because it's vastly superior to TrueNAS for the more general type hardware you have and then run a more focused NAS project like Openmediavault in a proxmox VM.
My recommendation, from experience, would be to setup a VM for anything touching hardware directly, like a NAS or Jellyfin (if you want to have GPU assisted transcoding) and I personally find it smoothest to run all my Docker containers from one Docker dedicated VM. LXCs are popular for some but I strongly dislike how you set hardware allocations for them, and running all Docker containers in one LXC is just worse than doing it in a VM. My future approach will be to move to more dedicated container setup as opposed to the VM focused proxmox but that is another topic.
I also strongly recommend using portainer or similar to get a good overview of your containers and centralize configuration management.
As for external access all I can say is do be careful. Direct internet exposure is likely a really bad idea unless you know what you're doing and trust the project you expose. Hiding access behind a VPN is fairly easy if your router has a VPN server built in. And WireGuard (like Netbird / tailscale / Cloudflare tunnels etc all use) is great if not.
As for authentication it's pretty tricky but well worth it and imo needed if you want to expose stuff to friends/family. I recommend Authentik over other alternatives.
Why would you virtualize a file server? You want direct access to disks for raid and raid-like things.
There's absolutely no issues whatsoever with passing through hardware directly to a VM. And Virtualized is good because we don't want to "waste" a whole machine for just a file server. Sure dedicated NAS hardware has some upsides in terms of ease of use but you also pay an, imo, ridiculous premium for that ease. I run my OMV NAS as a VM on 2 cores and 8 GB of RAM (with four hard drives) but you can make do perfectly fine on 1 Core and 2 GB RAM if you want and don't have too many devices attached / do too many iops intensive tasks.
Hmm. I strongly disagree. You've created a new dependency now for the fileserver to come up - a system that many other services will also depend on and which will likely contain backups.
A dedicated system is less likely to fail as it won't be sensitive to a bad proxmox upgrade or some other VM exhausting system resources on the host.
You can get cheap hardware if cost is an issue.
Sure, I'm not saying its optimal, optimal will always be dedicated hardware and redundancy in every layer. But my point is that you gain very little for quite the investment by breaking out the fileserver to dedicated hardware. It's not just CPU and RAM needed, it's also SATA headers and an enclosure. Most people doing selfhosted have either one or more SBCs and if you have more than one SBC then yeah the fileserver should be dedicated. The other common thing is having an old gaming/office PC converted to server use and in that case Proxmox the whole server and run NAS as a VM makes the most sense instead of buying more hardware for that very little gain.