Selfhosted

You would need to run the LLM on the system that has the GPU (your main PC). The front-end (typically a WebUI) could run in a docker container and make API calls to your LLM system. Unfortunately that requires the model to always be loaded in the VRAM on your main PC, severely reducing what you can do with that computer, GPU-wise.

No?

Do you have any links or guides that you found helpful? A friend wanted to try this out but basically gave up when he realized he'd need an Nvidia GPU.

Wasnt there a solution by AMD or someone close to them implementing a translation of CUDA for AMD hardware?

Very technical vs not can be very subjective.
It can be a 50 year old sysadmin vs Adam I pulled from the street or a graybeard linux admin vs a beginner sysadmin only in it for thr career instead of the passion (those can be very non-technical but good problem solver folks)

I know my comparison is flawed

you hear that said about AI because companies are desperately throwing more and more resources at it to get 0.3% better results, and people are collectively running an insane amount of prompts all the time.

but on a personal level it's not really any different from any other computations, people render videos all the time and no one complains about the resource usage from that, because companies aren't trying to sell bloated video rendering services to gardening businesses.

Not really, it uses some GPU power when it's actively generating a response, but otherwise it just sits idle.

I've been testing Ollama in Docker/WSL with the idea that if I like it I'll eventually move my GPU into my home server and get an upgrade for my gaming pc. When you run a model it has to load the whole thing into VRAM. I use the 8gb models so it takes 20-40 seconds to load the model and then each response is really fast after that and the GPU hit is pretty small. After I think five minutes by default it will unload the model to free up VRAM.

Basically this means that you either need to wait a bit for the model to warm up or you need to extend that timeout so that it stays warm longer. That means that I cannot really use my GPU for anything else while the LLM is loaded.

I haven't tracked power usage, but besides the VRAM requirements it doesn't seem too intensive on resources, but maybe I just haven't done anything complex enough yet.

You probably want 48gb of vram or more to run the good stuff. I recommend renting GPU time instead of using your own hardware, via AWS or other vendors - runpod.io is pretty good.

Together.ai has a number of uncensored models too. I’ve found that those are so cheap that it’s not worth trying to self just models unless you really need more privacy.

Open WebUI now has a docker environment variable so you can, by default, turn off the login page. You just declare it when you’re spinning up the container and you’re good to go.