for coding tasks you need web search and RAG. It's not the size of the model that matters, since even the largest models find solutions online.
this post was submitted on 16 Jun 2025
11 points (100.0% liked)
Selfhosted
48681 readers
1676 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
Any suggestions for solutions?
Open webui lets you install a ton of different search providers out of the box, but you do need sn API key for most and I haven't vetted them
I'm trying to get Kagi to work with Phi4 and not having success.
Thanks, when I get some time soon, I'll have another look at it and cherry ai with a local install of ollama
Not on top of my head, but there must be something. llama.cpp and vllm have basically solved the inference problem for LLMs. What you need is a RAG solution on top that also combines it with web search.
I have it roleplay scenarios with me and sometimes I verbally abuse it for fun.
Weirdly I'm polite to all LLMs, but Gemini sets me off and I end up yelling at it.
it's just so pushy and hard to remove. it's asking for abuse.
Learning/practice, and any use that feeds in sensitive data you want to keep on-prem.
Unless you’re set to retire within the next 5 years, the best reason is to keep your resume up to date with some hands-on experience. With the way they’re trying to shove AI into every possible application, there will be few (if any) industries untouched. If you don’t start now, you’re going to be playing catch up in a few years.
As cool and neato as I find AI to be, I haven't really found a good use case for it in the selfhosting/homelabbing arena. Most of my equipment is ancient and lacking the GPU necessary to drive that bus.
7b is the smallest I've found useful. I'd try a smaller quant before going lower, if I had super small vram.
It'll work for quick bash scripts and one-off things like that. But there's not usually enough context window unless you're using a 24G GPU or such.
Yeah shell scripts are one of those things that you never remember how to do something and have to always look it up!
Snippets are a great use.
I use StableCode on my phone as a programming tutor for learning Python. It is outstanding in both speed and in accuracy for this task. I have it generate definitions which I copy and paste into Anki the flashcard app. Whenever I'm on a bus or airplane I just start studying. Wish that it could also quiz me interactively.
Please be very careful. The python code it'll spit out will most likely be outdated, not work as well as it should (the code isn't "thought out" as if a human did it.
If you want to learn, dive it, set yourself tasks, get stuck, and f around.
I know what you mean. All the code generated with ai was loaded with problems. Specifically it kept forcing my api keys into the code without using environmental variables. But for basic coding concepts it has so far been perfect. even a 3b model seemingly generates great definitions
I've run a few models that I could on my GPU. I don't think the smaller models are really good enough. They can do stuff, sure, but to get anything out of it, I think you need the larger models.
They can be used for basic things, though. There are coder specific models you can look at. Deepseek and qwen coder are some popular ones
I installed Llama. I've not found any use for it. I mean, I've asked it for a recipe because recipe websites suck, but that's about it.
you can do a lot with it.
I heated my office with it this past winter.
view more: next ›