I use it for everything. I did move to ollama to podman and I replaced webUI with Alpaca
this post was submitted on 24 Jul 2024
53 points (90.8% liked)
Selfhosted
39085 readers
333 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
I use local AI for coding (more recently) and ML Photo storage facial recognition and security camera object detection (been using the later 2 for years now actually, don't want that kind of info out on someone else's cloud training on my images)
Roleplay (text adventures), a (stupid but occasionally funny) dungeon master, translation and help with creativity. These are the use cases I found. If you don't need that, you might get rid of it.
I use the Continue VS Code plugin with Ollama to use a couple of different models (deepseek-coder-v2 & starcoder2) to recreate a local only Github Copilot type experience for coding. This is on an M1 Apple Silicon though. For autocomplete the generation needs to be pretty brisk - I'm not sure how that would go in a VM without a GPU.
How well does the M1 chip keep up? What size models are you running with it? Interested in getting an M1 laptop and I am curious.
starcoder2:latest f67ae0f64584 1.7 GB 3 days ago
phi3:latest d184c916657e 2.2 GB 3 weeks ago
deepseek-coder-v2:latest 8577f96d693e 8.9 GB 3 weeks ago
llama3:8b-instruct-q8_0 1b8e49cece7f 8.5 GB 3 weeks ago
dolphin-mistral:latest 5dc8c5a2be65 4.1 GB 3 weeks ago
codeqwen:latest df352abf55b1 4.2 GB 3 weeks ago
llama3:latest 365c0bd3c000 4.7 GB 4 weeks ago
I mostly use starcoder2 with Continue for code autocomplete, the big deepseek coder is a bit slow (I can feel it thinking), but it and the regular llama3 are good for chatbot type programming questions.
I don't really have anything to compare the M1 performance to. I guess the 8GB models output text a little slower than the web versions of the same models, and the 4GB ones about the same. Using ollama in the terminal, there's sometimes a 0.5-2 second pause before it starts outputting. Not with phi3 though - it's surprisingly snappy for the quality of answers.
I have a 4070 sitting around collecting dust that I got from a trade, I've been thinking about setting it up with whispr and TTS and having a way to talk to my house.
I have a couple of smart home integrations, mostly air conditioning, light switches, security, and doors.
What I would like would be to have a few speakers on the walls that can talk to my server where I can say something like, hey computer, turn on the lights in the dining room and the lights in the dining room would turn on without transmitting that information to Google or Amazon.
I am really curious if you can get the traditional smart functionality along with a LLM. Maybe have some sort of keyword the prompts the AI. You also could write a custom generated system prompt that includes the weather, time and any other information
You can try integrating with SEPIA. Not that I used it befote but it surely looks promising.
Wonderful. I'll check it out. Thank you!
https://github.com/hendkai/paperless_sort_low_quality_ollama let ai tag your paperless ngx files base on content.
Think of LLMs like a stupid office worker. You wouldn't rely on them to make critical decisions, but they're valuable for tedious stuff.
For example, my calendar changed the way to enter new events breaking my workflow. Now I just type out a skeletal schedule and have LLM convert that into a .csv that I import.
I'm thinking of Ripping my CD collection again. I'm researching a way to use a LLM to tidy up the metadata.
I had a folder full of random stuff I've saved for years. Had a LLM organize and categorize it for me. I had to tweak the prompt enough that this was a medium difficulty task, but still way easier than doing it manually.
Can you share some info on how you did that folder organization? Did you provide the AI with a list of files?
I'm thinking of Ripping my CD collection again. I'm researching a way to use a LLM to tidy up the metadata.
If you ever figure out how to use AI to determine the genre(s) of a song, let me know. Have been looking for something like that for quite a while.
Nextcloud Recognize is supposed to do that, but I haven't tried it. You might try looking down that road.
Thanks for the tip! I took a look and it seems like Recognize uses this: https://github.com/jordipons/musicnn
Last update was 4 years ago but will give it a try this weekend.
Ollama without a GPU is pretty useless unless you're using with Apple silicon. I'd just get rid of it until you get a GPU.
Works fine on an 11th Gen i5. Not fast but not slow
I have never tested in on Apple silicon but it works fine on my laptop
What are your laptop specs?
Intel 12th gen i5
CPU is only one factor regarding specs, a small one at that. What kind of t/s performance are you getting with a standard 13B model?
I don't have enough ram to run a 13b. I just stick to Mistral 7b and it works fine.
view more: next ›