this post was submitted on 25 Dec 2023
25 points (93.1% liked)
People Twitter
4980 readers
2057 users here now
People tweeting stuff. We allow tweets from anyone.
RULES:
- Mark NSFW content.
- No doxxing people.
- Must be a tweet or similar
- No bullying.
- Be excellent to each other.
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I want said AI to be open source and run locally on my computer
This technology will be running on your phone within the next few years.
Yeah if your willing to carry a brick or at least a power bank (brick) if you don't want it to constantly overheat or deal with 2-3 hours of battery life. There's only so much copper can take and there are limits to minaturization.
It's not like that though. Newer phones are going to have dedicated hardware for processing neural platforms, LLMs, and other generative tools. The dedicated hardware will make these processes just barely sip the battery life.
wrong.
if that existed, all those AI server farms wouldn't be so necessary, would they?
dedicated hardware for that already exists, it definitely isn't gonna be able to fit a sizeable model on a phone any time soon. models themselves require multiple tens of gigabytes of storage space. you won't be able to fit more than a handful on even a 512gb internal storage. the phones can't hit the ram required for these models at all. and the dedicated hardware still requires a lot more power than a tiny phone battery.
Those server farms are because the needs of corporations might just be different from the needs of regular users.
I'm running a 8 GB LLM model locally on my PC that performs better than 16 GB models from just a few months ago.
It's almost as if technology can get better and more efficient over time.
I can run a pretty alright text generation model and the stable diffusion models on my 2016 laptop with two GTX1080m cards. You can try with these tools: Oobabooga textgenUi
Automatic1111 image generation
They might not be the most performant applications but they are very easy to use.
You seem to have missed the point a bit
“I wish I had X”
“Here’s X”
What point was missed here?
The post "I wish X instead of Y"
The comment: "And run it [X] locally"
The next comment: "You can run Y locally"
Also the one I told this literally admitted that I was right and you're arguing still
Funny how these comments appeared only today in my instance, I guess there are some federation issues still
new phone who dis?
It's getting there. In the next few years as hardware gets better and models get more efficient we'll be able to run these systems entirely locally.
I'm already doing it, but I have some higher end hardware.
Could you please share your process for us mortals ?
Stable diffusion SXDL Turbo model running in Automatic1111 for image generation.
Ollama with Ollama-webui for an LLM. I like the Solar:7b model. It's lightweight, fast, and gives really good results.
I have some beefy hardware that I run it on, but it's not necessary to have.