this post was submitted on 07 Feb 2024
196 points (95.4% liked)
Technology
59405 readers
2528 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter.
1 billion parameters ~ 2 Billion bytes ~ 2 GB.
From the name, this model has 72 Billion parameters, so ~144 GB of VRAM
Llama 2 70B with 8b quantization takes around 80GB VRAM if I remember correctly. I’ve tested it a while ago.
Though with quantisation you can get it down to like 30GB of vram or less.
Any idea what 8Q requirements would be? Or 4 or 5?
https://huggingface.co/senseable/Smaug-72B-v0.1-gguf/tree/main
About 44GB and 50GB for the Q4 and 5. You'd need quite some extra to fully use the 32k context length.
It's been discovered that you can reduce the bits per parameter down to 4 or 5 and still get good results. Just saw a paper this morning describing a technique to get down to 2.5 bits per parameter, even, and apparently it 's fine. We'll see if that works out in practice I guess
I'm more experienced with graphics than ML, but wouldn't that cause a significant increase in computation time, since those aren't native types for arithmetic? Maybe that's not a big problem?
If you have a link for the paper I'd like to check it out.
You can take a look at exllama and llama.cpp source code on github if you want to see how it is implemented.
My understanding is that the bottleneck for the GPU is moving data into and out of it, not the processing of the data once it's in there. So if you can get the whole model crammed into VRAM it's still faster even if you have to do some extra work unpacking and repacking it during processing time.
The paper was posted on /r/localLLaMA.
Ok but will this run on my TI-83? It's a + model.
Only if it’s silver.
Dang. So close.
My 83 was ganked by some kid I knew so my folks bought me a silver. He denied it. I learned that day to write my name in secret spots.
That kid you knew was a dick. At least he taught you a valuable lesson, I guess.
He absolutely was a dick. I stopped being mates with him after that. My school was like “yeah the cameras didn’t work that day actually”
Leads me to believe that the cameras never actually worked.
I believe that. Or they just didn’t want to be responsible for dealing with theft. Both ways make perfect sense to me.
Very true. Saying the cameras don't work is enough to keep the cops out of it. After all, hearsay between kids is never taken seriously. No footage, means no proof, means no cops. Then they can keep the facade of the school being crime-free.
Absolutely my dude! My school pretended the cams didn’t work when my calculator was stolen… they worked fantastically the next week.
That is some BS! Honestly, I probably would have dealt with that kid schoolyard-style (though, I'm not proud of that as an adult). My school never pretended the cameras were out of service. They just got rid of them after a teacher was accused of inappropriate behavior with a senior. Claimed they never worked. He "left for another state" later that week. I'm sure it was unrelated...
Hahaha he was a lil dude and he came from a rough family. Plus I was like 115-120lbs at the time, I ain’t no fightgurl. I knew my folks would get me a better one (it wasn’t even a plus model…) so I was easy on him.
Still stopped talking to him.
Also at your last coupe sentence… oh dear me. That’s… not good.
Sounds like he was trying to get what he needed to learn the maths, even if screwed over his friends. Hopefully he learned from it, improved himself, and didn't do something like that again. After all, we all did stupid things growing up. You seemed to have handled it really well. Good on you!
Yeah, it was really bad. And I wish I could say it was the first of its kind at that school. Or the last. But I wasn't either. That principal was a joke who had his boys club going strong. Last I heard, he was black-listed from all educational institutions, public and private. I hope that was a permanent ban. I'm not sure about the others who committed these acts, though.
no. but put this clustering software i wrote in ti-basic on 40 million of them? still no