this post was submitted on 13 May 2025
1 points (100.0% liked)

AI

5164 readers
6 users here now

Artificial intelligence (AI) is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans and animals, which involves consciousness and emotionality. The distinction between the former and the latter categories is often revealed by the acronym chosen.

founded 4 years ago
1
Local LLMs: 16gb vram model advice and quantization question (lemmy.ml)
submitted 1 month ago by [email protected] to c/[email protected]
0 comments fedilink hide all child comments
 

So, I have a 16GB vram GPU (4070 ti Super) and 32GB DDR4 RAM. The RAM is slow af and thus I tend to run models fully on GPU.

I can easily run up to 21b-ish models with Q4, sometimes high Q3.

I am testing various models out there but I was wondering if you guys have any reccommendation.

I am also really interested in understanding if quantization really decrease the model quality so much. Like, It would be better to have a Q6 12b model (like Gemma 3 12b), a Q2_K_L 32b model (such as QwQ 32b) or a Q3_XS model (such as Gemma 3 27b)?

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here