this post was submitted on 01 Feb 2024
1 points (100.0% liked)
Free Open-Source Artificial Intelligence
2886 readers
1 users here now
Welcome to Free Open-Source Artificial Intelligence!
We are a community dedicated to forwarding the availability and access to:
Free Open Source Artificial Intelligence (F.O.S.A.I.)
More AI Communities
LLM Leaderboards
Developer Resources
GitHub Projects
FOSAI Time Capsule
- The Internet is Healing
- General Resources
- FOSAI Welcome Message
- FOSAI Crash Course
- FOSAI Nexus Resource Hub
- FOSAI LLM Guide
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Do you know if there are any plans to quantize it? I'd love to test it, but my 3090 can't handle 70b models without quantization, unfortunately.
Only quantized versions of the model were leaked. If you see any unquantized version of it then it's something which was recreated from these, and not the original model. People have also requanted it from GGUF to EXL2 and probably other formats too.
There are quantized versions on hugging face. There's a q2 version, but idk how well that performs