200 tokens per second isn't achievable with a 1.5B even on low-midrange GPUs. Unless they're attaching an external GPU it's not happening on a raspberry pi.
This article is disjointed and smells like AI.
this post was submitted on 24 Jan 2025
1 points (100.0% liked)
Free Open-Source Artificial Intelligence
3119 readers
2 users here now
Welcome to Free Open-Source Artificial Intelligence!
We are a community dedicated to forwarding the availability and access to:
Free Open Source Artificial Intelligence (F.O.S.A.I.)
More AI Communities
LLM Leaderboards
Developer Resources
GitHub Projects
FOSAI Time Capsule
- The Internet is Healing
- General Resources
- FOSAI Welcome Message
- FOSAI Crash Course
- FOSAI Nexus Resource Hub
- FOSAI LLM Guide
founded 2 years ago
MODERATORS
I was using their 7B model and it was kinda poop. Gonna try the 14B one next when I get home
Just tried some of them today and they failed at trivial (for a human junior programmer) code modifications.
Yeah, my computer also runs a game at 200fps. But I'm not saying if it's Minesweeper or a recent AAA game...
Yea.. it's not quite the same thing to actually run DeepSeek R1, a 671B model, and for example DeepSeek-R1-Distill-Qwen-1.5B
A recent i7 on CPU only can manage qwen 1.5 in a satisfactory way, comparable to big online players. Curious about recent ultra Intel and snapdragons
How. I though u needed huge amounts of vram on exorbitantly prices GPUs to run LLM with decent capacity? Are the just running a really small model or is it hyper parametrised? Or is the "thinking" process just that effective u can make up for a weak LLM?
Even though it is the smallest of the distilled models that model still outperforms GPT 4o and Claude Sonnet 3.5.
The 7B parameter models crush the older models on performance benchmarks. The 14 billion parameter model is very competitive with OpenAI o1 mini in many metrics.
Yea sounds like it's their smallest model