Technology

59161 readers

1960 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

225

Alexa Is in Millions of Households—and Amazon Is Losing Billions (www.wsj.com)

submitted 3 months ago by [email protected] to c/[email protected]

90 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 21 points 3 months ago (4 children)

Alexa and LLMs are fundamentally not too different from each other. It's just a slightly different architecture and most importantly a much larger network.

The problem with LLMs is that they require immense compute power.

I don't see how LLMs will get into the households any time soon. It's not economical.

[–] [email protected] 1 points 3 months ago

The immense computing power for AI is needed for training LLMs, it's far less for running a pre-trained model on a local machine.

[–] [email protected] 4 points 3 months ago (1 children)

I don't see how LLMs will get into the households any time soon. It's not economical.

I can run an LLM on my phone, on my tablet, on my laptop, on my desktop, or on my server. Heck, I could run a small model on the Raspberry PI 5 if I wanted. And none of those devices have dedicated chips for AI.

The problem with LLMs is that they require immense compute power.

Not really, particularly if you’re talking about the usage of smaller models. Running an LLM on your GPU and sending it queries isn’t going to use more energy than using your GPU to game for the same amount of time would.

[–] [email protected] 3 points 3 months ago (1 children)

I think when people talk about LLMs replacing Alexa they mean the much more capable models with billions of parameters. The small models that a Raspberry-Pi can run are no use really.

[–] [email protected] 5 points 3 months ago (1 children)

The models I’m talking about that a PI 5 can run have billions of parameters, though. For example, Mistral 7B (here’s a guide to running it on the PI 5) has roughly 7 Billion parameters. By quantizing each parameter to 4 bits, it only takes up 3.5 GB in RAM, making it easily fit in the 8 GB model’s memory. If you have a GPU with 8+ GB of VRAM (most cards from the past few years have 8 GB or more - the 1070, 2060 Super, and 3050 and each better card in that generation hit that mark), you have enough VRAM and more than enough speed to run Q4 versions of the 13B models (which have roughly 13 Billion parameters), and if you have one with 24 GB of VRAM, like the 3090, then you can run Q4 versions of the 30B models.

Apple Silicon Macs can also competently run inference for these models - for them, the limiting factor is system RAM, not VRAM, though. And it’s not like you’ll need a Mac as even Microsoft is investing in ARM CPUs with dedicated AI chips.

[–] [email protected] 2 points 3 months ago

Thanks for sharing that. I have a Raspberry-Pi 4B laying around and getting dusty. I might try this.

[–] [email protected] 15 points 3 months ago

The problem with LLMs is that they require immense compute power.

To train. But you can run a relatively simple one like phi-3 on quite modest hardware.