LocalLLaMA

2249 readers

1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago

MODERATORS

[email protected]

Localllama setup for $100k. (programming.dev)

submitted 7 months ago by [email protected] to c/[email protected]

9 comments fedilink hide all child comments

Consider this hypothetical scenario: if you were given $100,000 to build a PC/server to run open-source LLMs like LLaMA 3 for single-user purposes, what would you build?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 0 points 7 months ago (3 children)

Why in the world would you need such a large budget? A mac studio can run the 70b variant just fine at $12k

[–] [email protected] 0 points 6 months ago* (last edited 6 months ago)

If possible, to run the upcoming llama 400B one. But this is just hypothetical.

[–] [email protected] 0 points 7 months ago

Depends on what you're doing with it, but prompt/context processing is a lot faster on Nvidia GPUs than on Apple chips, though if you are using the same prefix all the time it's a bit better.

The time to first token is a lot faster on datacenter GPUs, especially as context length increases, and consumer GPUs don't have enough vram.

[–] [email protected] 0 points 7 months ago (1 children)

So the answer would be "an alibi for the other $88k"

[–] [email protected] 0 points 7 months ago

I'll take 'Someone got seed funding and now needs progress to unlock the next part of the package' for $10 please Alex.