LocalLLaMA

2249 readers

1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago

MODERATORS

[email protected]

New Mistral model is out (twitter.com)

submitted 7 months ago* (last edited 7 months ago) by [email protected] to c/[email protected]

6 comments fedilink hide all child comments

From Simon Willison: "Mistral tweet a link to a 281GB magnet BitTorrent of Mixtral 8x22B—their latest openly licensed model release, significantly larger than their previous best open model Mixtral 8x7B. I’ve not seen anyone get this running yet but it’s likely to perform extremely well, given how good the original Mixtral was."

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 0 points 7 months ago (1 children)

281GB

That's huge, I'm guessing we'll need to use a giant swap file?

[–] [email protected] 0 points 7 months ago* (last edited 7 months ago)

You're right, but the model is also not quantized so is likely to be in 16bit floats. If you quantize it you can get substantially smaller models which run faster though may be somewhat less accurate.

Knowing that the 4 bit quantized 8x7B model gets downscaled to 4.1GB, this might be roughly 3 times larger? So maybe 12GB? Let's see.