290
No, you don't need a 'very bespoke AOSP' to turn your phone into a Rabbit R1 — here's proof
(www.androidauthority.com)
this post was submitted on 04 May 2024
290 points (95.9% liked)
Technology
59161 readers
2119 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
huggingface! i found it once and never could remember again who hosted all those models. thank you!
i use a different device than homelab, but now I am curious what I may be able to achieve with my syno system. it's hw weak, so probably not a lot. but i would like to give it a go. if it's decent, i may consider another device for the purpose.
Good luck! You can try the huggingface-chat repo, or ollama with this web-ui. Both should be decent, as they have instructions to set up a docker container.
I believe the Llama 3 models are out there in a torrent somewhere, but I didn't dig to find it. For the 70B model, you'll probably need around 64GB of RAM available, but the 7B one should run fine with just 8GB. It will be somewhat slow though, compared to the ChatGPT experience. The self-attention mechanism can be parallelized, which is why you will see much better results on a GPU. According to some others that tested it, if you offload some stuff to RAM, you could see ~10-12 tokens per second on an RTX 3090 for certain 70B models. But more capable ones will be at less than 1 token per second, all depending on the context window you use.
If you don't have a GPU available, just give the Phi-3 model a try :D If you quantize it to 4 bits, it can apparently get 12 tokens per second on an iPhone haha. It should play nice with pooling information from a search engine, or a vector database like milvus, qdrant or chroma.
Thank you for all this! Much appreciated!