Anyone interested in a local llm should check out Llamafile from Mozilla.
this post was submitted on 17 Mar 2024
126 points (90.4% liked)
Open Source
30364 readers
1943 users here now
All about open source! Feel free to ask questions, and share news, and interesting stuff!
Useful Links
- Open Source Initiative
- Free Software Foundation
- Electronic Frontier Foundation
- Software Freedom Conservancy
- It's FOSS
- Android FOSS Apps Megathread
Rules
- Posts must be relevant to the open source ideology
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon from opensource.org, but we are not affiliated with them.
founded 5 years ago
MODERATORS
So what exactly is this? Open-source ChatGPT-alternatives have existed before and alongside ChatGPT the entire time, in the form of downloading oogabooga or a different interface and downloading an open source model from Huggingface. They aren't competitive because users don't have terabytes of VRAM or AI accelerators.
Edit: spelling. The Facebook LLM is pretty decent and has a huge amount of tokens. You can install it locally and feed your own data into the model so it will become tailor made.
I think you mean tailor. As in, clothes fitted to you.
Exactly, my auto carrot likes Taylor.
It's basically a UI for downloading and running models. You don't need terabytes of VRAM to run most models though. A decent GPU and 16 gigs of RAM or so works fine.
What are the hardware requirements?
Depends on the size of the model you want to run. Generally, having a decent GPU and at least 16 gigs of RAM is helpful.
“100% Open Source“
[links to two proprietary services]
Why are so many projects like this?
I imagine it's because a lot of people don't have the hardware that can run models locally. I do wish they didn't bake those in though.
Other offline tools I've found:
GPT4All
RWKY-Runner
any feelings on what you like best / works best?
They all work well enough on my weak machine with an RX580.
Buuuuuuuuuut, RWKY had some kind of optimization thing going that makes it two or three times faster to generate output. The problem is that you have to be more aware of the order of your input. It has a hard time going backwards to a previous sentence, for example.
So you'd want to say things like "In the next sentence, identify the subject." and not "Identify the subject in the previous text."
I'm in the process of installing https://github.com/imartinez/privateGPT will check this one out afterwards.
The biggest difference seems to be that you can let privateGPT to let analyze your own files. Didn't see that functionality in Jan.
One difference is that Jan is increadibly easy to install, just download the AppImage, make it executable and start it.
@jeena And absolutely nothing can go wrong by downloading random files from the internet based on contemporary hype, making them executable and starting them...
As opposed to cloning a random repository and running make or something?
@xigoi Is that something you do?
How else would you install something that doesn’t happen to be in your favorite package manager?
@xigoi Are you actually trying to get malware into your computer? Don't install **random** shiny new things without maximum skepticism. Period. Just let some other fools "test" the minefield for you. Or do a proper inspection. Executing foreign code just because it had "GPT" in the name... and acting like there was no other option... yuck!
Does this differ from Ollama + Open WebUI in any way?
Ok I tried it out and as of now Jan has a better UI/UX imo (easier to install and use), but Open WebUI seems to have more features like document/image processing.
This is a desktop application, not something you need to host.
Depends. Are either of those companies bootstrapping a for-profit startup and trying to dupe people into contributing free labor prior to their inevitable rug pull/switcheroo?
Do explain how you dupe people into contributing free labor and do a switcheroo with an open source project. All the app does is just provide a nice UI for running models.
I think they meant to imply that the original post (not yours) had suspicious intentions, while the ones you cited were more trustworthy
having contributors sign a CLA is always very sus and I think this is indicative of the project owners having some plans of monetizing it even though it is currently under AGPLv3. Their core values of no dark patterns and whatnot seem like a sales argument rather than an actual motivation/principle, especially when you see that they are a bootstrapped startup.
I’m starting to come around to big corps running their custom enhanced versions while feeding their open source counterparts with the last gen weights. As much as I love open source, people need to eat.
As was mentioned, if they start doing something egregious, they’re not the only game in town, and can also be forked. Love it or hate it, a big corp sponsor makes Joe six-pack feel a little more secure in using a product.
Free as in freedom, not free as in beer.
GPLv3 allows you to sell your work for money, but you still have to hand over the code your customers purchased. You buy our product, you own it, as is. Do whatever you like with it, but if you sell a derivative, you better cough up the new code to whoever bought it.
Thanks for pointing that out—looks like they’re working on a Server Suite. I’d guess that they try to monetize that but leave the personal desktop version free
I mean anybody can fork it and keep developing it without a CLA under AGPL3.
Yeah it's easy to fall into a negativity bias instead of doing a risk benefit analysis , the company could be investing money and resources that could be missing from open source projects, especially professional work by non programmers (e.g. UX researchers) which is something that open source projects usually miss.
You could probably figure it out by going over the contributions.
Of course, I am not against software being open-source, and I much prefer this approach of companies making their software open-source, but it's the CLA that really bothers me. I like companies contributing to the FOSS ecosystem, what I don't like is companies trying to benefit from free contributions and companies having the possibility to change the license of the code from those contributors
view more: next ›