This is a most excellent place for technology news and articles.
It's pretty obvious that they will hit a ceiling.
Quick buck is over. And now it's time again for base research to create better approach.
I really wish we had a really advanced AI with reasonable resource consumption within my lifetime. I don't think it's unreasonable as we have got really far in the last 30 years of computational technology.
Looks, like AI buble is slowly coming to end just like what happned to crypto and NFT buble.
When did the crypto bubble end? Bitcoin is at an all time high....
Sure, except for the thousands of products working pretty well with current gen. And it's not like it's over, now we've hit the limit of "just throw more data at the thing".
Now there aren't gonna be as many breakthroughs that make it better every few months, instead there's gonna be thousand small improvements that make it more capable slowly and steadily. AI is here to stay.
Getting the GPU memory requirements down would be huge as well.
The bubble popping doesn't have to do with its staying power, just that the days of, "Hey, I invented this brand new AI ~~that's totally not just a wrapper for ChatGPT~~. Want to invest a billion dollars‽" are over. AGI is not "just out of reach."
I believe that the current LLM paradigm is a technological dead end. We might see a few additional applications popping up, in the near future; but they'll be only a tiny fraction of what was promised.
My bet is that they'll get superseded by models with hard-coded logic. Just enough to be able to correctly output "if X and Y are true/false, then Z is false", without fine-tuning or other band-aid solutions.
Seems unlikely as that's essentially what we had before and they were not very good at all.
Unlikely, but there's some percedent.
We've seen this pattern play out in video games a bunch of times.
Revolutionary new way to do things. It's cool, but not... You know...fun.
So we give up on it as a dead and and go back to the old ways for awhile.
Then somebody figures out how to (usually hard code) bumpers on the new revolutionary new way, such that it stays fun.
Now the revolutionary new way is the new gold stand and default approach.
For other industries, replace "fun" above with the correct goal for than industry. "Profitable" is one that the AI hucksters are being careful not to say...but "honest", "correct" and "safe" also come to mind.
We are right before the bit where we all decide it was a bad idea.
Which comes before we figure out hard-coding the bumpers can get us where we wanted, after a lot of work by really smart well paid humans.
I've seen industries skip the "all decide it was a bad idea" phase, and go straight to the "hard work by humans to make this fulfill the available promise" phase, but we don't actually look on track to, today.
Many current investors are convicned that their clever talking puppet is going to do the hard work of engineering the next generation of talking puppet.
I have some faith that we can reach that milestone. I'm familiar enough with the current generation of talking puppet to confidently declare that this won't be the time it happens.
My incentive in sharing all this is that I like over half of you reading there, and so figure I can give some of you a shot at not falling for this particular "investment phase" which is essentially, in practical terms, a con.
If you're referring to symbolic AI, I don't think that the AI scene will turn 180° and ditch NN-based approaches. Instead what I predict is that we'll see hybrids - where a symbolic model works as the "core" of the AI, handling the logic, and a neural network handles the input/output.
I understand folks don't like AI but this "article" is like a reddit post with lots of links to subjects which are vague and need the link text to tell us what is important, instead of relying on the actual article.
What the fuck you aren't kidding. I have comment replies to trolls that are longer than that article. The over the top citations also makes me think this was entirely written by an actual AI bot that was lrompted to supply x amoint of sources in their article. Lol
OpenAI, Google, Anthropic admit they can’t scale up their chatbots any further
Lol, no they didn't. The quotes this articles are using are talking about LLMs not chatbots. This is yet another stupid article from someone who doesn't understand the technology. There is a lot of legitimate criticism for the way this technology is being implemented but FFS get the basics right at least.
Claiming that David Gerrard an Amy Castor "don't understand the technology" is uh.... Hoo boy... Well it sure is a take.
Are you asserting that chatbots are so fundamentally different from LLMs that "oh shit we can't just throw more CPU and data at this anymore" doesn't apply to roughly the same degree?
I feel like people are using those terms pretty well interchangeably lately anyway
People that don't understand those terms are using them interchangeably
It's a known problem - though of course, because these companies are trying to push AI into everything and oversell it to build hype and please investors, they usually try to avoid recognizing its limitations.
Frankly I think that now they should focus on making these models smaller and more efficient instead of just throwing more compute at the wall, and actually train them to completion so they'll generalize properly and be more useful.
Though, I don't think that means they won't get any better. It just means they don't scale by feeding in more training data. But that's why OpenAI changed their approach and added some reasoning abilities. And we're developing/researching things like multimodality etc... There's still quite some room for improvements.
Though, I don't think that means they won't get any better. It just means they don't scale by feeding in more training data.
Agreed. There's plenty of improvement to be had, but the gravy train of "more CPU or more data == better results" sounds like it's ending.
I smell a sentient AI trying to throw us off it's plans for world domination..
Everyone ignore this comment please. I'm quite human. I have the normal 7 fingers (edit: on each of my three hands!) and everything.
Cylons. I knew it.
A 4 paragraph "article" lol
Are you suggesting “pivot-to-ai.com” isn’t the pinnacle of journalism?
It's absurd that some of the larger LLMs now use hundreds of billions of parameters (e.g. llama3.1 with 405B).
This doesn't really seem like a smart usage of ressources if you need several of the largest GPUs available to even run one conversation.
That's capitalism
Seeing as how the full unquantized FP16 for Llama 3.1 405B requires around a terabyte of VRAM (16 bits per parameter + context), I'd say way more than several.
I wonder how many GPUs my brain is
It's a lot. Like a lot a lot. GPUs have about 150 billion transistors but those transistors only make 1 connection in what is essentially printed in a 2d space on silicon.
Each neuron makes dozens of connections, and there's on the order of almost 100 billion neurons in a blobby lump of fat and neurons that takes up 3d space. And then combine the fact that multiple neurons in patterns firing is how everything actually functions and you have such absurdly high number of potential for how powerful human brains are.
At this point, I'm not sure there's enough gpus in the world to mimic what a human brain can do.
That's also just the electrical portion of our mind. There are whole levels of chemical, and chemical potentials at work. Neurones will fire differently depending on the chemical soup around them. Most of our moods are chemically based. E.g. adrenaline and testosterone making us more aggressive.
Our mind also extends out of our heads. Organ transplant recipricants have noted personality changes. Food preferences being the most prevailant.
The neurons only deal with 'fast' thinking. 'slow' thinking is far more complex and distributed.
42
Sounds generous.
You said GPUs, not CPUs and threading capabilities
The Answer to the Ultimate Question of Life, The Universe, and Everything
I don't think your brain can be reasonably compared with an LLM, just like it can't be compared with a calculator.
LLMs are based on neural networks which are a massively simplified model of how our brain works. So you kind of can as long as you keep in mind they are orders of magnitude more simple.
