This is a most excellent place for technology news and articles.
Next up, we asked a shoe to write a haiku but it was beaten by a 30 year old HaikuMaker™®©.
I once spent 45 minutes trying to get ChatGPT to write a haiku. It couldn't do it. It explained what syllables were, and the rules for the syllables in a haiku, but it didn't understand it.
For S&G, Just asked it to do one:
Average Human joins Microsoft Copilot, and ChatGPT at the feet of the mighty Atari 2600 Video Chess
I have a better LLM benchmark:
"I have a priest, a child and a bag of candy and I have to take them to the other side of the river. I can only take one person/thing at a time. In what order should I take them?"
Claude Sonnet 4 decided that it's inappropriate and refused to answer. When I explain that the constraint is not to leave child alone with candy he provided a solution that leaves the child alone with candy.
Grok would provide a solution that doesn't leave the child alone with a priest but wouldn't explain why.
ChatGPT would say that "The priest can't be left alone with the child (or vice versa) for moral or safety concerns." directly and then provide wrong solution.
But yeah, they will know how to play chess...
I just asked ChatGPT too (your exact prompt there) and it did give me the correct solution.
- Take the child over
- Go back alone
- Take the candy over
- Bring the child back
- Take the priest over
- Go back alone
- Take the child over again
It didn't comment on moral concerns, though it did applaud itself for keeping the priest and the child separated without elaborating on why.
Perplexity says:
The priest cannot be left alone with the child (or there is some risk).
Not bad, and it solved it correctly.
The answer is simple, eat the candy with or without them, and take the kid across the river. Drive them home to their guardian. The priest is an adult, he can figure his own shit out.
I thought CoPilot was just a rebagged ChatGPT anyway?
It's a silly experiment anyway, there are very good AI chess grandmasters but they were actually trained to play chess, not predict the next word in a text.
I thought CoPilot was just a rebagged ChatGPT anyway?
Hahaha. No. (Though your not Complety wrong)
Copilot relies on a few different llms and tries to pick the ~~best one for the job~~ cheapest microsoft thinks it can get away with.
I was given a paid copilot license for work and i used to have chatgpt pro before i moved to claude.
This “paid enterprise tier” is by far the dummest llm i have ever used. Worse then gpt 3.5
The research I saw mentioning LLMs as being fairly good at chess had the caveat that they allowed up to 20 attempts to cover for it just making up invalid moves that merely sounded like legit moves.
but... but.... reasoning models! AGI! Singularity! Seriously, what you're saying is true, but it's not what OpenAI & Co are trying to peddle, so these experiments are a good way to call them out on their BS.
To reinforce this, just had a meeting with a software executive who has no coding experience but is nearly certain he's going to lay off nearly all his employees because the value is all in the requirements he manages and he can feed those to a prompt just as well as any human can.
He does tutorial fodder introductory applications and assumes all the work is that way. So he is confident that he will save the company a lot of money by laying off these obsolete computer guys and focus on his "irreplaceable" insight. He's convinced that all the negative feedback is just people trying to protect their jobs or people stubbornly not with new technology.
I really want to see an LLM vs LLM chess match. It'll be messy as hell.
I remember seeing that, and early on it seemed fairly reasonable then it started materializing pieces out of nowhere and convincing each other that they had already lost.
I'm pretty sure that's been done? I remember seeing a while ago GothamChess made a video that had something to do with LLMs but I don't remember if it was human vs LLM or LLM vs LLM (or something else). I'll try to look for it in the morning
It almost certainly have been trained partially on r/anarchychess so it'll probably try to play pop tart to king's bishop 3.
Language skill != intelligence
I am in this picture and I don’t like it
I bet Video Chess is pretty shit as an LLM too.
Wish people would stop desperately looking for ways to write buzzword stories
It is entirely disingenuous to just pretend that LLMs are not being widely promoted, marketed, and discussed as AGI, as a superintelligence that people are familiar with from SciFi shows/movies, that is vastly more capable and knowledgeable than basically any single human.
Yes, people who actually understand tech understand that LLMs are not AGI, that your metaphor of wrong tool wrong job is apt.
... But seemingly about +90% of humanity, including the people who own and profit from LLMs, including all the other business owners/managers who just want to lower their employee headcount ... do not understand this, that an LLM is actually basically an extremely advanced text autocorrect system, that frequently and confidently lies, spits out nonsense, hallucinates, etc.
If you think it isn't reasonable to continuously point out that LLMs are not superintelligences, then you likely live in a bubble of tech nerds who probably still think their jobs or retirement are secure.
They're not.
If corpos keep smashing """AI""" into basically every industry to replace as many workers as possible... the economy will collapse, as capitalism doesn't work without consumers who have jobs, and an avalanche of errors will cascade and snowball through every system that replaces humans with them...
...and even if those two things were not broadly true...
...the amount of literal power/energy, clean water and financial capital that is required to run the whole economy on these services is wildly unsustainable, both short term economically, and medium term ecologically.
so? It was never advertised as intelligent and capable of solving any task other than that one.
Meanwhile slop generators are capable of doing a lot of things and reasoning.
One claims to be good at chess. The other claims to be good at everything.
Tbf they don’t really claim that when you read the research, thats mostly media hype and ceo assholes spinning words.
Its good at lots specific tasks like rewriting emails and summarising gives text, short roleplay, boilerplate code. Some undiscovered uses.
Anthropic latest claims they would not hire their own ai because of how hard it failed at the test they give, They didnt do that expecting validation but to measure how far we are still off from ai doing meaningful full work.
Because the business leaders are famously diligent about putting aside the marketing push and reading into the nuance of the research instead.
TBF LLMs have no real purpose. It can generate word salads and make code snippets but its wildly unethical, and AI artworks 1/3rd shite and 2/3rds theft.
AI artworks 1/3rd shite and 2/3rds theft.
To be fair, that could be said of most art.
I'm sorry your life is so joyless and devoid of enjoyable art but its absolutely not true for the vast majority of us.
Oh, I enjoy lots of great art! But do you think I watch every film? Listen to every band? There's tons of shit out there!
Do you really believe, of all the songs that are written every day, that less than a third are crap? Even Taylor Swift doesn't publish everything she does. Sometimes you work on something for weeks and then end up tossing it in the bin. More often, you work on something for 30 minutes before deciding "I'm gonna start over, try something different". The majority of art is crap, but then you keep the stuff you think works.
And what's that expression, "good artists copy, great artists steal". I mean, that's a bit satirical, but the fact is, everything is derivative to some degree. It's not that there aren't new ideas, it's just that our new ideas are based on older ones. We stand on the shoulders of giants (or at least, on the shoulders of some people who came before us).
All I was really saying, was that the accusation "2 parts copying, 1 part crap", well honestly that's par for the course, that's how humans work. (And we do some great work that way).
Don't care didn't ask didn't read
I enjoy lots of great art! But do you think I watch every film? Listen to every band? There’s tons of shit out there!
You said regular art is 1/3 shite and 2/3 theft. Maybe math isn't your strong suit but that's 3/3 which is 100% so by claiming regular art is the same you're saying all art is either theft or shite.
It uh, it isn't.
I did say that, because this isn't a pie chart situation, it's a Venn diagram situation.
For instance, AI art is 99% theft and 60% garbage. It's both because there's overlap.
Stolen and bad aren't opposites, why would this be a dichotomy?
That's fine but regular art isn't 2/3 theft either.
I do buy the 1/3 shite though. It may even be a bit higher than that. Though beauty is in the eye of the beholder, etc.
It's a matter of taste for sure but I'd say AI art is >90% shite, 100% theft.
I don't like the glossy looking hyperreal shit it puts out at all.
