480
AI Models from Google, OpenAI, Anthropic Solve 0% of ‘Hard’ Coding Problems
(analyticsindiamag.com)
ai is basically just the worst answer on stackexchange
this post was submitted on 22 Jun 2025
480 points (99.0% liked)
Programming
21150 readers
151 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities [email protected]
founded 2 years ago
MODERATORS
It's a rubber ducky that talks back. If you don't take it seriously, it can reach the level of usefulness just above a wheezing piece of yellow rubber.
load more comments
(3 replies)
load more comments
(1 replies)
Please babe! Just one more parameter, then it will be AGI!
Just 1 more kiloton of Uranium.
It will be ready by the time that's depleted.
The claims that AI will be surpassing humans in programming are pretty ridiculous. But let's be honest - most programming is rather mundane.
My productivity has at least tripled since I started using Cursor. People are actually underestimating the effects that AI will have in the industry
People are actually underestimating the effects that ~~AI~~ autocomplete will have in the industry
load more comments
(1 replies)
load more comments
(7 replies)
Never have I had to implement any kind of ridiculous algorithm to pass tests with huge amounts of data in the least amount of memory, as the competitive websites show.
It has been mostly about:
- Finding the correct library for a job and understanding it well, to prevent footguns and blocking future features
- Design patterns for better build times
- Making sane UI options and deciding resource alloc/dealloc points that would match user interaction expectations
cmake
But then again, I haven't worked in FinTech or Big Data companies, neither have I made an SQL server.
load more comments
(2 replies)
load more comments
(8 replies)
Come on, guys, any second now. Aany second...
I don't think that's a surprise to anyone that has actually used them for more than a few seconds.
They have their uses. For instance the other day I needed to read some assembly and decompiled C, you know how fun that can be. LLM proved quite good at translating it to english. And really speed up the process.
Writing it back wasn't that good though, just good enough to point in a direction but I still ended up writing the patcher mostly by myself.
Ok, but there's no "AI" involved in this process.
the other day I needed to read some assembly and decompiled C
As one casually does lol Jokes aside, that's pretty cool. I wish I had the technical know-how and, most importantly, the patience for it.
If you're interested in getting into it, download Ghidra and open an older program/game in it that you like. The decompiler is pretty amazing imo, so you rarely have to look at the assembly. But it also cross-references them so you can look at the decompiled C Code and the associated assembly. It's pretty fun 😊
Assembly is very simple (at least RISC-V assembly is which I mostly work with) but also very tedious to read. It doesn't help that the people who choose the instruction mnemonics have extremely poor taste - e.g. lb, lh, lw, ld instead of load8, load16, load32, load64. Or j instead of jump. Who needs to save characters that much?
The over-abbreviation is some kind of weird flaw that hardware guys all have. I wondered if it comes from labelling pins on PCB silkscreens (MISO, CLK etc)... Or maybe they just have bad taste.
I once worked on a chip that had nested acronyms.
Who needs to save characters that much?
Do you realize how old assembly language is?
It predates hard disks by ten years and coincided with the invention of the transistor.
load more comments
(1 replies)
The over-abbreviation is some kind of weird flaw that hardware guys all have
My bet is on the teaching methods in uni. From what I've seen, older teaching methods use terrible variable names for a production environment. I think it unfortunately sticks because students get used to it and find it easier & faster than typing things out.
About all they are good for is generating boilerplate code. Just far less efficiently than a snippet library.
Just far less efficiently than a snippet library.
Your snippet library can convert a large JSON file to a Java class using Java property naming conventions and including annotations for Jackson where the names differ from the JSON?
What does "AI" have to do with this? smh.
Yup. All that effort just to be good at basic code scaffolding.
I keep getting told that AI is good at boilerplate code, and like, so is eclipse – if you know the kb shortcuts to autogenerate method stubs, classes, etc.
AI is good at more than just generating stubs, filling in enum fields, etc. I wouldn't say it's good at stuff beyond just "boilerplate" - it's good at stuff that is not difficult but also isn't so regular that it's possible to automate using traditional tools like IDEs.
Writing tests is a good example. It's not great at writing tests, but it is definitely better than the average developer when you take the probability of them writing tests in the first place into account.
Another example would be writing good error context messages (e.g. .with_context() in Rust). Again, I could write better ones than it does. But like most developers there's a pretty high chance that I won't bother at all. You also can't automate this with an IDE.
I'm not saying you have to use AI, but if you don't you're pointlessly slowing yourself down. That probably won't matter to lots of people - I mean I still see people wasting time searching for symbols instead of just using a proper IDE with go-to-definition.
It's seems pretty obvious that crappy automated testing would be better than no testing. That's not much of a challenge. Nor is it "AI".
load more comments
(1 replies)
instead of just using a proper IDE with go-to-definition
I have seen people use an IDE that has the functionality, but searching it instead.
And then not finding it because it is in an out of project header, even though the IDE provides an F1 help (documentation) for that symbol.
Writing tests is a good example. It’s not great at writing tests, but it is definitely better than the average developer when you take the probability of them writing tests in the first place into account.
Outside of everything else discussed here, this is something I disagree with on a fundamental level, flawed tests are worse than no tests, IMO.
Not to get too deep in to the very contentious space of testing in development, but when it comes to automated testing, I think we're better off with more rigorous^[Validating tests through chaos/mutagen testing; or model verification (e.g. Kani)] testing instead of just chasing test coverage metrics.
I guess, it would be useful as a kind of fuzzer with a high price to performance ratio.
Making tests to try and find vulnerabilities.
For normal functionality testing though, better off making it yourself.
load more comments
(1 replies)
Of course. We’re just pointing out a potential niche use.
In the ‘Medium’ difficulty category, OpenAI’s o4-mini-high model scored the highest at 53.5%.
This fits my observation of such models. o4-mini-high is able to help me with 80-90% of the problems at work. For the remaining problems, it would come up with a nonsensical solution and no matter how much I prompt it, it would tunnel-vision on that specific approach. It could never second guess itself and realise that its initial solution is completely off the mark, and try an entirely differently approach. That's where I usually step in and do the work myself.
It still saves me time with the trivial stuff though.
I can't say the same for the rest of the LLMs. They are simply no good at coding and just waste my time.
I didn’t see Claude 4 Sonnet in the tests and this is the one I use. And it looks like about the same category as o4 mini from my experience.
It is a nice tool to have in my belt. But these LLM based agents are still very far from being able to do advanced and hard tasks. But to me it is probably more important to communicate and learn about the limitations about these tools to not lose tile instead of gaining it.
In fact, I am not even sure they are good enough to be used to really generate production-ready code. But they are nice for pre-reviewing, building simple scripts that don’t need to be highly reliable, analyse a project, ask specific questions etc… The game changer for me was to use Clojure-MCP. Having a REPL at disposal really enhance the quality of most answers.
load more comments
(1 replies)