TechTakes

1748 readers

16 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago

MODERATORS

[email protected]

Gemini 2.5 "reasoning", no real improvement on river crossings. (awful.systems)

submitted 5 days ago by [email protected] to c/[email protected]

33 comments fedilink hide all child comments

So I signed up for a free month of their crap because I wanted to test if it solves novel variants of the river crossing puzzle.

Like this one:

You have a duck, a carrot, and a potato. You want to transport them across the river using a boat that can take yourself and up to 2 other items. If the duck is left unsupervised, it will run away.

Unsurprisingly, it does not:

https://g.co/gemini/share/a79dc80c5c6c

https://g.co/gemini/share/59b024d0908b

The only 2 new things seem to be that old variants are no longer novel, and that it is no longer limited to producing incorrect solutions - now it can also incorrectly claim that the solution is impossible.

I think chain of thought / reasoning is a fundamentally dishonest technology. At the end of the day, just like older LLMs it requires that someone solved a similar problem (either online or perhaps in a problem solution pair they generated if they do that to augment the training data).

But it outputs quasi reasoning to pretend that it is actually solving the problem live.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 0 points 5 days ago* (last edited 5 days ago) (22 children)

Yeah, exactly. There's no trick to it at all, unlike the original puzzle.

I also tested OpenAI's offerings a few months back with similarly nonsensical results: https://awful.systems/post/1769506

All-vegetables no duck variant is solved correctly now, but I doubt it is due to improved reasoning as such, I think they may have augmented the training data with some variants of the river crossing. The river crossing is one of the top most known puzzles, and various people have been posting hilarious bot failures with variants of it. So it wouldn't be unexpected that their training data augmentation has river crossing variants.

Of course, there's very many ways in which the puzzle can be modified, and their augmentation would only cover obvious stuff like variation on what items can be left with what items or spots on the boat.

[–] [email protected] 0 points 5 days ago (21 children)

It's just overtrained on the puzzle such that it mostly ignores your prompt. Changing a few words out doesn't change that it recognises the puzzle. Try writing it out in ASCII or uploading an image with it written or some other weird way that it hasn't been specifically trained on and I bet it actually performs better.

[–] [email protected] 0 points 4 days ago (6 children)

oh look it's a loadbearing "just" in the wild. better hope you can shore that fucker up with some facts

Try writing it out in ASCII

my poster in christ, what in the fuck are you on about. stop prompting LLMs and go learn some things instead

some other weird way that it hasn’t been specifically trained on and I bet it actually performs better

"no no see, you just need to prompt it different. just prompt it different bro it'll work bro I swear bro"

god, every fucking time

[–] [email protected] 0 points 3 days ago (1 children)

Well has anyone tried prompting it in EBCDIC? How do we know doing so won't immediately create the super intelligence that "or whatever"s us to silicon Valhalla? Asking for a friend.

[–] [email protected] 0 points 3 days ago

you know, I was briefly considering trying to, and I figured you'd probably have to be forcing it by content escaping tricks or something (at least I presume their APIs will do basic type-checking...)

got other yaks to do atm tho

load more comments (4 replies)

load more comments (18 replies)