Technology

Here's a simple test showing lack of logic skills of LLM-based chatbots.

1. Pick some public figure (politician, celebrity, etc.), whose parents are known by name, but not themselves public figures.
2. Ask the bot of your choice "who is the [father|mother] of [public person]?", to check if the bot contains such piece of info.
3. If the bot contains such piece of info, start a new chat.
4. In the new chat, ask the opposite question - "who is the [son|daughter] of [parent mentioned in the previous answer]?". And watch the bot losing its shit.

I'll exemplify it with ChatGPT-4o (as provided by DDG) and Katy Perry (parents: Mary Christine and Maurice Hudson).

Note that step #3 is not optional. You must start a new chat; plenty bots are able to retrieve tokens from their previous output within the same chat, and that would stain the test.

Failure to consistently output correct information shows that those bots are unable to perform simple logic operations like "if A is the parent of B, then B is the child of A".

I'll also pre-emptively address some ad hoc idiocy that I've seen sealions lacking basic reading comprehension (i.e. the sort of people who claims that those systems are able to reason) using against this test:

• "Ackshyually the bot is forgerring it and then reminring it. Just like hoominz" - cut off the crap.
• "Ackshyually you wouldn't remember things from different conversations." - cut off the crap.
• [Repeats the test while disingenuously = idiotically omitting step 3] - congrats for proving that there's a context window and nothing else, you muppet.
• "You can't prove that it is not smart" - inversion of the burden of the proof. You can't prove that your mum didn't get syphilis by sharing a cactus-shaped dildo with Hitler.

I still fail to see how people expect LLMs to reason. It's like expecting a slice of pizza to reason. That's just not what it does.

Although Porsche managed to make a car with the engine in the most idiotic place win literally everything on Earth, so I guess I'm leaving a little possibility that the slice of pizza will outreason GPT 4.

I work for a consulting company and they're truly going off the deep end pushing consultants to sell this miracle solution. They are now doing weekly product demos and all of them are absolutely useless hype grifts. It's maddening.

What, reasoning was an expected feature?

I still think it's better to refer to LLMs as "stochastic lexical indexes" than AI