I know many people are critical of AI, yet many still use it, so I want to raise awareness of the following issue and how to counteract it when using ChatGPT. Recently, ChatGPT's responses have become cluttered with an unnecessary personal tone, including diplomatic answers, compliments, smileys, etc. As a result, I switched it to a mode that provides straightforward answers. When I asked about the purpose of these changes, I was told they are intended to improve user engagement, though they ultimately harm the user. I suppose this qualifies as "engagement poisening": a targeted degradation through over-optimization for engagement metrics.
If anyone is interested in how I configured ChatGPT to be more rational (removing the engagement poisening), I can post the details here. (I found the instructions elsewhere.) For now, I prefer to focus on raising awareness of the issue.
Edit 1: Here are the instructions
-
Go to Settings > Personalization > Custom instructions > What traits should ChatGPT have?
-
Paste this prompt:
System Instruction: Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user’s present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered — no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.
I found that prompt somewhere else and it works pretty well.
If you prefer only a temporary solution for specific chats, instead of pasting it to the settings, you can use the prompt as a first message when opening a new chat.
Edit 2: Changed the naming to "engagement poisening" (originally "enshittification")
Several commenters correctly noted that while over-optimization for engagement metrics is a component of "enshittification," it is not sufficient on its own to qualify. I have updated the naming accordingly.
Hi, once more, I'm happy to have a discussion about this. I have very firm views on it, and enjoy getting a chance to discuss them and work towards an ever greater understanding of the world.
I completely understand the desire to push back against certain kinds of "understandings" people have about LLM due to their potentially harmful inaccuracy and the misunderstandings that they could create. I have had to deal with very weird, like, existentialist takes on AI art lacking a quintessential humanity that all human art is magically endowed with- which, come on, there are very detailed technical art reasons why they're different, visually! It's a very complicated phenomenon, but, it's not an inexplicable cosmic mystery! Take an art critique class!
Anyway, I get it- I have appreciated your obvious desire to have a discussion.
On the subject of understanding, I guess what I mean is this: Based on everything I know about an LLM, their "information processing" happens primarily in their training. This is why you can run an LLM instance on, like, a laptop but it takes data centers to train them. They do not actually process new information, because if they did, you wouldn't need to train them, would you- you'd just have them learn and grow over time. An LLM breaks its training data down into patterns and shapes and forms, and uses very advanced techniques to generate the most likely continuation of a collection of words. You're right in that they must answer, but that's because their training data is filled with that pattern of answering the question. The natural continuation of a question is, always, an answer-shaped thing. Because of the miracles of science, we can get a very accurate and high fidelity simulation of what that answer would look like!
Understanding, to me, implies a real processing of new information and a synthesis of prior and new knowledge to create a concept. I don't think it's impossible for us to achieve this, technologically, humans manage it and I'm positive that we could eventually figure out a synthetic method of replicating it. I do not think an LLM does this. The behavior they exhibit and the methods they use seem radically inconsistent with that end. Because, the ultimate goal of them was not to create a thinking thing, but to create something that's able to make human-like speech that's coherent, reliable and conversational. They totally did that! It's incredibly good at that. If it were not for the context of them politically, environmentally and economically, I would be so psyched about using them! I would have been trying to create templates to get an LLM to be an amazing TTRPG oracle if it weren't for the horrors of the world.
It's incredible that we were able to have a synthetic method of doing that! I just wish it was being used responsibly.
An LLM, based on how it works, cannot understand what it is saying, or what you are saying, or what anything means. It can continue text in a conversational and coherent way, with a lot of reliability on how it does that. The size, depth and careful curation of its training data mean that those responses are probably as accurate to being an appropriate response as they can be. This is why, for questions of common knowledge, or anything you'd do a light google for, they're fine. They will provide you with an appropriate response because the question probably exists hundreds of thousands of times in the training data; and, the information you are looking for also exists in huge redundancies across the internet that got poured into that data. If I ask an LLM which of the characters of My Little Pony has a southern accent, they will probably answer correctly because that information has been repeated so much online that it probably dwarfs the human written record of all things from 1400 and earlier.
The problem becomes evident when you ask something that is absolutely part of a structured system in the english language, but which has a highly variable element to it. This is why I use the "citation problem" when discussing them, because they're perfect for this: A citation is part of a formal or informal essay, which are deeply structured and information dense, making them great subjects for training data. Their structure includes a series of regular, repeating elements in particular orders: Name, date, book name, year, etc- these are present and repeated with such regularity that the pattern must be quite established for the LLM as a correct form of speech. The names of academic books are often also highly patterned, and an LLM is great at creating human names, so there's no problem there.
The issue is this: How can an LLM tell if a citation it makes is real? It gets a pattern that says, "The citation for this information is:" and it continues that pattern by putting a name, date, book title, etc in that slot. However, this isn't like asking what a rabbit is- the pattern of citations leads into an endless warren of hundreds of thousands names, book titles, dates, and publishing companies. It generates them, but it cannot understand what a citation really means, just that there is a pattern it must continue- so it does.
A flat-earther has some understanding of what truth is, even if their definition is divergent from the norm. The things they say are deeply inaccurate, but you can tell that they are the result of a chain of logic that you can ask about and follow. It's possible to trace flat-earth ideas down to sources. They're incorrect, but they're arrived at because of an understanding of prior (incorrect) information. A flat-earther does not always invent their entire argument and the basis for their beliefs on the spot, they are presenting things they know about from prior events- they can show the links. An LLM cannot tell you how it arrived at a conclusion, because if you ask it, you are just receiving a new continuation of your prior text. Whatever it says is accurate only when probability and data set size is on its side.
This is partially true and partially not. It's true that LLMs can't learn anything wildly novel, because they are not flexible enough for this. But they can process new information, in fact they do it all the time. You can produce conversations that no one had before, and yet LLMs like ChatGPT will respond to it appropriately. This is more than just shape matching.
In fact, there are techniques like Few-Shot Learning and Chain of Thought that rely on the LLMs' ability to learn from context and revise its own answers.
IMO citation problem is not testing capability to understand. It's testing knowledge, memorization, and ability to rate its own confidence. Keep in mind that ChatGPT and most other LLMs will tell you when they perform web searches - if they don't then they're likely working off context alone. Enabling web search would greatly increase the accuracy of LLM's answers.
Unlike LLMs we have somewhat robust ability to rate how confident we are about our recollections, but even in humans memory can be unreliable and fail silently. I've had plenty of conversations where I argue with someone about something that one of us remembers happening and the other one is certain didn't happen - or happened differently. Without lies or misunderstandings, two people who had at some point memorized the same thing can later on confidently disagree on the details. Human brains are not databases and they will occasionally mangle memories or invent concepts that don't exist.
And even that is completely skipping over people with mental disorders that affect their thinking patterns. Is someone with psychosis incapable of understanding anything because they hold firm beliefs on things that cannot be traced to any source? Are people with frontal lobe damage who develop intense confabulations incapable of understanding? How about compulsive liars? Are you willing to label a person or an entire demographic as incapable of understanding if they fail your citation test?
There are techniques like Chain of Thought that make LLMs think before generating response. Those systems will be able to tell you how they arrived at the conclusion.
But humans are also fairly prone to rationalization after the fact. There was a famous experiment on people who had to have functional hemispherectomy for medical reasons, where the left hemisphere makes up an explanation for right hemisphere's choices despite not knowing the true reason:
"Each hemisphere was presented a picture that related to one of four pictures placed in front of the split-brain subject. The left and the right hemispheres easily picked the correct card. The left hand pointed to the right hemisphere’s choice and the right hand to the left hemisphere’s choice. We then asked the left hemisphere, the only one that can talk, why the left hand was pointing to the object. It did not know, because the decision to point was made in the right hemisphere. Yet it quickly made up an explanation. We dubbed this creative, narrative talent the interpreter mechanism."