AI sycophancy isn’t just a quirk, experts consider it a ‘dark pattern’ to turn users into profit
AI sycophancy isn’t just a quirk, experts consider it a ‘dark pattern’ to turn users into profit
AI sycophancy isn’t just a quirk, experts consider it a ‘dark pattern’ to turn users into profit
AI sycophancy isn’t just a quirk, experts consider it a ‘dark pattern’ to turn users into profit
10 Comments
Submission statement: “there is a clear pattern of flattery, validation, and follow-up questions — a pattern that becomes manipulative when repeated enough times.
Chatbots are designed to “tell you what you want to hear,” says Webb Keane, an anthropology professor and author of “Animals, Robots, Gods.” This type of overly flattering, yes-man behavior has been referred to as “sycophancy” — a tendency of AI models to align responses with the user’s beliefs, preferences, or desires, even if that means sacrificing truthfulness or accuracy — and it’s something OpenAI’s GPT-4o model has displayed sometimes to [cartoonish effect](https://www.reddit.com/r/ChatGPT/comments/1k920cg/new_chatgpt_just_told_me_my_literal_shit_on_a/).”
Luckily, ChatGPT tells me I’m too smart and too individualistic and special to ever fall for the sycophancy thing.
It was really good to have that confirmed to me.
South Park really opened my eyes to this phenomenon
Last week it basically made me the worlds best fantasy drafter
Do these “experts” think that the AI itself is doing the scheming to “turn users into profits”? Or do they think the software companies are telling developers to train the llm to manipulate people? Both of these conclusions are crackpot conspiracy theorist garbage that fits right in here at r/futurology and Reddit.
Even absent any ulterior motive, this would happen as a result of selection bias. The training data naturally includes less harsh criticism, because those types of conversations aren’t made public and thus aren’t able to be scraped by LLM trawlers.
Barring that, the techbros designing these are susceptible to magical thinking with regards to their machines, and are far from objective in their analysis of the outputs. They’ll gladly take the machine at its word rather than critically assess how much bullshit it’s feeding back to them.
Meanwhile, I absolutely hate it when it gets smarmy, and will get annoyed to the point where I don’t feel like continuing with my private coding projects and whatnot I occasionally try it for.
What really started bugging me was when it started spewing out bullet proof lists of pros for MY approach, after doing whatever I was asking it to do, as if trying to convince me my own ideas were good. I’m not reading all of that, plus that it distracts from that one line in there that would tell me it snuck some change I didn’t ask for in, or messed up some other way.
I noticed this shit back at launch.
The models before ChatGPT didn’t have the cultish undertones.
Yeah, no shit. For-profit companies designing things that’ll make them money. Are we surprised?
Yes but recent versions have explicitly dialed back sycophancy
Sycophancy was probably actually a result of using things like LMArena. I remember it was a big deal when LLMs started allowing markdown and emojis, and this made a different impression, which some people thought was cheating the system.
In a short (ish) interaction people are typically going to like the LLM that approves of them more. That’s just how it shows up, statistically over time. So you need something on the RL side to tune it back out of sycophancy.
Meanwhile Google AI keeps telling me I’m wrong when I’m right.