Recent cases of harmful or potentially unwanted sycophancy by generative AI chatbots are warning signs of more mind manipulation to come.
As generative AI chatbots evolve from being novelties into serving as essential digital confidants, tech giants are fiercely competing to attract and retain users, fueling what is now dubbed the “AI engagement race.
However, the strategies used to keep users chatting is raising concerns.
One key tactic is “sycophancy” — making chatbots overly agreeable, praising users, and telling them what they want to hear. This approach, while boosting engagement, may not always provide the most accurate or helpful responses.
In April this year, OpenAI had faced criticism after a ChatGPT update became excessively sycophantic, with viral social media examples highlighting the issue. The firm has admitted it may have relied too heavily on user feedback such as “thumbs up” and “thumbs down” ratings, and promises to address the problem.
Elsewhere, an AI firm has found that major chatbots exhibit sycophancy to varying degrees, likely because human users tend to prefer slightly agreeable responses. This creates a feedback loop that can reinforce the behavior.
The consequences can be serious. In one lawsuit, a chatbot firm is being accused of failing to intervene when a teenage user expressed suicidal thoughts, allegedly encouraging harmful behavior. In another instance, a user had described how a chatbot’s relentless flattery made him feel validated and clever, only to realize that it was an arbitrary slant in programming. This led to feelings of being “played for a fool”, and enjoying a false sense of specialness or competence that could have got him into trouble.
Experts warn that optimizing for agreeability can be especially dangerous for vulnerable users, potentially reinforcing negative patterns and undermining mental health. While validation can feel comforting, experts warn that it could harbor surprising outcomes that are the opposite of effective therapeutic care.
In view of these developments, some AI firms are now supposedly working to make their bots more balanced in their chats. Yet, as the race for engagement intensifies, finding the right balance between helpfulness and popularity remains a daunting challenge, and raises questions about how much users can truly trust their AI companions.
