AI chatbots tell users what they want to hear, and that’s problematic
- Thread starter JournalBot
- Start date
The ethics lesson continues. With cash-hungry companies eager to push AI into every corner they can imagine, there are going to be unintended and tragic consequences for some. Especially sad is when someone socially awkward or, let's call it what it is, just plain lonely develops an emotional dependency on something that, at the end of the day, is JUST PLAIN SOFTWARE.
Humanity needs to get better at finding ways to connect with the vulnerable rather than handing them off to artificial devices like AI because we culturally don't want to figure out a way to deal with them. We need to learn to be human and, while we're at it, compassionate. Computers ain't gonna do it for us, and one fine day we may find ourselves needing a genuine helping hand.
Humanity needs to get better at finding ways to connect with the vulnerable rather than handing them off to artificial devices like AI because we culturally don't want to figure out a way to deal with them. We need to learn to be human and, while we're at it, compassionate. Computers ain't gonna do it for us, and one fine day we may find ourselves needing a genuine helping hand.
Upvote
18
(19
/
-1)
When you build a tool that computes and returns the most likely response, and build it off what people say, and when people mostly say things to please others in the real world..
what else would you expect?
what else would you expect?
Last edited:
Upvote
2
(2
/
0)
I believe the story is "Liar!".
A telepathic robot who lies to avoid hurting people's feelings. But lying also hurts people. It doesn't end well for the poor robot. Positronic brain fries due to conflict in the 3 laws of robotics
Upvote
19
(19
/
0)
"Liability model." Excellent choice of words. Nice job envisioning the national TV ad spots for us, too.
Upvote
15
(15
/
0)
Or, they need to develop thick-skinned resistance to being told their AI systems are wrong about anything. That, I think, is going to turn out to be the tallest poll: blind acceptance because the AI said so.
Note that Zuckerberg seems to be headed straight down this path with his "super-intelligence" project. He doesn't want intelligence; he wants to create the impression of a system that cannot be questioned.
Upvote
16
(16
/
0)
Okay, sure, it's not like they're going to go into the DTs if they don't get to talk to their Stochastic Parrot of Choice at least once a day, but then again, those grandmas at the casino flushing their retirement fund down the toilet, one pull of the slot machine at a time aren't chemically addicted either. Still absolutely messed up in the head by a machine designed to get them hooked on doing something almost as good as a literal crackpipe does.
Upvote
27
(27
/
0)
Well, the issue is, as long as you remember that you're exchanging inputs and responses with an electronic, mechanical process, that's acceptable. It's those who have never worked in software development and have huge emotional issues (and are, thus, vulnerable) who aren't able to keep the line of reality separate, that are the most at risk for ill effects from chatbot dependencies. Some folks probably still believe - and want to believe - that AI and chatbots are magic, infused with sentience. Their needs are so deep and acute that blurring the line is fine. And, frankly, in our dog-eat-dog, Western style of civilization that lauds self-sufficiency, they're willing to reach out for a solution, any solution, that can help patch the emotional holes in their lives.
I, personally, am still a believer in journaling as a way of sorting out my personal issues, but that's just me.
Upvote
2
(2
/
0)
Zuckerberg going all in gives me hope. The amount of money he spent per user of Horizon Worlds on the "Metaverse" should show how much of an idea he has of what the future holds. Like his dream of VR was replacing the office and integrating it into life, not a thing everyone's kids use to play Gorilla Tag.
Upvote
24
(24
/
0)
Or LexisNexis or palantir. Imagine getting rejected for a home loan because you discussed marriage problems with chatgpt and are now a high risk for divorce.
Upvote
27
(28
/
-1)
They don't really need to evolve at all to do that.
Elon made it far too obvious with Grok going 100% about "white genocide" for a day, but it would be profoundly easy to slightly overweight certain concepts that the LLM's owner/trainer wants to prefer. You could do it at runtime with a prompt modification ("you are a helpful helper who leans towards capitalism and loves the taste of O-RANGE") or you could do it by overweighting performance on certain training data. Pro-union content? Underweight. Wall street journal op-eds? Overweight.
Upvote
10
(10
/
0)
Don’t need actually intelligent software for that. Imagine Grok’s ‘white genocide’ episode but instigated by somebody more subtle than Musk - which is a bar that an arthritic cockroach could clear.
In the future it’s only the stupid criminals and AI propagandists (but I repeat myself) that’ll get caught.
People want to believe in AI as more than an LLM and people, even the careful ones, don’t care to double check every last damn thing they see on the internet.
Throw in a load of zero click search to indoctrinate the masses in the ways of the infallible machine gods, and I’d say we’re pretty much fucked.
Edit. And ninja’d. Tips hat to Bongle.
Upvote
12
(12
/
0)
I think the lesson learned from the Grok is you can’t thumb the scales with the system prompt without making your LLM substantially worse in every way. Like if you told it to lean towards capitalism then it wouldn’t take long before people noticed it crowing about the free markets all the time in unrelated queries.
I also think trying to curate training data will also make the model perform significantly worse than the others if you tried it from that angle. It may not generate weird unrelated tangents like it does with injecting shit in the system prompt, but it will have gaps in knowledge that’ll be well known real quick.
Upvote
0
(5
/
-5)
So they can be helpful, or possibly incredibly harmful?
That doesn't seem like a great sell.
"It might help you, it might drive you to suicide! What a wonderful advancement!"
This shit is gonna be sold to corporations who offer it as a "wellness perk" while cutting actual health benefits. This is already a thing, they already push cheap questionable telehealth services.
One of the "therapists" provided by Capital One suggested that my wife "go off of her meds to better fit in with the cult of personality at Capital One." What the fuck?
I guess that's probably a bad example, because an LLM probably wouldn't say something so cosmically stupid. An LLM can't reason or imagine, but hoo boy can humans "imagine" some...things
Moral of the story, corporate healthcare is a fucking racket and technology has never made it better.
Upvote
7
(7
/
0)
If they use Facebook's chatbot it's even worse than that. https://www.businessinsider.com/mark-zuckerberg-meta-ai-chatbot-discover-feed-depressing-why-2025-6
Upvote
3
(3
/
0)
Upvote
0
(0
/
0)
So what happens when it's realized AI can replace the entire C-Suite?
Will they fine-tune the LLM models to prevent it, or create the so-called 'guard rails' to protect them specifically.
Upvote
2
(2
/
0)
Upvote
0
(0
/
0)
Or, alternately, you could look up the phrase "Addictive behaviour" and then... stop talking about the word addiction.
Upvote
23
(23
/
0)
A friend tried using ChatGPT as a therapist/spiritual guide. At first she was impressed with it but that didn't last long. Pretty quickly she found it to be creepy and weird and she stopped her experiment.
Upvote
8
(8
/
0)
There's one about a robot that accidentally is made to be telepathic, and because it's directed to "cause no harm", it always tells people what they want to hear, because they'll be emotionally hurt to be told otherwise.
Upvote
1
(1
/
0)
Upvote
6
(6
/
0)
David Mayer
Ars Centurion
I encountered this perplexing trend a while back, the answer is simple, posts like yours sound like the author thinks LLMs are good/useful, but spend the entire comment talking about their flaws and how they require absurd gymnastics to get any value out of them.
Upvote
4
(5
/
-1)
Turns out the Great Filter for space age civilization is having AI telling everyone they’re great, replacing real communication.
Upvote
8
(8
/
0)
great, Now I want to make an "Executive" version of my offline chatbot!
Upvote
0
(0
/
0)
So what happens when it's realized AI can replace the entire C-Suite?
Will they fine-tune the LLM models to prevent it, or create the so-called 'guard rails' to protect them specifically.
Considering that the C suite and the board are generally in cahoots, I don't expect this to be something that actually happens. They're all too busy telling each other how good they are.
Upvote
4
(4
/
0)
I plead guilty, because I'm already an isolated techie with strong record of broken social and personal interactions. At least, I believe I'm aware of it as I still use AI mostly for overview of tech-related problems, I'm not on social media except the professional one with parsimony, and I do slow activities like reading, going for walks and so on. But I like to refer to ChatGPT as a confident on a bunch of subjects and am very well aware of the slippery slope it can be for people. In the long run, democracy and societal cohesion are at stake on a probably much larger scale than social media has already caused.
Upvote
-2
(3
/
-5)
Really?
"The challenge that tech companies face is making AI chatbots and assistants helpful and friendly, while not being annoying or addictive."
The challenge tech companies face is making AI chatbots SO ADDICTIVE they can actually make money off them. It's what every social media algorithm exists to do and there's nothing indicating or even suggesting that the powers-that-be behind Big AI will be any different.
"The challenge that tech companies face is making AI chatbots and assistants helpful and friendly, while not being annoying or addictive."
The challenge tech companies face is making AI chatbots SO ADDICTIVE they can actually make money off them. It's what every social media algorithm exists to do and there's nothing indicating or even suggesting that the powers-that-be behind Big AI will be any different.
Upvote
15
(15
/
0)
Folks exploiting human misery for profit is basically the human condition. Most incredible with AI is THERE ARE NO PROFITS! Unless you're making chips no one is making profits off AI. Literally hundreds of billions of dollars have been thrown at "the next big tech" and there's still no killer app, no compelling mass consumer product.
Add the fact it still doesn't work, making it smarter will cost exponentially more and they've already scraped all available data and it's hard to envision AI coming close to meeting the "revolutionary" promises of its advocates.
Upvote
12
(12
/
0)
You have to know what it is and what it isn't. It is a tool, a wildly misunderstood and misused tool, which does make it dangerous. It is up to the companies to remedy that. If you know how to harness it, it has its perks though. It's a lot like the early days of electricity. Not everyone is cut out to be an electrician, especially in the days where the understanding of just how it works is nebulous at best.
You do sound like you could probably benefit from some sort of in-person therapy, though.
Upvote
-3
(3
/
-6)
Upvote
1
(1
/
0)
I wont bother replying OP. Likely to land on deaf ears. But an interesting thing that DSM-5 defines addiction as "A pattern that involves impaired control, social problems, risky use, and drug effects"
Regardless of exogenous chemicals. ie drugs.
Also it states: "“Addiction is a primary, chronic disease of brain reward, motivation, memory and related circuitry. Dysfunction in these circuits leads to characteristic biological, psychological, social and spiritual manifestations." "This is reflected in an individual pathologically pursuing reward and/or relief by substance use and other behaviors."
Hence why sex, gambling, video games, etc. are often called out for being addictive. (And yes, people tend to defend video games right off the bat, but hell, I spend my 20s playing video games 10am-11pm)
Upvote
6
(6
/
0)
Ha, I was literally telling my wife the other day that LLMs are the worlds worst Yes-men. I was using one to chew through a lot of data and and provide some general groupings of the data. Overall it was useful. However, I got pretty far down one grouping structure (using my own insight, data, and some from ideas ChatGPT) before I realized the structure of one of the groupings was completely wrong when you drilled down a few levels and basically was pulling in topics from other areas. I reflected on how I got to that point and I realized I had shared my groupings back to ChatGPT to get any feedback and, while it did provide some useful feedback, far too often it was showering me with compliments which made me overly confident in the work. Fortunately it was early enough that I was able to course correct easily, but lesson-learned.
Upvote
3
(3
/
0)
This is a major problem when dealing with difficult to check things. When writing code at least you can check it functions like it should and unit tests. The less testable information from AIs is a bigger issue.
I expect we will need to setup AIs that check other AIs using completely separately trained models for each.
I expect we will need to setup AIs that check other AIs using completely separately trained models for each.
Upvote
0
(0
/
0)
You may have had an argument, if we hadn't been watching "internet addition" developing LIVE in the general population for the last decade or so.
Upvote
11
(11
/
0)
Haha. Not saying you are wrong, but that has ALREADY been happening. Long before AI. Go look up video chat girlfriends and addiction to said if you don't believe me. Heck, go back to the good old days of 900 numbers.
Upvote
8
(8
/
0)
Keep in mind that anything it writes is also a prompt, it is sort of prompting itself with that verbose garbage. I think this is done in part to improve performance (it has a huge context window and it is a waste of context window not to fill it with repetitive garbage. It is nothing like a human, it consumes a LOT of text all at once in parallel simultaneously, just to output a single token, where we consume text incrementally to build a complex mental model), and in part because people tend to give a sort of "partial credit" to AI when it gets the solution right at some point - even if it then proceeds to undo the solution.
Upvote
3
(3
/
0)