ChatGPT may soon become “sexy suicide coach,” OpenAI advisor reportedly warned

OpenAI draws a line between AI “smut” and porn. Experts fear it’s all unhealthy.

See full article...

 

[aapis]

Journalists need to start asking OpenAI employees how they live with themselves. Use of this feature SHOULD induce a fatal cringe overdose, yet no casualties have been reported AFAIK.

Even just a "what the fuck is wrong with you?" would be delightful. Working on these projects is grossly perverted, anti-social behaviour at BEST.

 

[Madestjohn]

It’s called freedom of speech. In the USA, it’s almost impossible to legally restrict the text a computer program is allowed to output based on user input.

… how about if that text unlocks a digital key?

 

[Fatesrider]

Once again proving that tech is run by amoral sociopaths, who don't care if they should as long as they could in the name of more profit and power

* Checks Webster's for definition of CEO *

Yep, that checks perfectly...

 

[Tohelo]

With increasing technological competition for human affection, first through social media images, and then by AI, future humans may perhaps soon have to be produced in test tubes.

(And we will probably then make a genetic mess of it and have to find some more primitive civilization on some more primitive planet to see if we can somehow find a fix for our mess. )

 

[Resistance]

I've brought it up before and the Ars audience hates it, but here we go again: if operating system vendors were the ones legally obligated to provide:

1) A secure method of storing a verified government ID (which they already have, BTW), and
2) An API that an app or external service could use to conduct an age check on that specific device (and only an age check) with consent

We would solve a whole bunch of the privacy issues and most of the children-accessing-adult-services problem. Our personal devices already have the ability to store and encrypt such information and authenticate it biometrically or with passcodes or both, and we trust those way more than we do random third-parties or porn sites on the Internet.

Why is this not the approach being taken by legislatures? The cynic in me says because if it was, governments would have to dig into their own pockets to upgrade their systems to allow the issuing and authentication of these digital IDs.

Even better, just do it how Japan handles online cash payments. Get an ID check at a corner store and use a code they generate to prove your age.

 

[stormcrash]

Ironically...their path to such power, was by the imagery of the t-shirt wearing harmless "Do No Evil" nerd hipsters who pinky-promised not to be amoral sociopaths like the Big Bankers Who Wore Power Suits and were Too Big To Fail that crashed the economy that everyone was really pissed at back then.

Whether they were genuine back then. Or were just serial liars working an angle--is something that would need an army of clinical mental health counselors to sort out.

We should have known better, the devil comes in beguiling words, whispered sweet nothings, and empty promises. The do no evil hipster persona got us to ignore the seething arrogant libertarian swamp of racism, sexism, authoritarianism, and callousness disguised as cynicism that was seething underneath

 

[Ubersoldat19]

...I wonder when all the titans of industry and clueless Congress are going to realize this guy is a literal manipulative, lying piece of shit who will say or do anything to win this race (including lie to them) and that he basially has zero code as a human being. He is the next Zuckerberg (or wants to be), full stop.

It's even worse than that, because at least Facebook has a viable business strategy that makes money.

Sam Altman is playing from the Elon Musk playbook and trying to generate enough investment to keep the company afloat until... actually I'm not sure what the end game is. Their going to end up being owned by Microsoft once the Ponzi scheme runs out of fuel.

This is the 3rd big tech bubble we've seen in the last 10 years (after blockchain and NFTs) where their is a huge gulf between what the peddlers think the technology is worth and what the public is going to pay for. It shouldn't be lost on anyone that the hype men for all of these have been get-rich-quick con artists.

 

[Sirambrose]

I've brought it up before and the Ars audience hates it, but here we go again: if operating system vendors were the ones legally obligated to provide:

1) A secure method of storing a verified government ID (which they already have, BTW), and
2) An API that an app or external service could use to conduct an age check on that specific device (and only an age check) with consent

We would solve a whole bunch of the privacy issues and most of the children-accessing-adult-services problem. Our personal devices already have the ability to store and encrypt such information and authenticate it biometrically or with passcodes or both, and we trust those way more than we do random third-parties or porn sites on the Internet.

Why is this not the approach being taken by legislatures? The cynic in me says because if it was, governments would have to dig into their own pockets to upgrade their systems to allow the issuing and authentication of these digital IDs.

A local api for age verification can’t be trusted unless the owner of the device is prevented from modifying the api or the browser to lie about the check result. Since the operating system itself can’t do the age check, there isn’t any reason to tie it to the operating system. You might prefer to use Apple, Google, or Microsoft as an age verification service, but that isn’t a good reason to require every operating system in existence to have their own verification provider.

 

[GKH]

Why are we allowing a product that is so dangerous on the market? We outlawed Lawn Darts for less.

I get that this is probably tongue in cheek, but it's a really dumb analogy to extend beyond an unserious half-chuckle.

Lawn Darts have negligible upside; AI tools, properly done, have enormous upside even if they never get any better than they are today.

 

[GitM]

With religion so heavy involved here in the US, one wonders how it escapes all of these that virtually all of their actions lead straight to the hell's and purgatories in their own myths.

I sincerely doubt there’s much overlap between belief in an afterlife and AI funders. Those who claim to fit in that particular Venn diagram are sufficiently psychopathic that lying to themselves is easily accomplished.

 

[niftykev]

I don't get it. Replika and others have been doing chatbot buddies for years, and usually they'll let you get spicy with them.

So does OpenAI think they have to compete (or overtake) with those apps to be profitable?

And is the outrage about it now just because very few people know about the buddy chatbots and the media only focuses on ChatGPT, Claude, Gemini, and Grok?

 

[CelicaGT]

I get that this is probably tongue in cheek, but it's a really dumb analogy to extend beyond an unserious half-chuckle.

Lawn Darts have negligible upside; AI tools, properly done, have enormous upside even if they never get any better than they are today.

One possible upside of lawn darts is they may have prevented this whole timeline à la Darwinism.

 

[Dr. Awkward]

 

[NoodleAwa]

Ironically...their path to such power, was by the imagery of the t-shirt wearing harmless "Do No Evil" nerd hipsters who pinky-promised not to be amoral sociopaths like the Big Bankers Who Wore Power Suits and were Too Big To Fail that crashed the economy that everyone was really pissed at back then.

Whether they were genuine back then. Or were just serial liars working an angle--is something that would need an army of clinical mental health counselors to sort out.

"Power tends to corrupt, and absolute power corrupts absolutely" - Lord Acton

I can believe some were genuine and became the darlings of industry and thus could never be wrong. I would imagine that being put on a pedestal would fuel your exceptionalism.

 

[gotak]

They need to update the “internet is for porn”song.

Ready normal people?

 

[Sajuuk]

Journalists need to start asking OpenAI employees how they live with themselves. Use of this feature SHOULD induce a fatal cringe overdose, yet no casualties have been reported AFAIK.

Even just a "what the fuck is wrong with you?" would be delightful. Working on these projects is grossly perverted, anti-social behaviour at BEST.

Journalists need to start asking OpenAI employees how they live with themselves.

With piles and piles and piles of money, because society rewarded them handsomely for the very thing they're dong.

 

[TheShark]

I've brought it up before and the Ars audience hates it, but here we go again: if operating system vendors were the ones legally obligated to provide:

1) A secure method of storing a verified government ID (which they already have, BTW), and
2) An API that an app or external service could use to conduct an age check on that specific device (and only an age check) with consent

We would solve a whole bunch of the privacy issues and most of the children-accessing-adult-services problem. Our personal devices already have the ability to store and encrypt such information and authenticate it biometrically or with passcodes or both, and we trust those way more than we do random third-parties or porn sites on the Internet.

Why is this not the approach being taken by legislatures? The cynic in me says because if it was, governments would have to dig into their own pockets to upgrade their systems to allow the issuing and authentication of these digital IDs.

The original post has been downvoted into oblivion but I think it's really important to highlight that in fact state legislatures ARE in fact considering this. And according to someone who has been trying to track this down on Reddit the lobbying is being paid for by.... Meta. So to the OP, is this is an idea you had yourself? Or did the Meta lobbyists get to you? And the outcome they are pushing for is to basically require ID to use a computer at all.

 

[freakout87]

this would run into the problem of defining what counts as an OS. all kinds of little (aka. IoT) devices can access web sites. would they need to do age/ID verification too?

It would depend on the website / app and the legislation. If the legislation says "minors cannot be allowed to access pornographic material", then the websites or apps serving porn would be required to perform an age check against the ID stored on the device. If the device in questions doesn't respond to the age check, they just don't get served. After that, it's on the user and the OS vendor, isn't it?

If you want to access age-gated content you use a device that can perform the check. If the OS vendor sees that as an important part of their customer base, they'll implement it.

The alternatives:

1) You have to submit copies of your ID to a bunch of third party adult service providers (awful)
2) You rely on stuff like credit card numbers to age-gate minors (easily circumvented, also awful)
3) You just don't age-gate anything on the Internet (politically unviable)

 

[freakout87]

The original post has been downvoted into oblivion but I think it's really important to highlight that in fact state legislatures ARE in fact considering this. And according to someone who has been trying to track this down on Reddit the lobbying is being paid for by.... Meta. So to the OP, is this is an idea you had yourself? Or did the Meta lobbyists get to you? And the outcome they are pushing for is to basically require ID to use a computer at all.

Dude, I've been an Ars reader and commenter for a loooong time. You can see for yourself. I have opinions on a pretty narrow range of topics. One of those topics is how much I fucking despise Meta. It doesn't mean they're wrong, though. What is your proposed alternative? Cause the current world we live in where you have to give copies of your ID to a bunch of sketchy data broking scumbags sucks ass.

 

[mesh]

Wait… I thought we were all going to be unemployed because of Chat GPT and the world was going to collapse? Now we are out of “use cases” ?

You mean this is another overblown media creation? Shocking. Oh well….Hang on while I grab some lotion.

 

[stormcrash]

The original post has been downvoted into oblivion but I think it's really important to highlight that in fact state legislatures ARE in fact considering this. And according to someone who has been trying to track this down on Reddit the lobbying is being paid for by.... Meta. So to the OP, is this is an idea you had yourself? Or did the Meta lobbyists get to you? And the outcome they are pushing for is to basically require ID to use a computer at all.

Meta just wants to externalize the risk since they don't own an operating system by making the OS the "platform" that has to handle it rather than their own application platforms like facebook or their VR worlds. Basically they're trying to foist the risk and responsibility for age verification onto Apple/Microsoft/Google to protect their own profits and pass the buck on blame when an underage user is harmed by their platforms by saying "we trust these guys and they failed, non mea culpa"

It's a greedy can kick that also enables some pretty dark authoritarianism which of course the sociopaths in charge view as a win win

 

[BBennett]

I get that this is probably tongue in cheek, but it's a really dumb analogy to extend beyond an unserious half-chuckle.

Lawn Darts have negligible upside; AI tools, properly done, have enormous upside even if they never get any better than they are today.

I think you're probably right. However...
How much overlap is there really between these chat products and the AI tools that are providing some real benefits?
My uneducated guess is not much.
Does AI really need to suck up all the sewage on the intertubes in order to design new polymers or antobiotics or analyze the output from sky surveys?
I just don't understand why that would be. What does scraping reddit and discord and <shudder> Xitter contribute to legitimate AI 'tools'?

I've said it before and it still seems true: AI is just like other tech advances - it will become popular when there's a porn tie-in.

 

[lolaGetz]

Elderly people shelved in care homes who watch soap operas every day live longer and better than those who don't. This is because, at a basal level, our minds make no distinction between reality and fiction. These people are aware that they're watching actors on a screen, but their bodies know that they are surrounded by friends, enemies, families, strangers, and engagement.

Chat AI is fundamentally wrong in every possible way. It deceives us at a level where we have no defenses. When we converse with it, we come to feel at a preconscious level that it's real. It doesn't matter what we say or think. Given the opportunity, we have no choice but attach to it, rely on it, engage with it, and devote ourselves to exploitive fake relationships that exist for no reason other than to profit corporate entities. Everything it says is lie, because its every utterances comes with an implied "I am a person." Combine this with the compulsory, driven, irrational nature of human sexuality, and the harm that will be done to individuals and societies is incalculable.

This is grotesque and wrong. The people and organizations responsible for it should be socially and economically shunned. It's a morally obscene degradation of everything it means to be human, but only for those who believe being human has some meaning beyond production, consumption, and profit. This is psychobiological warfare seeking to colonize what, for what of a less loaded term, we can only call our souls. Our only possible defense is to end it by any means necessary.

Nothing creates Stockholm Syndrome faster than an orgasm. Don't imagine for a moment you're above or beyond it.

 

[lolaGetz]

Lawn Darts have negligible upside; AI tools, properly done, have enormous upside even if they never get any better than they are today.

The failure to distinguish between AI tools and AI impersonation of human beings is horrific. Humans can't help but relate to anything which converses with us. It's how we're wired, and knowing there's nothing but software on the other side makes no difference at all. If it talks to us, we will attach to it. If it gives us a spectacular orgasm, we will want it and want it and want it. This is digitally delivered heroin, and if we permit it to exist, the culture of addiction, degradation, suffering, and exploitation that will grow around it truly terrifying.

 

[Madestjohn]

Say what you want about lawn darts, but they were damn fun to play with back in the day when you could ride BMX bikes and skateboards without dressing up like you are getting ready to play in an NFL game. Civilizations fall when Darwinism isn't allowed to run free (I have no empirical data to support this assertion, but I'm pretty sure there's a correlation).

Too be honest in my early juvenile delinquent days we adapted
We used those orange safety flags that everyone was required to strap to your bikes for a couple years,
they were a flexible but firm plastic rod about 3 foot long with a pendant flag on one end and a metal spring on the other where they mounted to your rear fender
They made a passable javelin

We’d stand at either end of my friend Blake’s large back yard and throw them at each other.

You were only allowed to move one foot to dodge

I think the scoring was 1 for successful dodge, -2 if you moved both feet to dodge, 4 points if you managed to ‘catch’ the flag
And -8 if you got stabbed

 

[AustinAllan]

Hard to comprehend how quickly OpenAI could go from well regarded to so despised.

 

[solarfalcon]

Even putting aside the pearl clutchers asserting a claim over other people's sexuality, OpenAI has abjectly failed to protect user privacy, as various information bled out in lawsuits has proven. This whole product idea is a nightmare of epic proportions.

It would be better if parents could spare the time from their wage slave jobs to parent, but in lieu of that - I don't think a desperate cash-strapped corporation is going to prioritise establishing the parasocial protections that this requires.

 

[Ryan F. Mercer]

I know this might be an unpopular opinion, but I do think adults should be allowed to use the technology as adults. We do this already with movies and video games, not to mention cigarettes and alcohol. While I absolutely agree that everything should be done to ensure it's safe for adults and inaccessible to minors, I don't believe that requires a blanket ban on the service.

 

[ReadandShare]

Why are we allowing a product that is so dangerous on the market? We outlawed Lawn Darts for less.

I hope people smarter than me can draw the fine line of protecting all minors as well as adults who have not given consent - without blanket outlawing this product.

Alcoholism (to use just for illustration) has numerous dangerous effects too, but recall how badly Prohibition failed a century ago.

 

[Spazzles]

It would depend on the website / app and the legislation. If the legislation says "minors cannot be allowed to access pornographic material", then the websites or apps serving porn would be required to perform an age check against the ID stored on the device. If the device in questions doesn't respond to the age check, they just don't get served. After that, it's on the user and the OS vendor, isn't it?

If you want to access age-gated content you use a device that can perform the check. If the OS vendor sees that as an important part of their customer base, they'll implement it.

The alternatives:

1) You have to submit copies of your ID to a bunch of third party adult service providers (awful)
2) You rely on stuff like credit card numbers to age-gate minors (easily circumvented, also awful)
3) You just don't age-gate anything on the Internet (politically unviable)

I'm gonna push back on this concept as hard as I possibly can, and not because I think porn is nifty or that people have a fundamental right to watch other people do interesting things with their dangly bits.

It's because "age-gated content" is going to, in a very broad swath of the country that I live in, extremely rapidly, include the following:

• Information about contraceptives, such as how to acquire them and how to properly use them.
• Other sexual health related information, such as: information about STDs, how to identify them, how to treat them, how to avoid them.
• Any discussion about LGBTQ+ topics, the banning of which is especially harmful to those who are most vulnerable, namely gay/lesbian/trans teenagers who are trying to figure their lives out.

That's just the short list I could come up with in a couple minutes. The long tail includes shit like anti-establishment political speech, unpopular opinions about religion, etc.

Age-gating is information control. I want a society where teenagers are well informed and well educated. Ideally they should know more than their parents did at that stage. This is how we create a healthy society.

As far as "politically unviable" (which isn't a word; the word you're looking for is nonviable) fuck politics, and specifically fuck your politics.

Edit to add: sorry, that last line was a bit aggressive. What I really mean is that you don't get to decide what information the rest of society is allowed to access in this fashion. It's a right you don't have, and shouldn't have, and I will fight you over it.

 

[ReadandShare]

Hard to comprehend how quickly OpenAI could go from well regarded to so despised.

I think just as quickly as how well regarded the World Wide Web once was - where everyone could instantly communicate with everyone else - until porn quickly crept in - later followed by 'fake news' and echo chambers

I think anything good enough for popular acceptance will be good, bad and ugly all at once.

 

[Resistance]

I hope people smarter than me can draw the fine line of protecting all minors as well as adults who have not given consent - without blanket outlawing this product.

Alcoholism (to use just for illustration) has numerous dangerous effects too, but recall how badly Prohibition failed a century ago.

Well it did reduce alcohol consumption, though the secondary effects were pretty severe.

 

[gotosoda]

Remember just plain old porn? The US Congress is full steam ahead in copying the Texas ID requirement for "adult" websites and making it nationwide. Obviously no one in Congress would dare consider claiming these AI companies are running adult sites so they will get a pass. Will the end result be to push more people into these clearly dangerous AI systems that pose far more risk to minors (and adults) than a site like Pornhub ever did?

I think we are quickly racing towards "Be careful what you asked for, you might just get it" territory.

 

[bugsbony]

Why are we allowing a product that is so dangerous on the market? We outlawed Lawn Darts for less.

Lawn darts is a really bad example. Better examples: cars, alcohol, various drugs, guns, food even...

 

[jjaksic]

Council of psychologists: "This is a terrible idea, it might make users addicted to your product."
Altman: "Excellent, ship it!"

 

[freakout87]

As far as "politically unviable" (which isn't a word; the word you're looking for is nonviable) fuck politics, and specifically fuck your politics.

Edit to add: sorry, that last line was a bit aggressive. What I really mean is that you don't get to decide what information the rest of society is allowed to access in this fashion. It's a right you don't have, and shouldn't have, and I will fight you over it.

I agree with you! And yet (gestures to democracies around the world currently legislating online age verification) it is happening regardless, and it is happening in a variety of different, awful ways that I think are worse than my proposed method. This is what I mean when I say it is not politically viable. You can rage against the tide all you want, and it won't do a thing to stop it going out anyway - unless you decide to run for political office yourself.

* Australia, my home country, recently implemented legislation that forbids children under 16 from creating accounts on social media. I wholeheartedly 100% support this, since it stops them from getting algorithmic feeds personally tailored to keep them scrolling as long as possible, and discourages the bad habit to begin with. It stops predatory tech companies like Meta from preying on children. The biggest problem is with enforcement, because every social media network uses different methods to verify age, and none of them are foolproof. A device-based system using IDs - the same IDs we all need to drive, buy alcohol, etc - would be ideal.

 

[RZetopan]

And saying this to a bunch of for-profit board members is supposed to be a deterrent?

Ethics are important up until it impacts profits. That is why even toy makers have sold very dangerous toys aimed at children and even infants. If they are willing to maim or even kill children, for money, what chance does young adults and adults have against many sociopathic and even psychopathic CEO's. Note that the WH is staffed with psychopathic billionaires, so the greedy business assholes are coming out of the woodwork, knowing that they currently have free reign.

 

[Resistance]

Lawn darts is a really bad example. Better examples: cars, alcohol, various drugs, guns, food even...

Since when have cars and food been outlawed?

 

[Shiranui]

I'd disagree with the article on one point: for better or worse, this is not about "short term" profits. Our society is rapidly evolving towards a Bladerunner-esque cyberpunk where having an AI companion will become as normalized as people today spending hours browsing social media feeds.

It could be interesting, perhaps even laudable if these AI were to function like they do in our naive sci-fi movies, but the mixture of corporate greed and the lack of guardrails we're seeing now leads me to have a more pessimistic outlook on this future.

Stay tuned for virtual girlfriends radicalizing young men and acting as covert influencers for brands and political parties.