AI put “synthetic quotes” in his book. But this author wants to keep using it.

Steven Rosenbaum explains how inaccurate quotes got into his book <em>The Future of Truth</em>.

See full article...

 

[Dark Jaguar]

"There's an old saying in Tennessee - I know it's in Texas, probably in Tennessee - that says, fool me once, shame on - shame on you.... A fooled man can't get fooled again."

Edit: I had to find the EXACT wording for this.

 

[Tactical Finesse]

“As a writer, AI is often a delightful writing companion,” Rosenbaum told me. “When I say ‘writing companion,’ I don’t use that lightly. It’s strangely creative and crafty and unusual in all these ways… and then it betrays you in ways that are just really quite horrible.”

That is called anthopromophizing. And no it isn't creative. These are LLMs--they are limited to what they have been trained on.

 

[MilanKraft]

I'm sorry, while i don't know anything about this guy on a reputational level, at this point I trust ZE-RO non-fiction authors who have the stupid / cahones to use AI in their research, either to synthesize concepts for purposes of minor rephrasing and inclusion, or for finding out "what _____ said about _____ " and then using those quotes without directly verifying.

This clown absolutely deserves egg on his face. We've got along just fine all these centuries without authors needing a brainless chatbot to "do the work" for them or "reduce the boring work," so there's no excuse. It's not like researching online without using chatbots is difficult or heavily time consuming vs the old ways of using libraries (books, microfiche, etc). Do the fucking work / don't take lazy shortcuts with AI and this won't happen. Pretty simple.

 

[spopepro]

“Actually it’s good because it’s a valuable lesson for you” is a bold strategy.

Also, the delay in response was frustrating, but ars reporting on itself here is what I had hoped for going forward. Thanks.

 

[pauleyc]

As the old saying goes: "if you didn't bother to write (or properly research) it, I can't be bothered to read it". And using AI to "surface ideas [whut?], locate articles, summarize themes, identify people or papers" just sounds like laziness to me.

But good luck to Mr Rosenbaum, I'm sure the market for AI slop has a bright future ahead of it.

 

[beheadedstraw]

Here's an idea... just don't use AI when you need 100% accuracy.

 

[mmHmm]

Book about AI and truth contains AI fabrications.

Got to hand it to the publisher's marketing folks - this was a great PR stunt. I've seen articles about this everywhere.

 

[nash076]

Yet another "AI" evangelical who doesn't even understand how the technology works.

 

[beheadedstraw]

Yet another "AI" evangelical who doesn't even understand how the technology works.

Look at their history 5+ years ago and I'd almost guarantee you they were originally Crypto Bros turned AI bros, neither understanding how it works. Only this round you get even more Dunning-Kreuger thanks to AI.

 

[Jim Salter]

This reads like Hunter S. Thompson putting a D.A.R.E. PSA in the middle of Fear And Loathing.

Chatbot psychosis is very, very real, folks.

 

[Wheels Of Confusion]

Here's an idea... just don't use AI when you need 100% accuracy.

Better idea... just don't use AI.

 

[randomcat]

Never heard of this guy, but since he completely botched the entire point of his own book, doesn't really seem to have learned any real "lesson" from this, and fumbled that backwards-ass motorcycle analogy so badly... I don't think I'll be reading any of his stuff.

 

[TheOldChevy]

The author and the editor are responsible for what they publish, not the tool. I don't understand that they do not implement reliable verification process.

Using AI during the creative phase is one thing, trusting it is another.

 

[Kyle Orland]

Book about AI and truth contains AI fabrications.

Got to hand it to the publisher's marketing folks - this was a great PR stunt. I've seen articles about this everywhere.

It's possible that this was a pre-arranged PR stunt, but this whole affair is really stretching the "there's no such thing as bad press" maxim. The increased attention is probably overwhelmed by the massive lack of trust in the book/author from anyone paying attention. Spinning it as the kind of cautionary tale the book warns about after the fact is different from planning it ahead of time as a "PR stunt."

I tend to believe Rosenbaum when he says "I appreciate the book getting some attention, but this would not have been my choice about how to get it."

 

[Tactical Finesse]

Look at their history 5+ years ago and I'd almost guarantee you they were originally Crypto Bros turned AI bros, neither understanding how it works. Only this round you get even more Dunning-Kreuger thanks to AI.

Crypto could at least be explained as "everything money people don't understand about computers, combined with everything computer people fail to understand about money". The LLM bubble? It is just plagiarism on mass steroids.

 

[Platosbeard742]

It didn’t betray you. You failed. You failed in your responsibilities and you failed your ethical standards.

 

[dmsilev]

“I didn’t set out to fabricate anything,” Rosenbaum continued. “What happened is what increasingly happens to journalists, students, researchers, lawyers, and authors working with these systems every day: [There was] AI-generated information that looked authoritative, and some of it made its way too far downstream before being caught.”

It's kind of impressive, really. The old acronym GIGO started out as Garbage In Garbage Out. Then, when people got into the habit of blindly trusting computers even when the source data was crap it became Garbage In Gospel Out. Now, in the bright shining future of AI, those LLMs are trained on actual quotes and data and hallucinate their output, giving us Gospel In Garbage Out. Of course, the next frontier in AI is self-training models, feeding the output back in as training data. So then the acronymic circle will be complete.

 

[Flailsafe]

So much wrong with this.

Is your name on the book? - Yes
Did you state an AI disclaimer? - No
Are you still selling the book as is? -Yes
Have you learned any lesson whatsoever? - No

Get bent and fuck off Steven Rosenbaum.

Edit: Wrong about number #2

 

[iCowboy]

“synthetic quotes” - that's a Newspeak way of saying "lies".

 

[Albino_Boo]

Man writes book to complain about AI used for profit and uses AI to make a profit. Instant karma.

 

[MoranJ2000]

"Rosenbaum made it clear that using AI was the relatively safe “bicycle” option in this analogy. I responded that the supercharged efficiency and catastrophic risk inherent in using AI made it feel a bit more like the motorcycle. Rosenbaum said “that might be fair” and thanked me for “sharpening” his analogy."

This alone tells me that Rosenbaum doesn't understand AI at all. Or analogies...

 

[habilain]

The major problem with AI is that people treat it such as Rosenbaum treat it as "magical": it lets them do things they don't like or aren't particularly good at. In this case, it looks like for Rosenbaum it's research.

The description of "magical" is apt, because as all decent magicians will tell you, magic is all a trick. And if you're using AI to do something that you yourself aren't good at, you're not going to be able to call out when you end up tricked.

 

[jeffronicus]

If I want AI slop I can get it direct and free without paying a middleman like this guy.

 

[randomcat]

“The deck was 100 pages,” Rosenbaum said. “To cut and paste page by page, the text from each page would have been an hour’s worth of work, of mindless cutting and pasting. ChatGPT did it in about four seconds.”

There's no way such a straightforward project would take that long, unless he was incredibly bad at it. The average computer user could do that much "mindless" cutting and pasting, all 100 pages, in five minutes or less.

So he has AI come up with his ideas, and do his research, and he can't even be bothered to do the most basic writing tasks himself because he thinks they take too long at his outlandishly slow pace. Seriously, what does this guy think he is being paid for?

 

[VelvetRemedy]

AI put “synthetic quotes” in his book.​

No. The author did that. The particular mechanism (I refuse to call LLMs "tools") that he outsourced his thinking to matters less than the fact that he failed to practice his craft correctly.

 

[Zoc]

I'm not as much of an anti-AI absolutist as some people here, but this fool shows zero understanding of why he failed to use AI safely and is therefore extremely likely to do so again. It shouldn't be surprising that he then combines that with a total failure to take responsibility. Letting AI create a mountain of garbage for you and then trying to clean that garbage will never work.

 

[graylshaped]

Speaking to Ars in the wake of the controversy, Rosenbaum says he “learned a lesson” and is “going to be much more suspicious” and “reticent to trust” AI outputs going forward.

Nice try, homie. No.

[Looks at the author pic in the article]

Double no. Poser.

 

[Ianal]

A couple of thoughts on the article.

Rosenbaum used AI tools during his writing process, he told me, “to surface ideas, locate articles, summarize themes, identify people or papers I might want to look into.” He draws a hard line between this kind of research and the “actual reporting, narrative structure, interviews, arguments, and conclusions in the book,”

I'm finding that hard line kind of fuzzy to be honest, especially on the distinction between say, summarising themes or surfacing ideas, and narrative structure or arguments.

More generally - I really don't get Mr Rosenbaum's attitude towards AI tools. The whole interview is a litany of ways they got things wrong and were a pain to deal with, but somehow they're still magical, and super powerful and he can't quit using them. That doesn't strike me as a healthy attitude. Mind you, I see the same attitude in gods knows how many LinkedIn posts too (yeah, yeah, I know) - "haha, look at the mess AI made of this - but hey it's still totally awesome, and it's the future, and you'd better get on board with it, and yadda yadda yadda".

His comments on efficiency. I mean - possibly, but how efficient is it to deal with the fallout and reputational fallout from your AI confabulated quotes? Also, the specific example given:

“The deck was 100 pages,” Rosenbaum said. “To cut and paste page by page, the text from each page would have been an hour’s worth of work, of mindless cutting and pasting. ChatGPT did it in about four seconds.”

Oh the humanity. Not a whole hour of work on a project the size of writing a book? Oh noes.

And finally

Rosenbaum made it clear that using AI was the relatively safe “bicycle” option in this analogy. I responded that the supercharged efficiency and catastrophic risk inherent in using AI made it feel a bit more like the motorcycle. Rosenbaum said “that might be fair” and thanked me for “sharpening” his analogy.

Tell me that doesn't sound like a chatbot response to pushing back against something it's produced?

 

[rr6013]

The author and the editor are responsible for what they publish, not the tool. I don't understand that they do not implement reliable verification process.

Using AI during the creative phase is one thing, trusting it is another.

My takeaway was any use of AI, however trivial, warrants equal shared-author subtitle on the cover, Title page and references. AI has earned a “marque of disrepute” it deserves and publishing serves no function shoveling pulp fiction as non-fiction without due warning.

 

[Fred Duck]

When asked directly how he could succumb to some of the AI-related problems his own book warns about, Rosenbaum described what sounds like a dysfunctional relationship with a charming charlatan.

The reason(s) for overreliance on tech can be boiled down to one word: lazness.

 

[r0twhylr]

“Actually it’s good because it’s a valuable lesson for you” is a bold strategy.

It’s only a valuable lesson if you learn from it, and he doesn’t seem to be learning.

 

[cleek]

AI: successfully helping lazy mediocrity avoid competence

 

[barich]

There is no such thing as a "synthetic" quote. There are quotes, and there are lies.

 

[J.King]

“The deck was 100 pages,” Rosenbaum said. “To cut and paste page by page, the text from each page would have been an hour’s worth of work, of mindless cutting and pasting. ChatGPT did it in about four seconds.”

Hey, I've spent many, many, many hours doing mindless copy-and-paste drudgery (not to mention manual input because the source software had no means of putting information in a copy buffer), and it does suck, yes. But a lot of this stuff can be automated properly. If you want to be more efficient, learn how to (really) use a computer. Learn how to automate menial tasks with simple algorithms you can actually understand which will give predictable results. Bugs can still happen, of course, but you can get to something you have confidence in that you won't have to swear at, because you did it yourself only faster.

This is not a live-and-learn sort of thing. He should be mortified and aghast and should never want to touch ChatGPT for work ever again, I would think.

 

[archtop]

What a convenient way to try to rescue a book!

 

[nxg]

I think the more powerful and interesting quote is the one just before the motorbike analogy:

“I don’t do drugs, and I don’t drink, but I presume that that’s kind of the question an addict asks when they’re having one drink too many and they know they are,” Rosenbaum said. “I’ve never been in a place where I thought the tech that I was using was both intoxicating and dangerous

Rosenbaum's behaviour was stupid, but this seems a quite honestly self-interrogating way of framing his problem. I'm sure he knows what he has to do; but then, so do most folk cracking the seal on on their second vodka bottle of the day.

It also suggests that, just as there is a way of drinking responsibly and benevolently, which some folk just can't nail, and so have to cut booze right out, there may be some folk who should just hand their keys to their friend, when it comes to professional use of AI.

 

[Qwertilot]

The author and the editor are responsible for what they publish, not the tool. I don't understand that they do not implement reliable verification process.

Using AI during the creative phase is one thing, trusting it is another.

Because verification turns into an incredibly hard ask. Carefully check every fact, every quote? Because some percentage of them will have slipped in there but they'll all look entirely plausible.

Maybe one day they'll work out to have them know the limits of their certain knowledge, then they might be useful.

 

[ubercurmudgeon]

"No, I won't stop drinking this Kool-Aid."

-- Steven Rosenbaum

(Probably, if I prompt a chatbot enough times to come up with the sort of thing he might say.)