AI put “synthetic quotes” in his book. But this author wants to keep using it.

Steven Rosenbaum explains how inaccurate quotes got into his book <em>The Future of Truth</em>.

See full article...

 

[UweHalfHand]

I actually wonder if online gambling and now LLMs are, like video games for my generation, exposing people who have addictive personalities but who have avoided substance abuse for one reason or another...

Yes. And like any new drug, there’s a certain fraction of a population that’s especially susceptible, and on whom the effects will be catastrophic. I grew up in Alaska. Alaska Natives were never exposed to alcohol in pre-modern times, you can’t make weak beer on the tundra or an ice shore, and as a result they have vastly less tolerance for booze than people of European descent. Whiskey and other high-alcohol drinks have been, yeah, catastrophic for a lot of them. I do think something like that is what we’re seeing with LLMs for and on ourselves now. I agree with Sarty’s comment earlier, the more I see the less I want to experiment on myself. I fear where it could go.

 

[AdrianS]

Impressive for an author to unwittingly get caught by the technology he's writing about. From this we can deduce that:

• the author really doesn't understand AI
• he somehow did not take "AI delusions" seriously while writing about it
• he did not actually read the source material that he quotes
• there wasn't fact-checking as part of the writing or editing processes (despite his claims)

Sounds like a great, informative book /s

Sounds a pretty accurate re-run of what led to Benj's leaving Ars.

 

[Eldorito]

It's hard to keep holding this position in light of the current state of the models. Yes, limited to what they have been trained on, but they have been trained on intelligently producing chains of thought that, sometimes, produces things humans just haven't been able to do. See the below article for one of the most astonishing results, literally just published yesterday.

https://www.scientificamerican.com/...-erdos-problem-and-mathematicians-are-amazed/

One thing OpenAI has at its disposal is ridiculous amounts of computing power, and yes, AI can take existing mathematics and run through countless simulations and ideas. So this working isn't really surprising. It didn't come up with new mathematics, it just came up with a solution to an existing problem.

Absolutely nothing about that is "intelligent". We've been doing that with ML and throwing tonnes of hardware at problems for years now. The real question you should be asking is - why are we throwing hundreds of billions of dollars to write crap books instead of putting it towards medical and science breakthroughs?

 

[Redmess]

The irony in this story has layers, and it is delicious.

 

[Redmess]

Impressive for an author to unwittingly get caught by the technology he's writing about. From this we can deduce that:

• the author really doesn't understand AI
• he somehow did not take "AI delusions" seriously while writing about it
• he did not actually read the source material that he quotes
• there wasn't fact-checking as part of the writing or editing processes (despite his claims)

Sounds like a great, informative book /s

I don't know, it sounds like he does understand AI and its flaws and dangers, but he's too enamoured with it to take his own warnings seriously regarding his own work. He really does sound like an alcoholic who knows it is bad for him but just can't stop himself from drinking anyway.

 

[MilkyBarKid]

Was curious, so looked at what other works he's authored:
https://www.amazon.com/stores/author/B0F7J3KB8C
2 books on how to curate content, and a third book called Curation Nation, how to win in a world where consumers are creators.
And then this book.

I do like his hat.

My days of not taking him seriously have certainly reached a middle.

It seems like there’s a desire for a sort of quote-and-source tracking AI tool, to the extent that authors are instead misusing these generative tools for this task. They really aren’t well suited to it: when you ask an LLM to make sure to generate a real quote, it dutifully generates random text that looks like a real quote!

Is it really not possible to come up with some sort of RAG based system that actually looks through your papers and provides links instead of text? That doesn’t seem very difficult…

RAGs fail. They’re still probabilistic at their root.

Edit for less glibness: A good role of thumb for how likely a claimed capability for LLMs is is “If someone could get an LLM to do this reliably, would Sam Altman be able to shut up about it?” Getting the error rate low enough to not have to check is one of those things. If RAGs were as good as some people sold them, we’d see the resultant accuracy being sold in every model. In actuality, Anthropic changed some of the recall benchmarks to be more “agentic” rather than admit their ability to pull particular obscure facts from the context wasn’t improving.

 

[elboku]

Synthetic quotes?

Is that what liars are now calling their lies? The term AI is itself a lie. And that lie may become our ultimate downfall.

 

[Wheels Of Confusion]

The irony in this story has layers, and it is delicious.

I really don't find any of this "delicious." It's a sickness being pushed on all of us irresponsibly at every level.

 

[marsilies]

Of course her finds it delightful, it's designed to be sycophantic to a fault, this kind of people find that delightful.

I watched a recent video discussing Richard Dawkins recent gullible fall into thinking an LLM was sentient, and the quotes from his article and context shows he really just fell for the sychophancy of the LLM, while also presenting an intellectually dishonest argument.


View: https://www.youtube.com/watch?v=02pBnDkV0rQ


Link to Dawkin's article if you want the raw, unfiltered, absurdity:
https://archive.is/6RdK9

(I'd link to archive.org , but it didn't archive the whole article, just the paywall).

 

[marsilies]

I can't fathom running across so many examples of these tools fucking things up and still being like "yeah but it's faaast." It's just.... man, I don't even know. Like the "extract this quote verbatim" and the tool STILL doesn't do it, that really should be some kind of sign.

It reminds me of an old meme about Little Ceasar's Pizza:

With LLM companies, the variation seems to be:

LLM Companies: It's fast and cheap.

Us: Is it good?

LLM Companies: It's FAST. And it's CHEAP.

Of course, that "cheap" attribute is largely because they'd currently subsidizing the costs, and each new model gets ever more expensive to run, while only marginally more reliable/accurate.

 

[Grey Bird]

As far as LLM AI goes, "The only winning move is not to play." To quote Joshua/WOPR from War Games.

 

[J.King]

Link to Dawkin's article if you want the raw, unfiltered, absurdity:
https://archive.is/6RdK9

(I'd link to archive.org , but it didn't archive the whole article, just the paywall).

Unfortunately archive.is is playing its RT-redirection game again.

 

[acefsw]

I'm not saying it is a person or that you should consider it is intelligent in any sort of human sense, I'm saying you can't simply say "It's limited to what it is trained on", and wave it aside as if it can't be useful or produce any new knowledge. This is clearly false, the benefits of these general purpose system for producing knowledge has already been proven by this example, and it will likely only get faster from here. I.e., not intelligent, but "creative" in the sense that it can create new things that humans have not, including knowledge and mathematical proofs.

It didn't produce new knowledge and relied on known mathematical tools.

"Harvard University mathematician Melanie Matchett Wood says humans’ progress was probably limited by their belief that the conjecture was true. If all the experts assembled after the fact to parse the LLM’s answer had instead spent the same time seeking a counterexample, she says, they would have found one. “Maybe people should be spending more time, you know, playing devil’s advocate,” says Wood, who had also provided a commentary for OpenAI.

This is plausible because the AI’s solution was, in hindsight, a straightforward approach that no human had ever attempted despite the fact that the tools had already existed. Such circumstances are thought to be uncommon for major unsolved math problems. “I guess it got lucky that it found one of the cases where experts tried and missed something,” Litt says. Genuinely new, groundbreaking ideas remain beyond the reach of current LLMs, instead leaving the machines to mine the literature for rare gems where humans missed a relatively simple approach. "

Edit to add: that the results, like all such "incredible" results that people credulously gush over, required human intervention. A lot more than people likely know because OpenAI and such don't release the raw results, just summaries.

"The experts also hastened to add that, without humans intervening to “clean up” the AI’s work, the result wouldn’t be so convincing."

 

[k h]

If you sell it, list the model as author and yourself as co-author (or promptist).

I want to be a promptist when I grow up! <sigh>