Deloitte will refund Australian government for AI hallucination-filled report

Consulting firm quietly admitted to GPT-4o use after fake citations were found in August.

See full article...

 

[Stickmansam]

If an individual did this, they'd be in jail. I don't believe Australia has the concept of corporate personhood but here in the US, if corporations are people, why can't we throw corporations in jail? Cause there are many, many American companies that deserve decades in prison at this point (I'd give Deloitte maybe 4 years for this, they can be out in 2 years 3 months due to corporate prison overcrowding).

All the common law legal systems, and many civil law systems, have corporate personhood. It's because o ly a person can own property, enter contracts, owe debts, and so on.

So in order for a corporate entity to function and own property, enter contracts, owe debts, and so on, we gave it personhood and test it in part like a person. Associated to that, we give them rights similar to a person, the idea being that rights that can be exercised individually should be able to exercised collectively.

The issue is when the rights extended to corporations go too far and wonky interpretations of constitutional rights, most prevalent in the USA but by no means restricted to that jurisdiction. I won't get into the problems because they are many, but it's more a sign of a broken legal system that corporations have as many if not more rights than a regular person. Corporations should have only a subset of regular person hood rights, not in some cases more.

The equivalent to jail for corporations would be a consent order or various court orders to restrict their liberty and compel actions. Jail at it's core is seperation from society and a restriction on liberty. So the equivalent for a non physical entity is banking them from operating in certain jurisdictions for a period of time.

Probation or parole would be the equivalent of hiring independent monitors or being banned from certain types of operations or contracts. There is also the corporate death penalty of dissolving the corporation and devolving assets back to creditors and shareholders.

 

[Nefarious]

Don't these management consulting companies run on fresh-out-of-college "consultants" who have zero or close to zero real world experience?

ETA ninja'ed

but now we can do it with even cheaper less experienced 'prompt engineers' who didn't go to college and increase the profit margin!

 

[graylshaped]

All the common law legal systems, and many civil law systems, have corporate personhood. It's because o ly a person can own property, enter contracts, owe debts, and so on.

So in order for a corporate entity to function and own property, enter contracts, owe debts, and so on, we gave it personhood and test it in part like a person. Associated to that, we give them rights similar to a person, the idea being that rights that can be exercised individually should be able to exercised collectively.

The issue is when the rights extended to corporations go too far and wonky interpretations of constitutional rights, most prevalent in the USA but by no means restricted to that jurisdiction. I won't get into the problems because they are many, but it's more a sign of a broken legal system that corporations have as many if not more rights than a regular person. Corporations should have only a subset of regular person hood rights, not in some cases more.

The equivalent to jail for corporations would be a consent order or various court orders to restrict their liberty and compel actions. Jail at it's core is seperation from society and a restriction on liberty. So the equivalent for a non physical entity is banking them from operating in certain jurisdictions for a period of time.

Probation or parole would be the equivalent of hiring independent monitors or being banned from certain types of operations or contracts. There is also the corporate death penalty of dissolving the corporation and devolving assets back to creditors and shareholders.

Too many people don't want to understand that what you describe is foundational to modern civilization. The focus should be less on corporations as legal entities with "rights", and more on drawing a distinction between a corporation shielding an investor for legal liability beyond his or her investment, and the misbelief that the financial shield somehow magically protects its officers, agents, and trustees from any legal consequences for the impact of law-breaking they know or should have known was being perpetrated. The "should have known" there is important: if the law is broken on your watch and you should have known, then by definition you are not capable of being given that type of responsibility and should be barred from such.

 

[Malmesbury]

This. I absolutely cannot understand why a consultancy, who's entire business model is "pay us large sums of money for our expert's advice" would rely on a LLM for even as much as grammar advice.

If your expert is ChatGPT, why do I pay you? I can write prompts myself. This is an incredibly fast way to sink your entire business model- if I were McKinsey or one of the others I'd be out there advertising "We know what we're doing, we don't need AI to do it poorly"

I absolutely can understand why a consultancy would do this.

The way it works is that the salesdroid asks for an estimate of how much to do a project. They then bid 70% of that to get the work. Or even less.

So the managers on the project start out behind. So they pressure the inadequate number of staff to work harder.

The staff, battered by long hours being treated like shit, take a short cut. One that is provided by the company - all the big consultancies now provide an LLM as part of setup.

So they deliver a report on time. The manager get patted on the back. The partners get their pay days. The staff get told nice job and that pay rises are next year.

Yes, this is enshitification.

“But if I make my deadlines and cost recovers goals for 2 years, I get the promotion and the next guy…”

 

[Verumtamen]

Are they surprised really? These companies are funded by the cheap labor provided by fresh-out-of-college students. What did they think was going to happen?

 

[thawking]

Ars, i love you, but for the love of god stop referring to flat-out bullshit as "hallucinations"

 

[e.halap]

Expensive useless cunts in expensive suits. Their only skill is to impress the incompetent useless cunts in management.

 

[darkowl]

ChatGPT openly acknowledges, if you ask it, that it is best at summarizing short, clear content. Since that's the content you least need summarized, it seems to be a use-case in search of a use.

Well, Apple tried this with news headlines. We know how well it worked out.

 

[macosandlinux]

The astute reader will guess that yes, they did proceed with reckless abandon, shortly after I departed that company. One of my coworkers kept a copy of the presentation, and as they ran into each issue, annotated that with the date they did run into it, and some little screenshots from slack and emails of people freaking out, and then when it was all done, he realized it was 12 slides long, and printed it as a calendar and sent it to me

That's just golden. Best. Gift. Ever.

 

[macosandlinux]

Your process of 'baselining' exposes a fundamental flaw in many people's use of GenAI - potentially the same attitude that lead the authors of Deloitte's report to inserting a load of random citations.

The fact that you've checked a few summaries and they looked good should give you absolutely no confidence about the likelihood of future summaries being entirely accurate. That's simply not how current GenAI works. The risk of hallucination is inherent and continues to exist even in situations where the output is often correct.

While you can put in the effort to double-check that output is a representation of the text, how can you possibly know that nothing important has been missed without reading the whole thing?

If you need an accurate summary you can't trust GenAI. And if you're happy to accept the risks that the summary isn't accurate, then you don't need the summary.

My question is why would Deloitte use an off-the-shelf LLM implementation, and not a fine-tuned, custom trained model for their own consultancy needs? Too hard to do, takes too long, or don't they get the need for a Retrieval-Augmented Generation (RAG) database with the citation docs? Not sure how basic their workers implementation was, but damn, they sound dumber than they already are.

 

[macosandlinux]

Claude Haiku 3.5 | Here's a playful rhyme about Deloitte:

There once was a firm called Deloitte,
Where consultants would work day and night,
With spreadsheets galore,
And reports to the core,
Their PowerPoints always just right!

The rhyme pokes gentle fun at the consulting world, highlighting Deloitte's reputation for extensive reporting, long work hours, and meticulous presentation skills. It's a light-hearted take on the professional services giant, capturing the essence of corporate consulting with a touch of humor.

Note: The bold text is the actual humor to me.

 

[fuzzyfuzzyfungus]

Earlier this year, Deloitte declared it would start using generative AI for its reports as a way of enhancing the value provided to its clients. I don't remember if they said it in a specific report or not, but I recall seeing it.

The citation issue continues to trip people up across the spectrum, from lawyers to business analysts. It's striking how many supposedly smart people do not understand the limits of the tools they insist will deliver such amazing value.

What puzzles me is why fields like consultancy and law don't seem more concerned about detecting nonsense citations before they turn in their work.

Regardless of what you do or don't know about LLMs; dodgy citations are perhaps the single most damning flavor of error in what is supposed to be a document of authoritative reasoning because they combine almost the same level of triviality in detection as typos or misspellings(they don't just leap off the page; but someone who knows nothing about the subject can run your bibliography through a search engine one line at a time quite easily); while, unlike fiddly detail proofreading, casting substantial doubt on the material they are included in.

Unless someone is just preening; citations go where it's important. You are invoking an authority, or referring to the results of a study, or incorporating by reference a paper to justify your line of thought. Perhaps you are passing quickly over a deeper subject that the reader may seek clarification on. In all of those cases it's critical that what you are citing actually exist; because if it doesn't all those implied purposes are just lies.

It just seems weird to see fields that(ideally among other things; but definitely in part) trade in prestige and credibility just sort of...not...bothering to sanity check such a mistake when they probably would be concerned if the document was littered with typos or one of the figures was inadvertently inserted upside down; despite those sorts of errors potentially having to relation to the paper's quality of analysis at all.

 

[norton_I]

I sometimes think the only people hiring Deloitte are former employees who owe favors.. Look at the John Oliver episode - all they crank out is shit.

Guess who are former Deloitte consultants? Satya Nadella and Sundar Pichai.. Explains SO MUCH.

This sounds like conspiracy rant. Pichai worked for McKinsey not Deloitte, but for a short period of time, and to my understanding as an entry level consultant. Not a partner or anyone making strategic decisions for the firm. I haven't heard that Nadella worked for either, if so it would have been in the 90s.

In any case the idea that either would be acting to repay favors from such a job decades ago seems ludicrous. You can just blame them for their actions directly.

 

[mehaase]

While I am in favor of sticking it to people who use AI like this in any way possible, I think you'd probably find it hard to make the case that it's "defaming" her.(snip)

Edit: Due to the number of downvotes, I have to wonder if I just put my point across poorly or if there's just a bunch of AI defenders downvoting. (Snip)

You’re getting downvoted bc lay people have no idea how the law actually works but still fancy themselves experts.

 

[adamsc]

Consultants are most heavily used by managers who are out of their depth, in companies where almost everyone is out of their depth, where most of upper management is clinging onto their jobs for dear life. In such companies, the executives know that's the situation, because they're in the same boat, so they put no credence in the actual judgement of their team and want to see external advice. It's a sign that the organisation is rich on head-count, low on talent, and any trust in staff to make correct decisions has evaporated.

I’ve seen this most commonly but there was one other variant I saw at a large university. They had a toxic, dysfunctional middle management layer which always covered for each other (e.g. my boss showing up to work drunk was unsurprising and nothing came of it) but for various cultural reasons they weren’t willing to sack people. That left senior management aware that they were not getting an honest assessment of pretty much anything, so the solution was to pay for a bunch of 24 year olds in nice suits to talk to the actual workers without the management filter in-between saying that all of the failures and delays were due to the exceptional difficulties of the problem rather than, hypothetically, the super expensive Oracle software not actually being good despite what the salesperson said over thousand dollar lunches.

The reports were more accurate than the internal ones but the cost was staggering and one of the reasons why I left quickly was the realization that there was never going to be accountability for the people whose projects kept fizzling. There were just too many nepotism hires, too many people whose primary life achievement was going to the right school, for there to be much enthusiasm for measuring job performance on merit.

 

[adamsc]

Due to the number of downvotes, I have to wonder if I just put my point across poorly or if there's just a bunch of AI defenders downvoting. My point is that the law as it is written today in most countries requires defaming to somehow damage someone's reputation. That's what I'm saying would be hard to prove in this case. I'm all for the AI companies being charged with fraud for knowing that their software is riddled with bad output.

I think you’re seeing the hunger people have for balance against unethical companies. You’re quite right, though, that simply wanting something to be illegal doesn’t make it so – and that path leads to Trumpism so even joking about it carries a societal risk.

 

[Wybaar]

Deloitte business model: not faster, not cheaper, not better.

I know that people cite "Good, fast, cheap. Choose two." as a rule of thumb for project management. But this clearly supports my modification: "Good, fast, cheap. Choose at most two." They chose zero (or maybe one; I bet they 'wrote up' that report relatively quickly.)

 

[sixstringedthing]

Claude Haiku 3.5 | Here's a playful rhyme about Deloitte:

There once was a firm called Deloitte,
Where consultants would work day and night,
With spreadsheets galore,
And reports to the core,
Their PowerPoints always just right!

The rhyme pokes gentle fun at the consulting world, highlighting Deloitte's reputation for extensive reporting, long work hours, and meticulous presentation skills. It's a light-hearted take on the professional services giant, capturing the essence of corporate consulting with a touch of humor.

Note: The bold text is the actual humor to me.

Your poem is shit.
Fuck off with your A.I. Slop.
Okay thank you bye.

(Not meant to be overtly offensive but if we're going to Haiku then I can Haiku too.)

 

[jdale]

My question is why would Deloitte use an off-the-shelf LLM implementation, and not a fine-tuned, custom trained model for their own consultancy needs? Too hard to do, takes too long, or don't they get the need for a Retrieval-Augmented Generation (RAG) database with the citation docs? Not sure how basic their workers implementation was, but damn, they sound dumber than they already are.

Why would [subject of article] do [a thing that the article doesn't say they did]?

I think you're making two wrong and unsupported assumptions. First, about Deloitte's work practices, and second that a properly trained system wouldn't make these errors.

 

[Antron Argaiv]

The consulting industry is best understood as a way for the executive class to divert large sums of company resources to their friends and younger members of their class in exchange for applying the veneer of rigor to executive decisions. They not only don’t care how much of other people’s money they spend, it’s actually seen as better to spend a lot because when the business decision turns out to be flawed they have essentially prepaid for unlimited BS on demand to prevent accountability for the executives. In addition to the inexperienced recent grads, these companies maintain a stable of respectable senior guys of the right background who’ll show up in very nice suits and swear up and down nobody could reasonably have expected that pivot to blockchain to be anything less than a goldmine and that it’d be a major strategic error to factor the actual negative returns into someone’s bonus calculations. The overhead is paying for that service, too.

"A large, very well known, consulting organisation examined our situation and recommended..."

Cover for executives making decisions that could affect the company's profitability. "We spent half a million on this study. There's no way it could be wrong." (and even if it is, it's the consultants' fault, not mine)

 

[ezs]

The word "hallucination" in the context of LLM output should probably be deprecated.

Everything an LLM produces is a "hallucination". Some of the output so just happens to be in the training dataset or in the (search augmented) prompt. This is a direct consequence of the fact that an LLM is a hugely lossy representation of the dataset. It might be an impenetrably complicated and "clever" representation but a hugely lossy one nonetheless. Couple that with the stochastic sampling used for (practically all) generative tasks and it is obvious that no output can be trusted without significant additional verifications.

 

[GrimR3]

The entire report is suspect and should be refunded. In my own not a lawyer world I agree that publishing a report with face citations attributed to a real person is defamation. Sadly I suspect no one is adding Deloitte to a banned vendor list for defrauding the government.

 

[macosandlinux]

Why would [subject of article] do [a thing that the article doesn't say they did]?

I think you're making two wrong and unsupported assumptions. First, about Deloitte's work practices, and second that a properly trained system wouldn't make these errors.

Yep, that's fair. I don't know what exact setup they used, so that's up for debate. Maybe more details will come to light thanks to the reporting.

 

[10Nov1775]

It does worry me too that apparently nobody is sanity checking these reports before their general publication? Like if 10+ citations outright did not exist then Deloitte's analysis must have just been taken at face value with no serious review, and we are presumably using this to justify government policy?!

Even ignoring the generative AI aspect that strikes me as extremely concerning.

Yeah, it's the seemingly total lack of review by there organizations, rather than their use of an AI tool, that worries me the most.

LLMs have uses, but simply cannot be used for ANYTHING important without being thoroughly checked.

This is not necessarily a damning flaw; after all, whenever it costs less time and energy to check and correct something than it would to do it ourselves, even an error-prone tool could be useful.

But you have to check it.

 

[graylshaped]

Well, Apple tried this with news headlines. We know how well it worked out.

Apple said "Hmm. This is not ready for prime time," and turned it off.

OpenAI doubles down, yet again.

 

[Tofystedeth]

I almost want to contact the scam service from the scam poster above just to see if I can rustle someone's jimmies but man, it's a monday and I'm not ready for that yet

I've done it once, but only because it was a real doctor's office and not like a random drop shipper or recruiter or crypto company or whatever. Called them up, and left them a message that whoever they hired to do their marketing was spamming the forums of national publications for an office serving clientele in a single city. Which was both a waste of their money and just going to annoy people instead of being useful advertisement.
ETA: I framed in a "someone you paid to a do thing is doing it very wrong way" but I definitely had undertones of "I really hope you didn't choose this particular campaign because that would be stupid and evil."

 

[graylshaped]

Yeah, it's the seemingly total lack of review by there organizations, rather than their use of an AI tool, that worries me the most.

LLMs have uses, but simply cannot be used for ANYTHING important without being thoroughly checked.

This is not necessarily a damning flaw; after all, whenever it costs less time and energy to check and correct something than it would to do it ourselves, even an error-prone tool could be useful.

But you have to check it.

One of the hardest things to teach in business is appropriate delegation. Matching the demands of the task to the skills of the delegatee is not optional. This remains the core of my disgust for the people selling and shilling for these tools: they are not capable of reliably performing the tasks they are being sold to do.

The best managers are the best delegators. The best business leaders are the ones who employ, teach, and require effective delegation, which is the exact opposite of how the current "AI" industry is operating.

 

[sherkaner]

Society needs to start assigning responsibility for these things correctly.

If a human created a report by hand with a bunch of made up citations, there would be some serious repercussions. When that human chooses to use an LLM and gets that result, there should be no difference.

Instead we keep seeing LLM users issuing weak apologies, like there was nothing they could do – "silly LLMs, what ya gonna do, amiright?"

The LLM is a tool, and the user of the tool bears responsibility for the result.

 

[graylshaped]

Society needs to start assigning responsibility for these things correctly.

If a human created a report by hand with a bunch of made up citations, there would be some serious repercussions. When that human chooses to use an LLM and gets that result, there should be no difference.

Instead we keep seeing LLM users issuing weak apologies, like there was nothing they could do – "silly LLMs, what ya gonna do, amiright?"

The LLM is a tool, and the user of the tool bears responsibility for the result.

Well, the complication is that the company is telling its employees to use the CEO's idiot nephew for important work, and then turning around and blaming the employees for the mistakes of the idiot nephew while telling its customers how awesome the nephew is and how lucky the customers are for paying to have access to it.

edit: Had I known there was an idiot nephew hanging around willing to give me a butthurt downvote for my bingo card, I'd have offered that comment earlier. Thanks!

 

[squiggit]

One thing that strikes me about this story is how weirdly indifferent people at so many levels seem to be.

Consulting company is using nascent, still developing technology known to produce errors to help write their report... yet seemingly doesn't care enough to proofread or edit the document.

Government discovers their new report is based partially on fabricated data... but there's seemingly no investigation into the company that produced the fraudulent report...and it's only worth a partial refund because they might still use the rest of the report I guess.


I feel like ten years ago this story would have led to some kind of inquiry into whether the government was intentionally defrauded, maybe a lawsuit... but "oh well we used chat gpt" and suddenly nobody gives a shit? Even the victims of the misinformation seem weirdly okay with being lied to as long as it's an LLM doing it.

 

[marsiglio]

"the substance of the independent review is retained, and there are no changes to the recommendations." Fantastic - just what any PhD candidate would like to submit to their committee, or for that matter, a fifth grade book report. They get paid money for this crap?

 

[Faceless Man]

Don't these management consulting companies run on fresh-out-of-college "consultants" who have zero or close to zero real world experience?

ETA ninja'ed

As we've learned from the Brittany Higgins case, the government (or at least the previous LNP government) was run on fresh-out-of-college "senior advisors" who have zero or close to zero real world experience.

 

[Kenjitsuka]

used as part of the technical workstream to help "[assess] whether system code state can be mapped to business requirements and compliance needs."

Do they get their mumbo jumbo from old Dilbert comics mocking them?

 

[graylshaped]

As we've learned from the Brittany Higgins case, the government (or at least the previous LNP government) was run on fresh-out-of-college "senior advisors" who have zero or close to zero real world experience.

But did they have big balls?

 

[The Wonderful Wizard]

So, Deloitte are going to publish the system prompts used for this "tool chain" so people can check for biases that might have made it into the final "independent" report, right?

RIGHT?

That would undermine the main point of asking those consultancies for a report

 

[The Wonderful Wizard]

if I were McKinsey or one of the others I'd be out there advertising "We know what we're doing, we don't need AI to do it poorly"

McKinsey have been caught recycling reports and failing to swap out the names of the places in their recommendations.

 

[The Wonderful Wizard]

Interviewer: This report that Deloitte produced for Australia's Department of Employment and Workplace Relations this week...

Deloitte executive: The one with the false citations?

Interviewer: Yeah.

Deloitte executive: Yeah, that’s not very typical, I’d like to make that point.

Interviewer: Well, how was it un-typical?

Deloitte executive: Well there are a lot of these reports going around the world all the
time, and very seldom does anything like this happen. I just don’t want people
thinking that Deloitte's reports have false citations.

Interviewer: Did this report have false citations?

Deloitte executive: Well, I was thinking more about the other ones.

...

View: https://m.youtube.com/watch?v=3M7SzS_5PlQ

 

[Ma1achijr]

As an Aussie, there is nothing more Australian Government then paying consultants for nothing and trying to automate penalties. They need the automated penalty revenue to pay for more consulting on the automated penalty systems.

 

[zeroplusone]

Earlier this year, Deloitte declared it would start using generative AI for its reports as a way of enhancing the value provided to its clients. I don't remember if they said it in a specific report or not, but I recall seeing it.

The citation issue continues to trip people up across the spectrum, from lawyers to business analysts. It's striking how many supposedly smart people do not understand the limits of the tools they insist will deliver such amazing value.

Strictly speaking, Deloitte wasn't wrong; it delivered the final installment of value back to its client. LOL.

 

[qchronod]

I wish I hadn't taken yesterday off from surfing the internet all day.

As some who recently got downsized from Deloitte (because they bought and killed a successful Engineering Services company in just over 2 years), this doesn't surprise me at all. Since the beginning of 2024, they've been pushing the use of AI in pretty much every aspect of their "consulting". They even have a hilariously cringe training video about someone who waited until the last minute, then used AI to write up a report for them w/o checking sources, then magically was able to find new real sources that gave even better results.