Skip to content
NO “UNWANTED SLOP” HERE

Mozilla says 271 vulnerabilities found by Mythos have “almost no false positives”

The developer of Firefox says it has “completely bought in” on AI-assisted bug discovery.

Dan Goodin – | 138
Meet your new open source coding team! Credit: Getty Images
Meet your new open source coding team! Credit: Getty Images
Story text

The disbelief was palpable when Mozilla’s CTO last month declared that AI-assisted vulnerability detection meant “zero-days are numbered” and “defenders finally have a chance to win, decisively.” After all, it looked like part of an all-too-familiar pattern: Cherry-pick a handful of impressive AI-achieved results, leave out any of the fine print that might paint a more nuanced picture, and let the hype train roll on.

Mindful of the skepticism, Mozilla on Thursday provided a behind-the-scenes look into its use of Anthropic Mythos—an AI model for identifying software vulnerabilities—to ferret out 271 Firefox security flaws over two months. In a post, Mozilla engineers said the finally ready-for-prime-time breakthrough they achieved was primarily the result of two things: (1) improvement in the models themselves and (2) Mozilla’s development of a custom “harness” that supported Mythos as it analyzed Firefox source code.

“Almost no false positives”

The engineers said their earlier brushes with AI-assisted vulnerability detection were fraught with “unwanted slop.” Typically, someone would prompt a model to analyze a block of code. The model would then produce plausible-reading bug reports, and often at unprecedented scales. Invariably, however, when human developers further investigated, they’d find a large percentage of the details had been hallucinated. The humans would then need to invest significant work handling the vulnerability reports the old-fashioned way.

Mozilla’s work with Mythos was different, Mozilla Distinguished Engineer Brian Grinstead said in an interview. The biggest differentiating factor was the use of an agent harness, a piece of code that wraps around an LLM to guide it through a series of specific tasks. For such a harness to be useful, it requires significant resources to customize it to the project-specific semantics, tooling, and processes it will be used for.

Grinstead described the harness his team built as “the code that drives the LLM in order to accomplish a goal. It gives the model instructions (e.g., ‘find a bug in this file’), provides it tools (e.g., allowing it to read/write files and evaluate test cases), then runs it in a loop until completion.” The harness gave Mythos access to the same tools and pipeline that human Mozilla developers use, including the special Firefox build they use for testing.

He elaborated:

With these harnesses, so long as you can define a deterministic and clear success signal or task verification signal, you can just keep telling it to keep working. In our case when we’re looking for memory safety issues we have our sanitizer build of Firefox and if you make it crash you win. We point that agent off to a source file and say: “we know there’s an issue in this file, please go find it.” It will craft test cases. We have our existing fuzzing systems and tools to be able to run those tests. It will say: “I think there’s an issue here if I craft the HTML exactly so.” It sends it off to a tool, the tool says yes or no. If the tool says yes then there’s some additional verification.

The additional verification comes in the form of a second LLM that grades the output from the first LLM. A high score gives developers the same confidence they have when viewing reports generated through more traditional discovery methods.

“In terms of the bugs coming out on the other side, there are almost no false positives,” he said.

Thursday’s behind-the-scenes view includes the unhiding of full Bugzilla reports for 12 of the 271 vulnerabilities Mozilla discovered using Mythos and, to a lesser extent, Claude Opus 4.6. The test cases—meaning the HTML or other code that triggers an unsafe memory condition—are provided in each one and meet the same criteria Mozilla requires for all bugs to be considered security vulnerabilities in Firefox. At least one researcher said Thursday that a cursory look at the reports showed they were “pretty impressive.”

Unlike previous vulnerability disclosure slop, Grinstead said, the details provided by its harness-guided Mythos analysis, and confirmed by the second LLM, and ultimately included in the reports, provide a level of confidence his team didn’t have before.

“That’s the key thing that has unlocked our ability to operate at the scale we’ve been operating at now,” he said. “It gives the engineer a crank they can pull that says: ‘Yep, this has the problem,’ and then you can iterate on the code and know clearly when you’ve fixed it and eventually land the test case in the tree such that you don’t regress it.”

As noted earlier, Mozilla’s characterization of AI-assisted vulnerability discovery as a game changer has been met with massive, vocal skepticism in many quarters. Critics initially scoffed when Mozilla didn’t obtain CVE designations for any of the 271 vulnerabilities. Like many developers, however, Mozilla doesn’t obtain CVE listings for internally discovered security bugs. Instead, they are bundled into a single patch. Normally, Bugzilla reports detailing these “rollups” are hidden for several months after being fixed to protect those who are slow to patch. Now that Mozilla has revealed a dozen of them, the same critics will surely claim they too were cherry-picked and conceal less accurate results.

Of the 271 bugs found using Mythos, 180 were sec-high, Mozilla’s highest designation for internally reported vulnerabilities. These types of vulnerabilities can be exploited through normal user behavior, such as browsing to a web page. (The only higher rating, sec-critical, is reserved for zero-days.) Another 80 were sec-moderate, and 11 were sec-low.

The critics are right to keep pushing back. Hype is a key method for inflating the already high puffed-up valuations of AI companies. Given the extensive praise Mozilla has given to Mythos, it’s easy for even more trusting people to wonder: What’s it getting in return? Far from settling the debate, Thursday’s elaborations are likely to only further stoke the controversy.

To hear Grinstead tell it, however, the details are clear evidence of the usefulness of AI-assisted discovery, and Mozilla’s motivation is simple.

“People are a bit burned from the last year of these slop commits so we felt it was important to show some of our work, open up some of the bugs, and talk about it in a little more detail as a way to hopefully spur some action or continue the conversation,” he said. “There’s no sort of marketing angle here. Our team has completely bought in on this approach. We are trying to get a message out about this technique in general and not any specific model provider, company, or anything like that.”

Photo of Dan Goodin
Dan Goodin Senior Security Editor
Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82.
138 Comments
Staff Picks
f
frogstomp
Ok, this confirms what I suspected.
It works when given a very clear, easily machine verified goal. They've basically taken what they've learned from machine speed training and applied it to finding memory bugs, where basically if the process crashed, you've succeeded. So what is described here at least is quite a narrow window of capability, when given a very clear success/failure model which can be automatically marked by another process, model , or algorithm.
The harness is what turns an AI model onto an AI system, and it is absolutely key to success.
J
Jeff S
Because Mythos won't be used by bad actors at all.
Swear.
That's the thing about AI security vulnerability scanners - once they exist, you have to use them, because bad actors certainly will. But maybe the scanners can help you stay ahead of them?
Hacker Uno
Hacker Uno
I don't know how that's terribly different than the situation now - lots of internet connected software has been exploited by zero day vulnerabilities by malicious actors (state-sponsored or otherwise) for many years now?
Yes, but not at the speed and scale which AI will provide. I have serious doubts that even "AI Defenders" will be able to keep up.

It is a new arms race: How fast will the AI attack tools improve relative to the AI defense tools? If history is any lesson, the defenders will not be able to keep ahead of the attackers. I hope I am wrong, but history says I am correct.

Will AI change our future history? Only time will tell.
r
richardbartonbrown
Great work by Mozilla -- I hope they release the harness "code" and other non-standard setup, along with bug code.

And great article -- appropriately skeptical but not negative.
i
idpwf
The answer is pretty simple: access to Mythos, probably at far below the cost it takes to run Mythos, or indeed any frontier LLM.

LLMs, as massive pattern matching machines, may be pretty good at finding vulnerabilities (especially as vulnerabilities typically follow some form of pattern, e.g. buffer overflow or use after free). But the real test is going to be if LLMs are cost-effective. As @Kurenai said above, if to find bugs you have to burn more in tokens than it would cost in engineer salary... is there a point?

Another question would be given that Greg Kroah-Hartman is doing something similar with the Linux Kernel and a local LLM running on a Framework Desktop, is a frontier model like Mythos even necessary?

I suppose at least for now, during the heavily subsidized period of LLMs it might be a cost effective approach. But the economics of AI are... rather murky and it's difficult to see the current subsidy era continuing.
Because 1. even senior engineers make mistakes and 2. for Anthropic, donating the amount of tokens needed for this is probably a rounding error on their cash flow, while donating senior engineer time is... more complicated.

There is real public good achieved here, and that's exactly the way for commercial companies to earn good will. Especially since covering 20+ years of historical code is largely a one-off (yes, there will be even more competent models; and yes, running them on FF and other major / foundational OSS projects is a win-win)
b
bugsbony
Because Mythos won't be used by bad actors at all.
Swear.
What is your point. It's not just mythos, it's all the models getting better and better. For Mythos they did the responsible thing, give it to some people so that they can fix bugs early, while they are making it (among other things, it's a preview not final) somewhat safer by making it so that people have a harder time using it to find and exploit bugs when it's released. Yes they made some hype too.

The alternative was just releasing it once ready and getting blamed for bad people using it.

There is no alternative of stopping work on AI, other AIs are getting there too, some with less safeguards, and some will probably be open-weights for which any safeguards can probably be disabled.
A
A_Very_Tired_Geek
Will mozilla release the code behind the harness? Seems like they're hyping it up quite a bit.
Eventually very likely. People keep missing the part in the reveal that explicitly states Mozilla doesn't reveal the full details of bug sets for months after they release the patched version. That's Mozilla's long standing policy and I don't see them deviating from it. I read the original blog post and the detailed description of the patches they illuminated. None of them are stand-alone exploits, they all require chaining so their policy of not releasing usable exploit material is still intact. Sure, anyone can do a diff of 149 and 150 and get a shrewd idea of what changed, but it would take someone effort to put together workable exploits without an explanation of why XYZ changed. The logic walk through is as important as the code itself, although you can sometimes get the logic by reading the code. It's just not always the case without a lot of practiced skill in complex code trees.

For now, I think the reasonable stance here is to give Mozilla the benefit of the doubt and to point out that it's not just Mythos one has to be worried about. People have tunnel vision. The forest itself is changing. The newer models are all closing in on useful contributions when properly directed to detecting problems in existing code bases. That's what arm chair experts and luddites are missing. It's a paradigm shift much like automated fuzzers and automatically generated testing harnesses gave us a few years ago (and generated similar backlash). Conservative programmers can bury their head in the sand all they want, but *LM-users are going to blow right past them in the near future much like fuzzer users blew people away sticking to meticulously piecing through code in a debugger, or a skilled debugger user adeptly outperforming someone that never moved past inserting print statements in code. 30 years ago no college CS course taught how to build test harnesses for software nor bothered considering input sanitation as anything beyond a UX exercise. Now, building testing harnesses and security considerations, including fuzzing tools, language agnostic and specific advanced debugging techniques, and input management techniques are part of any well crafted CS course. The question isn't if, it's when managing *LM tooling becomes equally required in CS degrees.
Prev story
Next story