Ubuntu infrastructure has been down for more than a day

The outage has hampered communication concerning a critical vulnerability that gives root.

See full article...

 

[quamquam quid loquor]

Hopefully this is just run-of-the-mill DDOS and not anything concerning with respect to Ubuntu releases. All my servers run Ubuntu and I would be royally screwed on any supply chain attack.

 

[thadco]

I can't help but wonder if CopyFail made them turn off access to many, most, or all of their servers until they can patch. I think this because as Dan points out, DDOS protection does exist, so it's confusing why a DDOS attack would take everything down.

 

[benjaminoakes]

Is there a reliable alternative to Canonical's website to get checksums to validate ISOs? I downloaded one yesterday and now I'm suspicious.

 

[benjaminoakes]

There are times where I almost feel sorry for Ubuntu being a bit of a punching bag of the Linux community. Then there are times like right now.

Suck for the users a lot though.

Seems responsible if they (and other Linux vendors) can't mitigate quickly enough. Not great but better than the alternatives.

Edit: I read too fast and misunderstood. I was under the impression Canonical voluntarily took down services to mitigate CopyFail.

 

[polycyclicAnthrocene]

Interestingly Canonical's status page (https://status.canonical.com/) was up and working yesterday but today isn't displaying anything, although the page itself is being served.

 

[mikewhy]

Is there a reliable alternative to Canonical's website to get checksums to validate ISOs? I downloaded one yesterday and now I'm suspicious.

I'd assume any mirror would have them?

Like https://mirror.math.princeton.edu/pub/ubuntu-iso/resolute/SHA256SUMS

 

[Drts]

Some distinct but real possibilities that i’d put money on, neither of them good.

• overwhelm defenders who were likely already in crisis mode to cover for other hacking operations against ubuntu infrastructure
• take down the update and communication mechanism to buy runway to exploit copyfail on as many target hosts across the internet as possible

Opportunistic distraction using asymmetry of effort to maximize impact in both cases.

 

[Resistance]

Is there a reliable alternative to Canonical's website to get checksums to validate ISOs? I downloaded one yesterday and now I'm suspicious.

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served. Checksums should be hosted elsewhere with a different set of credentials required to update it than the credentials required to upload a new file.

That's like asking for two pieces of ID and counting the front and the back of a card separately.

 

[willdude]

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served. Checksums should be hosted elsewhere with a different set of credentials required to update it than the credentials required to upload a new file.

That's like asking for two pieces of ID and counting the front and the back of a card separately.

An unrelated but similar axe I've been grinding recently is the sheer number of financial web sites that have two-factor authentication, with the second factor being a texted/emailed code, but if you need to reset your password all you need to do is get a texted/emailed code. If the second factor can be used to reset the first factor, it's really only one factor!

 

[mikewhy]

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served. Checksums should be hosted elsewhere with a different set of credentials required to update it than the credentials required to upload a new file.

That's like asking for two pieces of ID and counting the front and the back of a card separately.

While I agree, isn't that there some security / verification that can be done from the .gpg file? Or is that moot too because compromised is compromised.

 

[vassago]

I ran into this yesterday when I went to check if the kernel update I just did included a fix for copyfail.
I was able to get the site earlier today, and the kernel version (6.8.0-111) doesn't have a fix, but kmod 31+20240202-2ubuntu7.2 amd64 (which I do have) mitigates it. This is for 24.04.4.

 

[poochyena]

BlueSky was under a DDoS attack recent too.

 

[yababom]

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served. Checksums should be hosted elsewhere with a different set of credentials required to update it than the credentials required to upload a new file.

That's like asking for two pieces of ID and counting the front and the back of a card separately.

It's because checksums weren't originally intended to avert supply chain attacks. They were intended to allow the user to quickly verify integrity of large files downloaded over analog modem connections.

 

[emertonom]

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served. Checksums should be hosted elsewhere with a different set of credentials required to update it than the credentials required to upload a new file.

That's like asking for two pieces of ID and counting the front and the back of a card separately.

I agree that it's not suitable for demonstrating that the file hasn't been tampered with before uploading. But I always thought the point was rather to check that it had downloaded correctly.

 

[Decoherent]

Yep, checksums are just for checking the download. Everything (should be) signed with a very public member of GPG keys owned by the release team, which are available from a number of sources; that's the actual verification.

 

[arsfez]

launchpad.net also appears to be affected

 

[Zukunftsweber]

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served.

You aren't supposed to blindly trust the checksum file. You're supposed to verify its signature first.

Fedora embeds the signature in the same file but for some reason it looks like Ubuntu has elected to put it in a separate file instead.

Admittedly that leads to the question of how you get a key you can trust in the first place, but TOFU is better than hoping an attacker can't compromise multiple hosts (especially when mirroring is in play). The original thought is that you'd trust it via the PGP web of trust, but that hasn't worked out.

In practice you trust the key because it's hosted by an https site owned by the same team and then that verifies the files wherever you find them. Then, since all updates are signed, you have a trusted path for all key updates. And if you can't trust that, you shouldn't use the software either. (Trust starts somewhere.)

 

[benjaminoakes]

An unrelated but similar axe I've been grinding recently is the sheer number of financial web sites that have two-factor authentication, with the second factor being a texted/emailed code, but if you need to reset your password all you need to do is get a texted/emailed code. If the second factor can be used to reset the first factor, it's really only one factor!

I definitely agree with this. It'd be great to have a "consensus checksum" service for well known files like this.

And for anyone who needs them, here are 2 sources:

https://mirrors.mit.edu/ubuntu-releases/26.04/SHA256SUMS
https://mirror.math.princeton.edu/pub/ubuntu-iso/resolute/SHA256SUMS

Both agree on these sha256sums:

487f87faaf547ea30e0aba4d5b53346292571256b25333a978db1692bcee9dd2 *ubuntu-26.04-desktop-amd64.iso
dec49008a71f6098d0bcfc822021f4d042d5f2db279e4d75bdd981304f1ca5d9 *ubuntu-26.04-live-server-amd64.iso
96c7f5fb28a7fe28245331f9bfbe4375f18dd29a4850116ad3c4f60f6700c55c *ubuntu-26.04-wsl-amd64.wsl

 

[TylerH]

I think whether the disclosure was “botched” is a matter of opinion.

What was definitely botched was the rollout of the kernel patch by many distributions.

Nah, you have it backward. The rollout of the kernel patch couldn't even be attempted because the people who published the exploit didn't even bother to notify the distros. QED, yes it was the disclosure that was botched.

 

[A_Very_Tired_Geek]

Hopefully this is just run-of-the-mill DDOS and not anything concerning with respect to Ubuntu releases. All my servers run Ubuntu and I would be royally screwed on any supply chain attack.

According to the article, it's just a DDOS attack coincidental to a somewhat overblown local-only privilege escalation attack.

For the latter, if no one on your payroll or access criteria isn't completely untrusted, there's no reason to panic. The attacker has to be an authenticated users (additional- they also have to have shell access - in the Unix world that's not always a given). No remote attack is possible with that exploit alone. The upstream kernels were patched in the 6.19 release cycle. If nothing else, many entities can roll out their own kernel package independent of Canonical if they're that concerned about the exploit*. I'm aware of the many nuances and caveats surrounding that statement. Yet, that's the important point of FOSS, unlike say MS Windows being largely dependent on Microsoft updates, no one must wait on Canonical, RedHat, Oracle, etc to patch a critical issue if it's already been patched upstream. It's just the norm/convenience/custom to do so.

I ended up doing something similar last month because I got tired of waiting for upstream (and downstream Linux Mint) to pick up some AMDGPU driver bug fix patches for driver bugs that had been a thorn in the side of desktop AMD GPU users since late 2024. Eventually I just switched said desktop over to Cachy-OS which pulled them into their repository already.

*This is complicated, however, by the fact that the patch doesn't integrate smoothly in older kernels. Normally, the kernel userspace ABI doesn't change between releases, but any environment that depends on specific internal kernel behavior and interfaces can be problematic.

 

[adespoton]

Is there a reliable alternative to Canonical's website to get checksums to validate ISOs? I downloaded one yesterday and now I'm suspicious.

But what would you be validating against? A supply chain attack would result in valid ISO xsums.

 

[StarturnCaproc]

There are times where I almost feel sorry for Ubuntu being a bit of a punching bag of the Linux community. Then there are times like right now.

Suck for the users a lot though.

There are many legitimate criticisms towards Canonical and Ubuntu, but I will always be thankful to them for getting me initiated with Linux systems back when I was a kid.

 

[Fatesrider]

Checkmarx, Bitwarden are recent supply chain attacks.

I’ll bet good money, future Ubuntu updates are going to be poisoned. That’s going to be a complete cluster-****

sudo apt upgrade // malware inbound!

I think you'd lose a lot of good money.

They're aware of the issue. A DDOS only shuts down a site, it doesn't compromise what's on it. The people who run these things are some of the best in the world, and will have taken the precautions necessary to ensure that updates are good rather than redirected.

In the meantime, since updates are 100% under user control, sysadmins can just pause them from happening automatically with a few clicks. And when the updates are known to be clean, they can restart them.

After all, Linux isn't run by totalitarians like Microsoft who think your property is theirs to do with as they please.

 

[etrask]

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served. Checksums should be hosted elsewhere with a different set of credentials required to update it than the credentials required to upload a new file.

That's like asking for two pieces of ID and counting the front and the back of a card separately.

Ideally the checksum file is signed with some PGP key that was widely available/verifiable.

 

[A_Very_Tired_Geek]

Nah, you have it backward. The rollout of the kernel patch couldn't even be attempted because the people who published the exploit didn't even bother to notify the distros. QED, yes it was the disclosure that was botched.

There were a number of complications with the disclosure. One of which is that the original POC was needlessly obfuscated by whatever means. Whether it was LLM written or deliberately obfuscated is left as an exercise to those skilled in unraveling code obfuscations. Tangentially, there are competitions for who can contrive the most obscured, but still functional, program code for just about any programming language. Second, while the upstream people were notified, the distributions mailing list wasn't, which is a much lower traffic list the Linux distributions and packagers follow, not necessarily the very high traffic LKML*. It's considered proper to notify both lists for major problems. Besides, it's not difficult to just notify Fedora/RedHat, Debian, Arch, and Canonical/Ubuntu (CC anyone?) thereby taking care of the vast majority of Linux users. Fedora would have coordinated with RedHat (or vice versa), Debian would have managed its downstream automatically, same with Arch, and Ubuntu. To say there are issues with how notification was handled, the notification text's obvious inaccuracies and bad grammar, and the obfuscated POC is an understatement.

*Linux Kernel Mailing List

 

[crepuscularbrolly]

I never understood why someone would trust the site providing the file to also provide the checksum when trying to determine if a file has been tampered with before it was served. Checksums should be hosted elsewhere with a different set of credentials required to update it than the credentials required to upload a new file.

That's like asking for two pieces of ID and counting the front and the back of a card separately.

They also provide a GPG-signed list of the checksums so you can assure yourself that the checksums are accurate. (Get the signing key from the usual GPG key distribution sites.)

e.g.

https://mirror.math.princeton.edu/pub/ubuntu-iso/resolute/SHA256SUMS.gpg

 

[Thorzdad]

Ubuntu and Canonical infrastructure went down hours after researchers released potent exploit code…

So, I’m not in any way conversant on how these things work, but why would researchers release exploit code into the wild? This seems to be a common thing, but it makes no sense to me.

 

[Tactical Finesse]

There are many legitimate criticisms towards Canonical and Ubuntu, but I will always be thankful to them for getting me initiated with Linux systems back when I was a kid.

They got me into Linux as well...and their forum-support model was much of why I very quickly stopped using it and went elsewhere. That and their backporting breakage that made distro upgrading a thing almost as unreliable as Windows upgrading.

 

[A_Very_Tired_Geek]

They also provide a GPG-signed list of the checksums so you can assure yourself that the checksums are accurate. (Get the signing key from the usual GPG key distribution sites.)

e.g.

https://mirror.math.princeton.edu/pub/ubuntu-iso/resolute/SHA256SUMS.gpg

FWIW, a properly executed full compromise of the supply chain would also alter the information on getting a proper GPG key to compare against. PGP doesn't protect you against first instance impersonation, nor can it. The only way you can be sure a public key is real is to receive it from the hands of the person or representative of the organization it represents. That's is why it's not uncommon for people who believe this matters to meet up at places like computer conferences and exchange thumb drives with their public key to prevent just that kind of impersonation. How they can verify the other person really is in fact the person is, of course, dependent on the circumstance.

 

[Bernardo Verda]

Why would "a group sympathetic to the Iranian government" give a damn about Ubuntu, let alone attack it ?

 

[AdamM]

So, I’m not in any way conversant on how these things work, but why would researchers release exploit code into the wild? This seems to be a common thing, but it makes no sense to me.

To light a metaphorical fire under ones ass to fix it.

 

[thearcher]

To light a metaphorical fire under ones ass to fix it.

It's "common" if the maintainer doesn't fix the issue in a reasonable amount of time. This requires the person finding the problem to discreetly report the problem to the maintainer to give them some time to release a fix. In this case, the reporter just released the proof-of-concept without warning.

 

[Decoherent]

FWIW, a properly executed full compromise of the supply chain would also alter the information on getting a proper GPG key to compare against. PGP doesn't protect you against first instance

That's why there are multiple keyservers, run by independent groups. One shouldn't be copy & pasting from the website, but rather using the key ID to check at least one extra server.
If your compromise has reached the point where the attacker has replaced the original key ID with one that's existing on the servers already, and is somehow also cosigned by the rest of the keyring, well, this is no longer the primary problem

 

[torndar]

In the header image caption "An iteration of what happens when your site gets shut down by a DDoS attack." "Iteration" should probably be "illustration".

 

[AstroTom]

Ummm.... are the Ubuntu updates I installed yesterday via 'Software Update' compromised?