New physical attacks are quickly diluting secure enclave defenses from Nvidia, AMD, and Intel

On-chip TEEs withstand rooted OSes but fall instantly to cheap physical attacks.

See full article...

 

[TheRealSimDan]

Curious, does this or a similar attack apply to Apple’s chips? All those iPhones would make attractive targets for hackers.

 

[Phone Sterilizer]

Sorry. Physical security is a thing. If people have direct access to hardware, all bets are off.

 

[breze]

Even Moore—a security veteran with more than three decades of experience—told me: “The surprising part to me is that Intel/AMD would blanket-state that physical access is somehow out of scope when it’s the entire point.”

This seems like a weird way of looking at it. The hardware vendors put physical access out of scope because it is more or less impossible to actually guarantee that somebody with physical access can’t break the security feature.

Others who misstate the TEEs’ protections provide more accurate descriptions elsewhere. Given all the conflicting information, it’s no wonder there’s confusion.

I guess you could call it conflicting information, but we have the information provided by the people who make the chips, and the information provided by software/hosting providers. The latter are telling us that they are providing services based on imaginary features that the former specifically says don’t exist. So, it shouldn’t really be too confusing!

 

[dangoodin]

This seems like a weird way of looking at it. The hardware vendors put physical access out of scope because it is more or less impossible to actually guarantee that somebody with physical access can’t break the security feature.


I guess you could call it conflicting information, but we have the information provided by the people who make the chips, and the information provided by software/hosting providers. The latter are telling us that they are providing services based on imaginary features that the former specifically says don’t exist. So, it shouldn’t really be too confusing!

It's confusing because the chipmakers don't make this exclusion explicit. I mean, can you find pages on any of their websites that spells this out? It's also unclear because, at least in the case of Nvidia, it seems to be saying Confidetial Compute DOES protect against physical attacks. Further, as noted, many TEE users are using them for things that are outside the threat model, or making assurances that the TEEs they use withstand attacks they can't. If Moore was surprised at the limitation, I think it's fair to say the chipmakers haven't done a good job making it explicit/well known. So yeah, conflicting, confusing, and misleading.

 

[dmsilev]

Curious, does this or a similar attack apply to Apple’s chips? All those iPhones would make attractive targets for hackers.

Since it works by putting an interposer in between the motherboard and the actual DRAM stick, adapting the attack to soldered-in RAM would be challenging. Not impossible I'd bet, but not "smuggle in a briefcase-sized thing into the same room as the target computer, and then done in a few minutes". More like "We stole your laptop or phone and can work on it at our leisure in a lab".

 

[alexvoda]

Sorry. Physical security is a thing. If people have direct access to hardware, all bets are off.

That kind of was part of the value proposition of TEE.

 

[Zeebee]

Curious, does this or a similar attack apply to Apple’s chips? All those iPhones would make attractive targets for hackers.

Well the attack also requires a compromised kernel... so in the case of an iPhone you'd have to unlock and first jailbreak the phone... at which point the rest is redundant.

While the attack is interesting from a research standpoint given that you need physical access and a compromised kernel (which is, itself, a pretty big problem), it is not an easy to attack to pull off.

 

[ArcaneTourist]

This is a bunch of noise. The reasoning about this hasn't changed in literal centuries. It has never been possible to guarantee security after physical compromise and never will be.

You can protect against man-in-the-middle attacks. Centuries ago this might have been by writing messages on a shaved head, using one-time cipher pads, or whatever. Today, we protect against man-in-the-middle attacks using cryptography.

But, the end-points? Centuries ago, if the recipient of your secret message was actually a double agent, you'd lost. Today, if your server or other endpoint is compromised, you've lost. And, no matter how far you go into the future, this will still be true. Simple logic tells us that you once you've taken control of a system, you can set it up to continue to do all its previous tasks but add anything else you want like logging, extra communication channels, manipulation of data, whatever.

 

[killerhurtalot]

Of all the things that AI/cloud providers do wrong, physical access to their server farms are one of the last things I would worry about...

 

[dangoodin]

While the attack is interesting from a research standpoint given that you need physical access and a compromised kernel (which is, itself, a pretty big problem), it is not an easy to attack to pull off.

As Moore notes, these types of attacks are a big threat to edge servers, which are often located in very remote facilities with less-than-great physical security. Three minutes is all the attacker needs. What's the "big problem"?

 

[eightycc]

Curious, does this or a similar attack apply to Apple’s chips? All those iPhones would make attractive targets for hackers.

Apple's Unified Memory Architecture (UMA) integrates DRAM and the SoC on a single carrier. The downside is that DRAM can't be upgraded, but it does introduce a barrier to the attack outlined in this article by making the wires attaching DRAM much less accessible.

 

[Rabid Badger]

It's almost as if running all of your infrastructure on computers you don't control and never see isn't a good idea.

Weird.

 

[FishInABarrel]

Of all the things that AI/cloud providers do wrong, physical access to their server farms are one of the last things I would worry about...

As the article notes, it doesn't matter how good your physical security is when the government shows up with a warrant.

 

[Danathar]

I'm going to get meta about this.

If something isn't physically with you, whether it's a computer, your car, or any piece of equipment, it's not secure. Period. You don't have control over it. Even if it's locked in a building you own on your property, it's still at risk. Unless someone is physically present who cannot be compromised, nothing stops a determined attacker from bulldozing through your walls and taking your equipment or modifying it for things you don't want it to do.

This is the reality cloud vendors happily hand wave away. When you surrender your infrastructure to someone else, you're no longer in charge. They are. If bad actors can reach your equipment, all bets are off.

 

[macr0t0r]

Is this an argument for the Framework Desktop design: soldered in RAM? Place a Portal Turret in the server room? Honestly, a server is just like a vault: no person will be able to casually grab the contents. But, nothing can survive an attacker if you give them physical access and plenty of time to work alone in an unguarded location. And, if the "attacker" is just an insider...or a government agency... I suppose this is why companies are talking about putting servers into low orbit.

 

[dangoodin]

Sorry. Physical security is a thing. If people have direct access to hardware, all bets are off.

This is a bunch of noise. The reasoning about this hasn't changed in literal centuries. It has never been possible to guarantee security after physical compromise and never will be.

You can protect against man-in-the-middle attacks. Centuries ago this might have been by writing messages on a shaved head, using one-time cipher pads, or whatever. Today, we protect against man-in-the-middle attacks using cryptography.

But, the end-points? Centuries ago, if the recipient of your secret message was actually a double agent, you'd lost. Today, if your server or other endpoint is compromised, you've lost. And, no matter how far you go into the future, this will still be true. Simple logic tells us that you once you've taken control of a system, you can set it up to continue to do all its previous tasks but add anything else you want like logging, extra communication channels, manipulation of data, whatever.

Yes, it's clear that you two fully understand the threat model. That's not what the article is about. The question is: do TEE users and even the chipmakers themselves get it? So many of them keep pumping out marketing and white papers and such making assurances that contradict the limitations. Some even use the TEEs for these purposes. Like the article says, a quick search turns up dozens of organizations, many mature and well-funded, who make statements that are at best misleading.

I don't understand why so many comments are critical of the story rather than of the many people who continue to market and use TEEs for protections they don't provide.

 

[jhodge]

It's interesting that this issue directly impacts the shared-responsibility model that the cloud depends on.

When we all had our own infra in our own building, it was clear that physical security was our responsibility.
When infra was hosted in a colocation facility, we toured the facility and inspected the security arrangements, but also trusted that the colo operator would follow procedures.
Now, it's "in the cloud" and the shared-responsibility model requires that we trust the cloud provider to ensure physical security (among other things).

If you don't or can't trust the provider to ensure physical security, the model breaks and your security is undermined. That's a deeply uncomfortable conclusion for companies that are not prepared to entertain a move away from the cloud to self-owned and managed infra.

 

[tim305]

This is a bunch of noise. The reasoning about this hasn't changed in literal centuries. It has never been possible to guarantee security after physical compromise and never will be.

You can protect against man-in-the-middle attacks. Centuries ago this might have been by writing messages on a shaved head, using one-time cipher pads, or whatever. Today, we protect against man-in-the-middle attacks using cryptography.

But, the end-points? Centuries ago, if the recipient of your secret message was actually a double agent, you'd lost. Today, if your server or other endpoint is compromised, you've lost. And, no matter how far you go into the future, this will still be true. Simple logic tells us that you once you've taken control of a system, you can set it up to continue to do all its previous tasks but add anything else you want like logging, extra communication channels, manipulation of data, whatever.

There are Hardware Security Modules and Secure Elements that are designed to protect against physical attack. You find these in credit cards, crypto hardware wallets, banking infrastructure, etc.. These may also be compromised in principle, but if the expense is high enough it will not justify the proceeds of any attack.

So, it is not noise to assess what is the effort level to compromise any particular technology, even if they all can be compromised in principle.

 

[bradfa]

Well the attack also requires a compromised kernel... so in the case of an iPhone you'd have to unlock and first jailbreak the phone... at which point the rest is redundant.

While the attack is interesting from a research standpoint given that you need physical access and a compromised kernel (which is, itself, a pretty big problem), it is not an easy to attack to pull off.

This is a very good and important point. If you need physical access AND the ability to compromise the OS kernel, this means you're going to have to either bypass secure boot mechanisms or find actual vulnerabilities in the exact kernel which is allowed to run that you can take advantage of. Both are possible, but definitely add to the complexity of pulling this off in the real world.

This seems like a totally reasonable defense-in-depth solution to TEE not being perfect. TEE is better at doing it's thing than not having TEE, and in order to support TEE staying secure you also need mechanisms like secure boot.

Ideally you'd also have your secure boot implementation verify all loaded kernel modules as well as verify every disk block for the root filesystem as further belt-and-suspenders defense in depth. Both are totally normal things to do today if you care about security and physical attacks are a concern.

 

[miken32]

“We don’t know where the hardware is,” Daniel Genkin, one of the researchers behind both TEE.fail and Wiretap, said in an interview. “From a user perspective, I don’t even have a way to verify where the server is. Therefore, I have no way to verify if it’s in a reputable facility or an attacker’s basement.”

If your service is so sensitive that your whole business model relies on the physical security of a server, of course you use your own hardware. But that requires a physical investment, and planning, and means setting up your "legitimate business" is riskier than dumping some code into a cloud instance. The article seems to imply that these poor tech bros are misunderstanding the limitations of TEEs; it seems more likely to me they're deliberately overlooking the limitations while hyping their product.

 

[DaveSimmons]

It's almost as if running all of your infrastructure on computers you don't control and never see isn't a good idea.

Weird.

No. AWS, Azure, Google cloud servers are very unlikely to fall victim to this kind of attack.

It's much more likely that one of your own employees will steal data or trash your infrastructure, and with cloud you are at least protected from them taking an axe to the server closet.

Or to have the servers stolen. We had a break-in at one of my old employers back before AWS was the way to go, and they trashed the office and stole a bunch of stuff.

 

[fuzzyfuzzyfungus]

What puzzles me about the 'TEE'/'confidential compute'/etc. claims, especially in the context of hyperscalers, is why people aren't more skeptical when (even if the implementation were perfect, which it isn't) the most they can do is change the party you are required to trust, not eliminate the trusted party.

If you are just renting a normal VM from me then my hypervisor can sniff secrets out of your RAM or whatever without any real trouble; that much is obvious; but if you are depending on some sort of TEE-backed enclave arrangement you don't actually have any way of knowing that there is a TEE, or that it has the properties it claims to have: you just get an attestation that ultimately chains up to a root cert from intel/amd/nvidia/etc. but doesn't actually prove that they did what they said they did. A random dude who buys in quantity 1 probably isn't going to get an exception made for them; but would you be as sure that hyperscalers who get custom SKUs, and frequently have their hardware, sometimes even their silicon, made to spec, is getting a TEE that fully restricts them, rather than one that has different behavior but attestation certs of the same format?

Now, for the minor quibble:

Where are they getting a 16902A for under a grand? Minor quibble; it's honestly kind of miraculous how cheap a logic analyzer that can work on DDR5 can be; but the used prices I'm seeing are more like $4k and up(not that the difference matters given the sort of secrets being discussed here).

 

[dangoodin]

If your service is so sensitive that your whole business model relies on the physical security of a server, of course you use your own hardware. But that requires a physical investment, and planning, and means setting up your "legitimate business" is riskier than dumping some code into a cloud instance. The article seems to imply that these poor tech bros are misunderstanding the limitations of TEEs; it seems more likely to me they're deliberately overlooking the limitations while hyping their product.

I think it's obvious from the article that the big tech bros are making promises that can't be kept, and as a result many smaller players struggle to fully understand what TEEs do and don't do. I don't get why anyone would think the post is being sympathetic to Big Tech.

Also:

If your service is so sensitive that your whole business model relies on the physical security of a server, of course you use your own hardware.

I can't think of a single service that isn't this sensitive. Can you name a few?

So many commenters aren't reading the plain words in the article and are instead splitting hairs about stuff that's tangential or even not claimed. Why are TEE users (e.g. Microsoft and Cloudflare) and the chipmakers themselves (e.g. Nvidia) holding out TEEs as a crucial protection for edge servers and burying the exclusions in fine print, if at all?

 

[Doomlord_uk]

If your service is so sensitive that your whole business model relies on the physical security of a server, of course you use your own hardware.

Physical security at the big three providers of online hardware is pretty excellent. I doubt the average SME could replicate that security.

 

[ranthog]

Personally, I wouldn't have expected these systems to be able to withstand a physical attack. As a rule of thumb, physical access has always overridden a lot of security mechanisms. If the hardware isn't in your hands personally, you absolutely have to trust those who have access to it.

If you can't trust the people hosting your servers, then you can't trust the servers.

 

[Nogami]

Great article as usual Dan!

I've always assumed that physical access to any device will eventually lead to compromise 100% of the time, the only question is "when".

 

[DaveSimmons]

What puzzles me about the 'TEE'/'confidential compute'/etc. claims, especially in the context of hyperscalers, is why people aren't more skeptical when (even if the implementation were perfect, which it isn't) the most they can do is change the party you are required to trust, not eliminate the trusted party.

If you are just renting a normal VM from me then my hypervisor can sniff secrets out of your RAM or whatever without any real trouble; that much is obvious; but if you are depending on some sort of TEE-backed enclave arrangement you don't actually have any way of knowing that there is a TEE, or that it has the properties it claims to have: you just get an attestation that ultimately chains up to a root cert from intel/amd/nvidia/etc. but doesn't actually prove that they did what they said they did. A random dude who buys in quantity 1 probably isn't going to get an exception made for them; but would you be as sure that hyperscalers who get custom SKUs, and frequently have their hardware, sometimes even their silicon, made to spec, is getting a TEE that fully restricts them, rather than one that has different behavior but attestation certs of the same format?

Now, for the minor quibble:

Where are they getting a 16902A for under a grand? Minor quibble; it's honestly kind of miraculous how cheap a logic analyzer that can work on DDR5 can be; but the used prices I'm seeing are more like $4k and up(not that the difference matters given the sort of secrets being discussed here).

It's reasonable to trust AWS since their entire business is based on not lying to you and if they did then one whistleblower could destroy their business. Keeping a secret like that would be almost impossible. You have to worry about incompetence (as we found out last week) but not malice.

This kind of attack is about the Mission Impossible scenario of someone else gaining access, attaching their hacking tech and stealing your stuff. Highly unlikely in an AWS data center, but as the article points out it becomes more plausible for some edge server set up away from the strong physical security of a data center.

Or a server in the home or business of one of the people who doesn't trust the cloud. If their office just has a locked door protecting their server a youtube video will show how to get past that without breaking the lock.

 

[Missing Minute]

Since it works by putting an interposer in between the motherboard and the actual DRAM stick, adapting the attack to soldered-in RAM would be challenging. Not impossible I'd bet, but not "smuggle in a briefcase-sized thing into the same room as the target computer, and then done in a few minutes". More like "We stole your laptop or phone and can work on it at our leisure in a lab".

You'd be surprised how quickly a skilled technician can replace soldered chips with relatively basic equipment. I doubt a field chip swap attack will ever happen, but it's not outside the realm of possibility.

 

[dangoodin]

Personally, I wouldn't have expected these systems to be able to withstand a physical attack. As a rule of thumb, physical access has always overridden a lot of security mechanisms. If the hardware isn't in your hands personally, you absolutely have to trust those who have access to it.

If you can't trust the people hosting your servers, then you can't trust the servers.

I've always assumed that physical access to any device will eventually lead to compromise 100% of the time, the only question is "when".

Lots of devices built on integrated circuits ARE designed to withstand physical attacks. And they do. Hardware Security Modules protecting TLS keys. TEEs in iPhones/iPads/Macbooks. Titan enclaves in Pixels. Yubikeys and other FIDO2-compliant physical keys. All are designed to withstand physical attacks and are mostly there to protect the encryption key of the storage.

Yes, an HSM and a Yubikey are very different than a TEE built into an Intel or AMD processor. Still, this notion from so many commenters that physical access to a chip or other hardware ALWAYS means game over -- or that you can't have one without the other -- isn't nearly as airtight as they say.

The problem is that many people are aware that some enclaves DO come with defense assurances against physical attacks. So when they hear Microsoft, Nvidia, Cloudflare, et al. making misleading statements, they believe them.

What I still don't understand is why so many commenters are calling out this article, rather than the many TEE users and makers who have been making false/misleading/confusing claims for years.

Updated to add "-- or that you can't have one without the other -- " in response to a comment below.

 

[A_Very_Tired_Geek]

For on-premise deployments, it may not be obvious that physical attacks (including side channels) are specifically out of scope.

Only for the particularly naive, which granted tends to include many upper management types who may be more focused on "investor relations" than the security of their business information.

Cyber security and physical security are inseparable. You can't have one without the other. If someone walks in the front lobby, sits down with a laptop in an unused conference room and plugs in an Ethernet cable to your internal network, but all of your security is at the external network perimeter, they just bypassed your entire security stack. Yet this is a common attack vector because a lot of locals criminals know many, if not most, organizations have extremely weak or non-existent internal security controls. That doesn't even get into the problem of insider threats that may deploy unsanctioned wireless end points in dusty closets near external walls.

Physical security, whether on prem or in the cloud, should never be an afterthought. Yet, ultimately, that physical security is going to boil down to how well trained and motivated the humans working in the physical plant are. There's no physical lock in the world that can't be jimmied in some fashion given time and motivation. This is the problem management often can't fathom, low morale will directly affect not only productivity, but also the security and safety of the organization in all respects.

 

[odikweos]

There's a separation of concerns issue here. Physical security is the datacenter's problem or the device owner's problem. Logical security is the firmware/software/chipmaker's problem. The article is conflating the two and expressing irritation that people in vastly different businesses don't care about unrelated concerns.

It's like blaming the janitor company for the locks sucking. The janitor company needs to screen the staff. The building management needs to secure the locks.

There is simply no way for companies operating in the logical sphere to have anything meaningful to say, or to help with meaningfully, physical security. Nor can they meaningfully defeat social engineering attacks or any number of other attack surfaces (such as repository attacks, network MiTM attacks, crypto attacks..)

 

[khumak50]

Honestly I'm not that worried about protecting against attacks that require physical access. If someone has physical access then you're pretty much hosed no matter what. So a vulnerability that requires physical access just means you're extra special hosed. Just don't allow physical access.

With physical access, even a mythical system with perfect, unbreakable security someone could guarantee with 100% certainty that they could still cause you problems by simply taking or destroying the servers that your data is on. They might not gain access to it but they can destroy your access to it.

I'm MUCH more concerned with vulnerabilities that can be exploited remotely, especially if it's one that does not require clicking on anything or downloading anything. Something like malware ads that just automatically run when you visit any site that has advertising (which is, um, basically all of them). This is why ad blockers are so important.

 

[informationsuperhighway]

Looks like Apple got it right with its own dedicated security processor running its own Secure Enclave.

 

[putzhobel]

Now, for the minor quibble:

Where are they getting a 16902A for under a grand? Minor quibble; it's honestly kind of miraculous how cheap a logic analyzer that can work on DDR5 can be; but the used prices I'm seeing are more like $4k and up(not that the difference matters given the sort of secrets being discussed here).

That was my first thought whan I saw the photo: I want that analyzer. With the cables, please - they tend to be more expensive than a used analyzer without cables...

 

[dangoodin]

Only for the particularly naive, which granted tends to include many upper management types who may be more focused on "investor relations" than the security of their business information.

Right, and that's why it's so counterproductive and harmful for organizations that should know better make confusing/misleading/inaccurate statements about TEE protections.

Cyber security and physical security are inseparable. You can't have one without the other. If someone walks in the front lobby, sits down with a laptop in an unused conference room and plugs in an Ethernet cable to your internal network, but all of your security is at the external network perimeter, they just bypassed your entire security stack. Yet this is a common attack vector because a lot of locals criminals know many, if not most, organizations have extremely weak or non-existent internal security controls. That doesn't even get into the problem of insider threats that may deploy unsanctioned wireless end points in dusty closets near external walls.

Physical security, whether on prem or in the cloud, should never be an afterthought. Yet, ultimately, that physical security is going to boil down to how well trained and motivated the humans working in the physical plant are. There's no physical lock in the world that can't be jimmied in some fashion given time and motivation. This is the problem management often can't fathom, low morale will directly affect not only productivity, but also the security and safety of the organization in all respects.

Again, this general truism that physical attacks ALWAYS means game over isn't nearly as airtight as you're framing it. Lots of devices DO promise physical attack defenses (see my comment above). The point is: lots of decision makers with big budgets are under informed and therefore influenced by marketing into believing these TEEs will do things they can't.

The industry has glossed over this problem for more than a decade. Given how relatively cheap and easy physical attacks are becoming, this needs to stop.

 

[A_Very_Tired_Geek]

Right, and that's why it's so counterproductive and harmful for organizations that should know better make confusing/misleading/inaccurate statements about TEE protections.


Again, this general truism that physical attacks ALWAYS means game over isn't nearly as airtight as you're framing it. Lots of devices DO promise physical attack defenses (see my comment above). The point is: lots of decision makers with big budgets are under informed and therefore influenced by marketing into believing these TEEs will do things they can't.

The industry has glossed over this problem for more than a decade. Given how relatively cheap and easy physical attacks are becoming, this needs to stop.

I never claimed physical attacks always means game over. I said that physical attacks are mitigated by an informed and motivated workforce. Physical barriers are meant to stop people long enough for intervention to occur. If no physical intervention occurs the likelihood of success in the typical scenario goes up precipitously. (Specifically talking about doors and door locks in this case, if nothing else a plasma lance will do the job if no one is around to intervene.)

Yes there are systems that are proven to withstand both cyber and physical attacks indefinitely, that's what EAL5 and higher provably promise. But the likelihood of a company to actually have those systems guarding their sensitive data is almost zero.

Edit to add: I agree with your last paragraph entirely though. It's long past time to pay attention to physical security in detail and stop with products that only give lip service to it, this includes physical barriers like trivially bypassed door locks, security modules easily bypassed with physical probes, etc.

 

[JWLong]

Nothing is ever one hundred percent secure, gee what a surprize. And then, you throw in tech bros whose job it is to cheapen up anything and everything they can to serve up dividends to stockholders (who are mostly "tech bros") and everything today is made with the cheapest end of the stick of manufacturing, then add in all the lies and bullshit from marketing and what do you get?

A big pile of brown organic matter that started that way and is going to stay that way.

Is anyone really fucking surprized by any of this, I don't think so!!!!!!!!!!

 

[Octavus]

You'd be surprised how quickly a skilled technician can replace soldered chips with relatively basic equipment. I doubt a field chip swap attack will ever happen, but it's not outside the realm of possibility.

Which is why companies like BlackBerry underfilled their memory and processors as desoldering an underfilled small pitch chip is not trivial.

 

[Cloudster]

No. AWS, Azure, Google cloud servers are very unlikely to fall victim to this kind of attack.

It's much more likely that one of your own employees will steal data or trash your infrastructure, and with cloud you are at least protected from them taking an axe to the server closet.

Or to have the servers stolen. We had a break-in at one of my old employers back before AWS was the way to go, and they trashed the office and stole a bunch of stuff.

Fortunately AWS and other cloud providers never go down.

Oh, wait a minute...