This is absolutely crazy stuff. Chinese hackers were able to get into a bunch of government email accounts by forging Microsoft access tokens, but how it happened is wild.
Apparently an internal Microsoft system responsible for signing consumer access tokens crashed, then a bug in the crash dump generator caused the secret key to be written to the crash dump. Microsoft's secondary system for detecting sensitive data in crash dumps also failed, allowing the crash dump to be moved from an isolated network to the corporate one. The Chinese hackers compromised a Microsoft engineer's account and were able to get a hold of the crash dump. They were not only able to find the key and figure out that it's responsible for signing consumer access tokens, but were also able to exploit a software bug to use it to sign enterprise access tokens too, basically giving them the keys to the kingdom.
So many security system had to fail for this to happen. Either the hackers were very lucky or extremely patient.
This is a testament to just how hard cybersecurity is. Microsoft had the forethought to not store keys into crash dumps, had the forethought to build a secondary system to double check them, had the forethought to store them on an isolated network, but a cascading failure basically blitzed through all their security controls and allowed nation/state hackers to walk off with critical signing keys.
@malwaretech they do appear to have been fairly comprehensive, with the thing that got them being (as it so often is) our good friend Hubris. Asserting that they could detect and scrub key material of such incredible sensitivity with sufficient reliability to mitigate the risk isn't a call I would have wanted to put my name to.
@malwaretech I think there is more to it because those government customers are not supposed to be simple enterprise customers but GCC (High?) customers, which is supposed to be an segregated instance, right?
@malwaretech does this report come solely from Microsoft? Has anyone questioned whether or not it's fully true? Microsoft is known to lie regularly for their own benefit. Could they be painting the picture to make themselves look diligent, while in fact it was blunders and bad judgement that led to this? Are they avoiding any liability with this report? As I understand, they didn't willingly investigate and write the report, but the USG forced them to (?).
@malwaretech@my_actual_brain I don’t get how they can forge keys for any M365 infrastructure just using the keys valid for that specific MS infrastructure, unless MS has some “passe-partout” wildcard key allowing access to any azure environment. But this would be as scary as the complete exposition of all azure-based systems worldwide, and possibly more.
But why were private keys allowed outside an HSM? They should never have that kind of keymat in the main memory of a PC because there is just so much attack surface.
@malwaretech sigh. I've wanted to do this in Linux for years, but I've got a different job, and nobody really likes being given ideas. Basically it's just an mprotect() flag to mark stuff as not dumpable, not mappable from /proc (or map-on-read zero pages), and show the flag in /proc/pid/maps.
@malwaretech When you lock all your doors but forgot you left your keys under your front mat. The more complex a system, the more likely something is forgotten.
@malwaretech Is it not a naive idea doomed to fail to scan memory dumps for sensitive data to begin with? Memory dumps have been considered highly sensitive for decades, probably since they exist. For it more sounds like Microsoft ignored common sense best practices to even allow memory dumps in production and then transfer them to developers.
There are times, when I feel it would be much easier if we went back to pen / paper / typewriters, and or non networked computers, and kept data on site in a secure data storage room that isn't connected to the internet.
@malwaretech wrong it's a case clearly stated in NSA best practices, DO NOT USE env vars for secrets, languages tend to dump env variables in case of a core dump.
@malwaretech or they had internal help, or they throw a very wide net and this is one of the rare catches we know of, or any combination of those or others we didn’t think of.
@malwaretech I am wondering about the implications for AI generated code for this sort of stuff. There are doubtless many examples of token claim validation that exist on github. But how many examples of core dump handling questions were uploaded to stack overflow and how much of the buggy windows kernel source was used to train Co pilot? An AI code generator model would need to be re-trained every time something like this is fixed, so the buggy code is a negative signal for the model. But then that's only one data point.
It seems to me the attackers must have been highly knowledgeable of these Microsoft systems already then , to have been able to utilize such a tiny piece of data in such huge haystacks of data. Or there must have been a huge sign pointing to SECRET DATA!
Has Microsoft not also installed cloud software inside China? Vianet? It seems likely it's architecture is known inside and out.
Only hardware devices can keep your keys leaking this simple.
@malwaretech A good reveal. I didn't get from the report how long the state actor was present in the MS prod environment employee's account undetected, itself being a significant concern. The reference to Apr 21 is vague. Large gap between that and the July 11th 23. If prolonged (allowing them to bide their time as some are suggesting) then why didn't some tripwires go off (presumably in sentinel). Shouldn't the crash dump of April 21 have been expired in terms of its usefulness/been deleted.
@malwaretech I wonder how similar their security setups in their China offices are to what are used in guarding the main parts? I.E. is this pure perseverance or do the APTs get insight from (I assume) having much greater access on what happens close by.
@malwaretech from a remediation/post mortem perspective, I literally do not know how you prevent this kind of thing from happening again. At least three layers of security, three direct failures.
@malwaretech o_O I’ve had vastly more solid attack chains dismissed as “too unlikely”. No more! 😈
But seriously, there has got to be more to this story.
"very lucky or extremely patient" Hmm- nah - it's a flawed model. It's not beyond the wit of man of man to keep the sensitive part of a memory dump in a separate bucket.
@malwaretech Definitely lucky.. but if this could have happened once in two years, could it have happened more than once, do you think?
I’m wondering if more keys could have gone the same way. Any ideas if the one MSA key was enough for any scope, or might other keys have covered other systems/accounts?
Add comment