Boffins convert typing sounds into text with 95% accuracy

Researchers in the UK claim to have translated the sound of laptop keystrokes into their corresponding letters with 95 percent accuracy in some cases.

That 95 percent figure was achieved with nothing but a nearby iPhone. Remote methods are just as dangerous: over Zoom, the accuracy of recorded keystrokes only dropped to 93 percent, while Skype calls were still 91.7 percent accurate.

In other words, this is a side channel attack with considerable accuracy, minimal technical requirements, and a ubiquitous data exfiltration point: Microphones, which are everywhere from our laptops, to our wrists, to the very rooms we work in.

elias_griffin,
@elias_griffin@lemmy.world avatar

Is it ignorance, indemnity, or conspiracy that this News Media Corporation didn’t give the primary mitigation?

A white noise generator.

brlemworld,

Article doesn’t say but I would guess they are testing with words and using that to build context for better accuracy. I imagine if you are typing some random password it would not be as accurate. Also the only password I type nowadays is the one to unlock the computer, everything else is in a password vault.

lolcatnip,

This is why I always make sure there are no boffins around before I start typing.

killeronthecorner,
@killeronthecorner@lemmy.world avatar

If there are boffins around, I start typing out the GDPR guidelines in full

disheveledWallaby,

What about Hornblower’s, Bolger’s, Took’s, Sackville’s or Grubb’s?

atrielienz,

How good does this work if there’s other noise pollution? Like music playing etc?

ultra,

Obligatory XKCD

https://imgs.xkcd.com/comics/security.png

reverendsteveii,

en.wikipedia.org/wiki/Rubber-hose_cryptanalysis

topinambour_rex,
@topinambour_rex@lemmy.world avatar

Use a speech to text and they won’t be able to hear your keyboard strokes. I know, I’m a genius.

ARk,

But instead they would hear the speech and translate that to text. No need to thank me.

popekingjoe,
@popekingjoe@lemmy.world avatar

Who are you, who are so wise in the ways of science?

kpw,

A duck.

Quexotic,

Got any grapes?

dukk,

I mean, I got lemonade…

stoy,

I wonder if you need to train it on a specific keyboard before it will work it.

Lemminary,

Most likely

stoy,

That would limit the practicallity quite a lot, as deskmats and typing style would change the sound of even a common keyboard.

I also notice that I slightly change my typing style between typing normally and entering my password.

Dark_Arc,
@Dark_Arc@social.packetloss.gg avatar

That would limit the practicallity quite a lot, as deskmats and typing style would change the sound of even a common keyboard.

Eh… I don’t know if it would be enough of a change. Also consider mass produced popular laptops (e.g. targeting the MacBook keyboard).

I also notice that I slightly change my typing style between typing normally and entering my password.

I don’t really think that’s normal… But hey, maybe it gives you some protection 🙂

UraniumBlazer,

I doubt so. Wouldn’t Zipf’s law be used for this?

NAXLAB,

I think I might have achieved security through obscurity. My custom keyboard is a unique shape and almost all the keys are one unit. Not only is it different enough from a traditional keyboard that the neural network probably won’t understand it, the function layers I use obscure whether I’m typing a letter at all.

Outsider9042,

Good luck listening to mine

Mr_Blott,

Does that come with free fingerless gloves?

Outsider9042,

No, but it comes with your choice of flavoured frozen yoghurt.

b000urns,
@b000urns@lemmy.world avatar

That’s good!

joel_feila,
@joel_feila@lemmy.world avatar

The yogurt contains potasium benzoate

MycelialMass,

That’d bad

A_Random_Idiot,

I have a headache just looking at that.

ReveredOxygen,
@ReveredOxygen@sh.itjust.works avatar

What keyboard is it, corne? I have to admit that your keycaps are incredibly cursed, how you have mixed caps from different layers

Outsider9042,

It’s a chocofi.

CTGAP on the base layer, and 6 layers on top of it, using a heavily modified version of Miryoku.

Most of the keycaps are correct, just for different layers. It helps prevent key peeking, plus I like the cursed aesthetic.

ReveredOxygen,
@ReveredOxygen@sh.itjust.works avatar

that’s a surprisingly cheap keyboard. I ended up ordering a zsa voyager a couple days ago because I wanted keys, but I couldn’t find any prebuilt split keyboards that had a base configuration below like $350. I might end up going with cursed keys on mine, it looks pretty cool

FrankTheHealer,

Can we normalise good but quiet keyboards. Like, I like the tactile feel of using a mechanical, but I hate the sound. Quieter mechanical keyboards aren’t a thing but they should be. Now as a security measure if nothing else.

Also Dvorak keyboards I guess

Pinecone,

There are tons of quiet mechanical keyboards. I’m using a low profile optical switch that’s quieter than my mouse clicks

FrankTheHealer,

Are those optical switches expensive though?

Pinecone,

No

xthexder,
@xthexder@l.sw0.com avatar

I went out of my way to find a keyboard with Cherry MX Clear switches. They’re basically a high-force tactile feel, but no clicky sound like MX Blue switches. I absolutely love them for typing, and I’ve been using them for years.

I’m not sure if there’s newer options now for silent switches? I know they had a couple models with extra internal damping.

murtaza64,

I used boba u4 silents on my custom keyboard. Absolutely love them. Wish they made a consumer-grade keyboard with them (or maybe they already do?) But I’ve been working on a MacBook recently and tbh the keyboard there is pretty good now. So next step for me is to build a low profile keyboard

VelociCatTurd,

There are definitely quiet tactile switches. The reason why they can still make sound is because they’re bottoming out which you don’t have to do.

Cethin,

As a partial solution, you can put o-rings in the keycaps. I had some of the bands for braces laying around at one point and used those, and it worked fairly well.

NAXLAB,

Dvorak is a cypher of Qwerty tho. Anything typed in Dvorak but transcribed as english can be reliably identified and decyphered

crispy_kilt,

Dvorak

vi tho.

murtaza64,

On my colemak keyboard I put arrow keys on another layer under where hjkl are on qwerty. Beyond that, most of the keys are remembered by mnemonic rather than position imo

FrankTheHealer,

That awkward moment when you use Nano

gatelike,

I always uninstall nano the first time it shows itself

Socsa,

Up against the wall

Garbanzo,

normalise good but quiet keyboards

Oh man, if Topre became popular enough to bring the price down through scale that would be pretty rad

laurelraven,

Not to be a jerk, but is this actually new? I’ve heard of this being done at least ten years ago…

On another note, one way to beat this (to a degree) would be to use an alternate keyboard like Dvorak (though you could probably code it to be able to detect that based on what’s being typed)

misophist,

Coding for alternate key mappings is almost as trivial as detecting other languages.

laurelraven,

Yeah, that’s what I figured

Buddahriffic,

It’s more trivial because it’s a 1:1 relationship. A is a, s is o, d is e, and so on. Detecting other languages is harder because there’s more of them and there isn’t a 1:1 conversation to English.

barsoap,

I think it’s largely been a state actor thing. Directional microphone to record your window from across the street, spend significant tax money on crunching numbers on a supercomputer to get at your password kind of thing, I think they already could do it in the 90s. Real-time 95% accuracy on a non-specialised device is a quite different ballpark: Now every skiddie can do it.

laurelraven,

Gotcha, that makes more sense

skulblaka,
skulblaka avatar

Now every skiddie can do it.

And this is the real, serious problem. Most people are pretty unlikely to stop a state sponsored spy operation no matter how careful they are. It's barely worth worrying about unless you know for a fact you're being tapped and that you will be killed about it, and even if you do know this the state can pull some space age bullshit out of their asses that doesn't yet have a counter. Top secret military industrial research goes into maintaining that exact advantage every year, if they really want to get you, you will get got. But if Joey Dickbeater and his school friends can just point a mic at your window and then upload it to the Pass-o-Gram to decode it, you have a real problem. It's like when TikTok kids figured out they can steal Kias with usb keys - if every teenager in America knows how to steal your car, its lifetime is going to be measured in minutes. Same with passwords.

Sounds like it's time to buy a bunch of random cherry switches and randomize them across my keyboard.....

ook_the_librarian,
@ook_the_librarian@lemmy.world avatar

Sounds like it’s time to buy a bunch of random cherry switches and randomize them across my keyboard…

And rotate them. While I don’t plan to waste my energy, having hot swap sockets and swapping a few around should thwart the attack. You would have to do it frequently enough that relevant training data gets wasted before it’s useful. I’m pretty paranoid, but not that much.

I’ll just consider it good security hygiene to get a new keyboard often :)

barsoap,

Have you considered only re-doing the tinfoil wrapper every day? It should crackle differently every time.

Socsa,

What it means is that NIST probably needs to update its security recommendations to require hardware keys for even low level systems. It’s going to be a huge pain in the ass though.

frezik,

There has been previous work on this, yes. It required a dictionary of suggested words. That would make it useful for snooping most typing, but not for randomly generated passwords. This new technique doesn’t seem to have that limitation.

laurelraven,

Okay, gotcha. I didn’t look that deeply into it previously so I never realized how limited that was

Dark_Arc,
@Dark_Arc@social.packetloss.gg avatar

So about those people that run around saying passphrases are better… 😅

ILikeBoobies,

Another advantage to the split keyboard

Pons_Aelius, (edited )

This attack is useless in the real world.

That said, what gives you the idea a split keyboard (if they had a sample of you typing on it etc) would be any different than a normal one?

It is just another keyboard with a different sound profile.

ILikeBoobies,

It was just a joke

em2, (edited )
@em2@lemmy.ml avatar

You can remap and customise keys to be whatever you want. There’s even auto shift, so if I hold certain key just a bit longer than a regular tap, it will automatically capitalise or whatever the shift + key combo would result in. There are also multiple layers you can easily activate with a press of a button, so the layout is something totally different.

Example: configure.zsa.io/moonlander/layouts/default/…/0/

Layer 0

Layer 1

Layer 2

paraphrand,

Neat, so when my friends are taking about satisfyingly clackety keyboards I can inform them it’s a security hazard.

ryannathans,

Good luck, I have a non standard key layout

Pheonixdown,

Dvorak?

Llewellyn,

It’s still vulnerable to dictionary attacks

ryannathans,

Except it’s not

Wilzax,

??? If you can map sound to qwerty keystroke placement, then it’s a simple matter of mono alphabetic substitution for other layouts to generate candidate texts. Using a dictionary attack to find more candidate layouts would absolutely work.

ryannathans,

No, all the timings change. You can’t just swap out the letters and hope it matches. Additionally I was responding to the poster claiming a dictionary attack on a password would work - only if it’s in the dictionary.

Wilzax,

The method is not based on timings. It is based on identifying the unique sound profile of each keystroke

ryannathans,

How can you make that claim? They used deep learning, does anyone know what characteristics the AI is using?

Wilzax,

I can only make that claim with as much confidence as yours that a non-standard keyboard layout would protect you. By your own argument, how can you make the claim that using a different keyboard layout will protect you? Everyone has a different typing style and their keyboards have different sound profiles, so clearly the AI needs to sample enough data from you and compare it to a dictionary to individually on audio collected of your typing. How do you know that the characteristics of the keyboard layout play an identifying role in this kind of attack when the AI has no concept of which sounds correspond to which keys until it has done its training?

rhandyrhoads,

A pretty simple deep learning approach would be to take a large sample and first identify the individual key sounds. From there it can start associating the most common letters with the most common sounds and switch it around until dictionary words start coming out. Once it can identify individual keys you could even brute force it in a pretty reasonable timeframe. The keyboard layout is the least important part because the individual key sound output is going to vary keyboard by keyboard and even potentially user by user. If you used a password without dictionary words and used a different keyboard layout exclusively for entering the password that would likely defeat this sort of attack.

AmberPrince,
AmberPrince avatar

I'll accept the risk. I need the clicky

Evotech,

Middle management will finally get rid of clacky keyboards with this weird trick

Duamerthrax,

Good luck making an acoustic map of the tens thousands of possible case, switch and key cap combinations.

helenslunch,

Someone explain how this works? Doesn’t make much sense to me how that’s even possible.

Pons_Aelius,

Because of different placement on the keyboard and different finger pressure, each key press has a slightly different sound.

The telling thing in this story is this

with 95 percent accuracy in some cases.

For some people (those with a very consistent typing style on a known keyboard) they were right 95% of the time.

In the real world this type of thing is basically useless as you would need a decent sample of the person typing on a known keyboard for it to work.

To go from keystroke sounds to actual letters, the eggheads recorded a person typing on a 16-inch 2021 MacBook Pro using a phone placed 17cm away and processed the sounds to get signatures of the keystrokes.

So to do this you need to have physical access to the person (to place a microphone nearby) and know what type of device they are typing on and for it to be a device that you have already analysed the sound profile of.

ILikeBoobies,

You don’t need physical access, just some malware that has access to the microphone

We would hope researchers “discovering” this wouldn’t have a production ready product as their proof of concept. So there is room from improvement but military contractors would love to invest in this

Pons_Aelius,

You don’t need physical access, just some malware

Which you still need to have previously installed...

If the person has allowed malware to be installed just install a keylogger (which gives you 100% accuracy every time) rather than jump through more hoops with this.

ILikeBoobies,

Different devices

I would have an easier time infecting someone‘s personal phone than a company machine

Pons_Aelius,

You would, would you?

Well, I must be talking to a leet hacker then.

Ok, install malware on my phone.

ILikeBoobies,

How did you get that from what I said?

Pons_Aelius,

I would have an easier time infecting someone‘s personal phone than a company machine

What did you mean by this then other than you, personally, are skilled at such things and have system penetration experience?

ILikeBoobies,

Easier doesn’t mean easy but I can send you an email/give you a link

The company email server should block it and the firewall should block the website

Sample

Check out this game! play.google.com/store/apps/details?id=com.robtopx…

But the page is actually

play.giggle.com/store/apps/details?id=com.robtopx…

Knowing this doesn’t make me 1337

agent_flounder,
@agent_flounder@lemmy.world avatar

The article says

The researchers note that skilled users able to rely on touch typing are harder to detect accurately, with single-key recognition dropping from 64 to 40 percent at the higher speeds enabled by the technique.

Hm. Sounds like “some cases” are hunt and peck typists or very slow touch typists.

I don’t know if training for each victim’s typing is really needed. I get the impression they were identifying unique sounds and converting that to the correct letters. I only skimmed and I didn’t quite understand the description of the mechanisms. Something about deep learning and convolution or…? I think they also said they didn’t use a language model so I could be wrong.

prole,

I imagine it probably also uses an algorithm to attempt to “guess” the next letter (or the full word itself, like your phone keyboard does) based on existing words. Then maybe an LLM can determine which of the potential words are the most likely being typed based on the context.

I dunno if that makes any sense, but that’s how I pictured it working in my brain movies.

Pons_Aelius,

The problems is that even with up to 95% accuracy that still means the with a password length of 10 there is a 50/50 chance that one character is wrong.

A password with one character wrong is just as useless as randomly typing.

Which character is wrong and what should it be? You only have 2 or 3 more guess till most systems will lock the account.

This is an interesting academic exercise but there are much better and easier ways to gain access to passwords and systems.

The world is not a bond movie.

Deploying social engineering is much easier than this sort of attack.

warrenson,

“Hearing” the same password twice drastically increases the accuracy, however, social engineering is indeed the most effective and efficient attack method.

0xD,

If the password is not random, as they seldomly are, you can just guess the last, or even the last few characters of they are not correct.

agent_flounder,
@agent_flounder@lemmy.world avatar

I was thinking of this attack in terms of grabbing emails, documents, stuff like that. Or snippets thereof.

prole,

The world is not a bond movie.

Deploying social engineering is much easier than this sort of attack.

Have you never seen a Bond movie? Yeah they always have a gadget or two, but the rest is basically him social engineering his way through the film. And shooting. Usually lots of shooting too.

helenslunch,

So basically if they know what type of hardware you’re using, and have training on that type of hardware, then it works. It can’t just be literally any keyboard, right?

That makes more sense.

catch22,

They’ll have modelled the acoustic signals to differentiate between different keys. Individual acoustic waves eminating from pressing a key will have features extracted from them to identify them. Opimal featues are then choose to maximise accuracy, such as features that still work when the signal is captured at different distances or angles. With all these types of singsl processing inference models, you never get 100 percent. The claim of 95 percent is actually very high.

9point6,

Every key is unique and at a different distance to the microphone and therefore makes tiny differences in noise.

Knowing this, and knowing the frequency distribution of letters in language (e.g. we know “e” is the most common letter) and some clever analysis over a large enough sample of typing, we can figure out what each key sounds like with a statically high level of probability. Once that’s happened it’s just like any other speech recognition software, except it’s the language of your keyboard.

TootSweet,

This is just me kindof guessing off the top of my head, but:

  • Depending where the mic is in relation to the keyboard, it can tell to some extent the relative distance from the key to the mic by volume of the keypress.
  • The casing of the keyboard has a particular shape with particular acoustic properties which would make certain keys sound different than others. (Maybe the ones toward the middle have a more bass sound to them as opposed to more treble in the keys closer to the edges of the keyboard.)
  • The surface on which the keyboard sits may also resonate differently with different keys.
  • There may be some extent to which the objects in the room (including the typist and monitor, etc) could have reflected or absorbed soundwaves in ways that would differ depending on the angle at which the soundwaves hit them, which would be affected by the location of the key.
  • Some keys like the spacebar and left shift almost always have a stabilizer bar which significantly affects the sound of the key for most keyboards.
  • For human typists, there are patterns in the timing of key presses. It’s quicker to type two keys in succession if those two keys are pressed by different fingers or different hands, for instance. Imaging typing the word “jungle”, for instance. “J”, “u”, and “n” are all pressed with the right index finger (for touch typists). So the first three letters would be slower to type than the rest of the letters.
  • I’d imagine this method also allowed the program to take into account various aspects of human language. (Probably English in this case, but it could just as well have been another language.) Certain strings of consonants just never appear consecutively. Certain letters are less frequently used. Things like that. Probably the accuracy would have been lower if the subjects were asked to type specific strings of random letters.
  • It may also be that this particular experiment involved fairly controlled circumstances. They always placed the mic 12cm from the keyboard, for instance. Maybe they also used the exact same keyboard on the exact same desk with the exact same typist for all tests and training. And it sounds like they trained it on known text for a good while before testing the AI by asking the AI to actually discern what was typed. That’s pretty perfect conditions that probably wouldn’t be realistic for an actual attack. Not to minimize the potential privacy imacts of this, though. I’d fully expect methods like this to be more accurate for a more generalized set of cases.

Now, the researchers didn’t sit down and list out all of these (or any other) ways in which software could determine what was typed from audio and compose an algorithm that accounted for all/most/some of these. They just kindof threw a bunch of audio with accompanying “right answers” at a machine learning algorithm and let the algorithm figure out whatever clues it could discern and combine those in whatever way it found most beneficial to come up with an (increasingly-more-accurate-with-every-training-set) answer. It’s likely the algorithm came up with different things than I did that helped it determine which key(s) were being pressed.

Waldowal,
@Waldowal@lemmy.world avatar

New policy from the corporate office: If you are working in a public place, like a coffee shop, please scream while typing your login password.

sanimalp,

I screamed my password and now I got hacked. Thanks for nothing!

tourist,
@tourist@lemmy.world avatar

use the onscreen keyboard

much more secure

why won’t my bank stop calling me

Dark_Arc,
@Dark_Arc@social.packetloss.gg avatar

On it boss youtu.be/HsvyjePPFRs?si=So4iKVWAUPXNjGVe

fruitycoder,

Might have to spend some time getting Easy Effects/Noise Torch set up on my systems again just to reduce the vectors again.

There is a good comment on this post on physical mitigation that seems helpful as well: www.reddit.com/r/Fedora/comments/…/i6p0jqa/?utm_s…

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • technology@lemmy.world
  • kavyap
  • mdbf
  • tacticalgear
  • osvaldo12
  • InstantRegret
  • DreamBathrooms
  • modclub
  • magazineikmin
  • thenastyranch
  • khanakhh
  • Youngstown
  • slotface
  • rosin
  • ethstaker
  • megavids
  • tester
  • GTA5RPClips
  • Durango
  • normalnudes
  • cubers
  • ngwrru68w68
  • everett
  • provamag3
  • cisconetworking
  • anitta
  • Leos
  • JUstTest
  • lostlight
  • All magazines
    •