Memory errors in consumer devices such as PCs and phones are not something you hear much about, yet they are probably one of the most common ways these machines fail.
I'll use this thread to explain how this happens, how it affects you and what you can do about it. But I'll also talk about how the industry failed to address it and how we must force them to, for the sake of sustainability. 🧵 1/17
First of all let's talk briefly about how memory works. What you have in your PC or phone is what we call dynamic random access memory. That is memory that stores bits by putting a minuscule amount of charge into vanishingly small capacitors (or not putting it in if we're storing a zero).
These capacitors continuously leak this charge, so it needs to be refreshed periodically - every few milliseconds - which is why it's called "dynamic". 2/17
This design is extremely analog in nature. When your machine needs to read some bits the capacitors holding them are connected to a bunch of wires. The very small voltage difference that happens in the wire is detected by the use of a circuit that turns it into a clear 0 or 1 value (this is called a sense amplifier). 3/17
So how can this fail? In a huge number of ways. Circuits age with time and use. The ability of the individual capacitors to hold the charge goes down slowly over time, the transistors in the sense amplifiers degrade, points of contact oxidize, etc... Past a certain point this can make the whole process end up outside of the thresholds required to reliably read, write and retain the bits in memory. 4/17
This can lead to different failures: a very common one is a stuck bit, which ends up being always read as 1 or 0, regardless of what was written into it. Another type is timing-dependent failures, which cause a bit to flip but only if it's not touched in due time by an access or a refresh. More catastrophic errors can affect entire lines - which is what happens when a sense amplifier starts to fail. 5/17
@ondra yes that happens, but that's a well understood and well studied phenomenon - especially in the context of data centers. When people talk about bit-flips that 's the first thing that comes to mind. I'd like to change this perception to make people realize that actual hardware faults are a lot more common than cosmic-ray hits.
Either way, even a single bit error which happens once in a blue moon is catastrophic to a consumer machine. Sometimes it will cause a pixel to slightly change color, but sometimes it will affect an important computation and lead to a crash. Or worse: it'll cause some user data to be corrupted before it's written to disk, and when it is, the damage has become permanent. 6/17
If your machine exhibits rare but hard-to-explain crashes, or if you're forced to reinstall programs - or even the operating system - because of mysterious failures, or experience random reboots or BSODs, then it's very likely that your memory is failing and you need to replace it. 7/17
It's not enough though, some errors can only be caught with more extensive testing. I recommend the open-source memtest86+ (https://memtest.org/) tool or the closed source memtest86 one (https://www.memtest86.com/) 8/17
I read talk, some years ago, about memtest+ not being maintained and actually not working at all for most cases (iirc it was in the context of the Ubuntu live cd boot option)
@kamstrup that was true for a while, then it was forked into PCMemTest by Martin Whitaker, and merged back with the memtest86+ codebase and now it's maintained by both the original author (Sam Demeulemeester) and the Martin Whitaker. The 7.0 major release dates back to this January.
@gabrielesvelto FWIW, I wrote a "memtest.js" version that runs in the browser. I even got some bad RAM sticks from Mozilla RelEng to verify that it could detect real failures!
@gabrielesvelto Of course there are some limitations, since JS (thankfully) doesn't have bare-metal access. But I wanted to see if periodically testing whatever chunks of memory the browser/OS gave out would work well enough.
That is, it can't say "all clear", but it can say "problems found". The idea being to eventually have the browser itself run a small background check, which over time should either detect any bad bits or give confidence that things seem OK.
(It was also an excuse to play with the then-new asm.js, for the hot bitwise-op loops. That code is lost, but IIRC it wasn't any faster because whatever JIT we used then already did a good job.)
@dolske thanks Justin! With @pbone we were thinking of doing just that, statistically testing crashy machines to figure out if we could spot some bad memory.
Naturally what happens on PCs also happens on phones, network devices, printers, TVs, etc... but you can't test them. This is a disaster because these failures are common, and they become more and more common as the device ages. If we want to have repairable devices that last for a long time, the industry will have to change its practices, but more about this later. 9/17
Now you might wonder: how often does this actually happen? The common wisdom on this topic is that hardware failures are so rare that software bugs will always dwarf them. As I found out this is demonstrably false.
While investigating Firefox crashes I've come to the conclusion that several of the most common issues we were dealing with were likely caused by flaky hardware. This led me to come up with a simple heuristic to detect crashes potentially caused by bit-flips. 10/17
Deploying this heuristic to Mozilla's crash reporting infrastructure has been eye opening: if I take the 10 most common crashes on Windows, 7 are out-of-memory conditions - that is, not bugs - and 3 are likely caused by bad memory.
You've read that right, three out of the ten most common reasons why Firefox crashes on Windows are caused by memory that's gone bad. 11/17
Now there's a few things that are worth mentioning: users with bad hardware will be over-represented in this category, their machines will crash far more often than others.
The second thing is that Firefox is exceptionally stable, we've driven down its crash rate by more than >70% in the last few years. But Firefox is also a 30 million-lines-of-code monster. There are bugs in there, but they're less common than hardware failures! 12/17
How can memory errors cause the same crash on two different computers unless they both have the same error at the same address? (Which is of course astronomically unlikely.)
@ali1234 because statistically they happen more frequently in pieces of code that touch a lot of memory. Firefox' JavaScript garbage collector is one such example. It traverses the heap using GC's typical mark & sweep behavior and touches thousands upon thousands of objects, crossing over an enormous amount of pointers. Because it's far more likely to hit a bad bit than the rest of the code it will show up far more often. Same for code that traverses huge hash tables, etc...
I see. A similar phenomenon was observed in initializing bitcoin full nodes, where verifying the full chain (currently 5TB and growing) exposed a lot of memory errors due to hash mismatches.
That happened essentially because everyone was doing the exact same calculations on the exact same data, in the exact same order, and expecting a known result.
I would have expected a lot more randomness in Firefox, but I forgot how much manual memory management it does.
Plotting these types of crashes against time yields interesting trends: the more machines age the more likely they are to encounter hardware-related failures. You might think that's obvious, and indeed it is, but until now the industry has looked the other way, based on the hand-wavy excuse that hardware failures were less common than bugs. 13/17
So what needs to change? First of all, error detection and correction must become commonplace. You can already build a desktop machine with ECC memory (https://en.wikipedia.org/wiki/ECC_memory), but it's uncommon in laptops, even mobile workstations, and completely absent on phones and other consumer appliances. This will measurably lengthen the usable life of these devices. 14/17
Note that detection is more important than correction. The user needs to know that there's something wrong without having to run a memory testing program. Think of the lights that turn on in cars if something's malfunctioning, or the error beeps that your washing machine makes when it thinks it's leaking water. These are extremely common, they need to be on computing devices too. 15/17
Finally hardware design must change to make devices repairable and prolong their useful life. Yes, I'm looking at non-ECC memories soldered on the motherboard or worse, on the same substrate as the CPU. 16/17
To end the thread I'd like to thank my colleagues Alex Franchuk and @willcage who did the implementation work and my boss Gian-Carlo Pascutto who plotted crashes against machine age. I'd also like to point out that we've got preliminary data on the topic, but I fully intend to write a proper article with a detailed analysis of the data. 17/17
@gabrielesvelto
Thanks for an interesting thread! It brought back memories from 90s computing that was full of repeatedly running memory checkers and reseating chips - and how that taught me for a while to only accept motherboards which supported ECC memory. Indeed no longer an option available for our devices.
@gabrielesvelto@willcage IBM designed parity checks everywhere in the System 360, so that the system would quickly stop in the event of hardware failure. ECC was implemented in main memory systems when it was discovered that solid state RAM was subject to transient memory failures from cosmic rays (really.) Early PCs used memory parity checks (which weren't adequate) until Apple abandoned them for cost reasons. Bad mistake.
@gabrielesvelto Working with existing non-ecc system, couldn't some of this be caught if the OS ran a low-priority process scrubbing memory, eg. writing and checking a random, but check-summed pattern to free pages (similar to ZFS scrubbing). Even better, important data structures should be check summed (something I actually did in a database engine I wrote many decades ago).
@tommythorn one of the things we plan to do is running a brief, low-priority scan in processes that crashed. If we find some bad memory we'll notify the user and flag future crashes from their machine as low-value so we don't spend too much time looking at them.
@gabrielesvelto Maybe saving someone else a Web search: Error correction code memory (ECC memory) is a type of computer data storage that uses an error correction code (ECC) to detect and correct n-bit data corruption which occurs in memory: https://en.wikipedia.org/wiki/ECC_memory.
@gabrielesvelto Really depressing that we've reached the physical limits of creating "memory we're confident that actually will store it's value reliably" :(.
We've went from PARITY CHECK 1/2 to "memory works fine without detection or correction" to "oh now not even parity check is enough". In that sense, it's WORSE than 40 years ago :P.
@gabrielesvelto@cr1901 maybe that's why it sometimes felt like those old machines were rock-solid in spite of their limitations: hardware has become less reliable faster than software became more reliable.
@gabrielesvelto I question if "out of memory conditions" are to be classified as "not a bug". Given how memory-hungry Firefox is, and given how jealously it clings to memory that it's been allocated by the OS (I had to install a third-party extension, Auto Tab Discard, to help combat this!), I would say that Firefox's thirst for memory is indeed, 100% a bug.
Of course if you're on-side with people being forced to continually update hardware...
@qqmrichter Firefox has limited control over what web pages do. When memory usage gets out of hand it's usually the page's fault. When I mentioned that Firefox is exceptionally stable I mean it, we've dramatically cut OOMs just a couple of years back, which is why we're now seeing even the hardware issues: https://hacks.mozilla.org/2022/11/improving-firefox-stability-with-this-one-weird-trick/
@gabrielesvelto You can choose, however, whether to degrade gracefully ("I'm sorry, this web page won't load because of memory issues") instead of crashing in a shower of bits (which I'll admit you're doing a bit better on now).
I mean worst comes to worst, have an LRU on off-screen assets and just drop them in order when you try to allocate memory and it fails. If they're needed again, reload them again. It's ludicrous that a web browser requires (WAY!) more space than a whole OS image.
@qqmrichter we already do that. We do a lot of stuff when memory is tight. We purge caches, trim buffers, unload tabs that haven't been used in a while, do aggressive garbage collection, etc... Sometimes there's just no way to avoid an OOM, especially in the case of pages with runaway memory consumption or if other applications are also using resources on the machine.
@gabrielesvelto@qqmrichter Sorry to hijack this thread; but since you appear to know a lot about this topic: is it possible (or even common) for websites to have memory leaks, e.g. in their JS code? And if so, do you have links to possible solutions (for me, as end-user)?
I suspect that some of the tabs I keep open for long time use up more and more memory (and they have lots of JS code); reloading the tabs appears to help, but I'm looking for an automatic solution.
Hi and thanks for your work.
Could there be an alert (I guess on the "about:performance" tab) for runaway memory consumption pages ?
It already shows the memory consumption but it's not clear when it's just that the page needs memory to do its work and when it's running amok.
Aldo, very empirically I believe that plug-ins can sometime provoke runaway problems and there's no real tool to monitor it.
@lienrag@qqmrichter it's hard to tell apart a page that uses a lot of memory from a page that's leaking. What we're doing though is making sure that pages that use a lot of memory and which you aren't looking at get unloaded eventually, freeing the memory they're holding to.
@gabrielesvelto I appreciate your focus on improving diagnosis. Those over-represented crashes distract people from real bugs. I don't understand why you included OOMs though -- allocation failures can and will happen, and it's our responsibility to handle them gracefully.
We already handle gracefully all the failures that we can realistically handle, but for a lot of them there's nothing we can do. That's especially true on non-Windows platforms where allocations never fails. Both on Linux and Android the kernel will kill processes to save memory without informing them or allowing for any type of reaction, so graceful handling is impossible.
@gabrielesvelto I believe that most bitflips happen in media files. That way, they go unnoticed. ECC becomes important pretty quickly when doing other things…
Peripherally related: Dram consumes considerable power whether you're actively using it or not. Are you aware of any research into disabling refresh on unused memory? Could reducing refreshes also extend DRAM useful life and reduce errors?
True, with modern cache design, pretty much all the RAM is used all the time, but "used" is a pretty fuzzy concept in this context. Moreover, swapping to SSD isn't anywhere near as expensive as swapping to a SATA disk.
@lolcat IIRC modern DRAMs have low-power modes which can be enable when utilization is low or none; I don't know if they can also shut down individual sections/banks but maybe they do? I remember reading about that as a theoretical possibility well over a decade ago so maybe yes
Add comment