@fell Disabling the oom killer on Linux is a thing.
I had this happen in my cluster that was perfectly fine until the security updates from ubuntu took all the remaining memory which then triggered ubuntu to kill the largest process (elasticsearch).
@jillesvangurp If I disable the oom killer, I risk rendering the system completely unresponsive. I limited ElasticSearch now. I just think it's stupit to let it take ALL of the memory by default.
@fell Depends how you start it. Using it on small instances things get a bit cramped. Are you using docker? If so, it should respect any limits you set there.
Anyway what happened here is not Elasticsearch's fault but your OS killing a process for no good reason whatsoever in a way that actively harmful. I've had some encounters with the OOM killer over the years and mostly it's a bit too trigger happy killing the most important processes first.
@jillesvangurp I disagree. The OOM killer is there to prevent a situation where there is no more memory available, which will render the system unresponsive to even SSH connections, forcing you to do a hardware reset.
In this particular case, the OOM killer correctly decided to kill ElasticSearch because it was taking nearly 100% (24 GB) of system memory for no reason.
I'm not using Docker, because I don't see any benefits in using containers other than containing poorly written software like ElasticSearch. I also simply do not trust Docker Inc.
But it doesn't matter. Any software that attempts to use all available memory is simply broken, and it's ridiculous that a default install of ElasticSearch does just that and nobody cares.
@jillesvangurp Exactly, a server is there to serve stuff. If one process eats all the memory and locks up the entire system, that's an even bigger problem. I rather have a single service killed than having ALL services fail.
Just to make that clear, there wasn't plenty of memory. The machine has 24 GB and ElasticSearch wanted all of it.
Add comment