Not sure if your distro version has a new enough version of systemd, but newer versions have a systemd-oomd service for that. It may not be enabled by default. On older versions you could try early-oom which is not part of systemd. OOM stands for Out-Of-Memory.