Saturday, August 16, 2008

Linux OOM-killer

Please make sure your OOM-killer settings are configured appropriately.

For us, MySQL is configured for ~40G of memory. A developer had a runaway PERL script that ate ~90G of memory before wiping out our production server. OOM-killer should have tagged that as bad and killed it. Unfortunately, it didn't - the server ran out of memory and cratered. Surprised it didn't sprout legs and go skynet on us.

Didn't help that upon reboot the RAID controller found a bad drive causing check/recover tables ot be run while in degrade mode.

No comments: