Linux Elevator Algorithm

There has been much debate over elevator algorithms concerning the 2.6 kernel.  For a database system, deadline and CFQ seem to be the most debated.  For a small database with relatively few drives in a RAID array, and relatively small memory, deadline works best.  One of my systems has a 48 x 15k drive SAS RAID array.  When you combine over 2GB/s of throughput, 128GB of memory, command queuing, etc - latency isn't the issue.  Leveraging caches to get a sequential R/W load is dramatically more important.  CFQ does this via implementation of R/B trees ordered by sector.  If you're periodically reshuffling the deck based on time expiration (deadline), you end up seeking all over the disk. 

Converting high performance servers to CFQ over deadline resulted in a 10-30% performance bump depending on load.

OLTP servers might have different experiences, but CFQ in an OLAP environment appears to be the way to go.  (for us! haha)

Linux OOM-killer

Please make sure your OOM-killer settings are configured appropriately.

For us, MySQL is configured for ~40G of memory. A developer had a runaway PERL script that ate ~90G of memory before wiping out our production server. OOM-killer should have tagged that as bad and killed it. Unfortunately, it didn't - the server ran out of memory and cratered. Surprised it didn't sprout legs and go skynet on us.

Didn't help that upon reboot the RAID controller found a bad drive causing check/recover tables ot be run while in degrade mode.

OLAP Linux filesystem settings

Before investing time in MySQL tweaks/settings -- properly configuring your I/O subsystem is key.

Regardless of underlying filesystem, if you write to many separate tables sequentially, you'll end up with disk fragmentation. Extent based allocation, and pre-alloc patches can serve to help, but increasing sequential disk accesses is critical.

If you're brave, one evil solution could be to place each table on its own partition.

A few other solutions:
1) noatime - a MUST
2) nodiratime - a MUST
3) turn off dir_index (ext3)
4) data=writeback if you have a battery backed RAID controller
5) Get your stride, RAID stripe, OS Page, and Database Page sizes aligned right. This is of huge importance.
6) Use memory/caches/anything to migrate your disks to sequential I/O. Prevent the head from seeking and you can improve throughput.
7) RAID10. One of my first databases utilized RAID6. The overhead regarding writes was abysmal.
8) RAID-N -- Have a hot spare. You don't want rebuilding to wait until you get around to finding a replacement drive.
9) RAID Controller - If you have a split Read/Write cache, allocate as much memory as possible to writes. If you have contiguous blocks, reads can be quick even if you have a cache miss.
10) Memory caching - Linux allows you to tune the dirty memory ratio. Do it! Standard installations are tuned towards desktop / OLTP servers.
11) Swappiness - Oh the religious debate this creates... Do you want your memory allocated for the application/database/data caching, or used so that the guy with a 8hr stale SSH session can have snappy performance?

MySQL and I/O Schedulers

Been doing some experimenting in the lab. Appears that deadline works great for a OLTP database load, but CFQ performs better in an OLAP environment.

Will post some empirical results eventually, but here's my $.02:

Let the OS reorder / batch anything headed to disk. Re-order as much as possible to reduce head movement.