[Discuss] memory management

[me at mattgillen.net (Matthew Gillen)]

I'm looking for some advice on tuning my linux box's memory management.
 I've got an older workstation that has merely 4GB of memory.  If I try
to run Firefox, and a few java apps (e.g., Eclipse), my machine thrashes
about and effectively locks up because of out-of-memory issues.

For example: the mouse will continue to move, but won't change it's icon
contextually.  If I hit cntl-alt-f2 and try to log in to a virtual
console, mgetty will eventually ask for the username, but after I hit
enter, it just hangs, not popping up the password prompt, and after 60
seconds the login times out.  Trying to ssh into the machine from
somewhere else ends up timing out.

After going on like this for literally 10 minutes, OOM-killer sometimes
kills the right thing (one of the two processes hogging the most memory:
firefox or eclipse), and the machine becomes usable again sometime later.

I have heftier workstations I can use, but this behavior is really
frustrating to me, because I'd like to think linux does good memory
management.  I've tried using huge swap (2x physical memory).  I've
tried with virtually no swap (on the theory that without swap, there
would be no thrashing and at least oom-killer would have to do its thing
without locking up the machine for 10 minutes first).  The problem there
was oom-killer making bad decisions about what to kill (e.g., the window
manager, and then whatever out-of-control process is sucking up memory
just sucks up whatever got freed, and nothing gets better).  At least
with some swap oom-killer seems to make better guesses about who to murder.

Does anyone have any tips on how to prevent linux from thrashing like
that?  The behavior when low on memory seems atrociously bad.

Thanks,
Matt