Back to the little things
About a year ago, my mail server started acting up.
The problem was subtle at first: mails to BJ's account wouldn't deliver, but mine would. Mostly. Kyle might have lost some emails, but I couldn't be sure. I also started receiving errors when viewing my mail via a web-based mail reader. The pages often wouldn't display at all.
Eventually, the problem became so bad I couldn't load more than one or two pages before I wasn't able to load any pages via the mail reader, and I had to check what was up. I had checked a few times, running "ps -ef | grep qmail" and looked at what was running. I'm not sure what prompted me to check, but I took off the qmail grep filter and looked to see what was up.
Turns out, the problem was each email I tried to delete (delete because I knew the email address was a spam address) caused a process to start up and not die. My system was running out of resources because of the undying processes. Instead of deleting the emails, I switched the processes to deliver to a dummy email account, which I could later view and filter.
When I figured out the fundamental problem (the delete email processes not dying), I recalled the change I had made a month before I started having the system problems. I, then, remembered my most recent lesson learned: listen to the little things, before they grow into big ones.
Clearly I need to listen better.