![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Swatting weird cron bugs
Feb. 13th, 2009 07:30 pmEarlier this week I installed a bandwidth monitoring tool on the Alpha DS10 at work because it came highly recommended by a couple of people. The Debian installation with synaptic, for whatever reason, was a flop. It didn't all get in the right places, wasn't functional, and I couldn't tell what exactly was wrong, so I "uninstalled" it. I figured I'd download the source, compile, and reinstall next week.
A couple of days later, I realized that my mailbox on that machine had hundreds of messages. Huh? There's no mail service to there, so they had to be internal. It turned out they were coming from crond, the scheduler that runs batch jobs at preset intervals. It was trying to run a reporting job from the uninstalled package, every five minutes. And failing, and failing, and failing...
Except, there was nothing in the crontab. Neither root nor my own account (the only two real non-daemonic users on that machine) had anything in crontab at all. The man pages for crontab suggested that the crond had its own internal storage that was updated only when crontab was updated. OK, so I figured the bad package had installed a crontab, and the uninstall had erased it, but crond never caught on and was still trying to run the now dead crontab. What to do?
Reboot? Stop the daemon and restart it? Neither one did anything. Hmm, how about if I make a new crontab, either empty or with some innocuous thing like "display the time at midnight each day" and let crond realize that the other job was no longer there? Nope, that didn't stop it either. Literal clockwork kept sending me the same error every five minutes. I purged the mailbox twice a day because it was getting so large.
Finally, today, I found it. Debian's man files are not always the best. The crond was using not just the crontabs in /var/spool/cron but also reading manually supplied tables in /etc/cron.d and of course, that's where the culprit command was coming from. Deleted it (as the uninstall should have done,) and the error mails finally stopped coming.
I'm inclined to think such an ill-behaved daemon should instead be named Crom, like Conan's god who lives on a mountaintop and ignores the prayers and lives of mortals. Or maybe I just should have called in Drax, that Timelord technician who was an expert in fixing disordered chronostats?
A couple of days later, I realized that my mailbox on that machine had hundreds of messages. Huh? There's no mail service to there, so they had to be internal. It turned out they were coming from crond, the scheduler that runs batch jobs at preset intervals. It was trying to run a reporting job from the uninstalled package, every five minutes. And failing, and failing, and failing...
Except, there was nothing in the crontab. Neither root nor my own account (the only two real non-daemonic users on that machine) had anything in crontab at all. The man pages for crontab suggested that the crond had its own internal storage that was updated only when crontab was updated. OK, so I figured the bad package had installed a crontab, and the uninstall had erased it, but crond never caught on and was still trying to run the now dead crontab. What to do?
Reboot? Stop the daemon and restart it? Neither one did anything. Hmm, how about if I make a new crontab, either empty or with some innocuous thing like "display the time at midnight each day" and let crond realize that the other job was no longer there? Nope, that didn't stop it either. Literal clockwork kept sending me the same error every five minutes. I purged the mailbox twice a day because it was getting so large.
Finally, today, I found it. Debian's man files are not always the best. The crond was using not just the crontabs in /var/spool/cron but also reading manually supplied tables in /etc/cron.d and of course, that's where the culprit command was coming from. Deleted it (as the uninstall should have done,) and the error mails finally stopped coming.
I'm inclined to think such an ill-behaved daemon should instead be named Crom, like Conan's god who lives on a mountaintop and ignores the prayers and lives of mortals. Or maybe I just should have called in Drax, that Timelord technician who was an expert in fixing disordered chronostats?
