Swatting weird cron bugs
Feb. 13th, 2009 07:30 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
altivoEarlier this week I installed a bandwidth monitoring tool on the Alpha DS10 at work because it came highly recommended by a couple of people. The Debian installation with synaptic, for whatever reason, was a flop. It didn't all get in the right places, wasn't functional, and I couldn't tell what exactly was wrong, so I "uninstalled" it. I figured I'd download the source, compile, and reinstall next week.
A couple of days later, I realized that my mailbox on that machine had hundreds of messages. Huh? There's no mail service to there, so they had to be internal. It turned out they were coming from crond, the scheduler that runs batch jobs at preset intervals. It was trying to run a reporting job from the uninstalled package, every five minutes. And failing, and failing, and failing...
Except, there was nothing in the crontab. Neither root nor my own account (the only two real non-daemonic users on that machine) had anything in crontab at all. The man pages for crontab suggested that the crond had its own internal storage that was updated only when crontab was updated. OK, so I figured the bad package had installed a crontab, and the uninstall had erased it, but crond never caught on and was still trying to run the now dead crontab. What to do?
Reboot? Stop the daemon and restart it? Neither one did anything. Hmm, how about if I make a new crontab, either empty or with some innocuous thing like "display the time at midnight each day" and let crond realize that the other job was no longer there? Nope, that didn't stop it either. Literal clockwork kept sending me the same error every five minutes. I purged the mailbox twice a day because it was getting so large.
Finally, today, I found it. Debian's man files are not always the best. The crond was using not just the crontabs in /var/spool/cron but also reading manually supplied tables in /etc/cron.d and of course, that's where the culprit command was coming from. Deleted it (as the uninstall should have done,) and the error mails finally stopped coming.
I'm inclined to think such an ill-behaved daemon should instead be named Crom, like Conan's god who lives on a mountaintop and ignores the prayers and lives of mortals. Or maybe I just should have called in Drax, that Timelord technician who was an expert in fixing disordered chronostats?
A couple of days later, I realized that my mailbox on that machine had hundreds of messages. Huh? There's no mail service to there, so they had to be internal. It turned out they were coming from crond, the scheduler that runs batch jobs at preset intervals. It was trying to run a reporting job from the uninstalled package, every five minutes. And failing, and failing, and failing...
Except, there was nothing in the crontab. Neither root nor my own account (the only two real non-daemonic users on that machine) had anything in crontab at all. The man pages for crontab suggested that the crond had its own internal storage that was updated only when crontab was updated. OK, so I figured the bad package had installed a crontab, and the uninstall had erased it, but crond never caught on and was still trying to run the now dead crontab. What to do?
Reboot? Stop the daemon and restart it? Neither one did anything. Hmm, how about if I make a new crontab, either empty or with some innocuous thing like "display the time at midnight each day" and let crond realize that the other job was no longer there? Nope, that didn't stop it either. Literal clockwork kept sending me the same error every five minutes. I purged the mailbox twice a day because it was getting so large.
Finally, today, I found it. Debian's man files are not always the best. The crond was using not just the crontabs in /var/spool/cron but also reading manually supplied tables in /etc/cron.d and of course, that's where the culprit command was coming from. Deleted it (as the uninstall should have done,) and the error mails finally stopped coming.
I'm inclined to think such an ill-behaved daemon should instead be named Crom, like Conan's god who lives on a mountaintop and ignores the prayers and lives of mortals. Or maybe I just should have called in Drax, that Timelord technician who was an expert in fixing disordered chronostats?
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2009-02-14 07:06 am (UTC)"If you won't help me...I spit on you!"
XD
no subject
Date: 2009-02-14 12:01 pm (UTC)no subject
Date: 2009-02-14 12:09 pm (UTC)Still wouldn't have been a problem if that particular package had been properly assembled. The real fault lies with those who put together the Debian installation for cacti. Both the installation and the uninstall are incomplete and poorly scripted.
no subject
Date: 2009-02-14 12:20 pm (UTC)no subject
Date: 2009-02-14 12:28 pm (UTC)I'll get the source, compile it, and install. Most likely that will come with configuration instructions too, which were omitted from the package.
Same problem with bandwidth-d. It installed without any documentation or configuration instructions at all.
no subject
Date: 2009-02-14 01:19 pm (UTC)no subject
Date: 2009-02-14 01:54 pm (UTC)no subject
Date: 2009-02-14 01:25 pm (UTC)no subject
Date: 2009-02-14 01:52 pm (UTC)