Why transaction logs are a good idea

[altivo]

I don't usually write about work. However, this is incredibly stupid and irritating and I have to complain somewhere.

On Friday Feb. 11 the shared library system (we share with eleven other libraries in our region) had problems. It was acting really quirky and we all knew it. The system managers waited until 4 in the afternoon to reboot the server when it was obvious by 10 in the morning that it needed this.

Whatever the problem was, today we find that no transaction that was made on Friday actually registered in the database. Books returned show as still checked out. Books checked out left no record of who has them. Books cataloged that day do not exist in the system. New library cards given out that day are unrecorded.

This "modern" client-server system keeps no transaction logs, and has no recovery method for these problems. Transaction logs would be easy to create, since the client programs spool their activity anyway and send it to the main server in small batches. But no, it has never been thought of apparently. The software vendor says it can't be done unless proposed and voted on as an "enhancement" -- that's a three year long process.

I do hope banks and retail establishments aren't following this same stupid methodology.

Edit an hour later: Only when confronted do we get the facts. The system was restored from Thursday's backup during the reboot late Friday afternoon. But they never told us until Monday afternoon, three days later. This is human misbehavior that compounds a computer problem.

Comments

Suddenly feeling very Luddite....

[songdogmi.livejournal.com]

What a horror story! Computer system designers still wonder why we want things like transaction logs and printouts. There's no wonder why a lot of people still don't want anything to do with computers and automation.

Re: Suddenly feeling very Luddite....

[altivo.livejournal.com]

Well, I hope that banks and hospitals do a better job of protecting the data in case of problems like this. Our situation was compounded by a "cover it up" attitude that prevails in the systems center. Like they thought if they didn't tell we'd never find out.

Turns out they were advised of people finding issues early this morning. But they waited until 3 pm to admit to the rest of us that yes, they restored the database late on Friday from a Thursday backup, so activities from Friday morning and afternoon are lost.

Apparently no conception at all on their part that the longer they wait, the more difficult it becomes to repair the loss. I guess they are assuming that we can't repair it at all, when in fact at least part of it can be caught. Paper records that haven't yet been filed can be re-entered, books not yet shelved can be checked in again just in case. Once shelved, there's no telling which ones need reentry. In other words, much of the problem is still human behavior, not the fact that a computer sneezed.

[songdogmi.livejournal.com]

At least if they'd owned up to it sooner, everyone could've not reshelved books and not filed paper records. And you wouldn't have to re-do work. Hm, wonder how the computer center would like it if you forced them to redo the work? Or at least buy you all lunch.

[altivo.livejournal.com]

No way would I let the database people redo that work. Getting it right depends on our own ability to recall the sequence of events on Friday, and pull together bits and pieces. For instance, three elementary school classes came in to the library. Each teacher received a list of the books their students checked out. We had to call the teachers and ask them for a copy of their list to use in reconstructing those particular checkouts.

Other reconstructions were based on the fact that Friday morning's interlibrary loan list came up again on Saturday, with the addition of further requests generated Friday night. And so forth. We'll never get it all, but with luck we got more than half.

[doco.livejournal.com]

Wow. This is a real-life implementation of a corrupted database. :)

(And no, banks wouldn't even *survive* something like that. Peer review is pretty strong in that field, and an incident like that would make their ratin g drop like a rock.)

[altivo.livejournal.com]

Sending a few real life alligators to chew on people's legs now. :)

A hardware problem, which this apparently was, cannot necessarily be blamed on anyone. Trying to conceal the effects though, can be. There will be some very strong words about this unless I miss my guess completely.

[doco.livejournal.com]

I guess libraries just aren't deemed important enough for measures like "2 out of 3" redundancy, which is what banks and other database-reliant businesses use.
Still, the potential mess that could be only undone by a *complete* manual audit frightens me.

[altivo.livejournal.com]

Well, books aren't considered as important as money. And having a system as secure as what a bank might have costs money, something libraries tend not to have. I don't get paid like a banker either, even though I have more education and experience than most of them. ;p

*looks around*

[dogteam.livejournal.com]

I'll be damned.
A horse that works in a library.

Now I know two.

It's a wiggly world, ain't it?

Re: *looks around*

[altivo.livejournal.com]

<horse-grin>

Of course. Horses like quiet spaces. And large herbivores are often deeper thinkers than folks expect. After all, we have lots of time to philosophize.

Saggittarius, the most equine of the zodiac signs, is often said to be a lover of books and reading. It may be a telling clue that of our library staff of nine people, four are born under that sign. ;)

[dasra.livejournal.com]

This is the kind of utter nonsense I have to deal with _daily_. And no matter how many times we tell the project teams to do things a certain way, we essentially get a "fu!" in our faces, and they do what they want. Then we have to scramble to fix their screw up, and we're held accountable for their issues.

Only reason I keep the job is so I can keep the horse.

[altivo.livejournal.com]

And why I got out of data processing. Unfortunately, every job seems to depend on the durn computers now in some way or other. ;p