Friday, June 30, 2006

$10,000 worth of radioactive materials and other work screw-ups

I don't tend to screw up at work (shut up Ryan!), but when I do, it usually takes quite an army to fix it. Plus, they always tend to occur when I'm under tremendous pressure to get the project done right or there's a tight deadline that I "just can't miss. "

Take yesterday for example. As most of you know, I've been working on a HUGE project that needed to be done by today. The project was going well, but lately we've noticed a number of data discrepancies between the new system and the old that, if left untreated, would cause a multitude of issues when we tried to do our post-project synch. I was left in charge of changing the enrollment schools of 44 students so that they would match. This, my friends, is supposed to be the world's simplest process. All I needed to do was identify the new school and write an update script that popped that new school into the old record. At the most, the process should have taken 30 minutes.

That was 11:00 yesterday morning;I finished at 7:30 this morning.

I still maintain that the completely snazzy, multi-part script that I wrote to accomplish the analysis and update in a single bound was some of my best work. I also maintain that the acoustics in Lincoln Center have nothing on this building as people came from all directions when the heard my scream when I saw this go by after I ran the UPDATE statement: 127808 records updated.

I had forgotten the "Where clause." (non-geek translation: the thingy that tells you WHAT specifically to update as you normally don't mean for the same edit to go to every record in the database) As a result, 128K students, from Preschool through 12th grade, were all enrolled at Sandalwood Elementary.

To their credit, Justin (the DBA) and Mark (my boss) didn't try to kill me. In fact both assured me that the problem wasn't as big as I thought it was. See, at least one or two of the karma Gods had been looking out for me, and I had timed my mistake exactly two minutes after the last scheduled backup. So, we had a really close point-in-time from which we could grab our data, and the users wouldn't have had much of a chance to make additional changes before we shut down the system. Plus, since the backup hadn't had an opportunity to get on to tape (or wherever back-ups go to die), so getting it back would be an "easy process."

Easy, my ass. The damn thing took 6 hours. Apparently the entire networking team subscribes to the Emerald City work-ethic. They get up at 12 and start to work at 1, take an hour (OR 5!!! Without answering their cell phones!!) for lunch and then at 2 they're done. It's jolly good fun, but not when they've left the back-up running with the wrong file name and they're the only people who can make the backups.

Since we didn't have anything else to do, Mark started telling me stories about the stupid things he did when he was a young pro -- like wiping out the entire financial spreadsheet system he designed about five minutes after he implemented it. I shared with him this story...

When I was 19, I landed a research job at Bowman Gray Medical Center at Wake Forest. My Grad Student, James* had made it very clear to us undergrads that after every experiment was complete, ALL WASTE was supposed to be cleaned up through this aspirating pipette that sucked it out of the building and probably into someone's pool in downtown Winston-Salem. Then, the jars were to immediately go into the thermal cleaner. If we were going to keep something, we were to clearly label it. After all, James threatened, we never knew when someone was going to come in and clean up.

Well, I knew, because I did it. That was the summer of me doing stupid things to lose weight (like eating lots of ramen noodles and drinking Kool-Aid -- OK OK, I was a terribly poor college student). That could have clouded my judgement, but I don't think so. Either way, instead of going to lunch one day, I started looking for ways to not be tempted by food. Therefore, I started cleaning up the lab. Since all the important stuff was labeled (including some of James's), it was more than abundantly clear what was being used and what wasn't.

When he returned, the lab was spotless...too spotless...the kind of spotless where you just start looking around for the one thing that you think that you forgot to label as "keep." Which he did....

I don't remember the details of the conversation, just the aftermath. Basically, he asked me where the UNMARKED jar of RADIOACTIVE materials had gone; I told him that I had sucked the entire contents of the jar into the aspirator (as instructed) and stuck the jar in the thermal cleaner (also as instructed). James then freaked. Not because I necessarily had done anything wrong, but because I had followed his instructions to the letter. While he was off lunching it up, this enterprising little co-ed had just sucked $10,000 worth of radioactive isotopes (and a good portion of his thesis work) literally down the tubes. What was worse, was that the aspirator that we had been told that ALL WASTE was to go down was actually, ALL NON-RADIOACTIVE WASTE.

I don't know how I actually kept my job that summer. James had a "discussion" with our boss that I wasn't privy too, but where I suspect he went in preparing to screw me and got screwed himself. Either way, it was a beautiful thing.

In the end, I (obviously) didn't go into medical research. I finished the biotech concentration because I had already started it, but I pretty much knew that I would never do anything with it. I did, however, take away a number of different life-lessons from that experience. The most important of these was: If you are going to insist all summer that you don't have your sister's sweater, don't wear it on the day that you have your security badge photo taken. The second most important though, and the one I still use today was: Follow your own rules because you never know who else will.

2 comments:

Kim said...

As a researcher who has worked with radioactive materials as a student, I can truly appreciate this story (although I tried to stay as far away from the stuff as possible, forget lab cleaning! ;) ). It's funny how every lab has its big "oops" story...

Jules said...

u"The most important of these was: If you are going to insist all summer that you don't have your sister's sweater, don't wear it on the day that you have your security badge photo taken."

Damn skippy there, sister of mine. Now, do I need to check your next DMV photo for my I <3 NY fitted t?