The Elusive Process Library

A couple of years ago I ran into an interesting situation at work.  It was one of those surreal things that kind of felt like it should have been in the movie "Office Space."  You run into this kind of stuff occasionally if you decide to sell your soul and work for a large organization.

I'm a Senior IT Engineer at a large healthcare organization in Nebraska.  For those of you who are surprised by this, yes there are actually large healthcare organizations in Nebraska.  Anyway, I was working on a project that required moving a medium-sized piece of equipment from one data center to another  data center across town.  By medium-sized I mean it was too big for one guy to move by himself, but not big enough to require a moving crew or any special equipment.

You would think that moving this thing would have been as simple as going into the data center, unhooking all the cables, removing it from its rack, and wheeling it out to a waiting Subaru to be transported to its new home.  Once at the destination the whole removal process would simply be done in reverse and the equipment would be in its new home whirring away and waiting to be called into action.  You would think this but you would be wrong.

The first problem with this assumption lies in the fact that we, as Senior Engineers responsible for installing, configuring, and operating these miracles of technology do not have access to the data centers where they are kept.  We have to get approval from Management and make arrangements with Operations to escort us to the device we need to work on.  If we bring someone to help us, they must do the same thing.  This is a security precaution implemented to prevent a rogue engineer from entering the data center and dismantling a production system before the Operations staff can wrestle him/her to the floor and secure the offender's wrists with cable ties behind his/her back while waiting for the authorities to arrive.  Instead, we manage these devices remotely. 

Think about that for a minute.  I can't walk into the data center with bad intent and a crowbar in hand, but I can sit undetected in the comfort of my home office at 2:00AM and slowly delete all the company data and render all the computing infrastructure useless with just a few mouse clicks.  Ok.

There are a number of other steps we must take that are security-related but I can't share them here.  Let's just say they are complex, take a lot of time, and some of them even involve biometrics.  They really want to make sure that the person they've worked with for the last 5 years is actually who they say they are. Being an astute reader, you have probably already guessed that most engineers really don't have the patience for much of this.  Management, in their ultimate wisdom, knew this and decided to boil it all down to one form for the engineer to fill out.  

This is how it works:  Once you answer about a page of questions, the online  form is routed to all the necessary approvers and eventually back to the Operations staff, who then contacts the engineer to schedule the activity.  All the engineer has to do is remotely shut down the equipment and Operations does the rest.  After the work is completed, Operations notifies the engineer that it is all done.  In the rare case that the engineer is required to be present while the work is done, he/she is notified again anyway.  This process works relatively well, although running cable for a new install or removing cable when a system is taken out of service can take some time.  That's an overview of the process for deploying something new or removing something from the data center.  Bear with me a little longer and soon we'll get to the Process Library part of the post.

With all this in mind, I decided to leverage the form and install/remove process  to get my system moved from one data center to the other one.  After all, it was basically just using the removal process and then immediately executing the installation process to achieve my objective of moving the system.  You would think the story ends there, but it's really only beginning.  This is where the Process Library part comes in.

I submitted my request on the appropriate form detailing what I wanted removed from one data center and where I wanted it to be reinstalled in the other data center.  It was immediately rejected.  Apparently it was too much information to go on one form and the person who receives the forms and then assigns the work thought it was confusing.  He said I should split it into two forms since it was actually two separate things I was asking Operations to do.  I complied and submitted two separate forms, both of which were immediately rejected.

The reason given was that two forms would generate two work tickets in their ticketing system and, although they were closely related, there was no way to tie the two work streams together to make sure they were done in the right sequence.  I asked if it would be possible to assign the two tasks to the same team and tell them something like "Do this one first and then immediately do the other one."  He told me that there was no way in their system to make sure the tasks would be assigned to the same team so there was no way to guarantee that they would be done in the correct order, which caused both forms to be rejected automatically.  After some discussion he eventually said "So what you really want is just to move the system from one data center to the other?"  I said "Yes, please.  I just need it moved.  Can we do that?"  I thought we were starting to get somewhere until he answered  "No, we don't have a process in our Process Library for that."

I asked him where the Process Library was kept and if it would be possible for me to write a process for them detailing the steps involved in the moving a system.  He informed me that I could certainly document the move process in the same format as the other processes in the Library, but it would have to be reviewed by the Process Engineering Committee and then approved by management before it could be placed in the library.

At this point I knew that documenting a new process would take less than an hour, but getting it reviewed and approved would likely take many years and I might not live long enough to finally celebrate its adoption.  As a point of reference, I’m 61 and in good health so my expiration date is well into the future.

I asked him again where I could find the Process Library so I could use a previously approved process as a template for my new one.  I really had no intention of writing a new process and then going through the long and grueling steps to get it approved, but by now I was curious about this Process Library thing and wanted to see it for myself.  By now I had decided that the best way to get my system moved would be to buy pizza and beer for the Operations staff and skip the form altogether.  I asked again where I could find the Process Library. 

Eventually I wore him down enough for him to admit that he’d never actually seen the process library himself but he had heard about it from others.  The perceptive readers are probably asking now that if he’s never actually seen the Process Library and didn’t know where it was kept, then how did he know that what I was requesting was not already in the Process Library?  I diplomatically asked that very question and he told me that he had a spreadsheet listing all of the processes in the Library.  I asked if I could see the spreadsheet and he told me that it was password protected and he was prohibited from sharing the password with me because it would be in violation of an Information Security policy. Ok. 

I decided that my best chance for locating the Process Library would be to ask one of the other IT employees who had been around longer than me.  I decided to ask Emily because she had been on the job for many years and used the word “process” in meetings more than anyone else.  If anyone knew the location of the Process Library it would be Emily.  I messaged her on Microsoft Teams and she immediately responded that she was in a meeting and she’d get back to me when her meeting was over.  I wondered who Emily was meeting with because no one really even knows what Emily does.  It must be important because she’s been doing it for a very long time, but no one really knows what function she performs. 

I decided to ask some other senior team members while I was waiting for Emily to call.  My next stop was Joe.  When I asked him about the Process Library he said that it was developed when management hired a consultant about twenty years ago and that person told management that they should have one.  Everyone was required to document what they did and how they did it and all of that information was eventually combined to build a Process Library.  That sounded plausible so I asked Joe where I could find it.  He told me that he didn’t know where it was kept and he had never actually seen it for himself, but he had once overheard some other employees secretly talking about it.  This didn’t help me but at least now I knew the history of the Process Library.

About the time I realized that I wasn’t making any headway with Joe, my phone rang. It was Emily. We exchanged the usual pleasantries and then I asked her where the Process Library is kept.  There was silence on the other end of the line for about 10 seconds as she recovered from the initial shock of hearing that question.  As she began to speak I noticed that the cordial tone of her voice was gone.  She was so uncomfortable with the direction of the conversation that I might as well have asked her what color lingerie she was wearing at the moment or if she was interested in coming over later and drinking mohitos naked with me in the hot tub.

Her voice got very soft and she spoke for quite some time, but I don’t think she actually said anything.  After a while she asked if there was anything else she could do for me.  I wasn’t interested in asking her about the hot tub thing so I thanked her and ended the call.  Back to square one.

My next stop was a Senior Analyst on our Application Development team.  I had worked with her several times and she had always been a good source of information.  I figured that most of our processes ultimately have something to do with applications at some point so someone from that team might know where to find the Process Library.  I reached her right away on Teams and asked her where I could find it.  She told me that she had asked the same question about a year earlier and was told to fill out a form requesting access to the Process Library.  She filled out the form but could never find out where to submit it.  She said everyone knew about the request form, but no one knew where to submit it.  I suggested that detailed instructions for requesting access to the Process Library are probably outlined in a process within the Library but no one could figure out how to get those instructions because no one actually has access to the Library.  It’s sort of like when you lock your only set of keys in your car.  All you really need to do to get them is to unlock the car and grab them and then you would be able to unlock your car.  She suggested that Judy, our Administrative Assistant, would certainly know how to gain access and that I should probably just ask her about it.  I thanked her and moved on.

Now you should know that Judy and I have a “history” and it involves a red Swingline stapler.  When I first started this job my wife bought me a red Swingline stapler like the one Milton had in the movie Office Space.  It was kind of an inside joke about me going to work for a very large corporate entity.  About a year into my employment, Judy came around to everyone and replaced their staplers with identical black ones.  I wanted to keep my red Swingline but Judy wasn’t going to stand for it.  Over the next couple of years she made many attempts to covertly replace my beloved red Swingline with a smaller generic black stapler so that everyone in the building would have exactly the same type of stapler. 

I wouldn’t have really cared if it wasn’t for the fact that my beautiful wife had given me the red Swingline.  In fact, I hardly ever use a stapler.  I’ve probably used less than 20 staples since I got the Swingline.  But it has sentimental value and I don’t want it to end up in a box of unused staplers stacked in some remote part of the basement storage area.  To protect it from Judy I usually locked it in a drawer or put it in my computer bag if I was going to be away from my desk for more than a few minutes.  This kept it out of Judy’s clutches and it was always available in case I happened to encounter some paper that was begging for a staple or two.

One afternoon Judy came by and demanded that I hand over the red Swingline.  I refused and Judy stormed off muttering something about a disciplinary action.  At this point the smart choice would have been for me to just take the stapler home and leave it there, but instead I decided to super glue it to my desk where Judy could see it anytime she was nearby.  My co-workers enjoyed this immensely because they got to watch Judy try to pry it loose several times until I finally removed it with a little acetone and took it home.

Ok, so back to the Process Library.  I went to Judy’s desk and asked for a request form for Process Library access and also how I should submit it once I filled it out.  I figured that once access was granted then someone would tell me its actual location.  Judy informed me that we were out of Process Library Access Request Forms and triumphantly handed me a Printed Form Request Form that I could use to request more of them.

I took it back to my desk and filled out both sides and had one of the other engineers initial the “Witnessed By” section.  After we both inspected it thoroughly for spelling and grammatical errors I took it back to Judy.  She looked it over and immediately rejected it and handed it back to me.  Judy then informed me that we no longer use Section R and I would have to do it over again, this time leaving Section R blank.

I made my way back to my desk and filled out the form again.  In the Quantity Requested field I specified 50 copies of the Process Library Access Request form and returned it to Judy.  She immediately rejected it, telling me that we have to order forms in increments of 100.  I went back to my desk again and filled out a third Printed Form Request Form, this time asking for a quantity of 100 copies of the elusive Process Library Access Request Form.  Miraculously it was accepted.  Judy said she would let me know when the forms arrived in 6-8 weeks.

At that point I went back to the Operations Control Center and simply asked the Operations staff if they would just move the system in question for me and afterwards I would buy them all the pizza and beer they could consume.  This approach worked and the system was moved by the end of the next day.  I fulfilled my end of the deal by taking them to a local pizza place and buying a couple of large pizzas and an insane number of PBR tall boys.

This is almost the end of the post, but I will say that the new forms eventually arrived and they had been printed on the wrong paper so they were returned.  After another 10 weeks the replacements arrived, but it was determined that the print shop had used the wrong version of the form and it contained incorrect contact information.  On the third try they were perfect so I filled one out and submitted it to Judy.  This time she accepted it and sent it on to the person who processes that particular form and then I never heard any more about it.  After almost 2 years I still have no idea where the elusive Process Library is located or what it contains. 

Later, I did run into a consultant working in our building who said he was retained as a Process Architect.  I asked him if he was working on the Process Library and he said “Not yet.  I just requested access to it.”  I think I will check with him in a few years and see how it’s going.

That’s the end of this post.  I think if there is a lesson to be learned here it would be that sometimes organizations become so focused on implementing and following processes that they become process bound and are unable to execute the basic things necessary to succeed in their primary mission.  I think there’s a sweet spot hiding somewhere in between the chaos of lean development/implementation methodologies and being process bound.  Very few organizations find this sweet spot and ours certainly isn’t one of them.

 Thanks for taking the time to read this exceptionally long post.  I promise to make the next one shorter.