How to Use IT Processes to Shoot Yourself in the Foot

I work in the Information Technology Department at a large regional teaching hospital. A colleague and I were recently automating a management function for one of our larger servers and we found ourselves in need of a service account to run some scheduled tasks overnight without human intervention.

If you're unfamiliar with Windows service accounts, they're basically a login to the server or domain that background or system processes use to implement the many different functions of a system. There are some built-in ones that Windows uses for all it's built-in background tasks, but if you want to add some software that runs in the background you need a service account to run it. In our case we needed a domain account because our software needed to access multiple servers. A domain account is used to log into multiple servers. A local account is good only for one specific server.

About two years ago our Information Security Department decided that engineers like me would not be allowed to set up service accounts anymore and that we would need to request all new service accounts through them. There was a lot of grumbling over that decision. Previously we had a script that we could use to create them in about two minutes.

We had also implemented a workflow management/help desk/request system called Service Now. This system is supposed to do everything but make coffee and, presumably, it was flexible enough to integrate with popular coffee makers and other kitchen appliances to make their operation much more efficient. It was also supposed to save us a ton of money. They didn't tell our management that they would need to hire a whole separate IT staff to configure it and keep it running.

Back to the service account. So now we have to request service accounts through Information Security. InfoSec had supposedly implemented a “Service Catalog” within Service Now where we could just select an item we needed from them and fill out a form. Then Service Now would generate a Service Now Request and route it to whoever needed to approve it and then after it was approved it would automagically be routed to the person within InfoSec who needed to fulfill the request.

This is where the problem started. It all sounds pretty great. Click on whatever you need, fill out a short form, click “OK” and then you get what you want. Unfortunately it didn't work out exactly like that.

Right out of the box Service Now doesn't know anything about service accounts or anything else so you have to get the Service Now IT staff you just hired to customize Service Now and implement the functionality you want. This process is apparently pretty difficult because it took them 18 months to implement an InfoSec Service Catalog with about a dozen items in it. On top of that, they used a development methodology popular for a short time in the 1990's called “Agile.”

I won't go into Agile in detail, but the idea of it is basically that developing something that's 100% perfect is a lot of hard work so it's better to implement something that's half-assed in a short timeframe, get paid for it, and then spend the rest of your natural life fixing the things you should have done right in the first place. Software companies largely abandoned Agile when they realized that it really pisses off your customers when they pay you a million dollars for software and you give them something that doesn't work. I'm not a genius, but I think that's a bad move that will cause your customers to never trust you again.

So, after 18 months, we had a Service Catalog with a lot of half-baked items in it that really didn't do much. As you can imagine, a teaching hospital doesn't stop everything and wait 18 months for software to be built.

While we were waiting, anything we needed from InfoSec had to be entered as a problem ticket and assigned to InfoSec. They have a guy in that department who is known as “The No Guy.” I wrote a whole blog post about him in 2019. Any work you send his way gets pushed back to you with a “Not my job” or an “I don't know” or an “I don't know what you want.” Once he surmises that you need him to actually do some work, he will simply reassign the ticket back to you with one of those short notes or perhaps just put an ambiguous comment and just close the ticket. Anything you need him to do usually requires management intervention. I've been dealing with him for ten years.

So for this particular service account we had a Service Catalog item that we could use and everything should have gone smoothly once that was done. It didn't. The first problem was that the InfoSec Service Catalog is not available to everyone. Access is granted on an “as needed” basis. My colleague and I spent about 2 hours searching the vast landscape of Service Now and never found it. I figured that for either of us to locate the elusive InfoSec Service Catalog I would purposefully enter the Service Account Request incorrectly as a Problem Ticket and the No Guy, not realizing that he was being played, would send me the correct link. It worked like a charm. Within 20 minutes I had the super secret highly classified link to the InfoSec Service Catalog along with a snarky message from the No Guy telling me my request was closed because I had entered it incorrectly.

Armed with the precious link, my colleague and I opened the elusive InfoSec Service Catalog and immediately found the Service Account Request item along with several other highly sought after request items. It took all of 30 seconds to fill out the fill out and submit the request. We thought we were home free. Not even close.

Once a service account is requested, it apparently gets automatically routed to a committee of InfoSec student interns and newly-graduated security engineers for their review. It is called the GRC committee. I don't know what that stands for and I really don't care. If it passes their intense scrutiny they send it over to the No Guy's team to possibly do the actual work.

This is where the Agile methodology hit us hard. After I submitted the form, instead of being routed to the GRC for approval, it was routed back to me for my approval. Therein lies the problem. I don't have the application permissions to approve anything. Unfortunately my colleague and I thought that we were home free so we patiently waited 10 days for InfoSec to spit out our service account. At that point I started asking questions and was told by InfoSec that I would have to approve the request so routing could continue, but they refused to give me approval permissions even just for the one minute it would take to get the ticket moving again.

We engaged the expert Service Now Team and they worked on it for a couple of days but could not free the ticket to get it moving again. I tried claiming extenuating circumstances to get the service account created but they weren't having it. After 19 days and about 50 man hours I was able to coerce someone with administrative privileges within Service Now to give me the necessary permissions for a few minutes so I approved the ticket. It immediately continued on it's seemingly endless journey and we are presently waiting to see what the No Guy does with it.

Remember: We used to be able to create these accounts in less than 2 minutes with a simple script. Now we're going on 21 days. Now that's efficiency at it's very best.