Beware the Silent T

 Something really strange is happening to the English language in the United States.  I first noticed it a little over a year ago.  At first I thought it was just one person who needed to go back to kindergarten for reprogramming, but now it is widespread.  It is in the news, TV shows, movies, and in daily face-to-face conversations.  I hear it more in the under 40 age group so us older people may be less likely to contract it.  You might be wondering if I'm describing a new virus or food-born illness, but I'm not.  It is even more insidious.

I'm talking about improperly placed silent Ts.  "What is that?" you may ask.  It happens when someone speaks and uses a word with the letter T as the first letter of the second (or sometimes the third or fourth) syllable and does not speak the T.  They just treat the venerable T as if it didn't even exist.

Let me give you a couple of examples so you'll know what to listen for.  We'll use the name of the college I attended since I hear this one all the time here in Omaha.  I went to Creighton University.  People with this affliction do not say the T that is the first letter of the second syllable in Creighton.  In real English, you would pronounce this as "CRAY-tun" (forgive the pronunciation notation if it's not correct, but you get the idea).  When pronounced properly, the second syllable would sound the same as when you say the word "ton."  They pronounce Creighton like "CRAY-un" and the second syllable sounds like the first syllable in the word "underwear."  "CRAY-un."  Like "CRAY-on" with a U instead of the O.

The strange thing is that between the syllables they still insert some kind of hard stop that allows the listener to clearly differentiate the two syllables rather than letting them run together.  "CRAY-un."  I can feel my face turn red with anger every time I hear that one.

The other example is the word "water."  These sick minds make the mouths connected to them say "WAH-urr."  Again, it makes my blood boil (no pun intended).  What was wrong with "WAH-tur?" When I hear one of the local news anchors speaking like this for the entire duration of a 30 minute news show I leave the room, go to my computer, and try to locate a local anger management class that I can immediately attend.

You might be wondering why this bugs me enough to write an entire blog post about it and, to be honest with you, I just don't know but it does.  You can't just change the English language because you're too fucking lazy to use it correctly.  It doesn't mean you're one of the cool kids.  It means you're fucking lazy and you need to repeat elementary school so you can get the basic things right.

Ok, that's enough of that for now.  I'm done ranting, but I'm still mad about the whole situation and if you speak to me like this during a conversation you can be sure I'm not listening to anything you say.  I'll be thinking about the fact that you deserve to have projectile diarrhea until you learn to speak properly again.

On another note, I hope everyone had a great holiday season...

The Mythical "Single Pane of Glass"

Since my very first days in IT almost 35 years ago I've heard people talk about something called "a single pane of glass."  It was said that somewhere an application must exist that would let IT people manage all the components of their hardware, data storage, backups, software deployment, licensing, performance management, capacity planning, and availability monitoring into a single application that an engineer or manager could use to manage the entire IT environment from one place.  Theoretically it would consist of a single screen showing the status of everything IT is responsible for and allow the user to drill down into any of them for more detailed information or to make changes to something without ever leaving the application.

The reason I'm dredging this up again is that I heard it again in our staff meeting last week when our new manager was talking about our team's upcoming initiatives.  This is an interesting concept, but it raises a couple of questions:  1) Does it actually exist, and 2) Should it actually exist?  Let's think about this for a minute.

All of those things listed above are important parts of my world in IT because I deal mostly with hardware and its deployment and operation.  All of those capabilities are available to me, but I have to go to a different application for each one of them.  The goal of "a single pane of glass" would be for me to see and manage all this stuff in one place.  

The Holy Grail would be for all these functions to be integrated and automatic.  For example, a performance management function might detect that a particular application is running slowly and automatically look at disk space consumption and memory/CPU utilization and determine that the performance problem is being caused by the disk containing the operating system being critically low on free space.  It would automatically respond by using its storage function to allocate a little more space to get the system running correctly again and then fire off an email to me advising me of the situation, how it was fixed, and that follow up would be needed to find out why it happened.  That would be pretty cool, especially if it was all done automatically without the need to wake me up at 2:00AM.  Let's pick this situation apart for a second and create a rough design of the "Single Pane of Glass" system.  That's a long name so we'll just call our new system Big Pane.

To get to Holy Grail status, this system would need to detect the problem, gather performance data from all the possible sources of the problem, sift through the data to determine the most likely cause, devise a solution or two, and then implement a possible fix.  We'll throw one more wrench into the works here and say that the system is running on Windows Server, so if storage was added then it would somehow need to notify the Windows operating system that the extra storage  resources were available for it to begin using.  

Detecting the problem would be easy.  We currently have very expensive software that can reach into another software system, perform various functions, measure how long it took, compare it to previous results, and determine if something isn't running as expected.  The challenge here is that software systems are not created equally so you have to check them all in different ways to get the true picture of how they're performing.  We have about 60 of these software systems and each one works only for a single software package.  There is no common denominator determining how performance is measured across all of them.  For this reason our new system would need different testing capabilities for each system.  Just for parity with my company's existing situation, we'll just say it would need 60 different testing functions.  If it actually worked, our management would require that it be expanded to accommodate our other 1440 application systems.

Next it would need to gather some data from all the possible sources of the problem.  Not a really big deal because just about everything has some kind of API (Application Programming Interface) and you can usually query them for the data you want using their API.  The problem is that they are all different so our system would need to know how to understand all of them.

Assuming we've gotten this far, Big Pane would need to determine the actual cause.  AI hasn't come far enough for a software system to think like an engineer with 30 years of experience, so it would most likely need to get this information from a knowledgebase somewhere.  Knowledgebases are not created equal, so you would probably do better creating your own, possibly from problems and solutions  you've seen before firsthand.  The solution could also be included in the knowledgebase so Big Pane wouldn't have to figure it out from scratch every time.  Each system would almost surely need its own knowledgebase entries.

Now that Big Pane knows how to fix the problem, it will have to implement the fix.  These days you can change almost anything in software so our Holy Grail system could make whatever changes are necessary just by going through the API of the problematic system.  Big Pane would need to contact our SAN management module in the SAN Director System and tell it to allocate some more disk to our system.  Luckily almost all of our systems use the same SAN for storage so this wouldn't be too difficult.  You'd just need to tell it what kind of storage is needed, how much you want, and what system it goes to.  

Now, you'd think that was all we needed to do but you'd be wrong.  You see, even the newest operating systems don't automatically recognize all changes, especially hardware changes like added disk space, memory, or CPU.  You have to restart them or notify them somehow that they can begin using these new resources.  The notification process is different for every operating system and sometimes different between different versions of the same operating system.  This part might not be a problem if a system is unusable due to a performance problem and will need to be rebooted anyway, but what about really specialized systems that are just running a little slowly?

One of the systems I support is used during open heart surgery.  A patient is hooked up to a bunch of probes, sensors, and cameras that collect various pieces of data the surgeon uses during procedures and then displays them for him in the operating room.  Some of these sensors are snaked up through an incision in the leg all the way to the heart.  If the system stops working during a procedure then the surgery has to pause until it is restarted.  In that case you have a bunch of doctors standing there as motionless as possible over a patient laying unconscious on the table with his/her heart fully exposed and several wires running through his veins.  You don't want this to happen for obvious reasons.  

The last task for Big Pane would be to send me a politely worded email advising me of the problem and how it was fixed.  This is always done the same way so it would be very easy.

Now that we have an idea of what this would take, imagine implementing it for 1500 different software systems simultaneously.  And imagine what it would take to keep  it up to date with all the regular changes in software, hardware, and configurations for all those applications.  Even with a huge investment in development, you'd end up with something of limited usability and it would never be truly complete.  And remember, the example we've discussed here is just for one scenario on a single system.  Most actual situations are much more complex than this one.

I'm not lazy and I'm not stupid either.  The effort and subsequent cost would be much larger than any reward from doing it could ever be.  The logical alternative to building this behemoth would be to hire a few good engineers and managers and just pay them well to use several different systems and be proactive in their daily work.  This is what we do today.  Now I would ask the second question "Should it exist," but you probably already know the answer to that by now.  Unfortunately, every time an IT executive goes to an industry conference he/she reads about some proposed Holy Grail system in a SkyMall magazine at the airport and we start this quest all over again.  If you're an IT executive and you're reading this, please stop.  SkyMall is not the place where you should get ideas for how you're going to spend your organization's IT budget.

Lean Development Methodology

 If you've decided to read this post then you probably recognize Lean Development Methodology as a development project methodology commonly practiced in IT organizations.  If you already know all about it then you might find this very boring and my feelings won't be hurt if you just skip to another article or go browse Amazon.

Assuming you're mildly interested, I'll tell you what I know about it.  The idea of the Lean Development Methodology is for a team to work feverishly on a project and bring it to market very quickly, often before it's completely finished.  This seems like a no brainer, but there's a little more to it than that.

Lean has a thing called Minimum Viable Product, or MVP.  Basically the idea of an MVP is that the system barely does what it is supposed to do in order to sell it, but not necessarily very well.  The idea if that if a company is producing a software product, then they want to start getting customers to pay for it as soon as possible.  If they can honestly say that it somehow performs the function it was designed for and they can get someone to buy it, then it is an MVP.

An MVP is almost certainly rough around the edges and may have a number of features that are not fully implemented yet.  It may have a ton of menu items that display "Not Yet Implemented" when you select them or maybe the whole system  needs to be restarted every 20 minutes.  It barely performs the function that it is being sold for but very little else.  The idea is that if the company can get customers to pay for it then they can use the money to finance finishing all the other parts that yet don't work and they can solicit feedback from the customers to help determine which features should be developed next.  Also, the theory is that if a customer invests money and time in buying an MVP, then they will be more likely to put up with some amount of deficiency and stick with it in order to protect their investment.

Let's look at an example.  This one doesn't involve developing software, but it illustrates the idea of Lean.  Suppose a company wants to add a rack of new servers to their data center.  There are a lot of things to consider, but let's focus on electrical power here.  A standard data center rack has a capacity of 42U in space for the servers.  This of a "U" as a slot in the rack.  You can fill 42 slots with servers and whatever else you need to put in there, but they all require electrical power and each one will have at least one power plug that needs to be plugged in.  Data center racks have these big surge strip things called Power Distribution Units, or PDUs.  It runs from top to bottom on both sides of the rack and has a bunch of electrical sockets on it.  Most PDUs can actually do much more than that, but we're just concerned about power capacity for this exercise.

Each rack probably has more than one electrical source running to its PDUs and there's a maximum capacity that those circuits will safely support.  Each device you put in the rack uses a chunk of this available capacity and it's really important not to exceed it.  For our example, the Data Center Manager has told us that the power capacity for the rack is 40,000 watts.  

So now we need to figure out how many watts each server consumes to get to how many servers we can safely put in the rack.  Those numbers are readily available from the manufacturers of each piece of equipment.  They are almost always in the products specs on the manufacturer's web site and it takes all of 2 minutes to click around and find them.  This is where we get to the Lean part.

Management, in it's vast collective wisdom, has decided that figuring out the total power requirement and determining how many servers can go in the rack should be a full-blown project rather than a 5 minute task involving adding up the numbers using the calculator built into someone's iPhone.  So next they assign a project manager, a system engineer, and four programmers to accomplish this seemingly monumental task.

After four weeks of meeting diligently on the project to develop a project mission statement, a charter, define deliverables, establish a schedule of status meetings and a reporting hierarchy to keep management informed of the team's progress, the programmers start coding the solution.  Immediately they discover that they have neglected to establish a SharePoint site and Microsoft Teams chat room for the project, so development stops for a couple of weeks and those pressing items are addressed.

The design that the team has chosen consists of a python program that reads an input file containing the wattages of each server, one per line, and adds those amounts together to produce a result.  The result is written as a single line to an output file where the user can go retrieve it.  The input file contains 42 lines, one for each slot in the rack.  If a slot is empty then there must be a zero on the line.  

After some minimal testing, the team decides that it has an MVP and decides to release it.  After a few more weeks it clears the Change Management process and a thorough review by Information Security and is moved onto a production system.  Users are notified that it is ready and where they can find it.

Now here is where the Lean Methodology MVP thing comes into play.  Despite the programmers' best efforts, they have only been able to get their creation to add up the first two numbers listed in the input file.  It simply ignores the rest of the rest of the file.  You might ask "Why did they release it?" and that would be a great question.  They released it because the project's requirements stated that it needed to add up some numbers and that requirement was met since it actually did successfully add the first two numbers.  They had an MVP.  

After a little feedback from the users they came up with a workaround for this obvious deficiency and this was it:

The user was to populate all 42 lines of the input file with either wattages of each device or a zero if the slot was empty.  When the program ran it would add the first two numbers and produce the output file.  The user was to take the result from the output file and copy it to the first line of the input file.  Then they would take the third line of the input file and move it to the second line and run the program again.  They would repeat this process for the fourth line, fifth line, and so forth until they had completed all 42 lines.  A future release of the program would address this issue if funding was approved for more development work, but for now it achieved it's intended purpose using a combination of less than clever coding and a lot of user intervention.  There you have it.  An MVP.

Sounds ridiculous, doesn't it?  Well that's because it is.  Software developed today using Lean often lacks the key features that would make it valuable to the customer.  The thought is to get customers to pay for it by selling them a rough draft of what the product will eventually become.  It's great for startups at first because it starts bringing income into the company sooner.  Using traditional development methodologies a company might have to lay out a year's worth of staff salaries before they ever see a single cent from selling a solid finished product.  By the time the customer realizes that they've been duped the software company is working hard to add the missing features or fix the stuff that doesn't work.  If they can get the customer to stick with them through that process then they will retain paying (but now not-so-trusting) customers and keep generating income.

I got to see this whole process in play when I worked briefly for a restaurant point of sale company.  They had a solid product but it was dated and didn't have the features of newer point of sale systems, so they decided to create a whole new one that kept customer data in the cloud and had a much friendlier color touch screen interface.  Some of the senior engineers figured out what it should look like and produced a mostly non-functional mock-up that the executives could show to our investors in order to get them to fund development of the new system.  The investors liked it and poured money into the company like crazy.

Next came the Lean part.  Some of the senior engineers decided that the mock-up, which was for show only and never intended to actually function, was a good enough starting place and they could get the new POS to market much sooner if they used it as the foundation and just added some features to it.  It was developed for presentation purposes only and consisted of random, half-functional chunks of javascript code cobbled together to run inside the user's Google Chrome browser.  They decided that it would be good enough to use as a foundation for building a new product.

As you can imagine, the developers struggled with this for months.  They discovered the limits of javascript running in a browser and found ways to work around most of them by adding yet another freeware code module or simply putting various point of sale features in their "parking lot" to be considered at some future date.  The final result was a house of cards built on a very shaky foundation.

Those of you who have ever developed anything in javascript probably know that javascript programs in a browser are usually run briefly just once or twice before the user moves along to something else, so efficient memory management / reclamation and variable scoping are not strengths of javascript.  You might also know that restaurant point of sale systems sometimes run for days, weeks, and even months before they are restarted.  You probably see where I'm going with this, but just in case you don't, I'll tell you that javascript programs can become less and less stable the longer you run them.  Javascript is not a good choice as the environment for a restaurant point of sale system.

With this in mind, our programmers forged ahead and developed the new POS in javascript anyway so they could get it to market sooner.  They released the new version in a few months and the reviews from customers were less than stellar.  If you've ever worked in the dining room of a restaurant as a server, bartender, or manager, you know that the ability to split checks is critical to the restaurant's workflow.  Unfortunately, this capability was one of the many "parking lot" items identified early in the development of the POS and it was not available in the initial release.  To make matters worse, the Split Ticket button was displayed, but when the user pressed it they would be presented with a small pop-up message advising them that this feature was not yet implemented.  The workaround that the developers advised was to void the ticket and then ring it in as separate  tickets.  As you can imagine, this was not well-received, especially by restaurants that had already replaced an exiting POS system with this one before noticing this deficiency.

Over the next nine months we lost a lot of customers and it became apparent that developing and selling a system way before it was completely finished was a bad idea.  I was happy the day my phone rang and found my manager and a Human Resources person on the line waiting to tell me that, due to an effort to streamline operations, they had decided to abandon doing business in the territory I covered and I had two weeks to wrap up any unfinished items before my last day of work.  This wasn't unexpected and I was glad to get out gracefully before the whole house of cards finished its inevitable collapse.

So, to sum up the Lean development methodology, it basically consists of building just enough functionality into a product to start getting people to pay you for it and then, once you are generating some sales, go back and finish the development tasks that should have been done in the first place.  As you have probably guessed by now, I'm not a fan.  I think it generates an element of distrust between the customer and the vendor.  Once that trust is lost it takes an extraordinary effort from the vendor to repair it and the relationship will never be as robust and collaborative as if the vendor had never broken that trust in the first place.

Warming Up for the 2024 Elections

 I saw something quite frightening when I was driving somewhere this morning.  It was a nice, new full-sized SUV with a "Trump 2024" bumper sticker.  If you've been reading my blog for any length of time you know I'm not a Trump fan.  For some reason I have a problem with the leader of the free world brazenly telling lies every time he opens his mouth.  For some reason there are a substantial number of people who think he should be elected President again.

Trump's only accomplishment as President was to normalize being mean and dishonest.  Other than that the only thing he did was systematically dismantle all the things that made the United States the best country in the world.  It was sad to watch what transpired over those four years.  When he lost the 2020 election I breathed a sigh of relief for about a week until I realized that he wasn't going to just go away quietly and disappear.  I soon realized that he would continue to dig his hooks into the soul of the country long after he was out of office and might well come back for a second shot at it in 2024.

The thing I want to write about here is that there are still many, many people who believe that Trump should be elected to another term as President.  The perplexing thing about this is that you know they heard all the lies during his first term and have chosen to just pretend he never even spewed them out to the world.  Many times we saw stories on the evening news where they showed video of him saying something and then followed it with another video saying something that directly contradicted what he said in the first one.  When he was challenged on the obvious contradiction he told the interviewer "That never happened."  Seriously.  We just saw it and heard it.  Knowing this, he just continued to say "That never happened."

Trump supporters ignored what they saw and heard themselves and swallowed up Trump's denials as if the heavens opened up and God spoke those corrupt words directly to them.  Classic example of gaslighting -- you didn't see or hear what you saw or heard.  At first I thought that these people were a special brand of stupid, but then I began to wonder if there was something more to it.  There were even people I knew to be very intelligent and aware of what's going on in the world around them who also bought into it.  Eventually I decided that Trump must have a special type of charisma that makes otherwise rational people become blind to factual information and makes it impossible for them to weigh facts against fiction.

Anyway, what has me worried today is that people are starting to do crazy things prior to 2024.  They're hoarding food and supplies, planning escape routes to get them out of the cities where they live, buying as many guns and as much ammunition as they can find, and learning survival skills.  Recently I've started to hear people talking about impending social unrest and possibly even civil war.  I even saw a news story from a major network (not Fox) where over 50% of people surveyed believed at least one state would secede during their lifetime.  What. The. Fuck.

It just seems like the country is dividing into one part where people are embracing the changes happening all over the world and another part where people are working hard to return us to the type of life we had in the 1950's.  If the country is basically split each way along geographical boundaries then maybe it really is time to consider splitting it into two entities.  Things change over time and sometimes it's better to just embrace the change and let everyone move on to whatever they believe will be better.  I don't know the answer to this.

I could ramble on about this stuff for a long time, but my greatest fear is that 2024 will bring a resurgence of extreme Trumpism and the ruthless cruelty that comes with it.  This will undoubtedly bring friction between both sides and I think the greatest losers will be ordinary Americans.  We stand to lose a lot while we are fighting the battles of our political leaders and I'm afraid many of us won't even understand what's happening because of Donald Trump's special type of charisma.  Hopefully we'll be able to survive as a country, but if not then we need to make whatever changes are necessary and move on.