The Dreaded Patch Upgrade

So that time has finally arrived. That version of software that you have been running so happily for the last five months since upgrading has a bug in it which has just shown up. It’s critical for year end processing (30 June) and it’s now the end of May.

The vendor, ever responsive in these situations, jumps to it and a week or two later announces they have a patch for you. Just as you are sighing with relief and thanking God for saving your career, you hear the off the cuff remark: but of course you will need to take all the other patches since your last upgrade!

Five months worth of patches averaging 10-20 a week to be implemented in live with less than a month to go. It normally takes you about a month to organise the business for an acceptance test and then another month to execute it, fix any new issues and get agreement to go live. Suddenly updating your LinkedIn profile seems incredibly important.

Out of desperation you call up your counterpart at a competitor seeking help to pressure the vendor into releasing a custom patch to match your old programs. His answer: not an issue. He has taken the patch yesterday, tested last night, identified two regression issues which should be fixed by Monday, retest Tuesday, live on Wednesday. Is this just reckless abandon or does he have a secret? The answer is Automated Regression Resting.

It seems so simple. The main reason a patch upgrade causes fear is the amount of time it takes to test them before going live and nobody wants to just put things in live. This elapsed time is smashed when using automated regression tests. The impact is apparent when you look at an example whereby a major international funds management group managed to cut the execution of their test scripts down from 6 weeks to 3 hours by introducing these tools.

What I can’t understand is why so few organisations put the effort into setting these things up. Over 20 years consulting to many different firms around the globe I have only seen it used in about 2-3 companies. Yet every company I have ever discussed it with recognises the value add but there is always a reason not to do it:

• “We don’t see ourselves upgrading in the near future!”
• “The maintenance challenges are too extensive!”
• “We don’t have the budget this year!”
  

In the example given above the firm had already documented its tests. It then took one person 3 months to automate them. Payback was achieved within the first 3 months.

So what is needed to become the hero of your business and be in a position whereby the mere mention of the wrd upgrade does not have you cringing under your desk?

First you must have some test scripts. Hopefully the last time you did a manual upgrade you documented with your business the tests that were necessary to satisfy them that the system was stable. These should be your base as the business has previously agreed they were acceptable. If you don’t have this then you need to set a team aside to document what tests are required.

Next you need to setup a database that is typical of your business. Real data is important rather than test data as it has a much better chance of identifying the unexpected changes that will really affect your business. This database is critical to the success of your testing so make sure you take some time to get good coverage.

Now automate the scripts using whichever tool you have chosen. This could be as simple as keystroke and mouse capture or a more complex scripting mechanism.

You can now create your baseline. Execute the tests across your database and store them away. It does not matter if the results are correct or not. The key for regression testing is to identify change. Change maybe wrong or it could be for the better. Once you have identified change then you work out which is correct and which wrong. Then you can update your base results set.

Your final piece to consider is maintenance. There is an ongoing cost to having an automated regression suite. As many changes that come through are deliberate you have to go through each of them and alter your scripts to cater for them. This can be a non trivial event but will still normally be offset by the gains from automation. To give you an indication of size, one vendor of asset management software took the one man month worth of effort to update between versions. Their test scripts at the time took 8 hours to run.

To be successful at automation consider the following:

• Use a professional test consultant to define your scripts. Experience at this point will make maintenance much easier.
• Check what automated checking tools your testing software has.
• Check what features your testing tool has to minimise maintenance. For example if one login screen is used for all tests can it be maintained in one place and not all test scripts.

In summary automated regression testing pays for itself in a very short time. On top of this though is the fact that it is one of the few ways to protect your business from the dangers of urgent patch upgrades. As such it would be hard to understand why those companies that don’t have it are not being negligent towards their business’ interests.

I would be interested to hear other people’s experiences with automated testing to see what benefits they have achieved and what pitfalls they have run into.