Reusable promo code, tricky bug and hell of a release: three stories about why you need to listen to your QA
Hi! I am the CEO of an outsourcing testing and QA agency «KVCHK». We work with a great number of various products, specializing in quality assurance for complex IT projects, e.g. airlines software. In this field, as well as in any other business, poor quality or even a single bug can cause huge financial losses and reputational crises.
However, airlines are huge and complex businesses, so market reputation and trust of the customer are extremely important for them. That is why even a small mistake in this field quickly gets in the spotlight.
Over the years of working with IT products for airlines, my team and I have gone through numerous complex cases, so I can say for sure that there are several important rules that insure quality:
1. The production team should trust their QA and listen to them.
2. All the communication and processes must be clear and transparent.
To support these statements with straight examples, I am going to tell you three cautionary stories from personal working experience.
1. Users are going to find everything
Once, our team had the task of testing the new logic of the promo code on the website of an airline. Such promo codes are often used on websites as a present for customers` loyalty. When the testers checked the main functionality, they found minor errors, about which they warned the developing team. However, they were ignored with something like: «This is just a testing environment, in the production everything will be fine». As a result, the testers found a very interesting bug on the release.
When booking a ticket, a client is likely to add some options. At the last step of the prepayment they enter a promo code, get a discount, and pay — everything has to be fine. But the testers, just like a cunning (and a little forgetful) client, reached the payment gateway, where the payment amount is already set considering the promo code.
At this stage they suddenly remember that they put an incorrect date of birth of a passenger. They come back, change the date and then get to the stage of payment again. And of course, they remove the old promo code in order to add a new one. Guess what they see? They see that the discount adds on.
And then both the client and the tester will feel active interest: «How many times can this work out?». The testers returned to the previous step and used the promo code again. The discount added on. Finally, it got to the point when the discount reached 100%. And that was a problem because a discount of even 50% might go unnoticed by the company.
Despite the suggestions of the testing team to fix this bug, the developers decided that this is an unrealistic case and no one will ever find it — it is way too difficult. But of course, as some time passed, users found this bug, and the developing team had to fix it in a rush.
This situation could have been prevented at the very beginning: fixing the bug was a matter of 30 minutes.
The point:
It is important to listen to your testing team because they look at the product from the user’s perspective. And if they are sure that even a minor bug can be discovered and eventually lead to business losses, it has to be fixed.
2. Listen to every person
The QA team was testing new integrations on the server. During the check, the team Lead found a bug due to which the fonts did not always adapt. The testers re-checked everything, but found no errors: everything was fine. The Lead suggested that the bug was unusual and most likely the reason was that the updates did not affect certain clusters. So the team decided to continue checking: after an update the fonts changed, but after another one it might not happen.
When the team ran some tests without VPN, the bug showed up, but it disappeared again after one more update. Testers used all kinds of browsers and even got to the point that they began testing different versions of Windows. It turned out that Lead`s guess was right: the updates simply did not affect a few clusters. When a user entered the site, they got to one of the clusters and then went further along the flow. If the cluster they got to was invalid, everything broke, and the fonts changed. It was difficult to catch the bug because the chance that you will get exactly to the problematic cluster was rather low.
This error might seem insignificant, but if it had not been found and fixed, it might have led to further malfunctions.
The point:
It is important to listen to all the members of the team, as well as to check and test hypotheses, even if at first they seem to be unimportant. Software quality is an area of constant doubt and critical thinking. And even a single doubt of any team member is a reason to check the product again.
3. Mistrust causes business losses
The testing team worked with a client who did not give access to the site administration system, even in the production. This, of course, increased the difficulty of work. The release was carried out, and for some time everything was fine. Soon after, the server crashed, followed by the production. The developers made a rollback and uploaded a new version. The production crashed again. All night long the team spent checking clusters and servers, looking at the operation of the site and testing it after each fix.
In the morning the developers uploaded a new version, and the site started working. Instead of celebrating the success, everyone went to bed, because it took about 18 hours to fix the problem. However, in the morning the server crashed again, so the team had to get back to work urgently.
The analysis showed that the server was crashing because of some changes. It turned out that the release had functionality for adding graphic elements to the site. At the time when the team was making the release, the manager was adding those elements, which led to an error related to the functionality. This error kept killing the entire system. And that is not all: later, in the morning, when the developers uploaded a new version, everything started working, but it happened simply because the manager went to bed.
At 10 a.m. when he returned and continued his work, the team lost the production again.
The point:
Trust your QA team. Their task is to help you release a user-friendly product of highest quality, not steal your data. If the customer had provided the testing team with the necessary access, they would have run the check and found the bug during the development process. Every minute of the downtime of the airline’s website is a big loss.
