Why so Expensive: How Much Is the Quality of an IT Product

Hi! I am the CEO and founder of an outsource testing agency — KVCHK.

During testing, we often encounter such bewilderment:«After all, this is testing, not development — so, why is it so expensive?». To understand what makes up a price, you need to do some research: What is the average price in the market right now? What are the criteria affecting the price and how do they apply to your product? What kind of specialists will conduct testing? And what are their skills? And so on. Yes, you really have a lot of information to explore.

Therefore, I have decided to analyze as clearly and honestly as possible how much the quality of an IT product can cost; what it depends on; how the price is formed within testing companies; and finally, how business owners can choose a testing company so that they could receive high quality without being financially hurt.

What Makes Up the Cost of Quality
for an IT Product under Our Project?

If you think it comes from our desire to make money on you, it is not true
(well, almost true). It comes from quite real metrics:

1. Project Context:

● Scale, scope, and workload. The larger the product scale, the wider its user audience, and the more sophisticated its content, the more time / resources it takes to develop it, and therefore, to test and support it.

● Area / domain. Developing and testing a simple website or a medical IT product require quite different amounts of time and resources. The latter product has more complex features, so it usually requires several types of testing, more time, and a good level of specialists’ expertise.

● Development quality. The quality of the code, its validity, and scalability — all affect the functional stability and, therefore, the amount of resources required for quality assurance and support.

● Availability of test documentation. Figuratively speaking, test documentation is a springboard for testing. If it is already available, this will reduce the time for immersion and work, while its absence will increase the timeline and budget, because we will have to spend time developing it.

2. Project Figures and Inputs:

● How many people use this service?

● How many people complain about this service?

● How many problems are we facing after the release?

● Is there an increase / decrease in business / money after the release?

● What are the deadlines and can we finish it by the release date?

● How satisfied are the top businesses with the product the IT department delivers? (Although this cannot be measured by a number, this indicator is also important.)

3. Dynamics

● How dynamic is the product: Update and release frequency.

What is important to understand here is that the quality cannot be fixed at one level. Quality, speed, user satisfaction from using the service — all this should grow. So, if a business has a growth ambition, then its product quality dynamics should be constant and positive.

The business generates ideas — the task of the development and QA departments is to implement them in the best possible way and deliver them to users on time. Therefore, the price of quality may vary depending on the complexity, tasks, and deadlines.

Does Good Tester Mean an Expensive Tester?

Does the price directly depend on the testers’ skills?

Let’s dive straight into it. Logically, the wages of specialists with good experience are much higher than that of beginners. Accordingly, we can assume that the higher the price set by QA agencies / companies, the more experienced the testers they have.

Because:

● Good specialists have bolder demands and higher wages;

● Skills and competencies, even with advanced specialists, need a constant upgrade. But they should come from somewhere — the company pays for this upgrade (at best), and the company also cannot create money out of thin air;

● Don’t forget to add the company’s indirect costs and margins.

However, there is no guarantee that you are being sold what you have been promised. It also happens that good specialists work with low wages. And there may be bad specialists with high wages. (The well-known truths!) Besides, specialists may not earn so much — the company may just have a big markup. Therefore, there is no clear dependence there.

However, the «trust but check it» rule can do you a great job! You can check the comments, and the good old jungle telegraph also works well. Take the trouble to interview the testers who will be working on your project to make sure that you are buying what you have been promised.

How is a Project Evaluated?

A testing company sells the most valuable resource they have: TIME. This means that the work is assessed in hours, which are calculated according to some criteria based on the project context and inputs:

● Number of browsers / devices to check;

● Complexity of features / makeup;

● Documents;

● Deadlines;

● Number of people involved in the project, etc.

As a rule, we ask our customers to send us some documents for the app, the terms of reference / layouts, the list of tasks, and even a link to the app — to assess the complexity of the upcoming work. Next, based on our experience in testing previous projects (often similar ones), we estimate an approximate range of time we need to test it in one environment. Then, we use the above criteria to estimate how much time we need in total.

A life hack to save your time: if you are reading this article and want to ask us how much it will cost to test your product, please send us the data immediately according to the list:

Here are some simple examples of what it might look like (based on average price):

A Minor Project:

Inputs:

● Let us assume you have an online store selling goods for cats;

● You have a product catalog, the customer’s personal account, and a promo page. All this is managed via a CRM system (no matter which one).

Task:

● Your developer has added a promo campaign offering a 15% discount with the promo code «cat»; it only applies to wet food.

Deadlines:

● The promotion starts tomorrow.

Total:

● An average price for such services per tester hour is $24;

● You need 4 hours to check, but you only have 2 hours until the end of the day;

● Therefore, you need 2 persons (we will share browsers / devices between them).

The estimated price is $96

A Major Project:

Inputs:

● Let us assume you have a web portal for your company where you want to find out the employees’ birthdays, to have your director’s business card so he/she could set it up nicely themselves, and also to have an internal library, and many other things.

Task:

● All this needs to be checked, and if something is changed, it must be checked automatically that the old items had not failed.

Deadlines:

● You are not in a hurry to launch it; you have time by the summer (not known which year).

Total:

● An average price for such services per tester hour is $24, per autotester — $30, the price per lead tester (who is to manage the work of the QA team so that no one is idle and does not waste money) is $45. But we provide the lead service free of charge, so we will not include its cost in our further calculations;

● You need 100 hours only for documents, another 500 hours for manual testing, and 500 hours for automation. And don’t forget the lead; let’s say that his/her engagement is estimated at 100 hours per project;

● Therefore, you need as many as 3 people for at least six months.
This is a full-time long-term job, which means you can request a discount ;).

The estimated price is $29 400 (don’t forget to request a discount!).

A difficult question is how much should you allocate for testing from the total budget; there is no universal formula for this. A software product is an organism with its individual type, with different stages of dynamics and development.

However, there are conventional estimates.

● A new project from scratch: 30%;

● A project with well-defined tests and a stable system with minor modifications: 15 to 20%;

● A critical system with low fault tolerance: 40 to 50%;

● An internal system with high fault tolerance: 5 to 10%.

Using this scheme, the budget is determined depending on the type of system supported.

How to Choose Proper Testers without Any Losing

How to know you are not overpaying:

● Cost

You just need to open the list of QA vacancies and look at the wages. Most likely, the agency will be cheaper. They already have materials and equipment that are spread out over the years, so you won’t need to:

● Spend money (and, last but not least, time) to find an employee;

● Spend money on employee indirect costs (workplace, development, leaves, etc.);

● Spend money on taxes, accounting, or manager’s work.

Again, there is no guarantee that the agency will be cheaper. Maybe, in your case, it would be more cost-effective to hire full-time testers.

So, there is only one advice: evaluate the scale of your project, do your market research, explore the testers’ wages and the price list of testing companies, apply your indirect costs to this, and figure out what is more profitable in terms of cost. It is important to understand that you may not win in quality, because QA agencies / companies have higher expertise, are more experienced, and have stronger skills.

● Quality

The quality is subject not only to the testers’ skills and professionalism but also to the services they use in their work: whether they have their own farm; whether they test using real mobile devices and real browsers; whether these farms work well — how good is the emulation of user’s actions, etc. Be sure to include KPIs in the contract and agree «on the shore» upon how you will apply them; for example, use the number and severity of detected / missing issues after testing completion.

● Interest in the Project

This seems to be the most common problem for outsourcing companies; that is, people act like robots: they perceive the project as a routine task that just needs to be done, without plunging headlong into the essence and objectives of your business. Or, for instance, they work only within their responsibility area, and if someone in the product team screws up, «Well… it’s not my problem». Although they are bound together: if you see a problem, report it!
Probably, the format «Do your job and don’t interfere» is cool in other areas but not in QA. The product quality is a minefield and a source of constant doubt: «But will it be user friendly?», «Will the user really like this feature?» etc. Therefore, for your user not to walk in this minefield, you should constantly look at what can be improved; you should constantly doubt and think about where and what can be «cleared», and what is equally important, you should ask yourself: «How will this help the business earn more?».

If your testers (or testing company) don’t ask questions; if they never annoyed the developers; if they didn’t offer to improve anything and didn’t even argue, you should think about their real involvement in the project. For example, we are a 100% outsource company, and we have many projects; but at the same time, we cultivate the attitude to your project within the company as if it were our own project. And it is a must-have for our colleagues to work for the company.

● Additional Bonuses and Customer Focus

This is rather an indirect, still important point. How much does the company strive to be involved in working with you? How quickly are they ready to fix a problem if they accidentally screwed up? Can they quickly adapt to project changes? Do they upgrade their employees’ skills to improve the quality of their work? What else are they willing to offer you as an «extra» to their services?

For example, we have such a courtesy: we provide QA lead support for each project free of charge. The QA lead’s time is paid out of our own pocket. We are idiots? No, it’s just important for us that our customer (even if there is only one tester in a project) could have full-fledged QA support and quality assurance. If an agency has such goodies, this is another plus for their karma and a sign that the guys are trying to do their best and are committed to long-term relations.

● Integration into Your Team

And of course, it is important that the testers / the QA team could quickly join your work process and establish clear and understandable interaction with your team. You can google a bunch of memes about how developers do not get along with testers. It’s a pretty thin line. Arguing in the context of «suggesting improvements based on objective data» is quite normal if it ultimately has a positive effect on the product. Arguing in the context of «squabbling» without particular grounds, crossing the lines of what is permitted, is lousy and unprofessional. By the way, an experienced QA lead can help here.

Some Important Tips:

Try to work with well-proven testers and be careful with newbies. Use the jungle telegraph to find a QA team; ask your friends for contacts or explore their customers / find customers’ contacts yourself to ask them about their experience with testers.

● Like in any other job, you need to control them, but don’t pester them with micromanagement. After all, you hired professionals, and if you cannot help but pester, figure out what is the reason for this: it shouldn’t be like that;

● If you don’t like something, speak up right away so they can fix it. If they don’t correct and don’t listen, run away from them like from a rabid dog;

● Never work without a contract where you must specify the acceptance criteria for the work performed.