The Cost of Quality - Bugs vs. Defects
When I was a QA Manager, I was frequently asked the question, “Why do I need to pay for testing?” Quality is a cost. It costs money to test and maintain those tests. The trick for project managers is to show the value of quality. One way is to display the metrics showing how bug counts have decreased over time. Another is to determine if bugs originate from recent code deployments or from lingering problems that are now emerging.
About a decade ago, I had a challenging client who consistently argued about paying for QA. They wanted to have pristine software delivered, but didn’t see a reason to pay the costs of testing it. So, I came up with a simple solution. We separated what was classified as a bug from what was classified as a defect.
Conceptually, what I am discussing here can get confusing depending on the tools your organization is using. Often, Bugs and Defects are interchangeable, and different systems classify them differently. So, for the context of this, I am going to define them:
Bug: Any problems found after the development team has released the code. This can be as early as user acceptance testing or as late as production.
Defects: Problems identified during testing of the current development work. Anything that hasn’t been released to the public.
Why would we do this?
This allowed us to add a new metric to provide to the customer. What did we catch before we handed it off to you?
Since quality assurance is an investment, it is also essential to demonstrate its value. The lowering of bug counts is a valuable metric, but it isn’t just testing that can bring that metric down. Better processes, team coordination, or developer experience can also bring it down. This means it doesn’t necessarily show the specific tester value. Recent code deployment is a similar situation.
The ability to show a client that what they received is superior to what they would have received without testing is a tried and true method of demonstrating the actual value of your testing processes and teams.
Incidentally, I have utilized this concept with JIRA and Azure DevOps, creating defects as subtasks while leaving the bugs as standard issues. We had a rule on the development teams that if a defect was found for a work item (user story, task, etc.) then it couldn‘t go forward until the defect was fixed.
Either way, the client I mentioned stopped asking why they had to pay for testing once we were able to show them precisely what we had caught before releasing it to them.
