Testing software is an important responsibility, but testing is not a synonym for quality. At Eventbrite, we are trying to dig deeper into quality and what it means to be a QA Engineer. This article is not just for QA engineers, it is for anyone who wants to better understand how to deliver higher quality products and better utilize QA resources. If you don’t have QA resources, by the end of this article you will have a better idea of what to ask for when you look to add a QA Engineer to your team.

Rethinking the role

When I sat down to write an updated job description for our QA Engineering position, I started my research by looking at job listings from similar companies. Most of the listings agreed on one thing: QA Engineers test. The specifics vary, but the posting would always include a range of automated and manual testing tasks.

While these testing tasks are worth doing, testing software doesn’t ensure that  the output is a high quality product. In practice, effective QA extends well beyond testing. QA Engineers should ensure teams develop products that work and address a targeted customer need.

The iron triangle

Being a strong advocate for quality requires understanding what could cause quality to suffer. I’d like to start this post by introducing the concept of “The Iron Triangle” The triangle is a visualization sometimes used to describe the constraints of a project, but it also works as a model for the challenges of maintaining quality.

The idea here is that we constrain the quality of a project by its scope, deadline, and budget (among other factors). Changes to one of these constraints then require balancing adjustments to the others, or quality suffers.

External quality

The team can’t control all of these constraints, but it is critical that they monitor them. These constraints directly impact the quality of work. This sort of quality is external because it is quality as understood by the customer.

Some scenarios

• A project has a broad scope. The timeline for the project is likely full of feature work, with limited time left for testing tasks. Intervention can mean working to carve out time to write and perform tests, advocating for a reduction in scope, or developing a testing approach that is lean without sacrificing too much coverage.
• A project has a tight budget. This type of project is likely to have even less time to spend on quality. In these cases, my preference is to establish clear goals and expectations with stakeholders for quality in the planning step. This process enables the team to pack their limited QA time with more precise and targeted testing tasks without misrepresenting how hardened our code may be when we finish the work.
• A project has an open timeline. This is less common but has its own challenges to quality. When we give plenty of time to projects, they naturally move more slowly. In these situations, it is essential to test along the way, because the closing days of this project can be hectic. I like to limit final testing before release as much as possible with incremental tasks and plenty of automated testing. That way, I can protect the development team from last-minute changes, complexity, and most major bugs.

External quality is linked directly to the success of the business and is everyone’s responsibility. All arms of the business are responsible for maintaining external quality and delivering functional products.

Beyond bugs

I loosely consider an issue a bug any time the software produces an incorrect or unexpected result or behaves in unintended ways. Bugs are going to happen, and minimizing their occurrence is why we test software. However, external quality can only cover so far as we understand how the product will be used. You cannot write a test to cover a use case you don’t understand or know about

If something works as expected but fails to meet the user’s need, this is still an issue of quality. However, this is not a bug. The QA team should bring knowledge of the product and the user to the entire development process. If QA is involved in the planning phase, and the testing phase of development, they can help with more than just finding bugs. They can help ensure developers more thoroughly understand how users employ the products they are building.

Internal quality

That said, there is also an internal, procedural component to quality. Writing code and building products in a way that minimizes technical debt and mitigates risk maintains internal quality. Being good at managing external quality does not make an organization good at managing internal quality.

A new scenario

• The development team is wrapping up a project and is ready to execute their test plan. Through testing, they uncover some bugs and edge cases that they didn’t think of when writing requirements for the project. To fix these issues, they need to add cyclomatic complexity. This could reduce internal quality and has downstream effects on external quality too. This issue could have been curtailed by involving QA in the writing of product requirements, or by being more deliberate when considering edge cases and architecting the feature.

Balancing external and internal quality

Good external quality is not an indication of good internal quality. Since QA Engineers are driving external quality, they need to be cognizant of increased complexity as an output of testing. Testing uncovers more than bugs, it also uncovers where the product we are building may be failing to meet user needs. Addressing these gaps is critical to quality, but can have a significant impact on timeline, budget, and scope. Our compromises are likely to produce technical debt

Technical debt

Technical debt should be a conscious compromise. The development team can give up some internal quality to make the project work within other constraints. Future work to pay off that technical debt often competes for the same development time as work done to fix a bug, and both issues concern overall quality. This can be a confusing number of plates to keep spinning at once. We should neglect neither type of quality work for the other, and understanding their relation to one another is crucial to preserving high overall quality.

One final scenario

• The business asks for a feature with very narrow scope, a small budget, and a tight deadline. The feature will require new development work on an old, neglected part of the codebase. The development team is worried about losing time to cleaning up technical debt around their integration points and bringing the old code in line with new standards and practices. Testing time for the new feature work is already tight, and the business wants the development team to prioritize keeping the existing feature set healthy. The team needs to make certain compromises to meet their target release date. One of those compromises is balancing investment in internal quality against the external quality of this new feature and the old code.

Protecting quality

While it is critical to be understanding and compromise during development, QA Engineers should remain biased toward quality. The organization has managers charged with protecting budget, scope, and deadlines – but quality should have an advocate too. QA Engineers should spend time encouraging and coaching development teams on bugs and testing tasks, but the real goal should be to encourage those teams to take ownership of quality.

When the user-need and gravity of testing is well-communicated and well-understood by developers, they write higher quality code. Developers that understand their users write better tests that leverage user stories, rather than the developer’s expectation for what their code does. Beyond testing functionality, they are making sure that what they have developed aligns with how the product is addressing targeted need.

Engaged developers make the best testers

To be clear, I am advocating that developers do their testing and own their quality. Outsourcing your testing to automation engineers or manual testers is an option, but comes with drawbacks. Developers bring vital skills for driving quality into the product at speed. Engineers are also uniquely positioned to solve problems with their code, and developers that write their tests are more vested in fixing them when they fail.

The QA team can and should assist with this process. They can help developers deliver higher quality products by making sure the project is testable upfront, and making sure the approach to testing is thorough and considerate of other constraints to development. Beyond just saying that “quality should be high”, the team should set expectations for quality within the context of other constraints. These expectations serve two purposes. Foremost, they helps with estimation. If you fail to consider QA tasks during estimation, then you have not made time for quality. Secondly, it binds quality to the development process, fostering ownership within the team. Teams that take ownership of their work are more invested in delivering higher quality products.

The new job description

QA Engineers should protect overall quality. They should work with teams to find the right balance of testing for each unique project. To do this, a good QA Engineer understands quality in the context of other constraints to development and is willing to compromise, but will never allow the business to concede quality. When a business delivers low-quality products, it fails.

What strategies do your teams use to assure quality? How do you leverage your QA team beyond testing? Tell us about it in the comments and drop me a line on Twitter @aqualityhuman.