Balancing Product Quality and Test Efficiency: A Better Solution

Source: TesterHome Community

 

---

 

Introduction

In the context of agile and lean R&D, balancing product quality with test efficiency is extremely important. Many real-world situations look like this:

• Project requirements are unclear, schedules are tight, resources are limited, and the demand is “to ensure launch readiness by XX date.”
• Testers lack experience and have an insufficient understanding of the product, leading to missed test coverage points.
• Project quality and schedule risks are high, making risk control a major challenge.

The Core Question

The question of “how to find as many important bugs as possible in as little time as possible” has been thrust into the spotlight.

Based on the Golden Circle proposed by Simon Sinek, we think from the inside out, starting with “why”. Improving R&D efficiency is a crucial strategic goal in the digital age. He Mian, the author of Lean R&D, points out that the essence of efficiency improvement lies in: “continuously and smoothly delivering effective value with high quality.”

For a tester, in this context, an important responsibility is to grasp the balance between product quality and test efficiency — achieving integration of quality and efficiency. Returning to the question: how to “find as many important bugs as possible in as little time as possible” — its essence is precisely the “path to balance” between product quality and test efficiency.

Seeking a Better Optimal Solution

For daily testing work, let’s look at a simplified process to understand the activities involved:

Understand Product Requirements → Determine Test Scope → Organize Test Points (Test Analysis) → Write Test Cases (Test Design) → Execute Test Cases → Product Launch

If you are a tester, you will be very familiar with this process. This is exactly what you do in every iteration; there is nothing special about it. And indeed, that is true.

The intention of this article is not to innovatively propose a new testing model to balance product quality and test efficiency. Its focus is on seeking a better optimal solution within the daily testing model.

Regarding how to seek this better optimal solution, the key inputs include:

• James Bach’s Heuristic Test Strategy Model (HTSM)
• Xiaomei Tai’s Pirate Testing Analysis – MFQ&PPDCS

Xiaomei Tai can be considered a student and peer of James Bach. Regarding the models and frameworks constructed by these two individuals, an important foundation is Context-Driven Testing thinking, and a key principle is Risk-Based Testing (RBT) .

 

Foundation: Context-Driven Testing Thinking

Context-Driven Testing thinking emphasizes paying attention to the project’s context, recognizing that the context can change, and that test strategies and methods should be formulated based on the context, and adjusted promptly and optimized continuously as the context changes.

Context-Driven Testing thinking is one of the main agile thinking methodologies and is also the foundation of test analysis in agile mode.

James Bach’s Contribution

James Bach is a renowned representative of the Context-Driven school. For many years, he and Michael Bolton have been practicing and developing Context-Driven thinking, including his proposed methodologies such as:

• Exploratory Testing
• Heuristic Test Strategy Model (HTSM)
• Session-Based Test Management (SBTM)
• Rapid Software Testing

It can be said that James Bach’s entire theoretical system of testing is built upon the foundation of Context-Driven Testing thinking.

The Heuristic Test Strategy Model (HTSM)

The Heuristic Test Strategy Model (HTSM) is a concrete practice based on Context-Driven Testing thinking. The context-driven heuristic test strategy focuses on considering the impact of three aspects — Quality Criteria, Project Environment, and Product Elements — on testing techniques, methods, and tools.

Each aspect contains multiple factors (i.e., various contextual factors), ultimately delivering a product that meets users’ quality requirements.

Key Insight: HTSM consists of a series of guiding words that allow testers to think about the product and testing from “high-level abstraction” down to “low-level details”. It doesn’t teach you how to test; rather, it inspires you how to think, thereby uncovering the test object and strategy.

Three Aspects of HTSM

1. Project Environment

To test a product or feature well, it is natural to understand the project’s background, especially the project elements related to software testing. You need to acquire, analyze, and comprehensively understand detailed information related to these elements to better clarify:

• Test goals
• Test scope
• Test schedule
• Test resources
• Test environment

James Bach, in his Heuristic Test Strategy Model, provides analytical guidance for the Project Environment. Combined with practical implementation, the analysis focuses on:

Component	Description
Customer	Project goals and user needs
Test Item	Test scope
Schedule	Progress and milestones
Developer Relationship	Collaboration with dev team
Test Team	Available test personnel
Equipment & Tool	Test environment and tools
Quality Requirements	Test exit criteria and deliverables

 

These parts form a crucial component of a test strategy.

2. Product Elements

The product is our test object, so it naturally demands even more attention. Once a project starts, testing should get involved as early as possible to understand:

• Product requirements
• Product architecture design
• Interface design
• Usability design
• Security design

Participating in relevant reviews helps you better grasp the system under test. The Product Element in HTSM is designed to facilitate better test analysis and confirm test coverage.

3. Quality Criteria

Quality Criteria are mainly considered from three aspects: the product’s target users, specific quality requirements, and which quality standards to reference. Essentially, it answers three key questions:

1. Who will use the software?
2. What are the users’ specific quality requirements?
3. Which quality standards or industry specifications should be referenced?

Who Will Use the Software?

Consider these factors:

• Who are the users?
• What are their pain points?
• What are the user scenarios?
• What are the user personas?
• What are the priorities among different users?
• What do users care about most?
• What is the users’ actual operating environment?

What Are the Users’ Quality Requirements?

The ISO/IEC 25010 software quality model defines a quality-in-use model and a product quality model. The software product quality model includes eight quality characteristics:

Quality Characteristic	Description
Functional Suitability	Does it meet stated needs?
Efficiency	Performance and resource utilization
Compatibility	Co-existence and interoperability
Usability	Ease of use and learnability
Reliability	Maturity, fault tolerance, recoverability
Security	Confidentiality, integrity, non-repudiation
Maintainability	Modularity, reusability, analyzability
Portability	Adaptability and installability

 

Based on these quality characteristics, conduct deeper analysis in conjunction with user, business, and product characteristics to understand users’ specific quality requirements and which quality characteristics need priority attention.

Which Quality Standards or Industry Specifications Should Be Referenced?

What standards or industry specifications must the product comply with? Whether in the aerospace, automotive electronics, finance, or transportation industries, in addition to general international/national standards, there are specific industry quality standards or specifications.

Example: If the project is an application system for the securities industry, it belongs to the financial industry, and there are more than 200 related specification standards.

Example: For a product supporting Bluetooth functionality, if you want to display the Bluetooth logo, you must pass BQB (Bluetooth Qualification Body) certification. Otherwise, the Bluetooth Technology Alliance will consider it an infringement. In this case, the company needs to consider how to conduct pre-tests to ensure the product obtains certification within the expected timeframe.

 

Principle: Risk-Based Testing (RBT)

What Is Risk-Based Testing?

Risk-Based Testing (RBT) is a testing method that uses software quality risks as the starting point for testing and the main reference for testing activities. It calculates the risk degree (risk level) based on the severity and probability of occurrence of risks, and then determines test priority and test coverage according to different risk levels.

Designing tests based on product risks is a common test design approach. In the complex real world, products face various risks. Only through comprehensive consideration and thorough testing can the serious consequences of risk exposure be avoided.

Core Insight: Recalling the goal – “to find as many important bugs as possible in as little time as possible” — its essence is making trade-offs after identifying and analyzing risks, finding that balance.

The most important viewpoint of RBT is to use the software’s potential risks as the basis and goal for test planning. Only through comprehensive analysis and understanding of current potential risks can you effectively design and organize appropriate tests, find as many potential defects as possible with as few testing resources as possible, maximize avoidance of potential risks, and reduce maintenance costs.

Why Use Risk-Based Testing?

Due to the defect cluster effect (a small number of modules typically contain most of the defects), compared to specification-based testing, RBT can achieve more efficient testing through precise risk identification.

Due to the inexhaustibility of testing, situations with insufficient testing resources and time are often encountered. Given limited time and resources, you can only find a balance between limited time, resources, and quality.

Adopting RBT helps test managers:

• Optimize testing activities
• Balance testing time, cost, scope, and quality
• Rationally formulate test plans and schemes
• Make reasonable arrangements for testing resources and priorities
• Make testing more targeted
• Improve testing efficiency and product quality

Moreover, RBT no longer forces everyone to make release decisions based on insufficient strategic metrics such as defect counts or test case counts. Instead, it works with stakeholders to decide on the acceptable level of residual risk, thereby making reasonable release decisions.

How to Implement Risk-Based Testing

The main process of RBT includes:

1. Information collection
2. Risk identification
3. Risk assessment
4. Risk control

These steps should run through the entire software lifecycle, involved in test preparation, test analysis and design, test execution, and other testing activities, forming an effective closed loop.

Important Note: The most important characteristic of risk is that it is constantly changing. Never expect to design perfect test cases in one go and then consider test analysis and design activities complete. Every step of each process involves countless speculations. Good test analysis and design activities are not one-time activities; they are destined to be iterative and continuous.

 

Practice: A Testing Model for Daily Work

Implementing Context-Driven Testing thinking and Risk-Based Testing as a sound testing model is extremely important. Just as RBT is not about following steps rigidly, it needs to be implemented as a habit and a way of thinking in the team’s daily testing work.

One of the best practitioners in this regard is Xiaomei Tai. She wrote the book Pirate Testing Analysis: MFQ&PPDCS. The focus of this book is not to explain various existing test design methods one by one, but to adhere to the idea of “starting from actual problems, not from methods.”

From the perspective of practical test analysis and design, it explains the patterns and methods that can be followed in software testing. For example, when a tester takes over a new system or feature under test, how to use various testing skills to analyze the test object step by step and ultimately complete the testing task.

Core Principles of a Pirate Tester

• Proactively gather information
• Provide early feedback
• Adjust strategies in time
• Test based on risk

The Five-Step Testing Model

The main content of Tai’s approach essentially elaborates a testing model for daily testing work:

Know Your Mission (KYM) → Test Coverage Outline (TCO) → Analysis Modeling (PPDCS) → Test Design → Test Execution

 

Step 1: Know Your Mission (KYM)

According to the Golden Circle, think from the inside out: Why, How, What.

KYM is heuristic. In RBT, it is an important action for collecting information, mainly including gathering information from four aspects:

• Understanding Customers
• Understanding the Project
• Understanding the Product
• Understanding the Mission

Why Do KYM?

Core viewpoint: Promote communication between testers and people around them, obtain valuable information in a timely manner, and identify risks early.

From daily testing, common problems include:

• Testers do not understand the context of real requirements, starting test design directly with requirements and design documents.
• Testers are too far away from users and indifferent to them — why was such a requirement raised, and what ultimate goal is it meant to achieve?

From a practical perspective, product R&D has a vision that defects should be discovered as early as possible. The discovery of these issues comes from decisions made based on information from understanding users, the project, and the product. This is precisely why KYM is important.

How to Do KYM?

Core viewpoint: The essence of KYM is to collect information and understand the context by continuously asking questions.

The process of obtaining information is a continuous process of asking questions. How to ask questions and what questions to ask are the essence of KYM. The ability to ask questions is very important for testers.

Xiaomei Tai drew on the Project Environment from James Bach’s HTSM to construct a guide for asking questions — the mnemonic CIDTESTD:

Letter	Stands For
C	Customer
I	Information
D	Developer Relationship
T	Test Item
E	Equipment & Tools
S	Schedule
T	Test Team
D	Description of Mission

 

When to Apply KYM?

Core viewpoint: KYM can be applied at any stage of the project. KYM runs through the entire testing project; it is not a one-time activity.

 

Step 2: Test Coverage Outline (TCO)

Why Do TCO?

Having performed KYM, you already have some understanding of the users, mission, and test object. But your understanding is still too coarse. If you only grasp one point and go deep, you may fall into the fog of “seeing the trees but not the forest.”

Drawing a TCO mainly helps achieve:

• Information refinement
• Information restructuring

How to Do TCO?

The Test Coverage Outline can be written in two ways:

Approach	Method	Use Case
Quick analysis	Based on Product Elements (SFDIPOT) from HTSM	Select test points for rapid exploratory testing
Deep analysis	Based on MFQ (MD + FI + QC)	Comprehensive testing and verification

Note on MFQ: MFQ includes:

• MD (Model-based single-function test analysis and design)
• FI (Function Interaction test analysis and design)
• QC (non-functional Quality Characteristics test analysis and design)

[Translator's Note: SFDIPOT represents Structure, Function, Data, Interface, Platform, Operations, Time dimensions in HTSM's Product Elements.]

 

Step 3: Analysis Modeling (PPDCS)

Why Choose the PPDCS Method?

Xiaomei Tai proposed using the PPDCS method for analysis modeling in Pirate Testing Analysis.

A model is an abstract and simplified representation of the test object. The test object can be represented by a model showing its internal logical relationships. Drawing a model is part of the test analysis process.

An important question: Faced with test objects of various characteristics and a wide variety of testing methods, which modeling method is most effective?

Carefully studying the basic testing techniques, we find they can generally be divided into several categories, each with its unique characteristics. When studying the test object, we also find that for a single function tested in isolation, its internal logic exhibits certain main characteristics.

When these two characteristics align, you can use that testing technique to analyze the single function. This is the origin of the PPDCS method.

[Translator's Note: PPDCS represents Parameter, Partition, Data, Combination, State dimensions for systematic test object analysis.]

How to Do PPDCS?

When implementing the PPDCS method, proceed from the following four aspects:

Aspect	Focus
Triggers	Identify what initiates the behavior
Essentials	Grasp the core functionality
Spanning Differences	Explore variations and edge cases
Targets	Define clear testing objectives

 

Summary

The starting point of this article was to think about how to improve the testing efficiency of team members, thereby extending to the question of how to “find as many important bugs as possible in as little time as possible.”

The core is to seek a better optimal solution within the daily testing workflow. Incorporating model frameworks from industry experts such as James Bach, Xiaomei Tai, and Zhu Shaomin, this article derives an excellent testing practice model grounded in:

• Context-Driven Testing thinking (the foundation)
• Risk-Based Testing (the guiding principle)