Valuable Insights: Best Practices for Enhancing QA at Tencent: Exploration and Construction of Intelligent Automated Testing

As people's demands for quality of life continue to rise, the frequency of product updates in the market also increases, which poses a significant challenge for product developers. The purpose of exploring automated testing is to help businesses capture the market with better quality and faster speed.

Compared to manual testing, automated testing offers higher test coverage and greater efficiency when testing TOP 300 and TOP 500 models. At the same time, automated testing can be integrated with continuous integration tools, enabling quick response versions, achieving a regression test with just one version, or even conducting regression tests with a single code submission. Moreover, automated testing can also integrate multiple specialized tests.

In recent years, the demand for automated testing within Tencent has continued to rise. From data between 2019 and 2020, the number of automated test cases for Tencent's major projects increased by 50%, and the frequency of automated testing rose by 200%. This blog will analyze in detail how Tencent has explored and built automated testing internally from various perspectives.

Exploration of Intelligent Automated Testing

Tencent has a large number of products, covering a wide range of types, with differences in access methods, testing scenarios, and testing needs. An excellent automated testing platform needs to help businesses solve the following problems:

• Efficiency: Business access is convenient and fast, supporting various access methods
• Availability: Test task execution stability> 99.9%+
• Effectiveness: Problem analysis and location are easy, and regression is simple

So, how does WeTest, as an all-in-one testing platform recognized by many of Tencent's well-known products, build the technical solution for intelligent automated testing?

1. Full-Link Testing Capabilities

The WeTest automated testing platform provides a series of automated testing capabilities from script writing to task triggering, task execution, task reporting, and finally problem debugging. It also offers access methods at different layers, including IAAS, PAAS, and SAAS.

At the IAAS layer, WeTest provides a series of hardware facilities, including signal shielding cabinets, constant temperature laboratories, and domestic and international dedicated networks.

At the PAAS layer, it offers different access methods.

At the SAAS layer, it provides full-link automated testing capabilities.

Businesses can access these underlying hardware and equipment through the IAAS layer, connect to the entire testing platform through the PAAS layer, or directly use the automated testing capabilities through the SAAS layer.

2. Multiple Access Mechanisms to Meet Various System Integration

To meet different business access requirements, WeTest provides various access methods. For example, the CI plugin is a component of CI automated testing, which can perform smoke or regression testing on version builds on a daily schedule. For businesses with testing platforms, WeTest offers platform-level integration via REST API. For different business needs, WeTest also offers automated testing that can be submitted with one click on a web page, and a command-line tool that allows automated testing to be submitted by executing a simple command.

3. Image-Based Testing Framework Deployment

When conducting automated testing, different business teams have varying automated testing technologies, with different script frameworks, script environments, and terminal device requirements. Some user scripts may even have security vulnerabilities.

To address these issues, WeTest provides an image-based deployment solution. When a business team deploys a testing framework, they only need to generate the corresponding image for use. The required scripts or testing framework libraries can be packaged into an image using a DockerFile, then uploaded to the platform's image repository. When a user submits automated testing, the platform will pull the required image from the user's image repository and run it in the entire K8S cluster.

Using a K8S cluster also automatically performs load balancing, ensuring that the task execution containers are evenly distributed across different nodes, which is also very convenient for scaling.

4. Configurable Test Scheduling

Devices, test cases, and scripts are all configurable resources. For example, when a user needs to schedule test cases, they simply need to configure the required execution cases and the device conditions needed to execute them. The platform will optimize the allocation of resources based on historical execution times, required conditions, estimated results, and execution methods, thus speeding up the test case execution. For device scheduling, the platform uses an algorithm to select devices that are both available and meet the user's testing requirements, improving testing efficiency.

5. Centralized Hardware Data Center, Networked Mobile Control, Clustered Execution Machines

Currently, most automated testing platforms in the industry use consumer-grade products to build their hardware infrastructure, which leads to the availability of mobile devices being dependent on the availability of all hardware in the entire link. When one piece of hardware fails, the mobile device cannot be used. This creates a bottleneck at the single failure point and makes the overall control link excessively long.

6. Comprehensive Test Reports

To allow businesses to quickly and conveniently discover issues, locate problems, and perform regression testing after completing tests, WeTest provides comprehensive test reports. These include device logs, performance data, network data, screenshot data, error stack information, and a playback of the entire test process. For issues found on individual mobile devices in the report, the platform also provides remote debugging functions, supporting high-definition image and audio input/output.

7. Private Cloud

For businesses with strong confidentiality requirements or special needs for device timeliness, the platform can provide a dedicated cloud that only [that] business can use. This not only meets confidentiality requirements and zero-wait device timeliness but also allows for special configuration and deployment of devices to meet specific application pre-installation, non-uninstall, persistent login status, automatic chatting, automatic review, and other requirements.

• Intelligent Automated Testing

To better guarantee product quality for users and provide a better platform experience, we have been continuously researching, exploring, and thinking about automated testing technologies, constantly striving for technological progress.

In the field of low-code tools and AI exploration, we developed a next-generation low-code automation tool for script recording—WeAutomator. This low-code tool includes various script recording functions, such as basic recording through image, spatial, and coordinate-based methods, as well as intelligent recording based on image edge algorithms. Additionally, the tool supports both scene and code modes, and is compatible with both Android and iOS.

• Case Sharing

Honor of Kings Skin Automated Testing

On the left side of the screen is the user's real game screen, while on the right side is a dedicated test map provided for automated testing of Honor of Kings. The time has been changed to a logical title, the noise in the map has been removed, the background is unified as blue, and unnecessary elements such as enemy heroes, health points, and hero names are hidden.

During the entire automated testing process, the system records the gameplay. After recording, every frame of each video is extracted using OCR recognition, and all image frames are saved. Different iterations of images are compared using ORB feature matching and precise image comparison from the image comparison library, filtering out images with differences and marking them to identify UI anomalies. The bug detection ability of this automated testing process is five times more effective than manual testing. Additionally, because Honor of Kings has many component resources, an automated test saves approximately 160 hours compared to manual testing.

WeChat Client Automated Testing Case

WeChat-related end-to-end UI automated testing mainly involves: basic WeChat client server, payment UAT, mini-program basics, search, and other business tests. Typically, UI automated testing involves four steps: selecting UI driver tools, selecting test case organization tools, writing test cases, and deploying test cases. Based on their business needs, the WeChat client project team chose 3 key points to intervene in automated testing during test case deployment.

1. Merge Check

Every time developers submit a MR, the compilation and startup issues are checked. This stage mainly verifies if WeChat can launch and send messages successfully using three test cases and login.

2. Red Teaming

This requires stable and usable basic functions, allowing for new features or some bugs in specific scenarios. After compiling the package, a selection of test cases (about 40 test cases) is run to ensure basic functions work and identify any significant user experience issues.

3. P0 Testing

Tested requirements are merged into the release branch, and after merging, the packaging is triggered. After packaging, P0 automated testing (about 400 test cases) is executed, which ensures the tests are completed within 26 minutes.

After WeChat integrates with WeTest's automated testing, it can distribute test cases, store test case results, and generate test reports, allowing the WeChat team to focus more on testing the business itself. WeTest provides highly stable devices, automated device operations and monitoring, with a testing stability of 99.68% and a 40% improvement in testing efficiency, making it easier for users to ensure product quality without worrying about device issues.

• Exploration Results

1. Device Dimension: High device stability, with a device success rate and task execution rate of 99.9% under an average monthly testing volume of over 1 million device instances.

2. Test Case Dimension: High test case coverage, dynamic distribution of test cases, shortened testing duration, and support for flexible test case distribution models.

3. Efficiency Dimension: More resources, faster speed, with a 40% improvement in testing efficiency.

4. WeTest now covers 6 major BGs at Tencent, with 23 key businesses.

Give it a try for free today！ Registration - WeTest-All Test in WeTest

For further information, we are glad to have a meeting with you: Calendly