Let Crashes "Speak": How AI Identifies Root Cause Signals From "Noise"

WeTest News 2025-12-23 15:44 214

As AI technology continues to evolve, game crash governance is shifting from passive response to proactive prevention. Tencent's CrashSight platform transforms game crash management by using intelligent clustering and a hybrid reasoning framework.

Introduction：

At the inflection point where the game industry is transitioning to long-term operations, we are witnessing unprecedented challenges: the exponential growth of open-world map sizes, the soaring complexity of real-time physics simulations, and the increasingly stringent requirements for cross-platform synchronization. In the process of game development and operation, crash issues have always been a persistent headache. Take a game with 500,000 Daily Active Users (DAU) as an example—a mere 1% crash rate would affect 5,000 devices every day, resulting in over 10,000 crash incidents. Faced with massive volumes of crash stack data, development teams often fall into an "Blind men describing an elephant" dilemma in their analysis.

01 The Pain Points of Crash Management: Why Do We Need AI?

Traditional crash management approaches face two significant challenges:

1 Crash Clustering Issues

Traditional stack classification rules are highly dependent on manual work, with outdated technical methods and poor business outcomes. Static stack hash grouping and fixed rules struggle to cope with complex and variable risk behaviors, leading to persistently high rates of false positives and false negatives.

(Data classification is fragmented, with excessive noise causing clustering failures)

2 Difficulties in Root Cause Analysis

On average, it takes troubleshooting across more than 10 modules to locate the root cause. 70% of repair solutions rely on the empirical judgments of senior engineers. The lengthy process of problem troubleshooting and verification results in hidden business losses.

(Traditional crash analysis workflow)

02 Technological Breakthroughs: Three Innovative AI-Driven Solutions

1 Intelligent Clustering and Deduplication: Extracting Valid Signals from "Noise."

Tencent's CrashSight platform leverages AI technology to achieve intelligent aggregation and deduplication of crash issues, effectively addressing the limitations of traditional classification rules. In practical applications, this technology has demonstrated remarkable results—the number of erroneous classifications has been reduced by up to 70.08% after optimization, significantly improving classification accuracy.

The core value of this intelligent clustering technology lies in its ability to automatically identify and filter stack information containing variables such as dynamic names and Universally Unique Identifiers (UUIDs), avoiding misclassification caused by fixed rules in traditional methods. Through machine learning algorithms, the system can understand the semantic meaning of stacks, rather than just performing superficial pattern matching.

(Abnormal stack similarity determination process diagram)

2 Reducing Manual Dependency: AI-Powered Root Cause Diagnostics

By automating pipelines to aggregate fragmented information, CrashSight utilizes knowledge graphs and intelligent algorithms to simulate the debugging logic of senior engineers. It tightly integrates code repositories with runtime data, ultimately achieving a qualitative leap from "observing the symptom" to "locating the root cause and suggesting fixes." This represents a deep practice of both DevOps and AIOps philosophies.

We have designed two components for root cause analysis: online analysis and offline analysis. The online component is based on a multi-step reasoning framework powered by Retrieval-Augmented Generation (RAG), which can analyze crash stacks layer by layer, track error propagation paths, and ultimately pinpoint the root cause. The offline component supports local deployment of code snippet acquisition modules, enabling fast matching and retrieval of issue-related code for analysis without uploading full code, thus maximizing the security of business code and core assets.

3 Limiting AI Hallucinations: Application of Hybrid Reasoning Framework

Due to scattered and incomplete log information, a large number of duplicate logs obscure key details. When large models process long contexts, they consume substantial tokens, which is prone to inducing AI hallucinations and compromising analysis accuracy.

CrashSight adopts an innovative hybrid reasoning framework that combines Tree of Thought with the Long-text Abstractive Selection (LATS) method, building a truly intelligent multi-step reasoning engine. The workflow of this engine starts with the generation and exploration of multi-branch hypotheses—the system simultaneously considers multiple potential crash causes instead of confining itself to a single direction. After hypothesis generation, the engine initiates an evidence-driven dynamic evaluation mechanism, verifying the rationality of each hypothesis using real-time collected crash data, log information and other evidence.

Next comes the recursive in-depth analysis phase, where the system conducts layered analysis on hypotheses that pass the preliminary evaluation, tracing back from surface phenomena to the fundamental root cause. This process simulates the troubleshooting logic of senior engineers, with greatly improved speed and precision. Finally, the system generates structured output results, ensuring that each conclusion is supported by a complete evidence chain and features strong interpretability.

It can not only locate the exact files, functions and crash points, but also provide directly usable patch code. Additionally, it delivers a prioritized list of repair solutions categorized into emergency fixes, root cause repairs and long-term reinforcement, truly realizing an intelligent leap from "phenomenon" to "truth".

03 Transforming from "Firefighters" to "Prophets"

We systematically promote the integration of AI and tools through a three-phase model:

In the tool enhancement phase, we automate single-point tasks to achieve "point-level" applications;
In the process optimization phase, we embed AI into the complete business workflow to achieve "line-level" applications;
Ultimately, in the strategic empowerment phase, we transform AI from "supporting business" to "driving business", realizing "full-scale" applications.

This evolution has enabled AI to become a "prophet" and "sentinel" in the quality defense line. With its intelligent core engine providing prediction, diagnosis and attribution capabilities, it seamlessly integrates with existing development toolchains, greatly enhancing developers' work efficiency and job satisfaction.

Conclusion：

In today's era of rapid AI development, following the trend may lead to detours, but ignoring it will surely result in missed opportunities. Mistakes can be corrected, but lost opportunities are gone forever! Game developers and technical teams should maintain an open mindset and actively embrace the transformative opportunities brought by AI technology.

As AI technology continues to evolve, game crash governance is shifting from passive response to proactive prevention. Tencent's CrashSight platform will continue to deepen its AI capabilities, providing game developers with more intelligent and efficient crash governance solutions.