Cross-Regional Multi-Active Project Testing: Financial Software QA Practices for Banking High Availability

1. Background | Financial Software Testing & Multi-Active Construction Demand

Financial software testing is one of the most vital segments in the global quality assurance and enterprise testing field, attracting massive professional testing engineers, QA specialists and fintech technical teams worldwide. With the rapid expansion of digital financial businesses, continuous upgrading of online banking service models, and growing demands for 7×24 uninterrupted financial services, financial institutions are raising higher standards for system stability, data security and disaster recovery capabilities.

Correspondingly, comprehensive financial software testing capabilities, multi-scenario testing skills and cross-regional multi-active testing experience have become core competencies for modern financial QA teams. Complex core banking systems require standardized testing processes, layered testing strategies and full-link risk verification to avoid fund risks, service interruptions and data inconsistency.

In this article, we invite Zhang Qing, Senior Specialist of the Online Payment Testing Team at the North R&D Testing Department of ABC R&D Center, to share practical multi-active project testing experience. The sharing is based on the real-world Super Online Banking cross-regional multi-active high-availability reconstruction project, focusing on replicable financial testing methods and multi-data-center testing best practices.

2. Project Overview | Super Online Banking Cross-Regional Multi-Active Project

The Online Payment Inter-Bank Clearing System, widely known as Super Online Banking, ranks among the eight critical and sensitive core systems of Agricultural Bank of China. As a key inter-bank financial service platform, it delivers round-the-clock 7×24 inter-bank transfer and clearing services for individual and enterprise end-users.

This core banking system features ultra-high daily transaction volume, complex accounting logic, strict financial risk control requirements and extreme system sensitivity. Any system failure, data synchronization error or switchover exception may trigger fund settlement risks and large-scale service anomalies, making rigorous multi-active testing indispensable for long-term stable operation.

Recently, ABC has fully completed phased development, iterative verification and full production launch of the Super Online Banking cross-regional multi-active high-availability construction project. After systematic architectural upgrading and multi-round testing validation, Super Online Banking has officially become ABC’s first front-line business system that realizes complete cross-regional multi-active deployment and high-availability reconstruction.

The multi-active transformation greatly strengthens cross-regional disaster recovery capability, improves peak transaction processing performance, and eliminates single data center operation risks. To support this large-scale architecture upgrade, the professional testing team sorted out existing testing assets, sorted core risk points, and launched targeted testing design for the application layer, data layer and disaster recovery scenario layer.

The team formulated exclusive solutions for functional testing, performance testing and cross-region switchover testing, and independently designed over 7,300 refined test scenarios. Full-coverage multi-active project testing effectively guaranteed stable delivery and smooth online deployment across four independent production launch windows. This article organizes the entire project testing process, core strategies and practical lessons to offer actionable guidance for financial multi-active transformation and QA teams.

3. In-Depth System Architecture Analysis & Multi-Active Project Testing Strategies

To build a scientific and high-coverage multi-active testing strategy, the testing team first conducted comprehensive dismantling and in-depth analysis of the overall Super Online Banking system architecture, business invocation links and cross-data-center interaction logic.

As a core inter-bank transfer channel for Agricultural Bank of China, Super Online Banking covers full-scenario core businesses, including cross-bank fund transfer, service signing authorization, inter-bank business inquiry, mobile number fast payment and same-customer cross-account inter-bank deduction.

In terms of system linkage, front-end banking channels and offline branch business systems serve as major upstream consumers. All service requests are unified through the enterprise integrated gateway platform for standardized traffic access and security verification. For external interaction, the system connects with the PBOC front-end machine to complete cross-bank data docking, and finally delivers transaction processing through the bank’s core accounting system. Basic modules such as customer information management and unified parameter configuration run through the entire business link, forming a highly coupled long-link financial architecture.

Based on the multi-center deployment characteristics of the new cross-regional multi-active architecture, the testing team built a three-dimensional testing system, including application layer testing, data layer testing and disaster recovery scenario testing.

3.1 Application Layer Testing Strategy

Application layer testing focuses on verifying business logic consistency and service compatibility after multi-center deployment, covering three core modules: public parameter management, cross-site cache synchronization and upstream consumer-side adaptation.

• Public ParametersUnified public parameters are the core foundation for consistent transaction rules in multi-active architecture. The key testing target is to maintain high parameter consistency across the Beijing headquarters center, Shanghai headquarters center and Shanghai local site during daily operation and cross-region switchover. After any parameter modification, the team adopts triple-write database verification, separately accesses databases of all three sites, and verifies parameter synchronization accuracy to prevent business rule deviations caused by asynchronous configuration updates in multi-active environments.
• Cache SynchronizationPartial core quota parameters of Super Online Banking are synchronized through the EXP platform. The testing team focuses on verifying cross-site cache consistency of transaction limit indicators, ensuring real-time data synchronization among three major nodes, and avoiding transaction rejection, quota judgment errors and user service exceptions caused by multi-center cache differences in the multi-active operating environment.
• Consumer-Side TestingIn this cross-regional multi-active upgrade, all external service interfaces, request protocols and response standards remain unchanged. Upstream channel systems and branch access terminals require no secondary modification or adaptation. Therefore, additional consumer-side transformation testing is not required, effectively reducing redundant testing scope and optimizing overall multi-active testing efficiency.

3.2 Data Layer Testing Strategy

Data layer stability and data consistency are the top priorities of financial multi-active testing, directly determining fund accounting accuracy and business compliance. Data layer testing mainly includes cross-region data synchronization, batch and online data file processing, and historical data migration verification.

• Data SynchronizationThe team focused on verifying data statistical accuracy for cross-region access businesses and reconstructed batch transaction links. A large number of combined test scenarios were designed, including single-node abnormal failures, multi-center concurrent exceptions and cross-region access interruptions. Through diversified abnormal verification, the data synchronization mechanism’s fault tolerance and anti-risk capability in complex multi-active scenarios are fully validated.
• Data File ProcessingFor online real-time transactions and offline batch accounting businesses involving multi-database joint operations, the testing team pre-arranged production-simulated test data in advance. Regression testing was implemented for independent partition data processing and mixed cross-partition data scenarios. All test baseline data strictly matches production volume and structure, ensuring effective deduplication of accounting batch tasks and preventing transaction response delay caused by frequent cross-database access in the new multi-active architecture.
• Data MigrationThis multi-active reconstruction focuses on deployment architecture optimization rather than core system replacement. The original data storage logic and historical business data remain fully compatible. Hence, large-scale data migration testing is not required in this project.

3.3 Disaster Recovery Scenario Testing Strategy

The core goal of Super Online Banking cross-regional multi-active construction is to realize active-active disaster recovery between Beijing and Shanghai data centers, ensure continuous service availability of the Beijing core node, and achieve non-sensing business switchover when the Shanghai regional center fails or goes offline. Meanwhile, the system must maintain stable message routing and orderly asynchronous transaction interaction in distributed multi-active deployment.

All disaster recovery scenario testing revolves around two core objectives:

1. Validate that when all business traffic is fully switched to the Beijing data center, system processing performance, hardware resource occupancy and concurrent transaction capacity fully meet production operation standards.
2. Simulate combined abnormal scenarios of dual-center switchover, to confirm that system recovery capability, business continuity and cross-center data consistency will not be affected under both normal switchover and abnormal fault conditions.

4. Core Measures to Ensure Multi-Active Testing Result Reliability

The cross-regional multi-active reconstruction project faced a tight 2-month testing cycle, accompanied by multiple concurrent iteration tasks and urgent release schedules. Under tight delivery pressure, the testing team adopted an early involvement and early testing mechanism, closely aligning all testing arrangements with agile development progress.

To further enhance testing accuracy, improve scenario coverage and guarantee high-quality delivery of multi-active projects, the team implemented four standardized and high-efficiency optimization measures, forming a complete quality assurance system for financial multi-active testing.

4.1 Deploy Automated Testing Tools to Boost Regression Efficiency

Facing frequent iteration and repeated regression demands in multi-active transformation, the team completed four rounds of comprehensive regression testing. The scope covers online transaction processing, offline batch scheduling, cross-center message routing, routing rule verification and post-switchover transaction validity check.

The project realized 100% automated coverage of core online transaction scenarios, with nearly 3,000 automated test cases executed in each regression round. Automated testing effectively replaces repetitive manual operations, ensures long-term transaction logic stability after multi-active architecture upgrading, and significantly shortens the overall testing cycle for large-scale financial transformation projects.

4.2 Full-Link Regression Testing for Complex Associated Transactions

Most inter-bank payment businesses belong to long-link cross-system transactions with complex interaction logic. For all associated businesses, the testing team adopted end-to-end full-link regression solutions to eliminate hidden risks at intermediate nodes.

Taking ABC payment agreement signing as a typical case:

This business process involves 5 independent participating systems and 14 cross-system message interactions. The testing scope covers internal system data storage log verification, cross-platform message parameter comparison, and end-to-end data docking inspection with external banks and PBOC platforms. Full-link testing ensures zero processing errors in the entire business chain under multi-active operating conditions.

4.3 Integrate Legacy Testing Assets + Chaos Simulation for Full Scenario Coverage

Leveraging mature testing assets accumulated from previous banking architecture upgrades, including account reconstruction, local dual-active transformation and cloud platform migration, the team collaborated with architecture and development teams to sort out potential fault risks in multi-center operation.

By introducing professional chaos engineering simulation tools, the team simulated database disconnection, routing failure, front-end machine abnormality and various switchover faults. Chaos testing complements traditional functional testing, realizes full-coverage verification of extreme abnormal scenarios, and greatly improves the risk prevention capability of multi-active system testing.

4.4 Strengthen Intelligent Routing Correctness & Stability Verification

As a real-time financial core system, Super Online Banking has strict requirements on transaction success rate and service stability. Intelligent routing is the core scheduling component of the multi-active architecture, undertaking transaction traceback, graylist interception, dynamic rule adjustment and traffic scheduling.

In functional testing, 7 core routing rules were fully regressed based on official interface documents to ensure rule matching accuracy. In performance testing, real production peak traffic was simulated to verify transaction integrity during routing switching, system load pressure and long-running fatigue stability. Comprehensive routing testing guarantees that multi-active architecture upgrades will not damage overall transaction service quality.

5. Conclusion | Summary of Financial Multi-Active Testing Practices

Effective tool management and standardized process optimization are the foundation of high-quality QA delivery for critical financial systems. For large-scale cross-regional multi-active transformation projects with complex architecture and high risk levels, testing teams need to decompose transformation objectives layer by layer, sort differentiated test points, and formulate targeted multi-active testing strategies.

This article fully summarizes the exploration, implementation and practical experience of Super Online Banking cross-regional multi-active testing. The layered testing framework, automated regression solution, chaos simulation testing method and full-link quality control logic summarized in the project can provide valuable references for global financial institutions and tech teams carrying out multi-center deployment, high-availability reconstruction and disaster recovery system upgrading.

In the future, financial software testing will continue to evolve alongside distributed architecture and cloud-native multi-active construction. Continuous optimization of dynamic testing strategies, closed-loop iteration of QA processes, and standardized promotion of multi-active testing specifications will further support financial enterprises in building more stable, secure and high-resilience core business systems.