Enhancing Differential Testing Through Domain-Specific Equivalence Rules

<p dir="ltr">The growing complexity of modern software systems in deep learning, distributed training, and quantum computing poses significant challenges for ensuring their reliability. Bugs in these systems can lead to severe consequences, particularly in critical applications like autonomous vehicles and medical diagnosis. While traditional testing approaches, such as formal verification or exhaustive test generation, are often impractical for such systems due to resource constraints, differential testing offers a feasible alternative. But its success is often constrained by the oracle problem: determining when two variants should match. This dissertation addresses that limitation by introducing domain-specific equivalence rules that preserve semantics while enabling systematic test generation.</p><p dir="ltr">This dissertation advances differential testing along two orthogonal axes. Horizontally, it demonstrates portability across domains. EAGLE first establishes 16 novel graph-level equivalence rules that expose hidden faults in single-device deep-learning frameworks; D3 expands the approach to distributed settings by crafting an equivalence rule for distributed DL training; and EriQua carries the idea into quantum computing through quantum equivalence rules and chained-API program synthesis. Vertically, the work loosens the strength of the equivalence constraint itself. While EAGLE assumes universal equivalence–outputs must match for every possible input–DEMI introduces equivalent-modulo-input (EMI) rules, which require agreement only for a subset of input. EMI rules are paired with pattern-guided module synthesis and symbolic tracing, allowing DEMI to stress-test DL compiler optimizations.</p><p dir="ltr">Together, these contributions show that carefully crafted equivalence rules, applied horizontally across distinct domains and vertically by relaxing universality to input-conditioned equivalence, transform differential testing into a powerful, practical methodology for uncovering hard-to-find bugs in todays most sophisticated software systems. By systematically resolving the oracle problem, this work advances the state of software testing and demonstrates the viability of differential techniques for complex, rapidly evolving, and safety-critical domains.</p>