Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels

Rethinking Precision in AI Computations

The development and deployment of advanced AI models, particularly large language models (LLMs), heavily rely on the accurate execution of tensor kernels. These kernels are the fundamental mathematical operations performed on specialized hardware like GPUs. Traditionally, ensuring the correctness of these kernels involves "all close-style" tests that compare results against a known correct implementation using fixed, often hand-picked, absolute and relative tolerances. The paper highlights a critical flaw in this common practice: these thresholds are frequently copied across different test cases and rarely re-evaluated, leading to potentially loose tolerances that miss subtle errors. This oversight can have significant implications for the reliability and performance of AI systems, as undetected numerical inaccuracies can propagate through complex models, leading to incorrect outputs or unstable training. The reliance on static, generalized tolerances fails to account for the inherent numerical properties and precision requirements of individual operators and data types, creating a gap in robust quality assurance.

Data-Driven Tolerance Calibration

To address the limitations of hand-picked tolerances, the research introduces an operator-aware mixed-precision tolerance calibration method. This approach moves away from arbitrary thresholds by empirically deriving tolerances based on observed error distributions. The authors mined element-wise error data from thousands of cloud GPU runs across a 26-entry gpuemu corpus and two data types, accumulating over 8,000 result rows. By analyzing what absolute tolerance a kernel would justify under its correct implementation, the study found that empirically derived tolerances are significantly tighter than current hand-picked values. For instance, the attention_triton fp16 kernel showed a remarkable 2,184x tightening of its absolute tolerance. This data-driven methodology provides a more scientific and precise way to define correctness criteria, ensuring that tests are sensitive enough to catch genuine numerical deviations without generating excessive false positives. The method essentially allows the kernels themselves to dictate their acceptable error bounds, reflecting their true operational characteristics.

Enhancing LLM Reliability and Development

The practical impact of this calibrated tolerance approach is substantial, particularly for the development of large language models. The paper specifically evaluated its method on seven LLM-style buggy variants, each paired with a correct counterpart. The results demonstrated a significant improvement in bug-detection recall, increasing from 73.2% (1,805 of 2,467) to 82.4% (2,034 of 2,467). This represents an absolute gain of 9.3 percentage points, translating to 229 new bug detections that would have been missed by traditional methods. While the control false-positive count did rise slightly from 0 to 20 out of 1,882 correct-control cases (a 1.1 percentage point increase), the substantial gain in true positive detections far outweighs this minor increase. This enhanced ability to identify and rectify numerical bugs early in the development cycle is crucial for building more robust, accurate, and trustworthy AI systems, ultimately accelerating the progress and reliability of LLM research and deployment.