Scaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 2

The Global Performance Tuner is a feature for automatic performance tuning released in NVIDIA TensorRT 11.1. It provides automatic, accuracy-aware and end-to-end performance tuning within TensorRT by customizing different optimizations for different models. For more details, see Part 1 of this series, which introduces the challenges associated with scheduling (a critical stage in deep learning compiler optimization), the motivation for developing the Global Performance Tuner feature in TensorRT, and the mathematical theories established to formalize the problem. This post, Part 2, uses the notation from Part 1 to describe the algorithm and architectural design of the Global Performance Tuner feature. It also provides best practices for using it. Designing the Global Performance Tuner feature in TensorRT The central design idea of the Global Performance Tuner is to represent alternative scheduling strategies as explicit and searchable build routes. A compiler knob (also called a compiler flag or compiler option) is a configuration setting that changes some aspect of the compilation pipeline, such as a heuristic, fusion decision, or code generation path. Each tunable knob exposes a finite set of allowed values and has a default value. A build route is a concrete combination of knob values. When no build route is specified, TensorRT uses the default build route, in which every knob retains its default value. Consider a hypothetical compiler consisting of three passes, (f_1), (f_2), and (f_3). Suppose a binary knob (p) controls the fusion strategy used by (f_2): when (p=1), the pass applies horizontal fusion; when (p=0), it applies vertical fusion: (f_2(s) = \begin{cases} f’_2(s), & \text{if } p = 1 \ f”_2(s), & \text{if } p = 0 \end{cases}) The two possible values of (p) define two build routes and therefore produce a candidate solution set of size two for the input model (m): (B(m) = {f_3(f’_2(f_1(m))),, f_3(f”_2(f_1(m)))}) Neither fusion strategy is universally better. Horizontal fusion may produce the faster engine for one model, while vertical fusion may be more effective for another. The result may also change across tensor shapes and target GPUs, even when the model itself remains unchanged. Selecting the better route therefore requires building both candidate engines and comparing their end-to-end performance. Even this two-route example captures the essential idea of tuning: the compiler exposes meaningful scheduling …