Skip to content

ub_overflow.md

Latest commit

 

History

History
78 lines (50 loc) · 2.75 KB

ub_overflow.md

File metadata and controls

78 lines (50 loc) · 2.75 KB
Raw

UB Overflow Troubleshooting Guide

Overview

UB space insufficiency is a common issue in Triton-Ascend development. This document provides detailed information about common causes, solutions, and debugging methods for UB Overflow.

Common Causes and Solutions

1. Using Interface Parameters That Increase UB Overhead

Certain interfaces automatically add additional processing logic under specific parameter configurations, resulting in increased UB space usage:

propagate_nan Parameter for tl.maximum, tl.minimum, and tl.clamp Interfaces

Issue Description: When setting propagate_nan=tl.PropagateNAN.NONE, the system automatically adds NaN value detection and processing logic.

Impact:

  • Significantly increases UB space usage
  • May cause performance degradation

Solutions:

  • If input data does not contain NaN values or strict NaN processing semantics are not required, consider adjusting the propagate_nan parameter value
  • In scenarios with limited UB space, prioritize parameter configurations that do not trigger additional NaN processing

2. Excessive Intermediate Variables

Problem: The kernel defines a large number of temporary tensors or intermediate computation results.

Solutions:

  • Reduce unnecessary intermediate variables
  • Reuse allocated buffers
  • Split large computations into multiple smaller kernels

3. Large Data Types and Shapes

Problem: Using larger data types such as fp64, bf16, or processing high-dimensional/large shape tensors.

Solutions:

  • Consider splitting large tensors into blocks for processing
  • Modify blocking strategies to reduce the size of each block
  • Use smaller data types (e.g., fp16 instead of fp32) while meeting precision requirements

4. Complex Control Flow or Loops

Problem: The kernel contains complex conditional statements or multi-level nested loops.

Solutions:

  • Simplify control flow logic
  • Reduce loop nesting levels or iteration counts
  • Split complex logic into multiple kernels

Debugging Recommendations

  1. Enable Detailed Logging

    • Use TRITON_DEBUG=1 to obtain detailed compilation information
    • Locate which specific operator causes UB overflow

  2. Step-by-Step Troubleshooting

    • Comment out parts of the code to locate the specific operation causing the issue
    • Use binary search approach to quickly identify problematic code sections

  3. Refer to Documentation

    • Check the "Special Limitations" section in each interface documentation
    • Understand parameter configurations that may increase UB overhead

  4. Optimization Strategies

    • Prioritize handling operators that consume significant UB space
    • Consider redesigning algorithms to reduce intermediate variables
    • Consider modifying blocking strategies