Skip to content

PT2/AOTI PyTorch backend reports incorrect compute_infer latency due to swapped ReportStatistics arguments #8874

Description

@yiran-zhao

Description

For PyTorch backend PT2/AOTI models, nv_inference_compute_infer_summary_us
can report extremely large values, e.g. around 18446744074s.

This appears to be caused by an argument ordering bug in the PT2 backend's
TRITONBACKEND_ModelInstanceReportStatistics call.

The API expects:
exec_start_ns, compute_start_ns, compute_end_ns, exec_end_ns

but pytorch_backend/src/pt2/model_instance_state.cc currently passes:
exec_start_ns, exec_end_ns, compute_start_ns, compute_end_ns

As a result, Triton core interprets exec_end_ns as compute_input_end_ns
and compute_start_ns as compute_output_start_ns. Then this subtraction
underflows as uint64_t (huge value 18446744074s).

Triton Information
What version of Triton are you using? 26.06

Are you using the Triton container or did you build it yourself? Container.

To Reproduce
Run PT2/AOTI pytorch model and observe nv_inference_compute_infer_summary_us.

Expected behavior

PT2/AOTI per-request statistics should pass timestamps in the same order as
legacy PyTorch backend:

TRITONBACKEND_ModelInstanceReportStatistics(
    TritonModelInstance(), request, responses[r] != nullptr,
    exec_start_ns, compute_start_ns, compute_end_ns, exec_end_ns)

For a PR, the actual fix is tiny in pytorch_backend/src/pt2/model_instance_state.cc: swap the last three timestamp args to match the API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions