Skip to content

test(telemetry): avoid requiring provider chunk intervals - #1626

Merged
planetf1 merged 1 commit into
generative-computing:mainfrom
planetf1:issue-1566
Sep 4, 2026
Merged

test(telemetry): avoid requiring provider chunk intervals#1626
planetf1 merged 1 commit into
generative-computing:mainfrom
planetf1:issue-1566

Conversation

@planetf1

@planetf1 planetf1 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Pull Request

Issue

Partly addresses #1566 (test case impact on other code)

Description

The live Ollama telemetry test required gen_ai.client.operation.time_per_output_chunk to exist for every streamed response. That metric records inter-chunk intervals, so a valid one-chunk response has no interval and leaves the histogram empty. Provider chunk framing is not deterministic, even when the prompt produces several output tokens, so the assertion caused unrelated CI failures.

Remove that provider-dependent assertion while retaining the live checks for token usage, request duration, and time to first chunk. Deterministic unit coverage for interval recording remains unchanged. The broader producer-side telemetry correction stays scoped to issue #1566.

Testing

  • Tests added to the respective file if code was changed
  • New code has 100% coverage if code was added
  • Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Verified locally:

uv run pytest test/telemetry/test_metrics_plugins.py test/helpers/test_async_helpers.py -q
18 passed, 1 skipped
uv run ruff check test/telemetry/test_metrics_backend.py
All checks passed!
uv run ruff format --check test/telemetry/test_metrics_backend.py
1 file already formatted

Attribution

  • AI coding assistants used

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

  • Component
  • Requirement
  • Sampling Strategy
  • Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

A streamed response may contain one provider chunk, in which case no inter-chunk interval exists and the optional time_per_output_chunk histogram is empty. Keep the live Ollama test focused on metrics guaranteed by the request while deterministic plugin tests cover interval recording.

Assisted-by: Codex
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
@planetf1 planetf1 self-assigned this Sep 4, 2026
@planetf1
planetf1 marked this pull request as ready for review September 4, 2026 14:54
@planetf1
planetf1 requested a review from a team as a code owner September 4, 2026 14:54

@AngeloDanducci AngeloDanducci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ajbozarth ajbozarth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok with removing this test assert if you've found it flaky.

@planetf1
planetf1 added this pull request to the merge queue Sep 4, 2026
Merged via the queue into generative-computing:main with commit 0430e87 Sep 4, 2026
12 checks passed
@planetf1
planetf1 deleted the issue-1566 branch September 4, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants