openetelemetry: Implement cmetrics max datapoints and its splitting - #12251
Open
cosmo0920 wants to merge 8 commits into
Open
openetelemetry: Implement cmetrics max datapoints and its splitting#12251cosmo0920 wants to merge 8 commits into
cosmo0920 wants to merge 8 commits into
Conversation
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
📝 WalkthroughWalkthroughOpenTelemetry metrics now support configurable data-point limits per export request. Oversized protobuf payloads are split across metric boundaries and posted sequentially. CMT and MessagePack inputs use the same batching APIs. Tests cover HTTP, gRPC, metric types, resource preservation, and partial responses. ChangesMetrics batching
HTTP/2 connection recycling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant MetricsInput
participant OpenTelemetryOutput
participant MetricsBatchAPI
participant HTTPOrGRPCReceiver
MetricsInput->>OpenTelemetryOutput: metric protobuf payload
OpenTelemetryOutput->>MetricsBatchAPI: split by metrics_max_datapoints
MetricsBatchAPI-->>OpenTelemetryOutput: bounded protobuf batches
OpenTelemetryOutput->>HTTPOrGRPCReceiver: post batches sequentially
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
cosmo0920
force-pushed
the
cosmo0920-implement-cmetrics-max-datapoints-and-its-splitting
branch
from
August 10, 2026 06:12
84e5469 to
0be6263
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 84e546954a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
tests/internal/opentelemetry.c (1)
2966-2994: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd coverage for metrics with no data points.
The splitting loop in
flb_opentelemetry_metrics_proto_batches_createhas a dedicated path fordata_point_count == 0(it callsbatch_add_metricwith count0), andmetric_data_point_counthas a path forOPENTELEMETRY__PROTO__METRICS__V1__METRIC__DATA__NOT_SET. Neither path is exercised.Extend this fixture with one metric that has
n_data_points == 0and one metric with an unsetdata_case. Then assert that the batch count and the data point totals stay unchanged, and that the empty metrics are still present in the output.Based on coding guidelines: "Validate both success and failure paths, including invalid payloads, boundary sizes, and null or missing fields."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/internal/opentelemetry.c` around lines 2966 - 2994, Extend the fixture constructing the metrics request to include one metric with n_data_points set to 0 and another whose data_case is OPENTELEMETRY__PROTO__METRICS__V1__METRIC__DATA__NOT_SET. Keep the existing batch totals and data-point assertions, add checks that the batch count and totals remain unchanged, and verify both empty metrics are retained in the resulting batches.Source: Coding guidelines
plugins/out_opentelemetry/opentelemetry.c (1)
768-780: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winReport which batch failed, and document the resend behavior.
If batch
nfails after batches0..n-1succeeded, the function returns the failure for the complete chunk. The engine then retries the complete chunk, so the already accepted batches are sent again. The backend receives duplicate data points.The at-least-once semantics are expected in Fluent Bit, but the current code gives no signal about it. Add an error log with the batch index and the total count, so operators can correlate duplicates with a partial export.
♻️ Proposed change
if (result != FLB_OK) { + flb_plg_error(ctx->ins, + "metrics batch %zu of %zu failed; previously accepted " + "batches will be resent on retry", + index + 1, batches->count); break; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/out_opentelemetry/opentelemetry.c` around lines 768 - 780, Update the batch loop in the surrounding export function to log an error when opentelemetry_post fails, including the failed batch index and total batches count. Keep returning the failure for the complete chunk so Fluent Bit retries it, and document that previously accepted batches may be resent under the existing at-least-once behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/opentelemetry/flb_opentelemetry_metrics_batch.c`:
- Around line 763-785: Update flb_opentelemetry_metrics_to_otlp_proto_batches
and flb_opentelemetry_metrics_msgpack_to_otlp_proto_batches to share a helper
for empty encoded payloads. When the encoder returns a zero-length payload,
return an empty batch collection with success instead of calling
flb_opentelemetry_metrics_proto_batches_create; preserve existing error handling
for NULL payloads and normal batching for non-empty payloads.
- Around line 342-344: Update metric_data_point_count to treat an unrecognized
data_case as zero data points instead of returning
FLB_OPENTELEMETRY_OTLP_PROTO_INVALID_ARGUMENT. Preserve the existing counts for
the five supported types so flb_opentelemetry_metrics_proto_batches_create
forwards unsupported metrics using the existing metric_view_create behavior.
---
Nitpick comments:
In `@plugins/out_opentelemetry/opentelemetry.c`:
- Around line 768-780: Update the batch loop in the surrounding export function
to log an error when opentelemetry_post fails, including the failed batch index
and total batches count. Keep returning the failure for the complete chunk so
Fluent Bit retries it, and document that previously accepted batches may be
resent under the existing at-least-once behavior.
In `@tests/internal/opentelemetry.c`:
- Around line 2966-2994: Extend the fixture constructing the metrics request to
include one metric with n_data_points set to 0 and another whose data_case is
OPENTELEMETRY__PROTO__METRICS__V1__METRIC__DATA__NOT_SET. Keep the existing
batch totals and data-point assertions, add checks that the batch count and
totals remain unchanged, and verify both empty metrics are retained in the
resulting batches.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5dc3e41f-55cd-44ad-925e-b1581cf94e64
📒 Files selected for processing (11)
include/fluent-bit/flb_opentelemetry.hplugins/out_opentelemetry/opentelemetry.cplugins/out_opentelemetry/opentelemetry.hsrc/CMakeLists.txtsrc/flb_http_client.csrc/opentelemetry/flb_opentelemetry_metrics_batch.ctests/integration/README.mdtests/integration/scenarios/out_opentelemetry/config/out_otel_grpc_metrics_max_datapoints.conftests/integration/scenarios/out_opentelemetry/config/out_otel_http_metrics_max_datapoints.conftests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.pytests/internal/opentelemetry.c
cosmo0920
force-pushed
the
cosmo0920-implement-cmetrics-max-datapoints-and-its-splitting
branch
from
August 10, 2026 07:02
0be6263 to
079d083
Compare
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
… points Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
…limitation Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
cosmo0920
force-pushed
the
cosmo0920-implement-cmetrics-max-datapoints-and-its-splitting
branch
from
August 10, 2026 07:04
079d083 to
aa2be0e
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
tests/internal/opentelemetry.c (1)
2757-2757: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse the required C control-block brace style.
Put the opening brace on the next line for each changed
forandifblock in this file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/internal/opentelemetry.c` at line 2757, Update the changed for and if control blocks in the OpenTelemetry test code to place each opening brace on the following line, including the loop beginning with index = 0. Preserve the existing block logic and indentation.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@tests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.py`:
- Around line 1030-1042: Strengthen the assertions in the test around
metrics_seen so the first two export requests each contain exactly four points
and their series sets are exactly {0,1,2,3} and {4,5,6,7}, respectively. Also
assert that series IDs 8, 9, and 10 do not appear in any exported request,
replacing the insufficient disjointness-only check.
- Around line 539-619: Validate resource and scope metadata preservation across
batches: in
tests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.py
lines 539-619, add distinct resource and scope schema URLs, have
iter_metric_points_with_resource return scope metadata alongside each point, and
assert every point retains its resource and scope association; in
tests/internal/opentelemetry.c lines 2900-3065, add distinct resource and scope
metadata to the input request and assert every decoded batch preserves it.
In `@tests/internal/opentelemetry.c`:
- Around line 2921-2955: Expand the test setup around the gauge, sum, histogram,
exponential histogram, and summary metrics so each contains more data points
than max_data_points, then exercise offset-based splitting for every metric
type. Assert each decoded batch retains the expected points and all associated
metric fields, including names, data cases, and type-specific values.
---
Nitpick comments:
In `@tests/internal/opentelemetry.c`:
- Line 2757: Update the changed for and if control blocks in the OpenTelemetry
test code to place each opening brace on the following line, including the loop
beginning with index = 0. Preserve the existing block logic and indentation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 2b960230-a856-479e-b220-cf35c41d36f8
📒 Files selected for processing (5)
plugins/out_opentelemetry/opentelemetry.csrc/opentelemetry/flb_opentelemetry_metrics_batch.ctests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.pytests/integration/src/server/otlp_server.pytests/internal/opentelemetry.c
🚧 Files skipped from review as they are similar to previous changes (2)
- plugins/out_opentelemetry/opentelemetry.c
- src/opentelemetry/flb_opentelemetry_metrics_batch.c
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@tests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.py`:
- Around line 1064-1074: Update the test around the batch_series assertions to
keep the receiver/service active through the configured retry interval before
taking the final requests snapshot and asserting no retry occurred. Ensure the
failed batch remains un-retried and validate both successful batches and the
retry behavior after the wait.
- Around line 1006-1028: Update the batch-validation test around the loop over
points to record series IDs separately for each received request, then assert
the requests arrive in order as {0,1,2,3}, {4,5,6,7}, and {8,9,10}. Preserve the
existing per-point resource and scope assertions, and add coverage for both
successful ordered batches and the failure path when ordering is violated.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 159b17ce-782b-4143-82e8-273bc499555e
📒 Files selected for processing (2)
tests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.pytests/internal/opentelemetry.c
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/internal/opentelemetry.c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented as a reusable Fluent Bit core feature, with no
lib/cmetricsor other bundled-library changes.Key changes:
struct cmt, and CMetrics MessagePack in flb_opentelemetry.h.metrics_max_datapoints;0retains existing unlimited behavior.Example:
[OUTPUT] Name opentelemetry Match * metrics_max_datapoints 1000Verification passed:
cmake --build build-codex-otel --target fluent-bit-bin flb-it-opentelemetry -j8ctest --test-dir build-codex-otel -R flb-it-opentelemetry --output-on-failure— passedtests/integration/.venv/Scripts/python.exe -m pytest tests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.py -k metrics_max_datapoints -q— 2 passed4,4, and3data points.git diff --check— passedCloses #12247.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Bug Fixes