Skip to content

JITSU-138: fix Datadog reserved-status collision; synthesize otlp message lines - #1475

Merged
absorbb merged 3 commits into
newjitsufrom
ildar/jitsu-138-otlp-datadog-presentation
Aug 20, 2026
Merged

JITSU-138: fix Datadog reserved-status collision; synthesize otlp message lines#1475
absorbb merged 3 commits into
newjitsufrom
ildar/jitsu-138-otlp-datadog-presentation

Conversation

@absorbb

@absorbb absorbb commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Follow-up to JITSU-138, from end-to-end testing against Datadog's agentless OTLP intake (which merges the structured body's top-level keys into root log attributes).

Architecture (reworked per review): the otlp destination stays fully payload-agnosticotlp.go is untouched relative to newjitsu. Body adaptation happens where envelopes are produced to the Kafka topic, and only for bodies the pipeline constructs itself:

  1. Go producer (eventslog/kafka_events_log.go), for bulker_batch / bulker_stream records (bulker-built bulker.State / stream-status shapes):
    • top-level statusrecord_status: COMPLETED/FAILED values collide with Datadog's reserved status attribute and made every record render as critical regardless of severity (reproduced in isolation with controlled probe payloads)
    • synthesized message when absent/empty/null: bulker_batch COMPLETED: 2 rows → events39, with a rune-safe-truncated error preview on failures
    • the eventId hash is computed from the unadapted body — ids and billing dedup unchanged
  2. Rotor producer (kafka-events-store.ts): dead-letter bodies (rotor-built {payload, error}) get their message line at construction.

Function-log bodies are user data and are never touched — no key renames, no synthesized fields.

Body stays structured (kvlist) — no stringification. Covered by TestKafkaEventsLogAdaptsOwnedBodies (rename, synthesis, truncation, function-body-untouched) and rotor deadLetterMessage tests (incl. surrogate-pair safety).

Docs note: jitsucom/websites#60.

🤖 Generated with Claude Code

…age lines for otlp records

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jitsu-code-review[bot]
jitsu-code-review Bot previously approved these changes Aug 20, 2026

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the OTLP body-shaping changes and the new coverage in otlp_test.go (including the status remap, synthetic message generation, and copy semantics). I found two potential regressions worth a quick follow-up in inline comments.

Comment thread bulker/bulkerlib/implementations/api_based/otlp.go Outdated
Comment thread bulker/bulkerlib/implementations/api_based/otlp.go Outdated
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jitsu-code-review[bot]
jitsu-code-review Bot previously approved these changes Aug 20, 2026

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the OTLP body-presentation changes in otlp.go and the new coverage in otlp_test.go. I left inline comments for a few correctness/regression risks worth clarifying: unconditional status remapping on generic OTLP payloads, message: null bypassing synthesis, and potential overwrite when record_status already exists.

Comment thread bulker/bulkerlib/implementations/api_based/otlp.go Outdated
Comment thread bulker/bulkerlib/implementations/api_based/otlp.go Outdated
Comment thread bulker/bulkerlib/implementations/api_based/otlp.go Outdated
…nation stays payload-agnostic

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the producer-side adaptation changes in and , plus their new tests. I focused on correctness, regressions, and user-visible behavior (status remapping/message synthesis scope and eventId/billing implications). I didn’t find actionable bugs or security issues in this diff.

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed producer-side adaptation changes in bulker/eventslog/kafka_events_log.go and services/rotor/src/lib/kafka-events-store.ts, plus the added tests. I focused on correctness, regressions, and user-visible behavior (status remapping/message synthesis scope and eventId/billing implications). I did not find actionable bugs or security issues in this diff.

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changes in bulker/eventslog and services/rotor for correctness, regressions, and security impact.

I focused on the producer-side body adaptation (statusrecord_status, synthesized message) and dead-letter message synthesis/truncation behavior. I did not find actionable bugs or user-visible regressions in this patch.

Sanity check: go test ./eventslog passes in bulker/.

@absorbb
absorbb merged commit 24391d1 into newjitsu Aug 20, 2026
13 checks passed
@absorbb
absorbb deleted the ildar/jitsu-138-otlp-datadog-presentation branch August 20, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant