Skip to content

Bootstrap OpenTelemetry in workers, Batch and Lambda - #1578

Merged
Nazehs merged 4 commits into
mainfrom
pro-648-bootstrap-otel-workers-batch-lambda
Aug 28, 2026
Merged

Bootstrap OpenTelemetry in workers, Batch and Lambda#1578
Nazehs merged 4 commits into
mainfrom
pro-648-bootstrap-otel-workers-batch-lambda

Conversation

@Nazehs

@Nazehs Nazehs commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Context

Bootstraps OpenTelemetry on the surfaces with no auto-instrumentation: the RQ worker, the Batch scripts, and the import/notifier Lambdas. Without a root span per run their logs can't be correlated. It's a drop-in seam guarded on the OTEL_ENABLED flag and a collector endpoint, so it stays a no-op until both are set.

Changes proposed in this pull request

  • Add a guarded OTel seam (bootstrap_otel, execution_span, flush_otel) to pipeline-common, the backend, and each Lambda.
  • Wrap each unit of work (RQ job, both Batch runs, all three Lambda handlers) in a root span carrying context_id and consultation_code, so logs correlate on the same id.
  • Flush at the execution boundary on every surface, since the worker idles and Batch/Lambda are short-lived.
  • Catch both ImportError and the endpoint-required RuntimeError in the Lambda bootstrap so a cold start degrades to a warning, not a crash.
  • Add unit tests for the backend and pipeline-common seams.

Things to check

  • I have added any new ENV vars in all deployed environments and updated the .env.test files in the repo

@linear-code

linear-code Bot commented Aug 25, 2026

Copy link
Copy Markdown

PRO-648

@Nazehs
Nazehs force-pushed the pro-648-bootstrap-otel-workers-batch-lambda branch from fac39ed to d314fe5 Compare August 25, 2026 12:32
@Nazehs
Nazehs force-pushed the pro-523-tune-sentry-sampling-deliberately branch 2 times, most recently from e10e4ac to 18fbd12 Compare August 25, 2026 15:37
@Nazehs
Nazehs force-pushed the pro-648-bootstrap-otel-workers-batch-lambda branch 2 times, most recently from db214e3 to cf4ca1c Compare August 25, 2026 16:04
@Nazehs Nazehs changed the title PRO-648: Bootstrap OpenTelemetry in workers, Batch and Lambda Bootstrap OpenTelemetry in workers, Batch and Lambda Aug 26, 2026
@Nazehs
Nazehs force-pushed the pro-523-tune-sentry-sampling-deliberately branch from 18fbd12 to 6a0c2e5 Compare August 26, 2026 10:37
Base automatically changed from pro-523-tune-sentry-sampling-deliberately to main August 26, 2026 10:47
@Nazehs
Nazehs force-pushed the pro-648-bootstrap-otel-workers-batch-lambda branch 5 times, most recently from fe2fda1 to 0cd73b4 Compare August 26, 2026 14:02
@Nazehs
Nazehs marked this pull request as ready for review August 26, 2026 14:27
@Nazehs
Nazehs requested a review from a team as a code owner August 26, 2026 14:27
@Nazehs
Nazehs requested review from ekin-odabas and removed request for a team August 26, 2026 14:27
@github-actions

github-actions Bot commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @Nazehs's task in 1m 29s —— View job

PR Review: Bootstrap OTel in workers, Batch and Lambda

Clean, well-guarded seam. The no-op path is correct and tests cover the key branches.

lambda/*/otel_bootstrap.py — all three Lambdas ship an identical file (same SHA f085e19c).

🔴 DO — eliminate the triplication. A copy step in the build or a shared Lambda layer means a single source of truth; right now a one-line fix requires three identical edits. Fix this →

⚪️ NIT — bootstrap_otel is silently skipped inside bootstrap_logger when service_name=None (pipeline-common/logging_bootstrap.py:46). A brief docstring note would prevent surprises.

Adds OTel spans to RQ jobs, Batch scripts, and all three Lambdas behind a dual-flag guard. No-op by default; flushes correctly at each execution boundary.

@Nazehs
Nazehs force-pushed the pro-648-bootstrap-otel-workers-batch-lambda branch from 0cd73b4 to 417d26f Compare August 27, 2026 09:49
@github-actions

github-actions Bot commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @Nazehs's task in 1m 34s —— View job

PR Review: Bootstrap OTel in workers, Batch and Lambda

  • Read changed files
  • Provide feedback

Triplication resolved correctly — lambda/shared/otel_bootstrap.py is now the single source, copied into each build by the Makefiles. Clean approach.

lambda/shared/otel_bootstrap.py vs pipeline-common/otel.py

🔵 CONSIDER — these two files are still ~identical: same otel_requested, execution_span, flush_otel logic, differing only in _TRACER_NAME and configure_otel vs configure_otel_for_lambda. If pipeline-common ever gains a Lambda target, consolidation becomes straightforward.

lambda/slack_notifier/code/main.py:161

⚪️ NIT — import_candidate_themes and import_response_annotations both call logger.set_context_field("context_id", context_id), but slack_notifier does not. Minor log-correlation gap for Slack notifications.

pipeline-common/src/pipeline_common/logging_bootstrap.py:47

⚪️ NIT (from previous review, still unaddressed) — bootstrap_logger silently skips OTel when service_name=None; a docstring note would prevent surprises.

Solid, well-guarded seam. No-op path is correct, flush happens at every execution boundary, and Lambda triplication is resolved.

@Nazehs
Nazehs force-pushed the pro-648-bootstrap-otel-workers-batch-lambda branch from 417d26f to de00aa5 Compare August 27, 2026 10:16
@github-actions

github-actions Bot commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @Nazehs's task in 1m 3s —— View job

PR Review: Bootstrap OTel in workers, Batch and Lambda

  • Read changed files
  • Provide feedback

Previous NITs addressed: logging_bootstrap.py docstring added; slack_notifier now sets context_id in both logger context and span (line 163 + 165).

backend/otel_bootstrap.py, pipeline-common/otel.py, lambda/shared/otel_bootstrap.py

🔵 CONSIDER — otel_requested(), execution_span(), and flush_otel() are identical across all three files; only _TRACER_NAME and the configure_otel* variant differ. If a shared utility layer is added later, these three collapse to one.

Clean implementation. No bugs found.

@Nazehs

Nazehs commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

CONSIDER — otel_requested(), execution_span(), and flush_otel() are identical across all three files; only _TRACER_NAME and the configure_otel* variant differ. If a shared utility layer is added later, these three collapse to one.

This is intentional to keep the duplication low and there is a ticket already created to address this duplication in tech debt later in the future and then this will be fully handled https://linear.app/iai-consult/issue/PRO-573/extract-shared-code-across-the-3-lambda-functions

Nazehs added 4 commits August 28, 2026 10:02
@Nazehs
feat(otel): add dormant OTel seam to pipeline-common
dc0255d 
Import-safe, endpoint-guarded OTel helper (bootstrap_otel, execution_span,
flush_otel) shared by the Batch scripts. Stays a no-op until
OTEL_EXPORTER_OTLP_ENDPOINT is set and the util [otel] extra is pinned, so
nothing changes at runtime yet. (PRO-648)
@Nazehs
feat(otel): wrap Batch find/assign runs in an OTel root span
4907dda 
Bootstrap OTel from bootstrap_logger and wrap each script's work in an
execution_span carrying context_id and consultation_code, flushing at the
process boundary. No auto-instrumentation exists for Batch, so this root span
is what makes the run's logs correlatable once the endpoint is wired. (PRO-648)
@Nazehs
feat(otel): bootstrap OTel in the RQ worker and span each job
b6e9916 
Worker-only OTel seam (gated on EXECUTION_CONTEXT == worker) that wraps each
RQ job in an execution_span carrying the same context_id the logs bind,
flushing at the job boundary since a worker can idle after. Import-safe and
endpoint-guarded, so it stays a no-op until the extra is pinned. (PRO-648)
@Nazehs
feat(otel): bootstrap OTel in the import and notifier Lambdas
42bede1 
Vendor the OTel seam into each Lambda and wrap the handler body in an
execution_span carrying context_id, flushing in finally so short-lived
invocations still ship telemetry. The bootstrap catches ImportError and the
endpoint-required RuntimeError, so a cold start never crashes before the
endpoint and util extra land. (PRO-648)
@Nazehs
Nazehs enabled auto-merge (rebase) August 28, 2026 09:02
@Nazehs
Nazehs force-pushed the pro-648-bootstrap-otel-workers-batch-lambda branch from de00aa5 to 42bede1 Compare August 28, 2026 09:02
@github-actions

github-actions Bot commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @Nazehs's task in 59s —— View job

PR Review: Bootstrap OTel in workers, Batch and Lambda

  • Read changed files
  • Provide feedback

Previous feedback addressed. No bugs found.

lambda/slack_notifier/code/main.py:167

⚪️ NIT — logger.info("✅ Slack message sent") sits outside the execution_span block, so the success log won't be attributed to the span.

Clean, no-op by default, flushes correctly at all execution boundaries.

@Nazehs
Nazehs merged commit e6be25f into main Aug 28, 2026
16 checks passed
@Nazehs
Nazehs deleted the pro-648-bootstrap-otel-workers-batch-lambda branch August 28, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ekin-odabas ekin-odabas ekin-odabas approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Nazehs @ekin-odabas