Skip to content

Blind review: production observability call sites - #403

Open
thebtf wants to merge 1 commit into
review-base/prc-sol-synthesis-r1-pre-observabilityfrom
review/ops-observability-callsites-r1
Open

Blind review: production observability call sites#403
thebtf wants to merge 1 commit into
review-base/prc-sol-synthesis-r1-pre-observabilityfrom
review/ops-observability-callsites-r1

Conversation

@thebtf

@thebtf thebtf commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Review-only PR for exact commit 888708b over d6b3375. Do not merge. Judge production behavior, OTel lifecycle concurrency, daemon-vs-shim ownership, auth/version/database/index diagnostics, low-cardinality security, and test sensitivity. Full focused/full/critical evidence is committed in the diff.

Summary by CodeRabbit

  • Новые возможности

    • Улучшена телеметрия: добавлены метрики gRPC для серверных и клиентских вызовов.
    • Добавлены диагностические события для ошибок авторизации, несовместимых версий, базы данных и фоновой индексации.
    • Телеметрия корректно различает daemon-режим и краткоживущие процессы.
    • Добавлена настройка имени сервиса в экспортируемых метриках.
  • Исправления

    • Улучшено завершение телеметрии при остановке и перезапуске daemon.
    • Подтверждена успешная работа критического набора из 197 тестов.

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Изменения расширяют observability для daemon lifecycle, gRPC transport/auth/version flows, фоновой индексации и database initialization. Добавлены контрактные тесты, OTLP resource identity, MeterProvider accessor и артефакты успешных test/review-прогонов.

Changes

Интеграция наблюдаемости

Layer / File(s) Summary
Контракт runtime и MeterProvider
go.mod, internal/module/obs/*
Добавлены выбор MeterProvider, InitForService и проверка service.name=engram-daemon.
Телеметрия daemon lifecycle
cmd/engram/*
Инициализация, shutdown и graceful restart используют observability runtime; daemon и shim-режимы покрыты тестами.
gRPC метрики и события
internal/grpcserver/*, internal/handlers/engramcore/*, internal/handlers/serverevents/*
Серверные и клиентские gRPC stats handlers экспортируют duration metrics, а auth/version ошибки записывают runtime-события.
События фоновых операций
internal/handlers/codeintel/*, internal/worker/*
Ошибки индексации и database initialization фиксируются через bounded runtime events и проверяются тестами.
Проверочные артефакты
.agent/reports/..., .agent/reviews/..., .agent/specs/..., scripts/production-smoke/*
Добавлены TDD/review-материалы и JSON-сводки успешных observability и critical-suite запусков.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: ai-review:active

Poem

Я, кролик, метрики в дашборд несу,
Прыжком проверяю успешность в JSON.
Daemon засыпает — flush завершён,
G RPC светится, тесты прошли.
В норе observability — порядок и сон!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 27.78% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Title matches the PR’s main focus on production observability call sites and review coverage.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch review/ops-observability-callsites-r1

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Comment @coderabbitai help to get the list of available commands.

@thebtf

thebtf commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review focus:behavioral correctness, lifecycle concurrency, security and low-cardinality telemetry, release regressions

@thebtf

thebtf commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown

@thebtf Understood — I'll perform a focused review of this PR with emphasis on:

  • Behavioral correctness of the observability call sites introduced/changed in this diff
  • OTel lifecycle concurrency — span/context creation, shutdown/flush ordering, and any races around start/stop
  • Daemon-vs-shim ownership — which component is responsible for initializing/tearing down telemetry resources
  • Security & low-cardinality telemetry — checking for high-cardinality or sensitive values (auth tokens, versions, DB/index identifiers) leaking into metric labels or span attributes
  • Test sensitivity — whether tests added/modified adequately cover the above, especially concurrency and negative/edge cases
  • Release regressions relative to the prior commit d6b3375

I'll flag anything relevant to auth, version, database, and index diagnostics call sites specifically, given the objectives noted for this blind review.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates OpenTelemetry gRPC instrumentation (otelgrpc) across client and server connections, separates daemon and server resource identities, and adds bounded runtime event tracking for authentication, version negotiation, database initialization, and background indexing. Additionally, daemon observability lifecycle management is added to the main entry point and graceful restart paths. Feedback is provided regarding a potential issue where the telemetry context is cancelled prematurely immediately after initialization, which could abort background connection attempts in the OTLP exporter.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread cmd/engram/main.go
Comment on lines +265 to +271
telemetryCtx, telemetryCancel := context.WithTimeout(context.Background(), 5*time.Second)
telemetry, err := initDaemonObservability(telemetryCtx)
telemetryCancel()
if err != nil {
logger.Error("observability initialization failed", "error", err)
os.Exit(1)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Cancelling the telemetryCtx immediately after initDaemonObservability returns can abort background connection and retry attempts in the OTLP exporter. Since otlpmetricgrpc.New initializes the connection asynchronously in the background, the context passed to it must remain active for the connection to be established and maintained.

Passing context.Background() directly avoids premature cancellation of the exporter's background connection.

	telemetry, err := initDaemonObservability(context.Background())
	if err != nil {
		logger.Error("observability initialization failed", "error", err)
		os.Exit(1)
	}

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
cmd/engram/main.go (1)

513-513: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Логируйте исходную ошибку shutdown.

Сейчас лог не содержит err, поэтому нельзя отличить timeout, TLS-ошибку и сбой exporter.

Предлагаемое исправление
- logger.Error("observability shutdown failed; check collector availability")
+ logger.Error("observability shutdown failed; check collector availability", "error", err)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/engram/main.go` at line 513, Update the observability shutdown error
handling around the logger.Error call to include the original shutdown err
alongside the existing context message. Preserve the current shutdown failure
path while ensuring the logged details distinguish timeout, TLS, and exporter
errors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
@.agent/reports/evidence/production-ready/observability/sol-synthesis-20260713-r1/summary.json:
- Around line 23-27: Update the evidence generation in verify-otlp.ps1 so
process_residue and container_residue are populated from actual process and
container checks rather than hard-coded empty arrays, and make verdict fail when
either contains residue. If these checks cannot be implemented, remove both
fields from the evidence contract and stop claiming their absence.

In
@.agent/reports/evidence/production-ready/release-gates-foundation/critical-suite-runner/sol-synthesis-20260713-observability-r2/commands.json:
- Around line 18-19: Normalize the evidence command records around the command
metadata and stdout/stderr artifact references: replace host-specific absolute
paths with repository-relative paths, explicitly record the working_directory,
and apply the same change to the additionally referenced entries. Preserve
stable artifact identification while ensuring commands are portable across hosts
and do not expose local machine structure.

---

Nitpick comments:
In `@cmd/engram/main.go`:
- Line 513: Update the observability shutdown error handling around the
logger.Error call to include the original shutdown err alongside the existing
context message. Preserve the current shutdown failure path while ensuring the
logged details distinguish timeout, TLS, and exporter errors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 23a76fab-a0c3-4857-aede-6956e783c2ed

📥 Commits

Reviewing files that changed from the base of the PR and between d6b3375 and 888708b.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (25)
  • .agent/reports/evidence/production-ready/observability/sol-synthesis-20260713-r1/summary.json
  • .agent/reports/evidence/production-ready/release-gates-foundation/critical-suite-runner/sol-synthesis-20260713-observability-r2/commands.json
  • .agent/reports/evidence/production-ready/release-gates-foundation/critical-suite-runner/sol-synthesis-20260713-observability-r2/go-test-summary.json
  • .agent/reports/evidence/production-ready/release-gates-foundation/critical-suite-runner/sol-synthesis-20260713-observability-r2/summary.json
  • .agent/reviews/2026-07-13-ops-observability-callsites-sol-review.md
  • .agent/specs/ops-observability-call-sites/evidence/OBS-CALLSITES-R1.tdd.json
  • cmd/engram/main.go
  • cmd/engram/observability_contract_test.go
  • go.mod
  • internal/grpcserver/observability_contract_test.go
  • internal/grpcserver/server.go
  • internal/grpcserver/version_negotiate.go
  • internal/handlers/codeintel/module.go
  • internal/handlers/codeintel/observability_contract_test.go
  • internal/handlers/engramcore/grpcpool.go
  • internal/handlers/engramcore/observability_contract_test.go
  • internal/handlers/serverevents/bridge.go
  • internal/handlers/serverevents/observability_contract_test.go
  • internal/module/obs/meter.go
  • internal/module/obs/metrics.go
  • internal/module/obs/runtime.go
  • internal/module/obs/runtime_test.go
  • internal/worker/database_observability_contract_test.go
  • internal/worker/service.go
  • scripts/production-smoke/verify-otlp.ps1

Comment on lines +23 to +27
"missing_tests": [],
"failed_tests": [],
"process_residue": [],
"container_residue": [],
"verdict": "PASS"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Не заявляйте отсутствие residue без фактической проверки.

process_residue и container_residue всегда записываются как пустые массивы в scripts/production-smoke/verify-otlp.ps1, а verdict не учитывает утечки процессов или контейнеров. Поэтому этот PASS-артефакт может пройти при оставшемся exporter/collector residue. Добавьте реальную проверку либо удалите эти поля из evidence-контракта.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
@.agent/reports/evidence/production-ready/observability/sol-synthesis-20260713-r1/summary.json
around lines 23 - 27, Update the evidence generation in verify-otlp.ps1 so
process_residue and container_residue are populated from actual process and
container checks rather than hard-coded empty arrays, and make verdict fail when
either contains residue. If these checks cannot be implemented, remove both
fields from the evidence contract and stop claiming their absence.

Comment on lines +18 to +19
"stdout": "D:\\Dev\\engram\\.agent\\worktrees\\prc-sol-synthesis-r1\\.agent\\reports\\evidence\\production-ready\\release-gates-foundation\\critical-suite-runner\\sol-synthesis-20260713-observability-r2\\go-test.stdout.jsonl",
"stderr": "D:\\Dev\\engram\\.agent\\worktrees\\prc-sol-synthesis-r1\\.agent\\reports\\evidence\\production-ready\\release-gates-foundation\\critical-suite-runner\\sol-synthesis-20260713-observability-r2\\go-test.stderr.log"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Нормализуйте пути в evidence-командах.

commands.json смешивает абсолютные пути D:\Dev\engram\... с относительным -InputPath, не фиксируя рабочий каталог. На другом хосте команда будет непереносимой, а артефакт раскрывает локальную структуру рабочей машины. Сохраняйте repo-relative пути и явно записывайте working_directory, либо используйте стабильные идентификаторы артефактов.

Also applies to: 27-34, 40-41

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
@.agent/reports/evidence/production-ready/release-gates-foundation/critical-suite-runner/sol-synthesis-20260713-observability-r2/commands.json
around lines 18 - 19, Normalize the evidence command records around the command
metadata and stdout/stderr artifact references: replace host-specific absolute
paths with repository-relative paths, explicitly record the working_directory,
and apply the same change to the additionally referenced entries. Preserve
stable artifact identification while ensuring commands are portable across hosts
and do not expose local machine structure.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 888708b3f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant