feat(observability): Adds OpenTelemetry tracing, metrics and logs with the Aspire Dashboard

.claude/rules/configuration.md

@@ -34,4 +34,6 @@ Proposed names (descriptive: once implemented, the code is the reference).
 | `Llm:Anthropic:ApiKey` | Matching.Worker | user-secrets only, never in the repo; required only when `Provider = Anthropic` |
 | `Llm:Anthropic:Model`, `Llm:Anthropic:MaxOutputTokens` | Matching.Worker | `claude-haiku-5-5`, 8000; the ceiling includes adaptive thinking |
 | `Llm:Anthropic:MaxRetries` | Matching.Worker | the SDK's retries of 408/409/429/5xx and connection failures; 2 |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | all three hosts | unset = no telemetry at all (tests, CI). Set only in the `launchSettings.json` profiles (`http://localhost:4317`, Aspire Dashboard), never in `appsettings*.json`: integration tests run in `Development` and load those files |
+| `OTEL_SERVICE_NAME` | all three hosts | `ingress-api`, `sync-worker`, `matching-worker`, next to the endpoint in `launchSettings.json`; required when the endpoint is set |
 | `Llm:Fake:Mode` | Matching.Worker | `Valid`, `InvalidJson`, `OutOfRange`, `Hang`; default when the CV has no `[llm:<Mode>]` marker |

.claude/rules/messaging.md

@@ -34,6 +34,9 @@ paths:
 No attributes, no payload body: the envelope is a notification. Workers fetch current state from the API
 (notify-then-fetch). `Contracts` references nothing.
 
+Transport metadata goes in `EventData.Properties`, never in the envelope. The only one is `traceparent`
+(`EventProperties.TraceParent`, ADR 010): the relay's send span, parent of every consumer attempt span.
+
 ## Ingress.Api
 
 1. Read the raw body (enable buffering). Verify signature v2 through the verifier interface from

.claude/rules/persistence.md

@@ -46,7 +46,8 @@ configurations take the schema as a constructor parameter.
 
 ## Tables (minimal columns, descriptive once migrations exist)
 
-- `Outbox`: `Id`, `EventId`, `CandidateId`, `Type`, `Envelope` (json), `CreatedAt`, `PublishedAt` (null until published).
+- `Outbox`: `Id`, `EventId`, `CandidateId`, `Type`, `Envelope` (json), `TraceParent` (W3C, nullable, ADR 010),
+  `CreatedAt`, `PublishedAt` (null until published).
   Filtered index on `PublishedAt IS NULL`.
 - `Inbox`: `Id`, `Source`, `EventId`, `ProcessedAt`.
 - `ParkingLot`: `Id`, `Source` (nullable), `EventId` (nullable), `PartitionId`, `SequenceNumber`, `Envelope`,

.claude/rules/testing.md

@@ -14,6 +14,7 @@ paths:
 | `TalentSync.Sync.Application.Tests`, `TalentSync.Matching.Application.Tests` | Unit, no I/O | Use cases: each fetch result and port result → use case result. Ports replaced by small hand-written stubs |
 | `TalentSync.Infrastructure.Teamtailor.Tests` | Unit, no network | Signature v2, webhook parsing, stage mapping, response mapping (404 → Deleted, 4xx → Rejected). Internals via `InternalsVisibleTo` |
 | `TalentSync.Infrastructure.Messaging.Tests` | Unit, no I/O | Consumer skeleton: inbox and parking lot mapping, failure classification, handler loop |
+| `TalentSync.Infrastructure.Observability.Tests` | Unit, no network | `AddObservability` with the SDK enabled on a test server (the request span is a root), `SpanExportPolicy` in front of an in-memory exporter. Internals via `InternalsVisibleTo` |
 | `TalentSync.Sync.Infrastructure.Tests` | Unit, no network | CRM client: status mapping (404 on delete = done, 4xx → Rejected), HTTP pipeline. Internals via `InternalsVisibleTo` |
 | `TalentSync.Matching.Evaluation.Tests` | Unit, no network | Prompt evaluation (ADR 009): metric calculations, golden set hash, case file validation, report format, the CI guard for the evaluation report. Never calls a model |
 | `TalentSync.IntegrationTests` | Acceptance | Scenarios from `docs/acceptance-scenarios.md` |
@@ -62,6 +63,13 @@ paths:
   The fixture sets `Llm:Timeout` to 2 s for M3. The circuit breaker is shared by all scenarios; only M3 fails a call,
   which stays below the configured failure ratio.
 
+## Traces
+
+Scenario O1 records spans with `SpanRecorder` (`Infrastructure/`), an `ActivityListener` on `TalentSync.*` and
+`Microsoft.AspNetCore` (request spans): all hosts run in the test process, so no exporter is needed. Our spans are
+matched by the `talentsync.event_id` tag, the request span by the webhook span's parent id. O1 runs without the SDK;
+the SDK's own handling of the request span is covered by `EdgeTraceTests` in `TalentSync.Infrastructure.Observability.Tests`.
+
 ## Prompt evaluation guard
 
 `EvaluationReportGuardTests` fails when `eval/reports/` has no report for the current `PromptVersion` and golden set

Directory.Packages.props

@@ -15,18 +15,26 @@
     <PackageVersion Include="Azure.Storage.Blobs" Version="12.23.0" />
     <PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.12" />
     <PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.12" />
+    <PackageVersion Include="Microsoft.Extensions.AI" Version="10.10.0" />
     <PackageVersion Include="Microsoft.Extensions.AI.Abstractions" Version="10.11.0" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.12" />
     <PackageVersion Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.12" />
     <PackageVersion Include="Microsoft.Extensions.Hosting" Version="10.0.12" />
     <PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.12" />
     <PackageVersion Include="Microsoft.Extensions.Http.Resilience" Version="10.10.0" />
     <PackageVersion Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.12" />
+    <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.19.1" />
+    <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.19.1" />
+    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.19.0" />
+    <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.19.0" />
+    <PackageVersion Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.19.0" />
     <PackageVersion Include="Polly.Core" Version="8.8.0" />
   </ItemGroup>
   <ItemGroup Label="Tests">
     <PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.12" />
+    <PackageVersion Include="Microsoft.AspNetCore.TestHost" Version="10.0.12" />
     <PackageVersion Include="Microsoft.Testing.Extensions.CodeCoverage" Version="18.12.0" />
+    <PackageVersion Include="OpenTelemetry.Exporter.InMemory" Version="1.19.1" />
     <PackageVersion Include="Testcontainers.EventHubs" Version="4.16.0" />
     <PackageVersion Include="Testcontainers.MsSql" Version="4.16.0" />
     <PackageVersion Include="xunit.v3" Version="4.0.2" />

README.md

@@ -58,7 +58,8 @@ dotnet user-secrets set "Llm:Provider" "Anthropic" --project src/Matching/Talent
 
 To click through the whole flow by hand (success, CRM outage, candidate without a CV), start the containers and the
 three hosts with `deploy/run-local.sh` (Git Bash; on Windows each host opens in its own Windows Terminal tab), then use
-[`docs/e2e.http`](docs/e2e.http).
+[`docs/e2e.http`](docs/e2e.http). Traces, metrics and logs of the three hosts are in the Aspire Dashboard at
+http://localhost:18888 (ADR 010).
 
 ## Simplifications
 

TalentSync.slnx

@@ -20,6 +20,7 @@
   <Folder Name="/src/Shared/">
     <Project Path="src/Shared/TalentSync.Contracts/TalentSync.Contracts.csproj" />
     <Project Path="src/Shared/TalentSync.Infrastructure.Messaging/TalentSync.Infrastructure.Messaging.csproj" />
+    <Project Path="src/Shared/TalentSync.Infrastructure.Observability/TalentSync.Infrastructure.Observability.csproj" />
     <Project Path="src/Shared/TalentSync.Infrastructure.Teamtailor/TalentSync.Infrastructure.Teamtailor.csproj" />
     <Project Path="src/Shared/TalentSync.Recruitment.Domain/TalentSync.Recruitment.Domain.csproj" />
   </Folder>
@@ -30,6 +31,7 @@
   </Folder>
   <Folder Name="/tests/">
     <Project Path="tests/TalentSync.Infrastructure.Messaging.Tests/TalentSync.Infrastructure.Messaging.Tests.csproj" />
+    <Project Path="tests/TalentSync.Infrastructure.Observability.Tests/TalentSync.Infrastructure.Observability.Tests.csproj" />
     <Project Path="tests/TalentSync.Infrastructure.Teamtailor.Tests/TalentSync.Infrastructure.Teamtailor.Tests.csproj" />
     <Project Path="tests/TalentSync.IntegrationTests/TalentSync.IntegrationTests.csproj" />
     <Project Path="tests/TalentSync.Matching.Application.Tests/TalentSync.Matching.Application.Tests.csproj" />

deploy/docker-compose.yml

@@ -1,5 +1,5 @@
 # Local infrastructure: SQL Server, Event Hubs emulator and Azurite (emulator metadata, later worker checkpoints),
-# plus the three simulators. The TalentSync hosts run on the host with `dotnet run`.
+# the three simulators and the Aspire Dashboard. The TalentSync hosts run on the host with `dotnet run`.
 # Secrets come from deploy/.env (gitignored, copy deploy/.env.example).
 name: talentsync
 
@@ -86,5 +86,16 @@ services:
     ports:
       - "5075:8080"
 
+  # Traces, metrics and logs of the three hosts (ADR 010), UI on http://localhost:18888. They export over OTLP/gRPC to
+  # 4317 (OTEL_EXPORTER_OTLP_ENDPOINT in their launchSettings.json). Data is in memory and is lost on restart.
+  # Anonymous access: local only, never expose this port.
+  aspire-dashboard:
+    image: mcr.microsoft.com/aspire/dashboard:13.6.1
+    environment:
+      ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS: "true"
+    ports:
+      - "18888:18888"
+      - "4317:18889"
+
 volumes:
   sql-data:

deploy/run-local.sh

@@ -30,13 +30,13 @@ if command -v wt.exe >/dev/null 2>&1; then
       cmd /k dotnet run --no-build --launch-profile http --project "${projects[$i]}"
     echo "Started ${titles[$i]}"
   done
-  echo "Click through the flow with docs/e2e.http."
+  echo "Click through the flow with docs/e2e.http. Traces and logs: http://localhost:18888"
 else
   trap 'kill $(jobs -p) 2>/dev/null' INT TERM EXIT
   for i in "${!titles[@]}"; do
     dotnet run --no-build --launch-profile http --project "${projects[$i]}" 2>&1 | sed -u "s/^/[${titles[$i]}] /" &
     echo "Started ${titles[$i]}"
   done
-  echo "Click through the flow with docs/e2e.http. Ctrl+C stops all three hosts."
+  echo "Click through the flow with docs/e2e.http. Traces and logs: http://localhost:18888. Ctrl+C stops all three hosts."
   wait
 fi

docs/acceptance-scenarios.md

@@ -193,3 +193,17 @@ Scenario: I4 Stale timestamp is rejected
   Then Ingress responds 401
   And ingress.Outbox has no row for that EventId
 ```
+
+## Observability
+
+The test records spans in process with an `ActivityListener`; no exporter, no dashboard.
+
+```gherkin
+# Illustrates: ADR 010 (one trace per webhook, propagated across the outbox and Event Hubs)
+Scenario: O1 One webhook produces one trace through Ingress, the relay and both workers
+  When candidate "c-o1" applies in FakeTeamtailor
+  Then the request span of its candidate.create webhook is the root of a trace
+  And the webhook span of that event is a child of the request span
+  And the send span of that event is a child of the webhook span
+  And every process span of that event in "sync-worker" and in "matching-worker" is a child of the send span
+```

docs/adr/010-observability.md

@@ -14,32 +14,76 @@ at Event Hubs (the envelope carries ids only) and in the retry loop (one call ca
 ## Decision
 
 - **OpenTelemetry over OTLP**, W3C trace context. Locally the Aspire Dashboard runs in docker compose. The exporter
-  is registered only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set, so tests and CI export nothing. Simulators are not
-  instrumented: they stand in for external systems.
+  is registered only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set, so tests and CI export nothing; `OTEL_SERVICE_NAME`
+  is then required. Simulators are not instrumented: they stand in for external systems.
 - **One project, `TalentSync.Infrastructure.Observability`** (`AddObservability`), referenced only by the hosts.
-  It holds one concern, so it is not a Common project (ADR 001). Our spans and metrics use the BCL `ActivitySource`
-  and `Meter` where the work happens; the SDK subscribes to `TalentSync.*`. No telemetry in `*.Domain` or `*.Application`.
-- **Explicit propagation, one trace per webhook.** Ingress stores the trace context in `ingress.Outbox.TraceParent`.
-  The relay's publish span is a child of it and writes `traceparent` to `EventData.Properties`, never to the envelope.
+  It holds one concern, so it is not a Common project (ADR 001). The SDK subscribes to `TalentSync.*`.
+  No telemetry in `*.Domain` or `*.Application`.
+- **One telemetry class per context, where the work happens:** `IngressTelemetry`, `ConsumerTelemetry`
+  (`Infrastructure.Messaging`), `MatchingTelemetry`. Each holds a static BCL `ActivitySource` and its instruments,
+  created through `IMeterFactory`, is a singleton injected into the code it observes, and never throws: an unknown
+  outcome is tagged `unknown`, never an exception that would fail the event.
+- **The LLM call is instrumented by `OpenTelemetryChatClient`** (`Microsoft.Extensions.AI`, gen_ai semantic
+  conventions: the `chat` span, token usage and duration) under the source name `TalentSync.Matching`.
+- **SQL calls are spans** through the SqlClient instrumentation, which also sees EF Core's commands.
+- **Names follow the semantic conventions** where they exist: `send ats-events` and `process ats-events` with
+  `messaging.*` attributes, `error.type`, `gen_ai.*`. Our own attributes are `talentsync.<concern>.<name>`
+  (`talentsync.consumer.outcome`, `talentsync.enrichment.outcome`); ids are `talentsync.event_id`,
+  `talentsync.event_type`, `talentsync.candidate_id` in spans and logs alike.
+- **Explicit propagation, one trace per webhook.** Our hosts are the edge and ignore the trace context of incoming
+  requests: the sender's context is untrusted, and one sender trace can carry several webhooks. So the webhook
+  request span is the root of a new trace, and the webhook span is its child.
+  Ingress stores the webhook span's context in `ingress.Outbox.TraceParent`.
+  The relay's send span is a child of it and writes `traceparent` to `EventData.Properties`, never to the envelope.
   The consumer span is its child. A missing or invalid value starts a new trace and never fails the event.
 - **One consumer span per attempt**, tagged with the outcome (`handled`, `skipped_duplicate`, `parked`, ...).
   A span is exported when it ends, so one span around the retry loop would stay invisible during an outage.
-- **Logs carry the trace** (OpenTelemetry logging provider) and a scope with `EventId`, `EventType`, `CandidateId`.
-- **Invariant 14 covers telemetry:** ids only; never the CV, the prompt, model output or response bodies.
+- **Metrics** answer what a trace cannot: consumer attempts by outcome, event age from webhook to effect, outbox
+  publish delay, enrichment calls by outcome, and the gen_ai token usage and call duration. The LLM call is a span;
+  a reused stored suggestion has none. Our histograms are in seconds and set their bucket boundaries as instrument
+  advice: the SDK's default buckets are sized for milliseconds.
+- **Logs carry the trace** (OpenTelemetry logging provider) and a scope with the event's ids, named like the span tags.
+- **Invariant 14 covers telemetry:** ids only; never the CV, the prompt, model output or response bodies. A failed span
+  carries the error status and `error.type`, never the exception message, which can echo a request: the message and
+  stack trace are in the log, which carries the trace id. `SpanExportPolicy` clears the status description of every
+  span before export, ours and the libraries' alike. `OpenTelemetryChatClient.EnableSensitiveData` is set to false
+  explicitly, so `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` cannot turn it on.
+- **A client span without a parent is not exported** (`SpanExportPolicy`): the relay polling the outbox, migrations.
+  It belongs to no flow, and every poll would be a trace.
+- **The checkpoint is our own span**, `checkpoint ats-events` (a settle operation), a child of the attempt that writes
+  it, with the Blob container that holds it in `talentsync.checkpoint.container`. The Blob SDK's HTTP requests are filtered out by their user agent: ownership polling and a checkpoint's
+  update-or-create round trips, whose first 404 per partition is expected, are the SDK's internals.
 
 ## Consequences
 
-- One trace shows the request, the outbox delay, both consumer groups and every HTTP and LLM call.
+- One trace shows the request, the outbox delay, both consumer groups, every HTTP, SQL and LLM call, and each
+  checkpoint.
 - A relay double publish shows up in the same trace as a `skipped_duplicate` branch. A webhook that Teamtailor sends
   twice is a new trace, found by the `EventId` tag.
 - One more column and migration in `ingress`. Rows published before it start their own trace.
 - Consumer spans can start long after their parent ended (outage, replay). Valid, but long traces are harder to read.
-- OpenTelemetry packages in one project, one more container in compose.
+- OpenTelemetry packages in one project, one more container in compose, `Microsoft.Extensions.AI` in
+  `Matching.Infrastructure`. The gen_ai conventions are still experimental, so their names can change with the package.
+- Our hosts ignore the sender's trace context only because the ASP.NET Core instrumentation does not extract it again
+  when the propagator is exactly `TraceContextPropagator`. That is the instrumentation's internal behaviour, pinned by
+  `EdgeTraceTests` with the SDK enabled; scenario O1 runs without it.
+- HTTP client metrics include the Blob SDK's polling: unlike spans, they cannot be filtered per request.
+- The checkpoint store's latency is in the `checkpoint` span only; a slow Blob request has no span of its own.
 
 ## Alternatives considered
 
 - **The Event Hubs SDK's own propagation.** Rejected: experimental switch, hidden in a library.
 - **Trace context in `EventEnvelope`.** Rejected: the envelope is a business contract, not transport metadata.
 - **A span link instead of a parent at the outbox.** Rejected: the dashboard would split one webhook into separate traces.
+- **Continuing the sender's trace.** Rejected: an external caller would choose our trace id and its sampling.
+- **A new trace at the webhook span, linked to the request span.** Rejected: dashboards show links only in span
+  details, so the request and the flow it starts would be separate traces.
 - **Setup duplicated in each host.** Rejected: three copies would drift.
+- **A hand-written gen_ai decorator.** Rejected: `OpenTelemetryChatClient` implements the conventions, metrics
+  included, for one package.
+- **EF Core instrumentation.** Rejected: still a beta package; the SqlClient instrumentation is stable and sees EF's
+  commands.
+- **Static `Meter`s.** Rejected: `IMeterFactory` ties the meters to the host's container, and tests create their own.
+- **The exception on the span (`AddException`).** Rejected: its message can carry data from the request (invariant 14),
+  and the log already holds it under the same trace id.
 - **Seq or Grafana.** Seq covers logs only; Grafana is heavier. Production only needs another OTLP endpoint.