Devops/ops004 Observation stack intergration: Serilogs with OpenTelemetry, Grafana, Loki, Tempo. Prometheus - #49
Conversation
…ki, Tempo, Prometheus, OTEL collector)
📝 WalkthroughWalkthroughThis pull request adds OpenTelemetry observability infrastructure to the SnakeAid.Api application. It introduces tracing and metrics collection through OpenTelemetry, configures exporters for Loki, Tempo, and Prometheus, and provides a complete Docker Compose stack with Grafana for visualization and monitoring. Changes
Sequence DiagramsequenceDiagram
participant API as SnakeAid API
participant Instrumentation as OpenTelemetry<br/>Instrumentation
participant OTLPCollector as OTLP Collector
participant Tempo as Tempo
participant Loki as Loki
participant Prometheus as Prometheus
participant Grafana as Grafana
API->>Instrumentation: Emit traces & metrics<br/>(ASP.NET, HTTP, Npgsql)
Instrumentation->>OTLPCollector: Send via OTLP<br/>(gRPC/HTTP)
OTLPCollector->>Tempo: Export traces
OTLPCollector->>Prometheus: Export metrics<br/>(scrape endpoint)
Note over API: Logs via Serilog<br/>GrafanaLoki sink
API->>Loki: Send structured logs
Grafana->>Tempo: Query traces
Grafana->>Loki: Query logs
Grafana->>Prometheus: Query metrics
Grafana->>Grafana: Visualize &<br/>correlate data
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Fix all issues with AI agents
In `@docker-compose.grafana.yml`:
- Around line 62-64: The compose file currently enables anonymous admin access
via GF_AUTH_ANONYMOUS_ENABLED=true, GF_AUTH_ANONYMOUS_ORG_ROLE=Admin and
disables login with GF_AUTH_DISABLE_LOGIN_FORM=true; change these for safety by
setting GF_AUTH_ANONYMOUS_ENABLED=false (or removing the env var), set
GF_AUTH_ANONYMOUS_ORG_ROLE to a least-privilege role like Viewer if anonymous
must remain, and enable the login form (GF_AUTH_DISABLE_LOGIN_FORM=false);
additionally move any dev-only anonymous settings into a
docker-compose.override.yml or an .env.dev file and ensure production
deployments do not include these vars.
In `@observability/grafana-datasources.yaml`:
- Around line 3-37: Tempo's tracesToLogs/tracesToMetrics fail because Prometheus
and Loki datasources lack explicit uids; add uid: 'Prometheus' to the Prometheus
datasource block and uid: 'Loki' to the Loki datasource block so the
tracesToMetrics.datasourceUid and tracesToLogs.datasourceUid resolve, and also
remove the irrelevant graphiteVersion entry from Prometheus.jsonData (the keys
to edit are the Prometheus and Loki datasource definitions and the jsonData
object under Prometheus).
In `@observability/loki-config.yaml`:
- Around line 30-31: The ruler configuration includes alertmanager_url pointing
to localhost which doesn't match our Docker Compose services; either remove the
entire ruler block (the "ruler:" stanza) from loki-config.yaml if you don't
intend to use Alertmanager, or update alertmanager_url to point to the
Alertmanager service name used in Compose (e.g., change alertmanager_url to
"http://alertmanager:9093") and add a matching Alertmanager service to Docker
Compose; ensure the symbol names "ruler" and "alertmanager_url" are updated
consistently with the Compose service name.
- Around line 20-28: The current schema_config only contains a boltdb-shipper
entry with schema v11 which is deprecated in Loki 3.x; to migrate safely, keep
the existing schema_config block exactly as-is (the boltdb-shipper / schema: v11
entry) and add a second schema_config entry with store: tsdb, schema: v13 and a
future from date (e.g., 2026-03-01) so new data uses TSDB while old queries
still work; also add tsdb_shipper settings in storage_config (configure
active_index_directory and cache_location) and ensure the compactor is
configured to run with TSDB for proper retention and query performance (avoid
setting the TSDB from date earlier than needed).
In `@observability/tempo-config.yaml`:
- Around line 24-25: The overrides block enables metrics_generator_processors
(service-graphs, span-metrics) but there is no metrics_generator configuration,
so add a metrics_generator block that defines at minimum
metrics_generator.storage.path for the WAL directory and
metrics_generator.storage.remote_write pointing to a Prometheus-compatible
remote write endpoint (e.g., http://prometheus:9090/api/v1/write); ensure the
new metrics_generator block is present alongside overrides so the service-graphs
and span-metrics processors can function correctly.
In `@SnakeAid.Api/appsettings.Example.json`:
- Around line 40-59: The example config's GrafanaLoki requestUri ("requestUri"
inside the "GrafanaLoki" entry) uses the Docker-only hostname http://loki:3100
which will fail for local (non-container) development; update the example to use
http://localhost:3100 as the default requestUri or add a clear inline comment
above the "GrafanaLoki" block explaining that http://loki:3100 is valid only
inside Docker Compose and developers running the API locally should use
http://localhost:3100 (refer to the "GrafanaLoki" name and its "Args.requestUri"
field).
In `@SnakeAid.Api/DI/DependencyInjection.cs`:
- Around line 289-321: The metrics builder in AddOpenTelemetryServices is
missing a ResourceBuilder so exported metrics won't have service.name; update
the metrics configuration (metricsProviderBuilder) to use the same
ResourceBuilder used for tracing (extract
ResourceBuilder.CreateDefault().AddService("SnakeAid.Api") into a local
variable) and apply it to the metrics provider just like
tracerProviderBuilder.SetResourceBuilder(...), ensuring the AddOtlpExporter call
on metricsProviderBuilder sends metrics with the service resource attached.
In `@SnakeAid.Api/Program.cs`:
- Line 108: Add validation around the OpenTelemetry endpoint inside the
AddOpenTelemetryServices implementation: check the configured string (used to
create the Uri in AddOpenTelemetryServices) with Uri.TryCreate and ensure
AbsoluteUri (or catch FormatException) and, if invalid, throw an
ArgumentException or log an explicit error describing the misconfigured
OpenTelemetry:Endpoint instead of letting new Uri(...) throw; update the code
path that currently does new Uri(otlpEndpoint) to use TryCreate and a clear
failure message indicating the invalid URI and the expected format.
🧹 Nitpick comments (7)
SnakeAid.Api/appsettings.Example.json (1)
70-72: Same Docker-internal hostname concern for OpenTelemetry endpoint.
http://otel-collector:4317only resolves inside Docker. TheDependencyInjection.cshas alocalhost:4317fallback, so this is partially mitigated, but the example config should note this is for the Docker Compose setup.observability/otel-collector-config.yaml (1)
19-28: Consider adding adebugexporter for collector troubleshooting.When debugging the observability stack locally, having a
debug(formerlylogging) exporter in the service pipelines can help diagnose data flow issues. This is optional for a dev-only stack.♻️ Example addition
exporters: otlp: endpoint: tempo:4317 tls: insecure: true prometheus: endpoint: "0.0.0.0:8889" namespace: "snakeaid_api" + debug: + verbosity: basic service: pipelines: traces: receivers: [otlp] processors: [batch] - exporters: [otlp] # To Tempo (via OTLP) + exporters: [otlp, debug] # To Tempo (via OTLP) metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] # Expose via Prometheus scrape endpointSnakeAid.Api/DI/DependencyInjection.cs (2)
294-294: MalformedOpenTelemetry:Endpointconfig will throw an unhandledUriFormatExceptionat startup.If someone misconfigures the endpoint value,
new Uri(otlpEndpoint)on lines 307/315 will throw with no clear indication of which setting is wrong. A guard or early validation with a descriptive error message would improve the developer experience.🛡️ Proposed validation
var otlpEndpoint = configuration["OpenTelemetry:Endpoint"] ?? "http://localhost:4317"; + +if (!Uri.TryCreate(otlpEndpoint, UriKind.Absolute, out var otlpUri)) +{ + throw new InvalidOperationException( + $"Invalid OpenTelemetry:Endpoint URI: '{otlpEndpoint}'. Expected an absolute URI (e.g. http://localhost:4317)."); +}Then use
otlpUriin both exporter configs instead ofnew Uri(otlpEndpoint).
300-300: Hardcoded service name — consider reading it from configuration or assembly metadata.
"SnakeAid.Api"is repeated inAddSource,AddService, andAddMeter. If the service name ever changes or you want per-environment naming, a single constant or config value would be cleaner.docker-compose.grafana.yml (3)
2-16:snakeaid-apihas nodepends_onforotel-collector— early traces/metrics may be lost.The API will start sending OTLP data immediately, but if the collector isn't ready yet, those initial spans and metrics will be dropped. Consider adding a dependency:
🔧 Proposed fix
snakeaid-api: image: thekhiem7/snakeaid-api:latest pull_policy: always build: context: . dockerfile: Dockerfile ports: - "8080:8080" environment: - DOPPLER_TOKEN=${DOPPLER_TOKEN} - DOPPLER_CONFIG=snake-aid/dev restart: unless-stopped + depends_on: + - otel-collector networks: - snakeaid-net
31-33: Bind-mounting./observability/tempo-datawill create a local directory with Tempo's data — make sure it's gitignored.This directory will accumulate trace data locally. If it's not in
.gitignore, it could accidentally get committed.#!/bin/bash # Check if tempo-data is gitignored rg -n 'tempo-data' .gitignore || echo "tempo-data is NOT in .gitignore"
19-19: All images use:latesttags — builds won't be reproducible.Pinning image versions (e.g.,
grafana/loki:2.9.4) ensures the stack behaves consistently across environments and avoids surprise breakage from upstream changes. This is especially relevant for the otel-collector, which has had breaking config changes between versions.Also applies to: 29-29, 44-44, 56-56, 69-69
| - GF_AUTH_ANONYMOUS_ENABLED=true | ||
| - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin | ||
| - GF_AUTH_DISABLE_LOGIN_FORM=true |
There was a problem hiding this comment.
Grafana is wide-open with anonymous Admin access — acceptable for local dev only.
Anonymous access with Admin role and login disabled means anyone with network access has full Grafana admin rights. This is fine for a local dev stack but ensure this compose file is never used in staging/production or exposed publicly (e.g., via a cloud VM with open ports).
🤖 Prompt for AI Agents
In `@docker-compose.grafana.yml` around lines 62 - 64, The compose file currently
enables anonymous admin access via GF_AUTH_ANONYMOUS_ENABLED=true,
GF_AUTH_ANONYMOUS_ORG_ROLE=Admin and disables login with
GF_AUTH_DISABLE_LOGIN_FORM=true; change these for safety by setting
GF_AUTH_ANONYMOUS_ENABLED=false (or removing the env var), set
GF_AUTH_ANONYMOUS_ORG_ROLE to a least-privilege role like Viewer if anonymous
must remain, and enable the login form (GF_AUTH_DISABLE_LOGIN_FORM=false);
additionally move any dev-only anonymous settings into a
docker-compose.override.yml or an .env.dev file and ensure production
deployments do not include these vars.
| datasources: | ||
| - name: Prometheus | ||
| type: prometheus | ||
| access: proxy | ||
| url: http://prometheus:9090 | ||
| isDefault: false | ||
| jsonData: | ||
| graphiteVersion: '1.1' | ||
|
|
||
| - name: Loki | ||
| type: loki | ||
| access: proxy | ||
| url: http://loki:3100 | ||
| isDefault: true | ||
|
|
||
| - name: Tempo | ||
| type: tempo | ||
| access: proxy | ||
| url: http://tempo:3200 | ||
| jsonData: | ||
| tracesToLogs: | ||
| datasourceUid: 'Loki' | ||
| tags: ['job', 'instance', 'pod', 'namespace'] | ||
| mappedTags: [{ key: 'service.name', value: 'app' }] | ||
| mapTagNamesEnabled: false | ||
| spanStartTimeShift: '0' | ||
| spanEndTimeShift: '0' | ||
| filterByTraceID: true | ||
| filterBySpanID: true | ||
| tracesToMetrics: | ||
| datasourceUid: 'Prometheus' | ||
| tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }] | ||
| queries: | ||
| - name: 'Sample query' | ||
| query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))' |
There was a problem hiding this comment.
Tempo's datasourceUid references won't resolve — datasource UIDs are not explicitly set.
Lines 24 and 33 reference datasourceUid: 'Loki' and datasourceUid: 'Prometheus', but neither the Loki nor Prometheus datasource definitions include an explicit uid field. Grafana auto-generates UIDs (they won't be "Loki" or "Prometheus"), so the traces-to-logs and traces-to-metrics links will silently fail.
🔧 Proposed fix — add explicit UIDs
- name: Prometheus
type: prometheus
access: proxy
+ uid: Prometheus
url: http://prometheus:9090
isDefault: false
- jsonData:
- graphiteVersion: '1.1'
- name: Loki
type: loki
access: proxy
+ uid: Loki
url: http://loki:3100
isDefault: trueAlso: the graphiteVersion under Prometheus jsonData is not relevant for a Prometheus-type datasource and can be removed.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| datasources: | |
| - name: Prometheus | |
| type: prometheus | |
| access: proxy | |
| url: http://prometheus:9090 | |
| isDefault: false | |
| jsonData: | |
| graphiteVersion: '1.1' | |
| - name: Loki | |
| type: loki | |
| access: proxy | |
| url: http://loki:3100 | |
| isDefault: true | |
| - name: Tempo | |
| type: tempo | |
| access: proxy | |
| url: http://tempo:3200 | |
| jsonData: | |
| tracesToLogs: | |
| datasourceUid: 'Loki' | |
| tags: ['job', 'instance', 'pod', 'namespace'] | |
| mappedTags: [{ key: 'service.name', value: 'app' }] | |
| mapTagNamesEnabled: false | |
| spanStartTimeShift: '0' | |
| spanEndTimeShift: '0' | |
| filterByTraceID: true | |
| filterBySpanID: true | |
| tracesToMetrics: | |
| datasourceUid: 'Prometheus' | |
| tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }] | |
| queries: | |
| - name: 'Sample query' | |
| query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))' | |
| datasources: | |
| - name: Prometheus | |
| type: prometheus | |
| access: proxy | |
| uid: Prometheus | |
| url: http://prometheus:9090 | |
| isDefault: false | |
| - name: Loki | |
| type: loki | |
| access: proxy | |
| uid: Loki | |
| url: http://loki:3100 | |
| isDefault: true | |
| - name: Tempo | |
| type: tempo | |
| access: proxy | |
| url: http://tempo:3200 | |
| jsonData: | |
| tracesToLogs: | |
| datasourceUid: 'Loki' | |
| tags: ['job', 'instance', 'pod', 'namespace'] | |
| mappedTags: [{ key: 'service.name', value: 'app' }] | |
| mapTagNamesEnabled: false | |
| spanStartTimeShift: '0' | |
| spanEndTimeShift: '0' | |
| filterByTraceID: true | |
| filterBySpanID: true | |
| tracesToMetrics: | |
| datasourceUid: 'Prometheus' | |
| tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }] | |
| queries: | |
| - name: 'Sample query' | |
| query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))' |
🤖 Prompt for AI Agents
In `@observability/grafana-datasources.yaml` around lines 3 - 37, Tempo's
tracesToLogs/tracesToMetrics fail because Prometheus and Loki datasources lack
explicit uids; add uid: 'Prometheus' to the Prometheus datasource block and uid:
'Loki' to the Loki datasource block so the tracesToMetrics.datasourceUid and
tracesToLogs.datasourceUid resolve, and also remove the irrelevant
graphiteVersion entry from Prometheus.jsonData (the keys to edit are the
Prometheus and Loki datasource definitions and the jsonData object under
Prometheus).
| schema_config: | ||
| configs: | ||
| - from: 2020-10-24 | ||
| store: boltdb-shipper | ||
| object_store: filesystem | ||
| schema: v11 | ||
| index: | ||
| prefix: index_ | ||
| period: 24h |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Loki 3.0 boltdb-shipper deprecated TSDB schema v13 migration
💡 Result:
In Loki 3.x, BoltDB / boltdb-shipper is deprecated, and Grafana recommends migrating indexing to TSDB (current development focus). Boltdb-shipper “may be removed in future major versions of Loki.” [1][2]
Recommended migration approach (boltdb-shipper → TSDB, schema v13)
1) Keep your existing boltdb-shipper period(s) unchanged (for old data)
Do not modify historical schema_config entries (store/schema/prefix/period). Those must remain as-is if you want to keep querying old logs. [2][3][4]
2) Add a new future-dated period_config that switches to TSDB + v13
Add another entry to schema_config.configs with:
from: a date/time in the future (relative to when you deploy the config)store: tsdbschema: v13(recommended in the TSDB migration docs)- same
object_store(unless you intentionally change it)
Example (adapt to your object store, prefixes, and dates): [1]
schema_config:
configs:
- from: 2023-01-01
store: boltdb-shipper
object_store: s3 # or gcs/filesystem/etc
schema: v11 # or your current schema
index:
prefix: index_
period: 24h
- from: 2026-03-01 # MUST be in the future when rolled out
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24hCommon failure mode: if the TSDB from date is wrong (too early), Loki will try to read older data using the TSDB rules and you’ll “lose” old logs in queries until you fix the date. [3]
3) Configure tsdb_shipper in storage_config
At minimum configure:
active_index_directorycache_location[1]
Example: [1]
storage_config:
tsdb_shipper:
active_index_directory: /data/tsdb-index
cache_location: /data/tsdb-cache4) Run the compactor (strongly recommended for TSDB)
Grafana strongly recommends running the compactor with TSDB for compaction/retention and better query performance. [1]
Notes specific to “schema v13”
- The migration docs explicitly show TSDB with
schema: v13as the target schema. [1] - Ensure your old period keeps its original schema (for example v11/v12), otherwise you can break querying of historical data. [4]
If you share your current schema_config + storage_config (and object store: S3/GCS/Azure/filesystem), I can point out the exact minimal diff for a safe cutover date.
boltdb-shipper with schema v11 is deprecated in Loki 3.x; migrate to TSDB with schema v13.
Do not replace the existing boltdb-shipper entry directly—this breaks queries on historical data. Instead:
- Keep the current
boltdb-shipperentry unchanged (required for querying old logs) - Add a new
schema_configentry withstore: tsdb,schema: v13, and a futurefromdate (e.g.,2026-03-01) - Configure
tsdb_shipperinstorage_configwithactive_index_directoryandcache_location - Consider running the compactor with TSDB for better retention and query performance
Setting the TSDB from date incorrectly (too early) will prevent queries on historical data until corrected.
🤖 Prompt for AI Agents
In `@observability/loki-config.yaml` around lines 20 - 28, The current
schema_config only contains a boltdb-shipper entry with schema v11 which is
deprecated in Loki 3.x; to migrate safely, keep the existing schema_config block
exactly as-is (the boltdb-shipper / schema: v11 entry) and add a second
schema_config entry with store: tsdb, schema: v13 and a future from date (e.g.,
2026-03-01) so new data uses TSDB while old queries still work; also add
tsdb_shipper settings in storage_config (configure active_index_directory and
cache_location) and ensure the compactor is configured to run with TSDB for
proper retention and query performance (avoid setting the TSDB from date earlier
than needed).
| ruler: | ||
| alertmanager_url: http://localhost:9093 |
There was a problem hiding this comment.
alertmanager_url references a non-existent service.
http://localhost:9093 points to an Alertmanager that isn't part of the Docker Compose stack. This will cause ruler errors if alerting rules are configured. Either remove the ruler block or add an Alertmanager service.
🤖 Prompt for AI Agents
In `@observability/loki-config.yaml` around lines 30 - 31, The ruler configuration
includes alertmanager_url pointing to localhost which doesn't match our Docker
Compose services; either remove the entire ruler block (the "ruler:" stanza)
from loki-config.yaml if you don't intend to use Alertmanager, or update
alertmanager_url to point to the Alertmanager service name used in Compose
(e.g., change alertmanager_url to "http://alertmanager:9093") and add a matching
Alertmanager service to Docker Compose; ensure the symbol names "ruler" and
"alertmanager_url" are updated consistently with the Compose service name.
| overrides: | ||
| metrics_generator_processors: [service-graphs, span-metrics] |
There was a problem hiding this comment.
❓ Verification inconclusive
Grafana Tempo metrics_generator configuration remote_write requirements
Tempo’s metrics-generator remote_write is essentially a Prometheus Agent that uses a WAL and then remote-writes generated metrics to a Prometheus-compatible endpoint. The practical requirements are:
- You must set a local WAL directory:
metrics_generator.storage.path(Tempo docs indicate the WAL path “must be set” / is required for this storage block). [1][2] - You must configure at least one remote_write endpoint under
metrics_generator.storage.remote_write, using standard Prometheusremote_writeconfig (at minimum, aurl, typically pointing to a Prometheus-compatible/api/v1/write). [1][2] - The remote_write receiver must accept what you send:
- If you enable exemplars (
send_exemplars: true), the backend must support ingesting exemplars. Example configs show this option commonly enabled. [2] - If you enable native histograms, the receiving metrics backend must be configured to ingest them. [3]
- If you enable exemplars (
- Multi-tenancy behavior:
- By default, the metrics-generator forwards
X-Scope-OrgIDto the remote_write endpoint (can be disabled withremote_write_add_org_id_header: false). [1][3] - If you need per-tenant auth headers, configure
remote_write_headersvia overrides (requires Tempo 2.4.0+). [4]
- By default, the metrics-generator forwards
Minimal example (illustrative, from Grafana Tempo docs):
metrics_generator:
storage:
path: /tmp/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true[2]
Sources:
[1] Grafana Tempo configuration (metrics-generator storage/remote_write) https://grafana.com/docs/tempo/latest/configuration/
[2] Grafana Tempo “Deploy on Linux” example config (metrics_generator.storage.path + remote_write.url) https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/locally/linux/
[3] Metrics-generator docs (remote write + native histograms note) https://grafana.com/docs/tempo/latest/metrics-from-traces/metrics-generator/
[4] Metrics-generator multitenancy support (remote_write_headers, Tempo >=2.4.0) https://grafana.com/docs/tempo/latest/metrics-from-traces/metrics-generator/multitenancy/
Add metrics_generator configuration block to enable metrics processors.
The metrics_generator_processors override enables service-graphs and span-metrics generation, but without a corresponding metrics_generator block, these processors won't actually function. Tempo requires at minimum:
metrics_generator.storage.path: WAL directory for metrics bufferingmetrics_generator.storage.remote_write: Prometheus-compatible remote write endpoint (e.g.,http://prometheus:9090/api/v1/write)
Without these, the processor setting is ineffective.
🤖 Prompt for AI Agents
In `@observability/tempo-config.yaml` around lines 24 - 25, The overrides block
enables metrics_generator_processors (service-graphs, span-metrics) but there is
no metrics_generator configuration, so add a metrics_generator block that
defines at minimum metrics_generator.storage.path for the WAL directory and
metrics_generator.storage.remote_write pointing to a Prometheus-compatible
remote write endpoint (e.g., http://prometheus:9090/api/v1/write); ensure the
new metrics_generator block is present alongside overrides so the service-graphs
and span-metrics processors can function correctly.
| { | ||
| "Name": "GrafanaLoki", | ||
| "Args": { | ||
| "requestUri": "http://loki:3100", | ||
| "labels": [ | ||
| { | ||
| "key": "app", | ||
| "value": "SnakeAid.API" | ||
| }, | ||
| { | ||
| "key": "environment", | ||
| "value": "Development" | ||
| } | ||
| ], | ||
| "propertiesAsLabels": ["Application", "Environment"], | ||
| "queueLimit": 10000, | ||
| "batchPostingLimit": 1000, | ||
| "period": "00:00:02" | ||
| } | ||
| }, |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Serilog.Sinks.Grafana.Loki requestUri endpoint configuration format
💡 Result:
In Serilog.Sinks.Grafana.Loki the configured uri/Url is the root/base address of the Loki server (scheme + host + optional port). The sink then posts to Loki’s push endpoint POST /loki/api/v1/push. [1] [2]
Example (appsettings.json)
{
"Serilog": {
"Using": [ "Serilog.Sinks.Grafana.Loki" ],
"WriteTo": [
{
"Name": "GrafanaLoki",
"Args": {
"uri": "http://localhost:3100"
}
}
]
}
}uri is documented as “the root URI of Loki”. [1] (and the README shows the same format). [2]
The actual HTTP endpoint Loki expects
Loki’s ingest endpoint is:
POST /loki/api/v1/push[3]
So with uri = http://localhost:3100, the sink will send to:
http://localhost:3100/loki/api/v1/push(conceptually: base +/loki/api/v1/push) [3]
Note if you’re sending to Grafana Agent / Alloy “loki_push_api”
Some agent configs use /api/v1/push (without the /loki prefix). If you point the sink at an agent endpoint that only accepts /api/v1/push, you can get 404s because the sink targets /loki/api/v1/push. [4]
Sources:
[1] Serilog.Sinks.Grafana.Loki wiki “Application settings” (parameter uri = root URI of Loki) (github.com)
[2] serilog-contrib/serilog-sinks-grafana-loki README (examples use "uri": "http://localhost:3100") (github.com)
[3] Grafana Loki HTTP API docs (ingest endpoint POST /loki/api/v1/push) (grafana.com)
[4] Issue report describing /api/v1/push vs /loki/api/v1/push mismatch with Grafana Agent (lightrun.com)
Citations:
- 1: https://github.com/mishamyte/serilog-sinks-grafana-loki/wiki/Application-settings
- 2: https://github.com/serilog-contrib/serilog-sinks-grafana-loki
- 3: https://grafana.com/docs/loki/latest/reference/loki-http-api/?utm_source=openai
- 4: https://lightrun.com/answers/serilog-contrib-serilog-sinks-grafana-loki-serilog-does-not-work-with-grafana-agent-due-to-invalid-loki-url?utm_source=openai
🏁 Script executed:
# Search for configuration files and patterns in the codebase
find . -name "appsettings*.json" -o -name "*.Example.json" | head -20Repository: Snake-AID/SnakeAid.Backend
Length of output: 109
🏁 Script executed:
# Look for Serilog configuration or documentation in the repo
rg -i "loki|grafana" --type json -A 3 -B 3 | head -100Repository: Snake-AID/SnakeAid.Backend
Length of output: 52
🏁 Script executed:
# Search for any comments or documentation about local development setup
rg -i "docker|localhost|environment" SnakeAid.Api/ | head -50Repository: Snake-AID/SnakeAid.Backend
Length of output: 2955
🏁 Script executed:
# Search for all appsettings files
fd "appsettings" SnakeAid.Api/Repository: Snake-AID/SnakeAid.Backend
Length of output: 52
🏁 Script executed:
# Check how appsettings are loaded in Program.cs
rg -A 10 "appsettings|Configuration" SnakeAid.Api/Program.cs | head -60Repository: Snake-AID/SnakeAid.Backend
Length of output: 2379
Loki requestUri uses Docker-internal hostname — won't work for local development outside containers.
The URI http://loki:3100 only resolves within the Docker network. Developers running the API locally will get connection failures. Since this is an example file, consider using http://localhost:3100 as the default, or add a comment noting this configuration is for Docker Compose environments only.
🤖 Prompt for AI Agents
In `@SnakeAid.Api/appsettings.Example.json` around lines 40 - 59, The example
config's GrafanaLoki requestUri ("requestUri" inside the "GrafanaLoki" entry)
uses the Docker-only hostname http://loki:3100 which will fail for local
(non-container) development; update the example to use http://localhost:3100 as
the default requestUri or add a clear inline comment above the "GrafanaLoki"
block explaining that http://loki:3100 is valid only inside Docker Compose and
developers running the API locally should use http://localhost:3100 (refer to
the "GrafanaLoki" name and its "Args.requestUri" field).
|
|
||
| #region OpenTelemetry | ||
|
|
||
| public static IServiceCollection AddOpenTelemetryServices(this IServiceCollection services, IConfiguration configuration) | ||
| { | ||
| var otlpEndpoint = configuration["OpenTelemetry:Endpoint"] ?? "http://localhost:4317"; // fallback for local without docker | ||
|
|
||
| services.AddOpenTelemetry() | ||
| .WithTracing(tracerProviderBuilder => | ||
| { | ||
| tracerProviderBuilder | ||
| .AddSource("SnakeAid.Api") | ||
| .SetResourceBuilder( | ||
| ResourceBuilder.CreateDefault() | ||
| .AddService("SnakeAid.Api")) | ||
| .AddAspNetCoreInstrumentation() | ||
| .AddHttpClientInstrumentation() | ||
| .AddNpgsql() | ||
| .AddOtlpExporter(opts => opts.Endpoint = new Uri(otlpEndpoint)); | ||
| }) | ||
| .WithMetrics(metricsProviderBuilder => | ||
| { | ||
| metricsProviderBuilder | ||
| .AddMeter("SnakeAid.Api") | ||
| .AddAspNetCoreInstrumentation() | ||
| .AddRuntimeInstrumentation() | ||
| .AddOtlpExporter(opts => opts.Endpoint = new Uri(otlpEndpoint)); | ||
| }); | ||
|
|
||
| return services; | ||
| } | ||
|
|
||
| #endregion |
There was a problem hiding this comment.
Metrics are missing ResourceBuilder — service name won't be attached to exported metrics.
Tracing configures .SetResourceBuilder(…).AddService("SnakeAid.Api") but the metrics builder does not. This means metrics exported via OTLP will lack the service.name resource attribute, making it harder to filter/correlate in Prometheus/Grafana.
🔧 Proposed fix
.WithMetrics(metricsProviderBuilder =>
{
metricsProviderBuilder
.AddMeter("SnakeAid.Api")
+ .SetResourceBuilder(
+ ResourceBuilder.CreateDefault()
+ .AddService("SnakeAid.Api"))
.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.AddOtlpExporter(opts => opts.Endpoint = new Uri(otlpEndpoint));
});Consider also extracting the shared ResourceBuilder into a local variable to avoid duplication:
var resourceBuilder = ResourceBuilder.CreateDefault().AddService("SnakeAid.Api");🤖 Prompt for AI Agents
In `@SnakeAid.Api/DI/DependencyInjection.cs` around lines 289 - 321, The metrics
builder in AddOpenTelemetryServices is missing a ResourceBuilder so exported
metrics won't have service.name; update the metrics configuration
(metricsProviderBuilder) to use the same ResourceBuilder used for tracing
(extract ResourceBuilder.CreateDefault().AddService("SnakeAid.Api") into a local
variable) and apply it to the metrics provider just like
tracerProviderBuilder.SetResourceBuilder(...), ensuring the AddOtlpExporter call
on metricsProviderBuilder sends metrics with the service resource attached.
| builder.Services.AddScoped<SnakeAid.Service.Implements.Email.EmailTemplateService>(); | ||
|
|
||
| builder.Services.AddServices(builder.Configuration); | ||
| builder.Services.AddOpenTelemetryServices(builder.Configuration); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -type f -name "*.cs" | xargs grep -l "AddOpenTelemetryServices" | head -20Repository: Snake-AID/SnakeAid.Backend
Length of output: 136
🏁 Script executed:
find . -type f -name "*DependencyInjection*" -o -name "*OpenTelemetry*" | head -20Repository: Snake-AID/SnakeAid.Backend
Length of output: 110
🏁 Script executed:
rg "AddOpenTelemetryServices" -A 30Repository: Snake-AID/SnakeAid.Backend
Length of output: 4821
The AddOpenTelemetryServices implementation handles missing configuration well but doesn't validate invalid URIs.
The method correctly uses a null-coalescing operator to default to http://localhost:4317 when OpenTelemetry:Endpoint is missing. However, if the configuration contains an invalid URI string (e.g., malformed value), the new Uri(otlpEndpoint) call will throw a FormatException at startup. Consider adding URI validation or error handling to fail gracefully with a clear error message when the endpoint is misconfigured.
🤖 Prompt for AI Agents
In `@SnakeAid.Api/Program.cs` at line 108, Add validation around the OpenTelemetry
endpoint inside the AddOpenTelemetryServices implementation: check the
configured string (used to create the Uri in AddOpenTelemetryServices) with
Uri.TryCreate and ensure AbsoluteUri (or catch FormatException) and, if invalid,
throw an ArgumentException or log an explicit error describing the misconfigured
OpenTelemetry:Endpoint instead of letting new Uri(...) throw; update the code
path that currently does new Uri(otlpEndpoint) to use TryCreate and a clear
failure message indicating the invalid URI and the expected format.
1 participant
This pull request introduces comprehensive observability support for the
SnakeAid.Apiproject by integrating OpenTelemetry tracing and metrics, enhancing logging with Grafana Loki, and providing a complete Docker Compose stack for local observability (Grafana, Loki, Tempo, Prometheus, and OpenTelemetry Collector). The most important changes are grouped below:OpenTelemetry instrumentation and configuration:
Npgsql.OpenTelemetry,OpenTelemetry.Exporter.OpenTelemetryProtocol,OpenTelemetry.Extensions.Hosting,OpenTelemetry.Instrumentation.AspNetCore,OpenTelemetry.Instrumentation.Http,OpenTelemetry.Instrumentation.Runtime) to bothDirectory.Packages.propsandSnakeAid.Api.csprojfor tracing and metrics support. [1] [2]AddOpenTelemetryServicesextension inDependencyInjection.cs, configuring tracing (ASP.NET Core, HTTP, Npgsql) and metrics, with OTLP exporter endpoint configurable via appsettings.Program.cs).appsettings.Example.jsonfor endpoint setup.Observability stack and configuration:
docker-compose.grafana.ymlfile to run the full observability stack (Grafana, Loki, Tempo, Prometheus, OpenTelemetry Collector, and the API itself) for local development and monitoring.observability/grafana-datasources.yamlfor Grafana data sources (Loki, Tempo, Prometheus)observability/loki-config.yamlfor Loki log storageobservability/tempo-config.yamlfor Tempo tracing backendobservability/prometheus.yamlfor Prometheus scraping OTEL Collector metricsobservability/otel-collector-config.yamlfor OpenTelemetry Collector pipelinesLogging enhancements:
Serilog.Sinks.Grafana.Lokito the logging configuration inappsettings.Example.jsonand provided a sample Loki sink configuration for structured log shipping. [1] [2]Documentation:
SnakeAid.Docssubmodule to the latest commit, which may include related documentation for observability.Summary by CodeRabbit
Release Notes
New Features
Chores