Skip to content

Devops/ops004 Observation stack intergration: Serilogs with OpenTelemetry, Grafana, Loki, Tempo. Prometheus #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
the-khiem7 wants to merge 6 commits into
base: dev
Choose a base branch
from
Open

Devops/ops004 Observation stack intergration: Serilogs with OpenTelemetry, Grafana, Loki, Tempo. Prometheus #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b389b41
docs: update SnakeAid.Docs submodule
the-khiem7 Feb 10, 2026
b245db0
Ignore local environment files
the-khiem7 Feb 11, 2026
908ba63
Add OpenTelemetry package versions and project references
the-khiem7 Feb 11, 2026
7a9a036
Register OpenTelemetry tracing and metrics and add default configuration
the-khiem7 Feb 11, 2026
78cc638
Configure Serilog to send logs to Grafana Loki
the-khiem7 Feb 11, 2026
1cd2c09
Add observability Docker Compose and configuration files (Grafana, Lo…
the-khiem7 Feb 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,9 @@ MigrationBackup/
# Fody - auto-generated XML schema
FodyWeavers.xsd

# Environment variables
.env

# appsettings.json

appsettings.json
Expand Down
6 changes: 6 additions & 0 deletions Directory.Packages.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@
<PackageVersion Include="MailKit" Version="4.11.0" />
<PackageVersion Include="MimeKit" Version="4.11.0" />
<PackageVersion Include="NETCore.MailKit" Version="2.1.0" />
<PackageVersion Include="Npgsql.OpenTelemetry" Version="8.0.5" />
<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.15.0" />
<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.15.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.15.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.15.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.15.0" />
<PackageVersion Include="Otp.NET" Version="1.4.1" />
<PackageVersion Include="PreMailer.Net" Version="2.6.0" />
<PackageVersion Include="RazorLight" Version="2.3.1" />
Expand Down
38 changes: 38 additions & 0 deletions SnakeAid.Api/DI/DependencyInjection.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@

using Serilog;
using Doppler.Extensions.Configuration; // Ensure this is available
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Exporter;
using Npgsql;

namespace SnakeAid.Api.DI;

Expand Down Expand Up @@ -281,4 +286,37 @@ public static IServiceCollection AddSwagger(this IServiceCollection services)
}

#endregion

#region OpenTelemetry

public static IServiceCollection AddOpenTelemetryServices(this IServiceCollection services, IConfiguration configuration)
{
var otlpEndpoint = configuration["OpenTelemetry:Endpoint"] ?? "http://localhost:4317"; // fallback for local without docker

services.AddOpenTelemetry()
.WithTracing(tracerProviderBuilder =>
{
tracerProviderBuilder
.AddSource("SnakeAid.Api")
.SetResourceBuilder(
ResourceBuilder.CreateDefault()
.AddService("SnakeAid.Api"))
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddNpgsql()
.AddOtlpExporter(opts => opts.Endpoint = new Uri(otlpEndpoint));
})
.WithMetrics(metricsProviderBuilder =>
{
metricsProviderBuilder
.AddMeter("SnakeAid.Api")
.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.AddOtlpExporter(opts => opts.Endpoint = new Uri(otlpEndpoint));
});

return services;
}

#endregion
Comment on lines +289 to +321

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Metrics are missing ResourceBuilder — service name won't be attached to exported metrics.

Tracing configures .SetResourceBuilder(…).AddService("SnakeAid.Api") but the metrics builder does not. This means metrics exported via OTLP will lack the service.name resource attribute, making it harder to filter/correlate in Prometheus/Grafana.

🔧 Proposed fix 
         .WithMetrics(metricsProviderBuilder =>
         {
             metricsProviderBuilder
                 .AddMeter("SnakeAid.Api")
+                .SetResourceBuilder(
+                    ResourceBuilder.CreateDefault()
+                        .AddService("SnakeAid.Api"))
                 .AddAspNetCoreInstrumentation()
                 .AddRuntimeInstrumentation()
                 .AddOtlpExporter(opts => opts.Endpoint = new Uri(otlpEndpoint));
         });

Consider also extracting the shared ResourceBuilder into a local variable to avoid duplication:

var resourceBuilder = ResourceBuilder.CreateDefault().AddService("SnakeAid.Api");
🤖 Prompt for AI Agents 
In `@SnakeAid.Api/DI/DependencyInjection.cs` around lines 289 - 321, The metrics
builder in AddOpenTelemetryServices is missing a ResourceBuilder so exported
metrics won't have service.name; update the metrics configuration
(metricsProviderBuilder) to use the same ResourceBuilder used for tracing
(extract ResourceBuilder.CreateDefault().AddService("SnakeAid.Api") into a local
variable) and apply it to the metrics provider just like
tracerProviderBuilder.SetResourceBuilder(...), ensuring the AddOtlpExporter call
on metricsProviderBuilder sends metrics with the service resource attached.

}
1 change: 1 addition & 0 deletions SnakeAid.Api/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ public static async Task Main(string[] args)
builder.Services.AddScoped<SnakeAid.Service.Implements.Email.EmailTemplateService>();

builder.Services.AddServices(builder.Configuration);
builder.Services.AddOpenTelemetryServices(builder.Configuration);

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

find . -type f -name "*.cs" | xargs grep -l "AddOpenTelemetryServices" | head -20

Repository: Snake-AID/SnakeAid.Backend

Length of output: 136

🏁 Script executed:

find . -type f -name "*DependencyInjection*" -o -name "*OpenTelemetry*" | head -20

Repository: Snake-AID/SnakeAid.Backend

Length of output: 110

🏁 Script executed:

rg "AddOpenTelemetryServices" -A 30

Repository: Snake-AID/SnakeAid.Backend

Length of output: 4821

The AddOpenTelemetryServices implementation handles missing configuration well but doesn't validate invalid URIs.

The method correctly uses a null-coalescing operator to default to http://localhost:4317 when OpenTelemetry:Endpoint is missing. However, if the configuration contains an invalid URI string (e.g., malformed value), the new Uri(otlpEndpoint) call will throw a FormatException at startup. Consider adding URI validation or error handling to fail gracefully with a clear error message when the endpoint is misconfigured.

🤖 Prompt for AI Agents 
In `@SnakeAid.Api/Program.cs` at line 108, Add validation around the OpenTelemetry
endpoint inside the AddOpenTelemetryServices implementation: check the
configured string (used to create the Uri in AddOpenTelemetryServices) with
Uri.TryCreate and ensure AbsoluteUri (or catch FormatException) and, if invalid,
throw an ArgumentException or log an explicit error describing the misconfigured
OpenTelemetry:Endpoint instead of letting new Uri(...) throw; update the code
path that currently does new Uri(otlpEndpoint) to use TryCreate and a clear
failure message indicating the invalid URI and the expected format.


// Register services using Scrutor
builder.Services.Scan(scan => scan
Expand Down
8 changes: 7 additions & 1 deletion SnakeAid.Api/SnakeAid.Api.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,13 @@
<PackageReference Include="Carter" />
<PackageReference Include="Doppler.Extensions.Configuration" />
<PackageReference Include="FirebaseAdmin" />
<PackageReference Include="Microsoft.AspNetCore.SignalR"/>
<PackageReference Include="Microsoft.AspNetCore.SignalR" />
<PackageReference Include="Npgsql.OpenTelemetry" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" />
<PackageReference Include="Serilog.AspNetCore" />
<PackageReference Include="Serilog.Enrichers.Environment" />
<PackageReference Include="Serilog.Enrichers.Span" />
Expand Down
24 changes: 24 additions & 0 deletions SnakeAid.Api/appsettings.Example.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"Serilog",
"Serilog.Sinks.Console",
"Serilog.Sinks.SQLite",
"Serilog.Sinks.Grafana.Loki",
"Serilog.Enrichers.Environment",
"Serilog.Enrichers.Span"
],
Expand All @@ -36,6 +37,26 @@
"outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Application} {Message:lj} {Properties:j}{NewLine}{Exception}"
}
},
{
"Name": "GrafanaLoki",
"Args": {
"requestUri": "http://loki:3100",
"labels": [
{
"key": "app",
"value": "SnakeAid.API"
},
{
"key": "environment",
"value": "Development"
}
],
"propertiesAsLabels": ["Application", "Environment"],
"queueLimit": 10000,
"batchPostingLimit": 1000,
"period": "00:00:02"
}
},
Comment on lines +40 to +59

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

Serilog.Sinks.Grafana.Loki requestUri endpoint configuration format

💡 Result:

In Serilog.Sinks.Grafana.Loki the configured uri/Url is the root/base address of the Loki server (scheme + host + optional port). The sink then posts to Loki’s push endpoint POST /loki/api/v1/push. [1] [2]

Example (appsettings.json)

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.Grafana.Loki" ],
    "WriteTo": [
      {
        "Name": "GrafanaLoki",
        "Args": {
          "uri": "http://localhost:3100"
        }
      }
    ]
  }
}

uri is documented as “the root URI of Loki”. [1] (and the README shows the same format). [2]

The actual HTTP endpoint Loki expects

Loki’s ingest endpoint is:

  • POST /loki/api/v1/push [3]

So with uri = http://localhost:3100, the sink will send to:

  • http://localhost:3100/loki/api/v1/push (conceptually: base + /loki/api/v1/push) [3]

Note if you’re sending to Grafana Agent / Alloy “loki_push_api”

Some agent configs use /api/v1/push (without the /loki prefix). If you point the sink at an agent endpoint that only accepts /api/v1/push, you can get 404s because the sink targets /loki/api/v1/push. [4]

Sources:
[1] Serilog.Sinks.Grafana.Loki wiki “Application settings” (parameter uri = root URI of Loki) (github.com)
[2] serilog-contrib/serilog-sinks-grafana-loki README (examples use "uri": "http://localhost:3100") (github.com)
[3] Grafana Loki HTTP API docs (ingest endpoint POST /loki/api/v1/push) (grafana.com)
[4] Issue report describing /api/v1/push vs /loki/api/v1/push mismatch with Grafana Agent (lightrun.com)

Citations:

🏁 Script executed:

# Search for configuration files and patterns in the codebase
find . -name "appsettings*.json" -o -name "*.Example.json" | head -20

Repository: Snake-AID/SnakeAid.Backend

Length of output: 109

🏁 Script executed:

# Look for Serilog configuration or documentation in the repo
rg -i "loki|grafana" --type json -A 3 -B 3 | head -100

Repository: Snake-AID/SnakeAid.Backend

Length of output: 52

🏁 Script executed:

# Search for any comments or documentation about local development setup
rg -i "docker|localhost|environment" SnakeAid.Api/ | head -50

Repository: Snake-AID/SnakeAid.Backend

Length of output: 2955

🏁 Script executed:

# Search for all appsettings files
fd "appsettings" SnakeAid.Api/

Repository: Snake-AID/SnakeAid.Backend

Length of output: 52

🏁 Script executed:

# Check how appsettings are loaded in Program.cs
rg -A 10 "appsettings|Configuration" SnakeAid.Api/Program.cs | head -60

Repository: Snake-AID/SnakeAid.Backend

Length of output: 2379

Loki requestUri uses Docker-internal hostname — won't work for local development outside containers.

The URI http://loki:3100 only resolves within the Docker network. Developers running the API locally will get connection failures. Since this is an example file, consider using http://localhost:3100 as the default, or add a comment noting this configuration is for Docker Compose environments only.

🤖 Prompt for AI Agents 
In `@SnakeAid.Api/appsettings.Example.json` around lines 40 - 59, The example
config's GrafanaLoki requestUri ("requestUri" inside the "GrafanaLoki" entry)
uses the Docker-only hostname http://loki:3100 which will fail for local
(non-container) development; update the example to use http://localhost:3100 as
the default requestUri or add a clear inline comment above the "GrafanaLoki"
block explaining that http://loki:3100 is valid only inside Docker Compose and
developers running the API locally should use http://localhost:3100 (refer to
the "GrafanaLoki" name and its "Args.requestUri" field).

{
"Name": "SQLite",
"Args": {
Expand All @@ -46,6 +67,9 @@
}
]
},
"OpenTelemetry": {
"Endpoint": "http://otel-collector:4317"
},
"Jwt": {
"SecretKey": "mysuperlongsecretkeythatishard2guess123!",
"RefreshSecretKey": "anotherultrasecretkeythatmustbe32chars!!",
Expand Down
2 changes: 1 addition & 1 deletion SnakeAid.Docs
Open in desktop
Submodule SnakeAid.Docs updated 21 files
+106 −0 Docs/05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/emergency-rescue.roadmap.md
+273 −0 ...05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/live-tracking/live-tracking.architecture.md
+77 −0 ...05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/live-tracking/live-tracking.introduction.md
+78 −0 Docs/05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/live-tracking/live-tracking.plan.md
+68 −0 Docs/05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/live-tracking/live-tracking.prompt.md
+153 −0 Docs/05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/live-tracking/live-tracking.sourcecode.md
+139 −0 Docs/05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/live-tracking/live-tracking.usageguide.md
+92 −0 ...-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/rescue-trigger/rescue-trigger.introduction.md
+85 −0 Docs/05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/rescue-trigger/rescue-trigger.plan.md
+74 −0 Docs/05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/rescue-trigger/rescue-trigger.prompt.md
+266 −0 ...05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/rescue-trigger/rescue-trigger.sourcecode.md
+151 −0 ...05-Backend/01-flows/P1-emergency/S1-identified/emergency-rescue/rescue-trigger/rescue-trigger.usageguide.md
+0 −47 Docs/05-Backend/01-flows/P1-emergency/S1-identified/rescue-trigger/rescue-trigger.introduction.md
+0 −166 Docs/05-Backend/01-flows/P1-emergency/S1-identified/rescue-trigger/rescue-trigger.sourcecode.md
+0 −130 Docs/05-Backend/01-flows/P1-emergency/S1-identified/rescue-trigger/rescue-trigger.usageguide.md
+210 −0 Docs/05-Backend/02-layers/grafana-otel-serilogs/grafana-otel-serilogs.introduction.md
+80 −0 Docs/05-Backend/02-layers/grafana-otel-serilogs/grafana-otel-serilogs.plan.md
+333 −0 Docs/05-Backend/02-layers/grafana-otel-serilogs/grafana-otel-serilogs.prompt.md
+4 −0 Docs/05-Backend/AGENTS.md
+83 −56 Docs/_sidebar.md
+38 −2 Docs/index.html
90 changes: 90 additions & 0 deletions docker-compose.grafana.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
services:
snakeaid-api:
image: thekhiem7/snakeaid-api:latest
pull_policy: always
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
environment:
- DOPPLER_TOKEN=${DOPPLER_TOKEN}
- DOPPLER_CONFIG=snake-aid/dev

restart: unless-stopped
networks:
- snakeaid-net

loki:
image: grafana/loki:latest
command: -config.file=/etc/loki/local-config.yaml
ports:
- "3100:3100"
volumes:
- ./observability/loki-config.yaml:/etc/loki/local-config.yaml
networks:
- snakeaid-net

tempo:
image: grafana/tempo:latest
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./observability/tempo-config.yaml:/etc/tempo.yaml
- ./observability/tempo-data:/var/tempo
ports:
- "14268" # jaeger ingest
- "3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
- "9411" # zipkin
networks:
- snakeaid-net

prometheus:
image: prom/prometheus:latest
command:
- --config.file=/etc/prometheus/prometheus.yml
- --web.enable-lifecycle
volumes:
- ./observability/prometheus.yaml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
networks:
- snakeaid-net

grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
volumes:
- ./observability/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
Comment on lines +62 to +64

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Grafana is wide-open with anonymous Admin access — acceptable for local dev only.

Anonymous access with Admin role and login disabled means anyone with network access has full Grafana admin rights. This is fine for a local dev stack but ensure this compose file is never used in staging/production or exposed publicly (e.g., via a cloud VM with open ports).

🤖 Prompt for AI Agents 
In `@docker-compose.grafana.yml` around lines 62 - 64, The compose file currently
enables anonymous admin access via GF_AUTH_ANONYMOUS_ENABLED=true,
GF_AUTH_ANONYMOUS_ORG_ROLE=Admin and disables login with
GF_AUTH_DISABLE_LOGIN_FORM=true; change these for safety by setting
GF_AUTH_ANONYMOUS_ENABLED=false (or removing the env var), set
GF_AUTH_ANONYMOUS_ORG_ROLE to a least-privilege role like Viewer if anonymous
must remain, and enable the login form (GF_AUTH_DISABLE_LOGIN_FORM=false);
additionally move any dev-only anonymous settings into a
docker-compose.override.yml or an .env.dev file and ensure production
deployments do not include these vars.

networks:
- snakeaid-net

otel-collector:
image: otel/opentelemetry-collector:latest
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./observability/otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "1888:1888" # pprof extension
- "8888:8888" # Prometheus metrics exposed by the collector
- "8889:8889" # Prometheus exporter metrics
- "13133:13133" # health_check extension
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP http receiver
- "55679:55679" # zpages extension
depends_on:
- loki
- tempo
- prometheus
networks:
- snakeaid-net

networks:
snakeaid-net:
driver: bridge
37 changes: 37 additions & 0 deletions observability/grafana-datasources.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
apiVersion: 1

datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: false
jsonData:
graphiteVersion: '1.1'

- name: Loki
type: loki
access: proxy
url: http://loki:3100
isDefault: true

- name: Tempo
type: tempo
access: proxy
url: http://tempo:3200
jsonData:
tracesToLogs:
datasourceUid: 'Loki'
tags: ['job', 'instance', 'pod', 'namespace']
mappedTags: [{ key: 'service.name', value: 'app' }]
mapTagNamesEnabled: false
spanStartTimeShift: '0'
spanEndTimeShift: '0'
filterByTraceID: true
filterBySpanID: true
tracesToMetrics:
datasourceUid: 'Prometheus'
tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }]
queries:
- name: 'Sample query'
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
Comment on lines +3 to +37

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Tempo's datasourceUid references won't resolve — datasource UIDs are not explicitly set.

Lines 24 and 33 reference datasourceUid: 'Loki' and datasourceUid: 'Prometheus', but neither the Loki nor Prometheus datasource definitions include an explicit uid field. Grafana auto-generates UIDs (they won't be "Loki" or "Prometheus"), so the traces-to-logs and traces-to-metrics links will silently fail.

🔧 Proposed fix — add explicit UIDs 
   - name: Prometheus
     type: prometheus
     access: proxy
+    uid: Prometheus
     url: http://prometheus:9090
     isDefault: false
-    jsonData:
-      graphiteVersion: '1.1'

   - name: Loki
     type: loki
     access: proxy
+    uid: Loki
     url: http://loki:3100
     isDefault: true

Also: the graphiteVersion under Prometheus jsonData is not relevant for a Prometheus-type datasource and can be removed.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: false
jsonData:
graphiteVersion: '1.1'
- name: Loki
type: loki
access: proxy
url: http://loki:3100
isDefault: true
- name: Tempo
type: tempo
access: proxy
url: http://tempo:3200
jsonData:
tracesToLogs:
datasourceUid: 'Loki'
tags: ['job', 'instance', 'pod', 'namespace']
mappedTags: [{ key: 'service.name', value: 'app' }]
mapTagNamesEnabled: false
spanStartTimeShift: '0'
spanEndTimeShift: '0'
filterByTraceID: true
filterBySpanID: true
tracesToMetrics:
datasourceUid: 'Prometheus'
tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }]
queries:
- name: 'Sample query'
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
datasources:
- name: Prometheus
type: prometheus
access: proxy
uid: Prometheus
url: http://prometheus:9090
isDefault: false
- name: Loki
type: loki
access: proxy
uid: Loki
url: http://loki:3100
isDefault: true
- name: Tempo
type: tempo
access: proxy
url: http://tempo:3200
jsonData:
tracesToLogs:
datasourceUid: 'Loki'
tags: ['job', 'instance', 'pod', 'namespace']
mappedTags: [{ key: 'service.name', value: 'app' }]
mapTagNamesEnabled: false
spanStartTimeShift: '0'
spanEndTimeShift: '0'
filterByTraceID: true
filterBySpanID: true
tracesToMetrics:
datasourceUid: 'Prometheus'
tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }]
queries:
- name: 'Sample query'
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
🤖 Prompt for AI Agents 
In `@observability/grafana-datasources.yaml` around lines 3 - 37, Tempo's
tracesToLogs/tracesToMetrics fail because Prometheus and Loki datasources lack
explicit uids; add uid: 'Prometheus' to the Prometheus datasource block and uid:
'Loki' to the Loki datasource block so the tracesToMetrics.datasourceUid and
tracesToLogs.datasourceUid resolve, and also remove the irrelevant
graphiteVersion entry from Prometheus.jsonData (the keys to edit are the
Prometheus and Loki datasource definitions and the jsonData object under
Prometheus).

31 changes: 31 additions & 0 deletions observability/loki-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
auth_enabled: false

limits_config:
allow_structured_metadata: false

server:
http_listen_port: 3100

common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory

schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
Comment on lines +20 to +28

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

Loki 3.0 boltdb-shipper deprecated TSDB schema v13 migration

💡 Result:

In Loki 3.x, BoltDB / boltdb-shipper is deprecated, and Grafana recommends migrating indexing to TSDB (current development focus). Boltdb-shipper “may be removed in future major versions of Loki.” [1][2]

Recommended migration approach (boltdb-shipper → TSDB, schema v13)

1) Keep your existing boltdb-shipper period(s) unchanged (for old data)

Do not modify historical schema_config entries (store/schema/prefix/period). Those must remain as-is if you want to keep querying old logs. [2][3][4]

2) Add a new future-dated period_config that switches to TSDB + v13

Add another entry to schema_config.configs with:

  • from: a date/time in the future (relative to when you deploy the config)
  • store: tsdb
  • schema: v13 (recommended in the TSDB migration docs)
  • same object_store (unless you intentionally change it)

Example (adapt to your object store, prefixes, and dates): [1]

schema_config:
  configs:
    - from: 2023-01-01
      store: boltdb-shipper
      object_store: s3   # or gcs/filesystem/etc
      schema: v11        # or your current schema
      index:
        prefix: index_
        period: 24h

    - from: 2026-03-01   # MUST be in the future when rolled out
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

Common failure mode: if the TSDB from date is wrong (too early), Loki will try to read older data using the TSDB rules and you’ll “lose” old logs in queries until you fix the date. [3]

3) Configure tsdb_shipper in storage_config

At minimum configure:

  • active_index_directory
  • cache_location [1]

Example: [1]

storage_config:
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location: /data/tsdb-cache

4) Run the compactor (strongly recommended for TSDB)

Grafana strongly recommends running the compactor with TSDB for compaction/retention and better query performance. [1]

Notes specific to “schema v13”

  • The migration docs explicitly show TSDB with schema: v13 as the target schema. [1]
  • Ensure your old period keeps its original schema (for example v11/v12), otherwise you can break querying of historical data. [4]

If you share your current schema_config + storage_config (and object store: S3/GCS/Azure/filesystem), I can point out the exact minimal diff for a safe cutover date.

boltdb-shipper with schema v11 is deprecated in Loki 3.x; migrate to TSDB with schema v13.

Do not replace the existing boltdb-shipper entry directly—this breaks queries on historical data. Instead:

  1. Keep the current boltdb-shipper entry unchanged (required for querying old logs)
  2. Add a new schema_config entry with store: tsdb, schema: v13, and a future from date (e.g., 2026-03-01)
  3. Configure tsdb_shipper in storage_config with active_index_directory and cache_location
  4. Consider running the compactor with TSDB for better retention and query performance

Setting the TSDB from date incorrectly (too early) will prevent queries on historical data until corrected.

🤖 Prompt for AI Agents 
In `@observability/loki-config.yaml` around lines 20 - 28, The current
schema_config only contains a boltdb-shipper entry with schema v11 which is
deprecated in Loki 3.x; to migrate safely, keep the existing schema_config block
exactly as-is (the boltdb-shipper / schema: v11 entry) and add a second
schema_config entry with store: tsdb, schema: v13 and a future from date (e.g.,
2026-03-01) so new data uses TSDB while old queries still work; also add
tsdb_shipper settings in storage_config (configure active_index_directory and
cache_location) and ensure the compactor is configured to run with TSDB for
proper retention and query performance (avoid setting the TSDB from date earlier
than needed).


ruler:
alertmanager_url: http://localhost:9093
Comment on lines +30 to +31

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

alertmanager_url references a non-existent service.

http://localhost:9093 points to an Alertmanager that isn't part of the Docker Compose stack. This will cause ruler errors if alerting rules are configured. Either remove the ruler block or add an Alertmanager service.

🤖 Prompt for AI Agents 
In `@observability/loki-config.yaml` around lines 30 - 31, The ruler configuration
includes alertmanager_url pointing to localhost which doesn't match our Docker
Compose services; either remove the entire ruler block (the "ruler:" stanza)
from loki-config.yaml if you don't intend to use Alertmanager, or update
alertmanager_url to point to the Alertmanager service name used in Compose
(e.g., change alertmanager_url to "http://alertmanager:9093") and add a matching
Alertmanager service to Docker Compose; ensure the symbol names "ruler" and
"alertmanager_url" are updated consistently with the Compose service name.

28 changes: 28 additions & 0 deletions observability/otel-collector-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
receivers:
otlp:
protocols:
grpc:
http:

processors:
batch:

exporters:
otlp:
endpoint: tempo:4317
tls:
insecure: true
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "snakeaid_api"

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp] # To Tempo (via OTLP)
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus] # Expose via Prometheus scrape endpoint
8 changes: 8 additions & 0 deletions observability/prometheus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
global:
scrape_interval: 15s

scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['otel-collector:8889']
25 changes: 25 additions & 0 deletions observability/tempo-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
server:
http_listen_port: 3200

distributor:
receivers:
otlp:
protocols:
grpc:
http:

ingester:
trace_idle_period: 10s
max_block_bytes: 1_000_000
max_block_duration: 5m

storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks

overrides:
metrics_generator_processors: [service-graphs, span-metrics]
Comment on lines +24 to +25

@coderabbitai coderabbitai Bot Feb 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

❓ Verification inconclusive

Grafana Tempo metrics_generator configuration remote_write requirements

Tempo’s metrics-generator remote_write is essentially a Prometheus Agent that uses a WAL and then remote-writes generated metrics to a Prometheus-compatible endpoint. The practical requirements are:

  • You must set a local WAL directory: metrics_generator.storage.path (Tempo docs indicate the WAL path “must be set” / is required for this storage block). [1][2]
  • You must configure at least one remote_write endpoint under metrics_generator.storage.remote_write, using standard Prometheus remote_write config (at minimum, a url, typically pointing to a Prometheus-compatible /api/v1/write). [1][2]
  • The remote_write receiver must accept what you send:
    • If you enable exemplars (send_exemplars: true), the backend must support ingesting exemplars. Example configs show this option commonly enabled. [2]
    • If you enable native histograms, the receiving metrics backend must be configured to ingest them. [3]
  • Multi-tenancy behavior:
    • By default, the metrics-generator forwards X-Scope-OrgID to the remote_write endpoint (can be disabled with remote_write_add_org_id_header: false). [1][3]
    • If you need per-tenant auth headers, configure remote_write_headers via overrides (requires Tempo 2.4.0+). [4]

Minimal example (illustrative, from Grafana Tempo docs):

metrics_generator:
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

[2]

Sources:
[1] Grafana Tempo configuration (metrics-generator storage/remote_write) https://grafana.com/docs/tempo/latest/configuration/
[2] Grafana Tempo “Deploy on Linux” example config (metrics_generator.storage.path + remote_write.url) https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/locally/linux/
[3] Metrics-generator docs (remote write + native histograms note) https://grafana.com/docs/tempo/latest/metrics-from-traces/metrics-generator/
[4] Metrics-generator multitenancy support (remote_write_headers, Tempo >=2.4.0) https://grafana.com/docs/tempo/latest/metrics-from-traces/metrics-generator/multitenancy/

Add metrics_generator configuration block to enable metrics processors.

The metrics_generator_processors override enables service-graphs and span-metrics generation, but without a corresponding metrics_generator block, these processors won't actually function. Tempo requires at minimum:

  • metrics_generator.storage.path: WAL directory for metrics buffering
  • metrics_generator.storage.remote_write: Prometheus-compatible remote write endpoint (e.g., http://prometheus:9090/api/v1/write)

Without these, the processor setting is ineffective.

🤖 Prompt for AI Agents 
In `@observability/tempo-config.yaml` around lines 24 - 25, The overrides block
enables metrics_generator_processors (service-graphs, span-metrics) but there is
no metrics_generator configuration, so add a metrics_generator block that
defines at minimum metrics_generator.storage.path for the WAL directory and
metrics_generator.storage.remote_write pointing to a Prometheus-compatible
remote write endpoint (e.g., http://prometheus:9090/api/v1/write); ensure the
new metrics_generator block is present alongside overrides so the service-graphs
and span-metrics processors can function correctly.