feat: add prometheus monitoring stack and grafana dashboard for k8s - #269
Open
escoutdoor wants to merge 14 commits into
Open
feat: add prometheus monitoring stack and grafana dashboard for k8s#269escoutdoor wants to merge 14 commits into
escoutdoor wants to merge 14 commits into
Conversation
escoutdoor
added
enhancement
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (24)
✅ Files skipped from review due to trivial changes (6)
🚧 Files skipped from review as they are similar to previous changes (10)
📝 WalkthroughWalkthroughAdds end-to-end Prometheus/Grafana observability across all three Go services. Each service gets a dedicated metrics package (HTTP counters/histograms, active-request gauge, domain-specific event counters), a shared Gin middleware, a ChangesPrometheus/Grafana Monitoring Stack
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@build/prometheus.yaml`:
- Around line 18-32: The prometheus.yaml file has hardcoded targets using
host.docker.internal with ports 3800, 3850, and 3900, but when this
configuration is used with deploy/compose/compose.local.yaml, the actual service
ports are different (8082, 8080, 8081), causing Prometheus to fail scraping.
Create a new file prometheus.compose.local.yaml specifically for the compose
deployment topology where the targets use service names (guest-api, admin-api,
business-api) on their internal ports (3800, 3850, 3900) instead of hardcoded
host addresses, and update the compose.local.yaml file to mount this new
configuration instead of the original prometheus.yaml.
In `@cmd/admin-auth-api/main.go`:
- Around line 82-90: The metrics middleware is currently being registered after
ErrorMiddleware, which causes it to record HTTP status codes before the error
handler has a chance to set the correct status. Move the metrics middleware
setup (the call to router.Use(metricsMiddleware)) to occur before the
ErrorMiddleware is registered so that error responses are properly recorded with
their correct status codes instead of 200. This reordering needs to be applied
in three files: cmd/admin-auth-api/main.go, cmd/business-api/main.go, and
cmd/guest-api/main.go.
In `@cmd/business-api/main.go`:
- Around line 80-93: The middleware registration order is incorrect:
ErrorMiddleware is registered before metricsMiddleware, but since Gin executes
response handlers in reverse registration order, metricsMiddleware records the
status code before ErrorMiddleware can set error codes like 404 or 400. Swap the
registration order so that metricsMiddleware is registered first (before
ErrorMiddleware), which will cause ErrorMiddleware to execute first during
response handling and set the correct error status code, allowing
metricsMiddleware to then record the accurate status in the metrics.
In `@cmd/guest-api/main.go`:
- Around line 91-103: The metrics middleware needs to be registered before the
error middleware to ensure it captures the correct HTTP status codes. Move the
router.Use(metricsMiddleware) call and the metrics setup (the
prometheus.NewRegistry(), metrics.New(), and middleware.Metrics() calls) to
occur before the router.Use(guest_middleware.ErrorMiddleware()) call. This
ensures the metrics middleware reads the final status codes after the error
middleware has translated errors to appropriate HTTP status codes.
In `@deploy/k8s/monitoring/share-bite-services.json`:
- Line 68: The share-bite-services.json dashboard file has multiple panels all
assigned the same ID value of 1, which prevents proper panel identification and
breaks editing functionality. Replace each panel's "id" field with a unique
sequential integer value, starting from 1 and incrementing for each subsequent
panel throughout the entire dashboard definition to ensure stable panel identity
and correct behavior during import/provisioning.
In `@deploy/k8s/monitoring/values.yaml`:
- Line 7: The `adminPassword: admin` field in the Grafana configuration is a
hardcoded weak credential stored in version control, which is a security risk.
Remove the plaintext `adminPassword: admin` entry from the values.yaml file and
instead configure the Grafana admin password through a Kubernetes Secret or via
CI/CD pipeline-provided values that are injected at deployment time. This
ensures sensitive credentials are not stored in git and can be properly managed
through your secrets management system.
In `@docs/k8s/monitoring.md`:
- Line 51: The documentation file contains an incorrect file path reference to
the dashboard JSON file on line 51. Replace the current path reference from
`docs/k8s/monitoring/share-bite-services.json` with the correct path
`deploy/k8s/monitoring/share-bite-services.json` so that developers looking at
the documentation can accurately locate the actual dashboard source file.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bc7b312d-6afd-41cf-b720-9b70518cc7a2
📒 Files selected for processing (31)
.env.example.gitignoreMakefilebuild/compose.infra.yamlbuild/prometheus.yamlcmd/admin-auth-api/main.gocmd/business-api/main.gocmd/guest-api/main.godeploy/compose/compose.local.yamldeploy/k8s/admin-auth/admin-auth-service.yamldeploy/k8s/admin-auth/admin-auth-servicemonitor.yamldeploy/k8s/admin-auth/kustomization.yamldeploy/k8s/business/business-service.yamldeploy/k8s/business/business-servicemonitor.yamldeploy/k8s/business/kustomization.yamldeploy/k8s/guest/deployment.yamldeploy/k8s/guest/kustomization.yamldeploy/k8s/guest/service.yamldeploy/k8s/guest/servicemonitor.yamldeploy/k8s/infra/configmap.yamldeploy/k8s/infra/kustomization.yamldeploy/k8s/kustomization.yamldeploy/k8s/monitoring/kustomization.yamldeploy/k8s/monitoring/share-bite-services.jsondeploy/k8s/monitoring/values.yamldocs/k8s/local-kubernetes.mddocs/k8s/monitoring.mdinternal/admin-auth/metrics/metrics.gointernal/business/metrics/metrics.gointernal/guest/metrics/metrics.gointernal/middleware/metrics.go
escoutdoor
force-pushed
the
feat/add-metrics
branch
from
June 22, 2026 10:57
919e52b to
c2094b1
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@build/prometheus.local.yaml`:
- Line 21: The prometheus.local.yaml configuration uses host.docker.internal as
the target hostname on lines 21, 28, and 35, but this assumes Docker DNS
resolution which is not portable across all systems (particularly breaking on
Linux Docker). To fix this, add extra_hosts configuration to the Prometheus
service in the docker-compose file that uses this prometheus.local.yaml config
by specifying extra_hosts: ["host.docker.internal:host-gateway"] to enable
proper hostname resolution across different Docker environments. Alternatively,
document this as a runtime requirement that must be enforced when running the
services.
In `@Makefile`:
- Around line 220-221: The `kind-load` target hardcodes `:latest` as the image
tag for all four images (guest-api, business-api, admin-auth-api, and migrator),
which causes it to load images with the wrong tag when a custom TAG variable is
provided. Replace each `:latest` suffix with `$(TAG)` in the `kind load
docker-image` command so that the tag used matches the TAG variable that was
passed to the build process.
- Around line 238-240: The monitoring-down target's helm uninstall command for
kube-prometheus-stack will fail if the release is already absent. Add the
--ignore-not-found flag to the helm uninstall kube-prometheus-stack command to
make it idempotent and prevent failures when the release doesn't exist, similar
to the approach already used with kubectl delete namespace monitoring.
- Around line 223-236: The monitoring-up target applies the grafana-secret.yaml
before the monitoring namespace is created, causing the secret application to
fail. Either add a kubectl create namespace monitoring command before the
kubectl apply line that references deploy/k8s/monitoring/grafana-secret.yaml, or
modify the kubectl apply command to include the --create-namespace flag. Ensure
the namespace exists before attempting to apply the secret that declares
namespace: monitoring.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 1a815556-ea7e-4128-8e9f-304dbab83099
📒 Files selected for processing (32)
.env.exampleMakefilebuild/compose.infra.yamlbuild/prometheus.local.yamlbuild/prometheus.yamlcmd/admin-auth-api/main.gocmd/business-api/main.gocmd/guest-api/main.godeploy/compose/compose.local.yamldeploy/k8s/admin-auth/admin-auth-service.yamldeploy/k8s/admin-auth/admin-auth-servicemonitor.yamldeploy/k8s/admin-auth/kustomization.yamldeploy/k8s/business/business-service.yamldeploy/k8s/business/business-servicemonitor.yamldeploy/k8s/business/kustomization.yamldeploy/k8s/guest/deployment.yamldeploy/k8s/guest/kustomization.yamldeploy/k8s/guest/service.yamldeploy/k8s/guest/servicemonitor.yamldeploy/k8s/infra/configmap.yamldeploy/k8s/infra/kustomization.yamldeploy/k8s/kustomization.yamldeploy/k8s/monitoring/grafana-secret.yamldeploy/k8s/monitoring/kustomization.yamldeploy/k8s/monitoring/metrics-values.yamldeploy/k8s/monitoring/share-bite-services.jsondocs/k8s/local-kubernetes.mddocs/k8s/monitoring.mdinternal/admin-auth/metrics/metrics.gointernal/business/metrics/metrics.gointernal/guest/metrics/metrics.gointernal/middleware/metrics.go
✅ Files skipped from review due to trivial changes (15)
- deploy/k8s/business/business-service.yaml
- deploy/k8s/monitoring/kustomization.yaml
- deploy/k8s/infra/kustomization.yaml
- build/prometheus.yaml
- deploy/k8s/admin-auth/admin-auth-servicemonitor.yaml
- deploy/k8s/admin-auth/admin-auth-service.yaml
- deploy/k8s/monitoring/metrics-values.yaml
- deploy/k8s/business/business-servicemonitor.yaml
- deploy/k8s/monitoring/grafana-secret.yaml
- deploy/k8s/infra/configmap.yaml
- .env.example
- docs/k8s/monitoring.md
- deploy/k8s/guest/service.yaml
- deploy/k8s/guest/deployment.yaml
- deploy/k8s/admin-auth/kustomization.yaml
🚧 Files skipped from review as they are similar to previous changes (15)
- internal/middleware/metrics.go
- deploy/k8s/guest/servicemonitor.yaml
- deploy/k8s/kustomization.yaml
- deploy/k8s/business/kustomization.yaml
- docs/k8s/local-kubernetes.md
- deploy/compose/compose.local.yaml
- internal/admin-auth/metrics/metrics.go
- deploy/k8s/guest/kustomization.yaml
- internal/guest/metrics/metrics.go
- build/compose.infra.yaml
- cmd/guest-api/main.go
- internal/business/metrics/metrics.go
- cmd/business-api/main.go
- cmd/admin-auth-api/main.go
- deploy/k8s/monitoring/share-bite-services.json
escoutdoor
force-pushed
the
feat/add-metrics
branch
from
June 22, 2026 11:31
567bc74 to
febb8b6
Compare
escoutdoor
marked this pull request as ready for review
June 22, 2026 11:35
escoutdoor
requested review from
DmyMi,
IlyaMaluk,
MihuNt3r and
isaistvo
and removed request for
DmyMi,
mblinovv and
viktorzhabskyi
June 22, 2026 11:36
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deploy/k8s/monitoring/share-bite-services.json`:
- Around line 1549-1734: Several dashboard panels in share_bite_services.json
have overlapping or out-of-bounds gridPos values, which breaks Grafana’s
24-column layout. Update the affected panel definitions so each row uses
non-overlapping widths that stay within the grid, following a consistent pattern
like 0/8, 8/8, 16/8 for the grouped panels. Check the panel objects around the
timeseries and piechart sections, and adjust their x, w, and y values so the
affected rows at y=56, y=64, and y=72 align cleanly without collisions.
In `@Makefile`:
- Around line 227-230: The monitoring-up bootstrap steps are not safely
rerunnable because the helm repo setup can fail on an existing
prometheus-community entry and the kubectl namespace creation currently hides
real errors. Update the Makefile target around the helm repo add/update and
kubectl create namespace command so repeated runs succeed without error, and
remove the unconditional ignore behavior so bootstrap failures in these steps
surface immediately. Use the monitoring-up target and the prometheus-community
helm repo / kubectl namespace creation commands to locate the fix.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 357c89f3-3e5a-4ad6-b6e9-9a5ca3b386c6
📒 Files selected for processing (49)
.env.exampleMakefilebuild/compose.infra.yamlbuild/prometheus.local.yamlbuild/prometheus.yamlcmd/admin-auth-api/main.gocmd/business-api/main.gocmd/guest-api/main.godeploy/compose/compose.local.yamldeploy/k8s/admin-auth/admin-auth-deployment.yamldeploy/k8s/admin-auth/admin-auth-service.yamldeploy/k8s/admin-auth/admin-auth-servicemonitor.yamldeploy/k8s/admin-auth/kustomization.yamldeploy/k8s/business/business-service.yamldeploy/k8s/business/business-servicemonitor.yamldeploy/k8s/business/kustomization.yamldeploy/k8s/guest/deployment.yamldeploy/k8s/guest/kustomization.yamldeploy/k8s/guest/service.yamldeploy/k8s/guest/servicemonitor.yamldeploy/k8s/infra/configmap.yamldeploy/k8s/infra/kustomization.yamldeploy/k8s/kustomization.yamldeploy/k8s/monitoring/grafana-secret.yamldeploy/k8s/monitoring/kustomization.yamldeploy/k8s/monitoring/metrics-values.yamldeploy/k8s/monitoring/share-bite-services.jsondocs/k8s/local-kubernetes.mddocs/k8s/monitoring.mdinternal/admin-auth/handler/admin/handler.gointernal/admin-auth/handler/auth/handler.gointernal/admin-auth/metrics/metrics.gointernal/business/handler/business/create-org.gointernal/business/handler/business/create_box.gointernal/business/handler/business/create_location.gointernal/business/handler/business/create_post.gointernal/business/handler/business/handler.gointernal/business/handler/business/reserve_box.gointernal/business/metrics/metrics.gointernal/guest/handler/collection/create_collection.gointernal/guest/handler/collection/handler.gointernal/guest/handler/collection/invite_collaborator.gointernal/guest/handler/follow/follow.gointernal/guest/handler/follow/handler.gointernal/guest/handler/post/create.gointernal/guest/handler/post/handler.gointernal/guest/handler/post/like.gointernal/guest/metrics/metrics.gointernal/middleware/metrics.go
✅ Files skipped from review due to trivial changes (17)
- deploy/k8s/business/kustomization.yaml
- deploy/k8s/monitoring/kustomization.yaml
- internal/business/handler/business/create_location.go
- deploy/k8s/kustomization.yaml
- deploy/k8s/monitoring/grafana-secret.yaml
- internal/guest/handler/collection/invite_collaborator.go
- deploy/k8s/guest/service.yaml
- deploy/k8s/business/business-service.yaml
- internal/guest/handler/follow/follow.go
- internal/guest/handler/post/like.go
- deploy/k8s/admin-auth/admin-auth-service.yaml
- deploy/k8s/monitoring/metrics-values.yaml
- docs/k8s/local-kubernetes.md
- deploy/k8s/guest/kustomization.yaml
- deploy/k8s/admin-auth/kustomization.yaml
- docs/k8s/monitoring.md
- .env.example
🚧 Files skipped from review as they are similar to previous changes (9)
- deploy/k8s/guest/servicemonitor.yaml
- deploy/k8s/admin-auth/admin-auth-servicemonitor.yaml
- deploy/k8s/infra/kustomization.yaml
- build/prometheus.local.yaml
- build/prometheus.yaml
- deploy/k8s/guest/deployment.yaml
- deploy/k8s/business/business-servicemonitor.yaml
- build/compose.infra.yaml
- deploy/compose/compose.local.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #268
What
Adds centralized observability for the Kubernetes setup.
Changes
/metricswith HTTP request counters, response time histograms, and Go runtime stats (goroutines, heap, GC)ServiceMonitorresources for all services so Prometheus auto-discovers endpointsGo runtimestats. Includes a service selector dropdownConfigMapwithgrafana_dashboardlabeldocs/k8s/monitoring.mdexplaining how the stack works and how to add monitoring to new servicesmake kind-loadfor Kind users and updatedlocal-kubernetes.mdWhy
Before this change, debugging performance required checking individual pod logs. Now the team has a single URL with live metrics for all services.
How to verify
make monitoring-up— deploysPrometheus+Grafanamake monitoring-forward-grafanaopens Grafana at localhost:3000make monitoring-forward-prometheusopens Prometheus at localhost:9090guest-api,business-api, andadmin-authdocs/k8s/monitoring.mdcontains the setup instructions.Summary by CodeRabbit
Release Notes
/metricsendpoints and request/latency tracking across services, plus business event counters.PROMETHEUS_PORTdefault.kind, including image loading.k8s-upto wait for the CNPG cluster to become ready.