From 687c916203bfa3286cc1c3e4f1b7f3f937b20cb6 Mon Sep 17 00:00:00 2001 From: Vladislav Antonov Date: Sat, 14 Mar 2026 17:35:02 +0200 Subject: [PATCH 1/2] Add remote service metrics setup --- .env.remote.example | 4 ++ .env.server-logs.example | 3 + README.md | 63 ++++++++++++++++++++- docker-compose.remote.yml | 71 ++++++++++++++++++++++++ docker-compose.server-logs.yml | 20 +++++++ observability/prometheus.remote.yml.tmpl | 17 ++++++ observability/promtail.remote.yml.tmpl | 21 +++++++ 7 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 .env.remote.example create mode 100644 .env.server-logs.example create mode 100644 docker-compose.remote.yml create mode 100644 docker-compose.server-logs.yml create mode 100644 observability/prometheus.remote.yml.tmpl create mode 100644 observability/promtail.remote.yml.tmpl diff --git a/.env.remote.example b/.env.remote.example new file mode 100644 index 0000000..9a2eb50 --- /dev/null +++ b/.env.remote.example @@ -0,0 +1,4 @@ +REMOTE_SERVER_ADDRESS=example.com +REMOTE_APP_SCHEME=https +REMOTE_APP_PORT=8000 +REMOTE_METRICS_PATH=/metrics \ No newline at end of file diff --git a/.env.server-logs.example b/.env.server-logs.example new file mode 100644 index 0000000..205f34b --- /dev/null +++ b/.env.server-logs.example @@ -0,0 +1,3 @@ +REMOTE_LOKI_PUSH_URL=http://logs.example.com:3100/loki/api/v1/push +REMOTE_LOG_HOST=commerce-remote +REMOTE_LOG_JOB=docker \ No newline at end of file diff --git a/README.md b/README.md index c28c42f..f5837dd 100644 --- a/README.md +++ b/README.md @@ -522,6 +522,62 @@ Note: * Compose now includes a one-shot `migrate-indexes` service that runs `scripts/migrate_indexes.py` after PostgreSQL becomes healthy. * The `app` service waits for this migration to complete successfully before starting. +#### Option B: Run observability locally against a remote app and database + +Use the remote compose file when the API service and PostgreSQL already run on another host and you only want the local monitoring stack. + +1. Create a dedicated env file for the remote target: + +```bash +cp .env.remote.example .env.remote +``` + +2. Set the remote host and, if needed, the exposed app port: + +```bash +REMOTE_SERVER_ADDRESS=api.example.com +REMOTE_APP_SCHEME=https +REMOTE_APP_PORT=8000 +REMOTE_METRICS_PATH=/metrics +``` + +3. Start the remote-targeted observability stack: + +```bash +docker compose -f docker-compose.remote.yml --env-file .env.remote up -d +``` + +This stack does not start the `app`, `migrate-indexes`, or `db` services locally. Prometheus scrapes `REMOTE_APP_SCHEME://REMOTE_SERVER_ADDRESS:REMOTE_APP_PORTREMOTE_METRICS_PATH`, while Grafana, Tempo, Loki, and the OpenTelemetry Collector continue to run locally. + +Remote limitations: +* Metrics work immediately as long as the remote app exposes `/metrics` publicly or over a reachable private network. +* Traces appear only if the remote app is configured to send OTLP traffic to a collector endpoint reachable from that remote host. +* Logs require a separate `promtail` agent on the remote Docker host because the local stack can only scrape local container log files. + +Optional server-side log shipping: + +1. On the remote Docker host, create a log shipper env file: + +```bash +cp .env.server-logs.example .env.server-logs +``` + +2. Point it to a Loki endpoint reachable from that server: + +```bash +REMOTE_LOKI_PUSH_URL=https://logs.example.com/loki/api/v1/push +REMOTE_LOG_HOST=commerce-prod-01 +REMOTE_LOG_JOB=commerce-remote +``` + +3. Start the remote log shipper on that server: + +```bash +docker compose -f docker-compose.server-logs.yml --env-file .env.server-logs up -d +``` + +This ships Docker JSON logs from the remote host into Loki. If Loki runs on your workstation, expose it through a tunnel or reverse proxy first because the remote server must be able to reach `REMOTE_LOKI_PUSH_URL` directly. + Stop services: ```bash @@ -534,7 +590,7 @@ Stop services and remove the PostgreSQL volume: docker compose down -v ``` -#### Option B: Run app locally against PostgreSQL +#### Option C: Run app locally against PostgreSQL Before starting the server, ensure PostgreSQL is running. You can use Docker for the database only: @@ -751,6 +807,10 @@ Useful endpoints: * Tempo API: `http://127.0.0.1:3200` * Loki API: `http://127.0.0.1:3100` +For a remote-hosted API and database, use `docker-compose.remote.yml` instead. It keeps Grafana, Prometheus, Tempo, Loki, and the collector local, but scrapes the application metrics endpoint from `REMOTE_SERVER_ADDRESS:REMOTE_APP_PORT` configured through `.env.remote`. + +To ingest logs from that remote host as well, run `docker-compose.server-logs.yml` on the remote Docker server with `REMOTE_LOKI_PUSH_URL` set to a Loki endpoint the server can reach. + Relevant configuration files: * `app/observability/logging.py` configures structured JSON logging and trace/span correlation fields * `app/observability/setup.py` initializes tracing, metrics, middleware, and the `/metrics` endpoint @@ -759,6 +819,7 @@ Relevant configuration files: * `observability/otel-collector-config.yaml` configures OTLP ingestion and exporter pipeline * `observability/loki-config.yaml` configures Loki storage and ingestion * `observability/promtail-config.yaml` configures container log scraping and shipping to Loki +* `observability/promtail.remote.yml.tmpl` configures the remote-server Promtail agent for Loki shipping * `observability/prometheus.yml` configures scrape jobs and alert rule loading * `observability/grafana/provisioning` provisions Grafana datasources and dashboards diff --git a/docker-compose.remote.yml b/docker-compose.remote.yml new file mode 100644 index 0000000..ec1d599 --- /dev/null +++ b/docker-compose.remote.yml @@ -0,0 +1,71 @@ +services: + otel-collector: + image: otel/opentelemetry-collector-contrib:0.104.0 + container_name: commerce-otel-collector-remote + command: ["--config=/etc/otelcol-contrib/config.yaml"] + volumes: + - ./observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro + depends_on: + - tempo + ports: + - "4317:4317" + - "4318:4318" + - "8888:8888" + + tempo: + image: grafana/tempo:2.6.1 + container_name: commerce-tempo-remote + command: ["-config.file=/etc/tempo.yaml"] + volumes: + - ./observability/tempo.yaml:/etc/tempo.yaml:ro + ports: + - "3200:3200" + + loki: + image: grafana/loki:3.1.1 + container_name: commerce-loki-remote + command: ["-config.file=/etc/loki/config.yaml"] + volumes: + - ./observability/loki-config.yaml:/etc/loki/config.yaml:ro + - loki_remote_data:/loki + ports: + - "3100:3100" + + prometheus: + image: prom/prometheus:v2.53.1 + container_name: commerce-prometheus-remote + entrypoint: ["/bin/sh", "-c"] + environment: + REMOTE_SERVER_ADDRESS: ${REMOTE_SERVER_ADDRESS:?REMOTE_SERVER_ADDRESS is required} + REMOTE_APP_SCHEME: ${REMOTE_APP_SCHEME:-http} + REMOTE_APP_PORT: ${REMOTE_APP_PORT:-8000} + REMOTE_METRICS_PATH: ${REMOTE_METRICS_PATH:-/metrics} + command: + - 'sed -e "s|__REMOTE_SERVER_ADDRESS__|$${REMOTE_SERVER_ADDRESS}|g" -e "s|__REMOTE_APP_SCHEME__|$${REMOTE_APP_SCHEME}|g" -e "s|__REMOTE_APP_PORT__|$${REMOTE_APP_PORT}|g" -e "s|__REMOTE_METRICS_PATH__|$${REMOTE_METRICS_PATH}|g" /etc/prometheus/prometheus.remote.yml.tmpl > /tmp/prometheus.yml && /bin/prometheus --config.file=/tmp/prometheus.yml --storage.tsdb.path=/prometheus --web.console.libraries=/etc/prometheus/console_libraries --web.console.templates=/etc/prometheus/consoles' + volumes: + - ./observability/prometheus.remote.yml.tmpl:/etc/prometheus/prometheus.remote.yml.tmpl:ro + - ./observability/prometheus-alerts:/etc/prometheus/alerts:ro + depends_on: + - otel-collector + ports: + - "9090:9090" + + grafana: + image: grafana/grafana:11.1.0 + container_name: commerce-grafana-remote + environment: + GF_SECURITY_ADMIN_USER: admin + GF_SECURITY_ADMIN_PASSWORD: admin + volumes: + - ./observability/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro + - ./observability/grafana/provisioning/dashboards/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml:ro + - ./observability/grafana/dashboards:/var/lib/grafana/dashboards:ro + depends_on: + - prometheus + - tempo + - loki + ports: + - "3000:3000" + +volumes: + loki_remote_data: \ No newline at end of file diff --git a/docker-compose.server-logs.yml b/docker-compose.server-logs.yml new file mode 100644 index 0000000..a303fb3 --- /dev/null +++ b/docker-compose.server-logs.yml @@ -0,0 +1,20 @@ +services: + promtail: + image: grafana/promtail:3.1.1 + container_name: commerce-promtail-remote-agent + restart: unless-stopped + entrypoint: ["/bin/sh", "-c"] + environment: + REMOTE_LOKI_PUSH_URL: ${REMOTE_LOKI_PUSH_URL:?REMOTE_LOKI_PUSH_URL is required} + REMOTE_LOG_HOST: ${REMOTE_LOG_HOST:-commerce-remote} + REMOTE_LOG_JOB: ${REMOTE_LOG_JOB:-docker} + command: + - 'sed -e "s|__REMOTE_LOKI_PUSH_URL__|$${REMOTE_LOKI_PUSH_URL}|g" -e "s|__REMOTE_LOG_HOST__|$${REMOTE_LOG_HOST}|g" -e "s|__REMOTE_LOG_JOB__|$${REMOTE_LOG_JOB}|g" /etc/promtail/promtail.remote.yml.tmpl > /tmp/promtail.yml && /usr/bin/promtail --config.file=/tmp/promtail.yml' + volumes: + - ./observability/promtail.remote.yml.tmpl:/etc/promtail/promtail.remote.yml.tmpl:ro + - /var/lib/docker/containers:/var/lib/docker/containers:ro + - /var/run/docker.sock:/var/run/docker.sock:ro + - promtail_remote_positions:/var/lib/promtail + +volumes: + promtail_remote_positions: \ No newline at end of file diff --git a/observability/prometheus.remote.yml.tmpl b/observability/prometheus.remote.yml.tmpl new file mode 100644 index 0000000..a2ad1c4 --- /dev/null +++ b/observability/prometheus.remote.yml.tmpl @@ -0,0 +1,17 @@ +global: + scrape_interval: 5s + evaluation_interval: 5s + +rule_files: + - /etc/prometheus/alerts/*.yml + +scrape_configs: + - job_name: "commerce-app-remote" + scheme: __REMOTE_APP_SCHEME__ + metrics_path: __REMOTE_METRICS_PATH__ + static_configs: + - targets: ["__REMOTE_SERVER_ADDRESS__:__REMOTE_APP_PORT__"] + + - job_name: "otel-collector" + static_configs: + - targets: ["otel-collector:8888"] \ No newline at end of file diff --git a/observability/promtail.remote.yml.tmpl b/observability/promtail.remote.yml.tmpl new file mode 100644 index 0000000..68a260f --- /dev/null +++ b/observability/promtail.remote.yml.tmpl @@ -0,0 +1,21 @@ +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /var/lib/promtail/positions.yaml + +clients: + - url: __REMOTE_LOKI_PUSH_URL__ + +scrape_configs: + - job_name: docker-logs + static_configs: + - targets: + - localhost + labels: + job: __REMOTE_LOG_JOB__ + host: __REMOTE_LOG_HOST__ + __path__: /var/lib/docker/containers/*/*-json.log + pipeline_stages: + - docker: {} \ No newline at end of file From 48339abb2d5477b00426ae6e9a37e2a82181eb97 Mon Sep 17 00:00:00 2001 From: Vladislav Antonov Date: Sat, 14 Mar 2026 17:38:59 +0200 Subject: [PATCH 2/2] Add remote example configuration --- README.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f5837dd..20e71c2 100644 --- a/README.md +++ b/README.md @@ -541,13 +541,32 @@ REMOTE_APP_PORT=8000 REMOTE_METRICS_PATH=/metrics ``` +Example for this deployed site (`https://commercesystemdemo.onrender.com/`): + +```bash +REMOTE_SERVER_ADDRESS=commercesystemdemo.onrender.com +REMOTE_APP_SCHEME=https +REMOTE_APP_PORT=443 +REMOTE_METRICS_PATH=/metrics +``` + 3. Start the remote-targeted observability stack: ```bash docker compose -f docker-compose.remote.yml --env-file .env.remote up -d ``` -This stack does not start the `app`, `migrate-indexes`, or `db` services locally. Prometheus scrapes `REMOTE_APP_SCHEME://REMOTE_SERVER_ADDRESS:REMOTE_APP_PORTREMOTE_METRICS_PATH`, while Grafana, Tempo, Loki, and the OpenTelemetry Collector continue to run locally. +This stack does not start the `app`, `migrate-indexes`, or `db` services locally. Prometheus scrapes `REMOTE_APP_SCHEME://REMOTE_SERVER_ADDRESS:REMOTE_APP_PORT${REMOTE_METRICS_PATH}`, while Grafana, Tempo, Loki, and the OpenTelemetry Collector continue to run locally. + +Quick verification for the Render example: + +```bash +curl -sS https://commercesystemdemo.onrender.com/metrics | grep '^commerce_' | head +``` + +Then open local dashboards: +* Grafana: `http://127.0.0.1:3000` +* Prometheus targets: `http://127.0.0.1:9090/targets` Remote limitations: * Metrics work immediately as long as the remote app exposes `/metrics` publicly or over a reachable private network.