Add remote metrics setup

.env.remote.example

@@ -0,0 +1,4 @@
+REMOTE_SERVER_ADDRESS=example.com
+REMOTE_APP_SCHEME=https
+REMOTE_APP_PORT=8000
+REMOTE_METRICS_PATH=/metrics

.env.server-logs.example

@@ -0,0 +1,3 @@
+REMOTE_LOKI_PUSH_URL=http://logs.example.com:3100/loki/api/v1/push
+REMOTE_LOG_HOST=commerce-remote
+REMOTE_LOG_JOB=docker

README.md

@@ -522,6 +522,81 @@ Note:
 * Compose now includes a one-shot `migrate-indexes` service that runs `scripts/migrate_indexes.py` after PostgreSQL becomes healthy.
 * The `app` service waits for this migration to complete successfully before starting.
 
+#### Option B: Run observability locally against a remote app and database
+
+Use the remote compose file when the API service and PostgreSQL already run on another host and you only want the local monitoring stack.
+
+1. Create a dedicated env file for the remote target:
+
+```bash
+cp .env.remote.example .env.remote
+```
+
+2. Set the remote host and, if needed, the exposed app port:
+
+```bash
+REMOTE_SERVER_ADDRESS=api.example.com
+REMOTE_APP_SCHEME=https
+REMOTE_APP_PORT=8000
+REMOTE_METRICS_PATH=/metrics
+```
+
+Example for this deployed site (`https://commercesystemdemo.onrender.com/`):
+
+```bash
+REMOTE_SERVER_ADDRESS=commercesystemdemo.onrender.com
+REMOTE_APP_SCHEME=https
+REMOTE_APP_PORT=443
+REMOTE_METRICS_PATH=/metrics
+```
+
+3. Start the remote-targeted observability stack:
+
+```bash
+docker compose -f docker-compose.remote.yml --env-file .env.remote up -d
+```
+
+This stack does not start the `app`, `migrate-indexes`, or `db` services locally. Prometheus scrapes `REMOTE_APP_SCHEME://REMOTE_SERVER_ADDRESS:REMOTE_APP_PORT${REMOTE_METRICS_PATH}`, while Grafana, Tempo, Loki, and the OpenTelemetry Collector continue to run locally.
+
+Quick verification for the Render example:
+
+```bash
+curl -sS https://commercesystemdemo.onrender.com/metrics | grep '^commerce_' | head
+```
+
+Then open local dashboards:
+* Grafana: `http://127.0.0.1:3000`
+* Prometheus targets: `http://127.0.0.1:9090/targets`
+
+Remote limitations:
+* Metrics work immediately as long as the remote app exposes `/metrics` publicly or over a reachable private network.
+* Traces appear only if the remote app is configured to send OTLP traffic to a collector endpoint reachable from that remote host.
+* Logs require a separate `promtail` agent on the remote Docker host because the local stack can only scrape local container log files.
+
+Optional server-side log shipping:
+
+1. On the remote Docker host, create a log shipper env file:
+
+```bash
+cp .env.server-logs.example .env.server-logs
+```
+
+2. Point it to a Loki endpoint reachable from that server:
+
+```bash
+REMOTE_LOKI_PUSH_URL=https://logs.example.com/loki/api/v1/push
+REMOTE_LOG_HOST=commerce-prod-01
+REMOTE_LOG_JOB=commerce-remote
+```
+
+3. Start the remote log shipper on that server:
+
+```bash
+docker compose -f docker-compose.server-logs.yml --env-file .env.server-logs up -d
+```
+
+This ships Docker JSON logs from the remote host into Loki. If Loki runs on your workstation, expose it through a tunnel or reverse proxy first because the remote server must be able to reach `REMOTE_LOKI_PUSH_URL` directly.
+
 Stop services:
 
 ```bash
@@ -534,7 +609,7 @@ Stop services and remove the PostgreSQL volume:
 docker compose down -v
 ```
 
-#### Option B: Run app locally against PostgreSQL
+#### Option C: Run app locally against PostgreSQL
 
 Before starting the server, ensure PostgreSQL is running. You can use Docker for the database only:
 
@@ -751,6 +826,10 @@ Useful endpoints:
 * Tempo API: `http://127.0.0.1:3200`
 * Loki API: `http://127.0.0.1:3100`
 
+For a remote-hosted API and database, use `docker-compose.remote.yml` instead. It keeps Grafana, Prometheus, Tempo, Loki, and the collector local, but scrapes the application metrics endpoint from `REMOTE_SERVER_ADDRESS:REMOTE_APP_PORT` configured through `.env.remote`.
+
+To ingest logs from that remote host as well, run `docker-compose.server-logs.yml` on the remote Docker server with `REMOTE_LOKI_PUSH_URL` set to a Loki endpoint the server can reach.
+
 Relevant configuration files:
 * `app/observability/logging.py` configures structured JSON logging and trace/span correlation fields
 * `app/observability/setup.py` initializes tracing, metrics, middleware, and the `/metrics` endpoint
@@ -759,6 +838,7 @@ Relevant configuration files:
 * `observability/otel-collector-config.yaml` configures OTLP ingestion and exporter pipeline
 * `observability/loki-config.yaml` configures Loki storage and ingestion
 * `observability/promtail-config.yaml` configures container log scraping and shipping to Loki
+* `observability/promtail.remote.yml.tmpl` configures the remote-server Promtail agent for Loki shipping
 * `observability/prometheus.yml` configures scrape jobs and alert rule loading
 * `observability/grafana/provisioning` provisions Grafana datasources and dashboards
 

docker-compose.remote.yml

@@ -0,0 +1,71 @@
+services:
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:0.104.0
+    container_name: commerce-otel-collector-remote
+    command: ["--config=/etc/otelcol-contrib/config.yaml"]
+    volumes:
+      - ./observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    depends_on:
+      - tempo
+    ports:
+      - "4317:4317"
+      - "4318:4318"
+      - "8888:8888"
+
+  tempo:
+    image: grafana/tempo:2.6.1
+    container_name: commerce-tempo-remote
+    command: ["-config.file=/etc/tempo.yaml"]
+    volumes:
+      - ./observability/tempo.yaml:/etc/tempo.yaml:ro
+    ports:
+      - "3200:3200"
+
+  loki:
+    image: grafana/loki:3.1.1
+    container_name: commerce-loki-remote
+    command: ["-config.file=/etc/loki/config.yaml"]
+    volumes:
+      - ./observability/loki-config.yaml:/etc/loki/config.yaml:ro
+      - loki_remote_data:/loki
+    ports:
+      - "3100:3100"
+
+  prometheus:
+    image: prom/prometheus:v2.53.1
+    container_name: commerce-prometheus-remote
+    entrypoint: ["/bin/sh", "-c"]
+    environment:
+      REMOTE_SERVER_ADDRESS: ${REMOTE_SERVER_ADDRESS:?REMOTE_SERVER_ADDRESS is required}
+      REMOTE_APP_SCHEME: ${REMOTE_APP_SCHEME:-http}
+      REMOTE_APP_PORT: ${REMOTE_APP_PORT:-8000}
+      REMOTE_METRICS_PATH: ${REMOTE_METRICS_PATH:-/metrics}
+    command:
+      - 'sed -e "s|__REMOTE_SERVER_ADDRESS__|$${REMOTE_SERVER_ADDRESS}|g" -e "s|__REMOTE_APP_SCHEME__|$${REMOTE_APP_SCHEME}|g" -e "s|__REMOTE_APP_PORT__|$${REMOTE_APP_PORT}|g" -e "s|__REMOTE_METRICS_PATH__|$${REMOTE_METRICS_PATH}|g" /etc/prometheus/prometheus.remote.yml.tmpl > /tmp/prometheus.yml && /bin/prometheus --config.file=/tmp/prometheus.yml --storage.tsdb.path=/prometheus --web.console.libraries=/etc/prometheus/console_libraries --web.console.templates=/etc/prometheus/consoles'
+    volumes:
+      - ./observability/prometheus.remote.yml.tmpl:/etc/prometheus/prometheus.remote.yml.tmpl:ro
+      - ./observability/prometheus-alerts:/etc/prometheus/alerts:ro
+    depends_on:
+      - otel-collector
+    ports:
+      - "9090:9090"
+
+  grafana:
+    image: grafana/grafana:11.1.0
+    container_name: commerce-grafana-remote
+    environment:
+      GF_SECURITY_ADMIN_USER: admin
+      GF_SECURITY_ADMIN_PASSWORD: admin
+    volumes:
+      - ./observability/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro
+      - ./observability/grafana/provisioning/dashboards/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml:ro
+      - ./observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    ports:
+      - "3000:3000"
+
+volumes:
+  loki_remote_data:

docker-compose.server-logs.yml

@@ -0,0 +1,20 @@
+services:
+  promtail:
+    image: grafana/promtail:3.1.1
+    container_name: commerce-promtail-remote-agent
+    restart: unless-stopped
+    entrypoint: ["/bin/sh", "-c"]
+    environment:
+      REMOTE_LOKI_PUSH_URL: ${REMOTE_LOKI_PUSH_URL:?REMOTE_LOKI_PUSH_URL is required}
+      REMOTE_LOG_HOST: ${REMOTE_LOG_HOST:-commerce-remote}
+      REMOTE_LOG_JOB: ${REMOTE_LOG_JOB:-docker}
+    command:
+      - 'sed -e "s|__REMOTE_LOKI_PUSH_URL__|$${REMOTE_LOKI_PUSH_URL}|g" -e "s|__REMOTE_LOG_HOST__|$${REMOTE_LOG_HOST}|g" -e "s|__REMOTE_LOG_JOB__|$${REMOTE_LOG_JOB}|g" /etc/promtail/promtail.remote.yml.tmpl > /tmp/promtail.yml && /usr/bin/promtail --config.file=/tmp/promtail.yml'
+    volumes:
+      - ./observability/promtail.remote.yml.tmpl:/etc/promtail/promtail.remote.yml.tmpl:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - promtail_remote_positions:/var/lib/promtail
+
+volumes:
+  promtail_remote_positions:

observability/prometheus.remote.yml.tmpl

@@ -0,0 +1,17 @@
+global:
+  scrape_interval: 5s
+  evaluation_interval: 5s
+
+rule_files:
+  - /etc/prometheus/alerts/*.yml
+
+scrape_configs:
+  - job_name: "commerce-app-remote"
+    scheme: __REMOTE_APP_SCHEME__
+    metrics_path: __REMOTE_METRICS_PATH__
+    static_configs:
+      - targets: ["__REMOTE_SERVER_ADDRESS__:__REMOTE_APP_PORT__"]
+
+  - job_name: "otel-collector"
+    static_configs:
+      - targets: ["otel-collector:8888"]

observability/promtail.remote.yml.tmpl

@@ -0,0 +1,21 @@
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /var/lib/promtail/positions.yaml
+
+clients:
+  - url: __REMOTE_LOKI_PUSH_URL__
+
+scrape_configs:
+  - job_name: docker-logs
+    static_configs:
+      - targets:
+          - localhost
+        labels:
+          job: __REMOTE_LOG_JOB__
+          host: __REMOTE_LOG_HOST__
+          __path__: /var/lib/docker/containers/*/*-json.log
+    pipeline_stages:
+      - docker: {}