Add request-scoped DB query tracing to metadata service

[vagxrth]

Summary

Closes #5.

This PR adds request-scoped DB query tracing to the metadata service using a ContextVar-backed tracer plus an aiohttp middleware.

The tracing is fully opt-in:

• deployment must enable QUERY_TRACING_ENABLED=1
• a request must send X-Metaflow-Trace-DB: 1

When enabled for a request, the metadata service logs:

• query_count
• total_rows
• db_time_ms
• request_time_ms

at the end of the request, without changing any API payloads or schemas.

What Changed

Request lifecycle tracing

• Added a new shared tracing module for metadata DB request tracing.
• Added an aiohttp middleware that:
  • checks the opt-in tracing header
  • creates a request-local tracer
  • resets tracer state in finally
  • emits one INFO summary log per traced request
  • emits per-query DEBUG logs only when DEBUG logging is enabled

DB instrumentation

• Instrumented the shared execute_sql() path used by metadata-service read APIs.
• Query timing is measured only around:
  • execute(...)
  • fetchall()
• Python row serialization is intentionally excluded from db_time_ms.

Failure handling

• Failed queries are still traced.
• On query failure, the trace captures:
  • table name
  • elapsed DB time
  • row_count=0
  • error_type
• The original exception flow is preserved.

Safe SQL logging

• Per-query SQL detail is only retained/logged when DEBUG is enabled.
• SQL previews are truncated.
• Bound parameter values are never logged; only the SQL template is recorded.

Why This Approach

This implementation keeps the strong parts of the linked PR direction (ContextVar + middleware), but tightens the behavior in a few important ways:

• tracing is per-request, not “all requests whenever enabled”
• DB timing reflects actual DB work, not Python serialization overhead
• failed queries are included in traces
• detailed SQL logging is DEBUG-only to reduce overhead and noise
• the summary log format is stable and easy to use for follow-on benchmarking work

Logging Behavior

For traced requests, the service logs one request summary like:

• method
• path
• status
• query_count
• total_rows
• db_time_ms
• request_time_ms

When the QueryTracing logger is in DEBUG, it also logs one line per query with:

• table
• success/failure
• row count
• DB time
• error type
• truncated SQL template

Tests Added

Unit tests

Added coverage for:

• tracer context setup/reset
• env/header activation behavior
• INFO-only summary logging
• DEBUG per-query logging
• failed query tracing
• SQL template safety (no bound values logged)
• ContextVar isolation across concurrent tasks

Integration tests

Added coverage for:

• traced GET /flows/{flow_id}/runs
• traced GET /flows/{flow_id}/runs/{run_number}/steps/{step_name}/tasks
• no trace when the header is absent
• no trace when tracing is disabled

Local Verification

Passed locally:

• python3 -m py_compile ...
• git diff --check
• ./.venv/bin/pytest services/metadata_service/tests/unit_tests/query_tracing_test.py -q

Integration tests are included, but could not be completed end-to-end in this environment because the repo’s integration suite expects:

• a dedicated test Postgres at db_test
• the migration tooling (goose) available in the test environment

Notes

This PR is intentionally scoped to metadata-service read tracing through execute_sql(). It does not expand tracing to:

• mutation paths
• healthcheck/direct cursor paths
• response metadata changes
• benchmark/reporting endpoints