feat: support custom HTTP headers (API gateway / IBM APIC)

[robinliubin]

User description

Summary

Adds support for sending customer-defined HTTP headers on every request the SDK makes to the Galileo API. This is required when the Galileo API is fronted by an API gateway (e.g. IBM APIC) that expects client credentials such as ibm_client_id / ibm_client_secret on each call.

The feature is header-agnostic: the customer defines both the header names and values via a single config knob.

What changed (this repo)

The distributed/streaming ingestion client (IngestTraces) builds its own httpx.AsyncClient and bypasses galileo-core's ApiClient, so it needs the headers wired in explicitly:

• IngestTraces now accepts extra_headers and merges them into its request headers, with Galileo's own headers (Galileo-API-Key, Content-Type, X-Galileo-SDK) taking precedence on key collisions.
• GalileoLogger._create_traces_client passes config.extra_headers.
• GalileoPythonConfig inherits the new extra_headers field from galileo-core, so the standard API and batch-ingestion paths are already covered by the core change.

Configuration (customer-facing)

export GALILEO_EXTRA_HEADERS='{"ibm_client_id": "my_client_id", "ibm_client_secret": "my_client_secret"}'

or programmatically:

from galileo.config import GalileoPythonConfig
GalileoPythonConfig.get(extra_headers={"ibm_client_id": "...", "ibm_client_secret": "..."})

Dependency

⚠️ Depends on the matching galileo-core change: rungalileo/orbit#1376 (feat(core): support custom extra headers on every request). That PR adds the extra_headers field to ApiClient / GalileoConfig and applies it to all core egress paths (resource management, batch ingestion, and pre-auth login/healthcheck). This SDK PR only covers the distributed-ingestion path that bypasses core, and must land/release after the core change is published to PyPI.

Testing

• Added TestIngestTracesExtraHeaders in tests/test_traces_client_headers.py covering: extra headers present, Galileo headers win on collision, and default (no extra headers).
• Runtime-verified the header merge with the concrete IBM APIC headers.

🤖 Generated with Claude Code

---

Generated description

Below is a concise technical summary of the changes proposed in this PR:
Add support for customer-defined HTTP headers across the SDK's distributed ingest flow by wiring extra_headers into IngestTraces, GalileoLogger, and the ingest health probe. Update the configuration docs and tests so gateway credentials like ibm_client_id / ibm_client_secret are sent on every relevant request while Galileo's own headers still win on collisions.

Topic	Details
Custom headers	Propagate customer headers through IngestTraces and GalileoLogger, and key the ingest health check cache by api_url plus extra_headers so gateway-authenticated probes rerun correctly. Modified files (3) AGENTS.md src/galileo/logger/logger.py src/galileo/traces.py Latest Contributors(2) User Commit Date bin@galileo.ai fix(logger): key inges... July 30, 2026 david@galileo.ai fix: Handle checking/u... June 01, 2026
Header tests	Cover extra-header merging, collision handling, and probe cache behavior with tests for the ingest client and logger health check. Modified files (2) tests/test_logger_distributed.py tests/test_traces_client_headers.py Latest Contributors(2) User Commit Date bin@galileo.ai fix(logger): key inges... July 30, 2026 namrata.ghadi@galileo.ai feat: add agentcontrol... May 11, 2026

_{Review this PR on Baz | Customize your next review}

[codecov]

Codecov Report

❌ Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.62%. Comparing base (2b2a3ee) to head (198514b).

Files with missing lines	Patch %	Lines
src/galileo/logger/logger.py	93.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #629      +/-   ##
==========================================
+ Coverage   83.53%   83.62%   +0.09%     
==========================================
  Files         124      124              
  Lines       11029    11037       +8     
==========================================
+ Hits         9213     9230      +17     
+ Misses       1816     1807       -9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[robinliubin]

✅ End-to-end validation against a live Galileo backend

Before release, this change was validated end-to-end through a mock IBM APIC gateway fronting a real Galileo API, proving the SDK carries the custom headers on every egress path.

 local Galileo SDK  ──►  mock APIC gateway  ──►  Galileo API
 (this branch's code,   (validates the two      (real backend, TLS)
  extra_headers)         IBM headers on every
                         request; 401 if absent)

The gateway enforces X-IBM-Client-Id / X-IBM-Client-Secret on every request and forwards valid ones upstream — the exact failure mode a real APIC gateway produces if a header is missing.

Results — all PASS

Scenario	Result
With IBM headers (programmatic extra_headers={...})	✅ SDK logged in; current_user via gateway returned test@galileo.ai
With IBM headers (env var GALILEO_EXTRA_HEADERS)	✅ Same — env-var path works
Without IBM headers (negative control)	✅ Gateway returned 401; SDK init failed as expected

Proof — gateway log shows every SDK egress path carried the headers

200-path forwarding GET  /healthcheck   -> https://api:8088/healthcheck
200-path forwarding POST /login/api_key -> https://api:8088/login/api_key
200-path forwarding GET  /current_user  -> https://api:8088/current_user
401 rejecting       GET  /healthcheck — IBM headers invalid (id_present=False secret_present=False)

Every pre-auth (/healthcheck), login (/login/api_key), and authenticated (/current_user) call arrived at the gateway already carrying valid IBM headers and was forwarded to the Galileo API. The one request sent without headers was rejected with 401.

_{Validated with the local working copies of both galileo and galileo-core (this branch), not the PyPI releases.}

[robinliubin]

✅ Trace logging verified end-to-end through DevStack mock APIC

Building on the earlier gateway validation, I took the simplest run-Galileo demo (notebooks/1_chatbot.ipynb, cell 20 — a manual trace with one LLM span) and pushed a real trace through the mock IBM APIC gateway into a live Galileo backend, using this branch's SDK.

 GalileoLogger  ──►  mock APIC gateway  ──►  Galileo API
 (extra_headers =    (validates the two      (real backend;
  IBM headers)        IBM headers, forwards)   trace persisted)

The push

SDK config:
  api_url       : http://localhost:8087/          ← the APIC gateway
  extra_headers : {'X-IBM-Client-Id': ..., 'X-IBM-Client-Secret': ...}
RESULT: PASS — trace pushed via APIC gateway to Galileo API

Gateway log — every logging call routed through APIC → Galileo API

GET  /projects                        -> https://api:8088/...
POST /projects                        -> https://api:8088/...   (created project)
POST /projects/{id}/log_streams       -> https://api:8088/...   (created log stream)
POST /v2/projects/{id}/traces         -> https://api:8088/...   (INGESTED THE TRACE)

Each call required valid IBM headers to pass the gateway and was forwarded to the real Galileo API.

Persistence confirmed — queried the trace back through the gateway

traces/search -> OK via gateway; records: 1
   input   : I need help with my billing statement.
   tags    : ['prod', 'support', 'apic-header-test']
   metadata: {'customer_tier': 'premium', 'region': 'us-west', 'gateway': 'mock-apic'}

The exact record pushed — input, tags, and metadata — was read back, proving the trace was truly logged, not merely accepted.

Takeaway

The full customer scenario works: the Galileo SDK, configured with extra_headers, sends traces through an IBM APIC gateway (which enforces the two headers) and successfully logs them to the Galileo API — with no change required on the Galileo side.

_{Ingestion used the standard Traces client (core ApiClient, which carries extra_headers) since the Go ingest service wasn't live in this env; the IngestTraces path is also wired for extra_headers (galileo-python #629).}

[robinliubin]

Addressed review feedback (commit ecb412c)

Thanks @baz-reviewer — all three findings were valid and are now fixed.

1 & 2 — Headerless /ingest/healthz probe (Logical Bug / Breaking Change, high).
_is_ingest_service_available() now reads config.extra_headers (via getattr, so it degrades safely on older galileo-core) and forwards them on the httpx.get(.../ingest/healthz) probe. A gateway that enforces the IBM APIC headers on every request — health checks included — no longer 401s the probe and mistakenly reports the ingest service as down. This makes the runtime behaviour match the "headers applied to all egress paths" contract documented in AGENTS.md.

3 — Case-insensitive header collision (Security, low).
IngestTraces.__init__ now normalizes (trim + lowercase) both the reserved set (content-type, galileo-api-key, x-galileo-sdk) and the caller's extra_headers keys, dropping any case-insensitive collision before Galileo's canonical headers are appended. A caller passing content-type/Galileo-API-Key in different casing can no longer put two copies on the wire. Added test_extra_headers_case_insensitive_collision_dropped covering case + whitespace variants.

The identical case-insensitive guard was applied to galileo_core.ApiClient (the standard/pre-auth path) in the paired orbit PR, so all egress paths behave consistently.

Tests: tests/test_traces_client_headers.py + tests/test_logger_distributed.py (new test_is_ingest_service_available_forwards_extra_headers) — 46 passed, ruff clean.

[robinliubin]

Addressed (commit 198514b)

• Stale ingest cache ignores headers (logger.py): the /ingest/healthz result is no longer stored in a single process-wide slot. It's now keyed by the effective probe configuration — (api_url, sorted(extra_headers)) via the new _ingest_cache_key helper — so a GalileoPythonConfig.reset() or a later get(extra_headers=...) that changes where/how we probe re-runs the check instead of reusing an availability result gathered under a different gateway or auth headers. Added test_is_ingest_service_available_reprobes_when_config_changes (same URL, different headers → two probes, each carrying its own headers).
• Sentence-case Given description (test_logger_distributed.py): tightened the Given/When/Then comments to single-clause sentence case per AGENTS.md.

47 tests pass across test_logger_distributed.py + test_traces_client_headers.py, ruff clean.

[fercor-cisco]

🤖 This review was generated by the Astra agent (claude-opus-4-8). It may contain mistakes.

Verdict: approve — Well-implemented, thoroughly tested feature; all prior review comments addressed and verified. Only a non-blocking dependency-pin concern remains.

General Comments

• 🟡 minor (design): This feature is only functional when galileo-core actually provides the extra_headers field, but pyproject.toml still pins galileo-core = "^4.4.0" and the code uses getattr(config, "extra_headers", None) fallbacks everywhere. If a user resolves/locks to a core release without the field, setting GALILEO_EXTRA_HEADERS silently no-ops on every path (distributed ingest via the getattr fallback; resource-management/batch/login via core) with no warning or error — a hard-to-diagnose failure mode for a gateway-credentials feature, since the gateway will simply return 401. The PR body acknowledges the release ordering, but the version constraint doesn't enforce it. Recommend bumping the galileo-core minimum to the release that adds extra_headers (in both the main and test dependency entries) once it's published, so a fresh resolve can't pick an incompatible core.

Follow-ups

Suggested follow-up work that could be tracked as Jira tickets:

• pyproject.toml:39-57: After the corresponding galileo-core change is published to PyPI, bump the galileo-core version constraint (lines 39 and 57) from ^4.4.0 to the release that introduces the extra_headers field, so the SDK can't resolve to a core that silently ignores GALILEO_EXTRA_HEADERS. At that point the defensive getattr(config, "extra_headers", None) fallbacks in logger.py/traces.py could also be simplified to direct attribute access.