Vertex: Agent.chat context canceled — http2 "no cached connection was available" fetching oauth2 token (intermittent)

[PrenSJ2]

Environment

• SDK version: google-antigravity 0.1.6 (also reproduced on 0.1.5)
• Backend: Vertex AI (LocalAgentConfig(vertex=True, project=…, location=…)), ADC
• Auth: user ADC (gcloud auth application-default login) and service account (Cloud Run Workload Identity / metadata server) — both reproduce
• OS/Python: macOS + Python 3.14 (local) and python:3.11-slim on Cloud Run — both reproduce
• Models: gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash-001 — all reproduce

Summary

On the Vertex backend, Agent.chat(...) intermittently fails: the harness's streamGenerateContent call is context canceled, usage_metadata is None, no result is produced (and agent.chat() can hang). It works when "fresh" but, under sustained use, degrades to a persistent failure window that recovers on its own after some time. Read/navigation runs and file-edit runs both fail once in a bad window (it is not task-specific). The same code works on the Gemini Developer API (api_key) path.

Root cause (from GODEBUG=http2debug=1)

Running with GODEBUG=http2debug=1 shows the real error immediately before the cancel:

http2: Transport failed to get client conn for oauth2.googleapis.com:443:
       http2: no cached connection was available
received model response error: doRequest: error sending request:
  Post ".../gemini-2.5-pro:streamGenerateContent?alt=sse":
  Post "https://oauth2.googleapis.com/token": context canceled

So the bundled localharness Go binary's HTTP/2 client fails to obtain a connection to the OAuth token endpoint (http2: no cached connection was available — a golang.org/x/net/http2 connection-pool failure mode where a pooled h2 conn goes stale/GOAWAY and the transport does not dial a fresh one). That cancels the token fetch, which fails the model request. On Cloud Run the same context canceled appears on the metadata token endpoint (169.254.169.254).

GODEBUG=http2client=0 (force HTTP/1.1) does not help — the binary appears to use a custom http2.Transport rather than net/http's, so the flag doesn't apply.

Proof it is the harness, not Vertex / creds / quota

Same machine, same ADC, same project, same minute:

Path	Result
direct google.auth + REST …:generateContent	3/3 HTTP 200
direct google.auth + REST …:streamGenerateContent?alt=sse (harness's exact endpoint), incl. a ~40K-token prompt	3/3 HTTP 200, streams
Agent.chat(...) on the same model/project	fails (context canceled, usage=None)

So Vertex, credentials, quota, the streaming endpoint, and request size are all healthy — only the harness's own HTTP client fails. Retrying Agent.chat() within a bad window does not recover (8/8 fail); it recovers only after an extended idle period.

Minimal repro

import asyncio
from google.antigravity import Agent, LocalAgentConfig

async def main():
    cfg = LocalAgentConfig(vertex=True, project="<PROJECT>", location="us-central1",
                           model="gemini-2.5-pro", workspaces=["/tmp/repo"])
    async with Agent(cfg) as agent:
        r = await agent.chat("Create hello.py with def hi(): return 'hi'. Use your file tools.")
        print(r.usage_metadata)   # None once in a failure window; run under GODEBUG=http2debug=1 to see the http2 error

asyncio.run(main())

Ask

Please look at the OAuth/token HTTP client's HTTP/2 connection lifecycle on the Vertex path — e.g. enabling h2 health checks (http2.Transport.ReadIdleTimeout / PingTimeout) or retrying on no cached connection was available / stale-conn errors. Exposing model-call retry/backoff (#124) would also give callers a mitigation.

Related

• parent_idle latches True #117 — another context canceled bug, but a different cause (parent-idle latch in sub-agent fan-out); this one has no sub-agents and is the HTTP/2 token-connection failure.
• Expose model-call retry/backoff configuration through the SDK (HarnessConfig) #124 — retry/backoff config request; a built-in retry would mitigate this.

[PrenSJ2]

Additional evidence — this is not specific to Vertex or the oauth2 token endpoint; it's the harness's HTTP/2 client degrading under sustained use for any googleapis host.

Under load (many runs in a window), the failure appears simultaneously on both backends:

• Vertex (aiplatform.googleapis.com, oauth2 ADC): fails context canceled (with the http2: no cached connection was available on oauth2.googleapis.com shown under GODEBUG=http2debug=1).
• Developer API (generativelanguage.googleapis.com, api_key, no oauth2 token fetch): fails …:streamGenerateContent: context canceled at the same time — 6/6.

Meanwhile, at the same instant, a plain httpx/google.auth client on the same machine gets 3/3 HTTP 200 to the identical …:streamGenerateContent?alt=sse endpoint. So the endpoints, credentials, quota, and network are healthy — only the bundled localharness HTTP/2 client fails.

It's windowed: works when "fresh", degrades to a persistent failure after sustained use, and recovers on its own after an idle period. Retrying Agent.chat() within a bad window does not recover it (8/8 fail).

This suggests the fix is in the harness's shared HTTP/2 transport / connection-pool handling (stale-conn / GOAWAY recovery, ReadIdleTimeout), not anything auth- or backend-specific.

[PrenSJ2]

Re-tested on 0.1.7 with the global endpoint — still reproduces, unchanged.

• SDK: 0.1.7 (earlier reports above were 0.1.5/0.1.6)
• Endpoint: locations/global — so the failure is region-independent as well
• Models: all five this project can reach on global — gemini-3.5-flash, gemini-3-flash-preview, gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite — 5/5 fail identically (sequential runs, macOS local, user ADC):

received model response error: doRequest: error sending request:
  Post "https://aiplatform.googleapis.com/v1beta1/projects/<PROJECT>/locations/global/publishers/google/models/gemini-3.5-flash:streamGenerateContent?alt=sse":
  Post "https://oauth2.googleapis.com/token": context canceled

Control, same minute, same ADC: direct REST generateContent to the identical global endpoint returns HTTP 200 for all five models.

One data point against the "recovers after idle" framing: this machine had been idle ~2 days since the last antigravity run, and the first run today failed immediately — the bad window here now looks persistent rather than load-induced.

Updated matrix: fails independent of model (2.5 line, 3-flash, 3.5-flash), region (us-central1, global), auth (user ADC, SA/metadata), environment (macOS, Cloud Run cold), and SDK version (0.1.5–0.1.7). Developer-API path remains the only working one.