Skip to content

SDK client init hangs 40-90s inside K8s pods on Debian 13 (kernel 6.12); succeeds on host with identical static binary #277

Description

@gokhan-uysal

Environment

  • Operator versions tested: v1.12.0 and v1.10.1 (both fail with the same signature)
  • Bundled SDK (both versions): github.com/1password/onepassword-sdk-go v0.3.1
  • Chart: 1Password/connect-helm-charts connect v2.4.1 with operator.authMethod: service-account
  • Kubernetes: v1.36.2 (kubeadm)
  • Container runtime: containerd 1.7.24
  • Node OS: Debian GNU/Linux 13 (trixie), kernel 6.12.95+deb13-amd64
  • CNI: Calico v3.30.4, IPIP mode
  • Auth method: 1Password Service Account (per USAGEGUIDE.md)

Summary

With operator.authMethod: service-account, the operator fails to start. sdk.NewClient(ctx, WithServiceAccountToken) blocks and eventually errors out with a WASM stack trace pointing at Extism / tetratelabs/wazero's HTTP host function while calling https://my.1password.com/api/v2/auth/verify (or /v1/auth, /v3/auth/start depending on SDK version — same class of failure).

Isolation: the same static Go SDK binary that fails inside a pod on this cluster succeeds in ~4 s on the host (same LAN, same egress). Same 1P endpoint responds to curl in 400 ms from inside the failing pod. Same Service Account token works with op CLI inside the same pod. Only the WASM+wazero HTTP path stalls, and only inside a K8s pod on this Debian 13 node.

Operator log

INFO  setup  Operator Version: 1.12.0
INFO  setup  Go Version: go1.24.13
INFO  setup  Using Service Account Token
[~90 seconds silence]
ERROR setup unable to create 1Password client
{"error": "1Password sdk error: error initializing client:
 Post \"https://my.1password.com/api/v2/auth/verify?\": context canceled
 (recovered by wazero)
 wasm stack trace:
   extism:host/env.http_request(i64,i64) i64
   op_extism_core.wasm._ZN10extism_pdk4http7request...
   op_extism_core.wasm._ZN12op_b5_client9auth_flow20authenticate_session...
   op_extism_core.wasm.init_client() i32"}

Container exits, CrashLoopBackOff.

Diagnostic tests (all ruled out)

# Test Result
1 Token is valid → op CLI (image 1password/op:2) inside a pod in the operator's namespace with the token from the same K8s Secret runs op vault list Lists the accessible vault immediately
2 Endpoint reachable → curl -X POST -m 15 https://my.1password.com/api/v2/auth/verify from a pod on the same node HTTP 401 in ~400 ms
3 DNS works → getent ahosts my.1password.com from pod Resolves; A records reachable
4 Try operator v1.10.1 (older) Same failure (same bundled SDK)
5 Force IPv4 with hostAliases (single 1P IPv4) Same 40s timeout
6 Increase pod resources to 2 CPU / 2.5 Gi Same 40s timeout
7 hostNetwork: true (bypass Calico overlay) Same 40s timeout
8 Swap base to alpine:3.20 with CA certificates (not distroless) Same 40s timeout
9 Minimal Go program: sdk.NewClient + Vaults().List, SDK v0.3.1, CGO_ENABLED=0 static build Same failure inside pod
10 Same program with SDK v0.4.0 (CGO enabled) Same failure inside pod
11 Same static binary from #9 on the host (Arch Linux, kernel 7.1, same LAN) Client created OK in ~4.09 s, vault listed

Minimal reproduction

package main

import (
    "context"
    "fmt"
    "os"
    "time"

    onepassword "github.com/1password/onepassword-sdk-go"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    start := time.Now()
    client, err := onepassword.NewClient(ctx,
        onepassword.WithServiceAccountToken(os.Getenv("OP_SERVICE_ACCOUNT_TOKEN")),
        onepassword.WithIntegrationInfo("sdktest", "0.0.1"),
    )
    if err != nil {
        fmt.Printf("NewClient error after %v: %v\n", time.Since(start), err)
        os.Exit(1)
    }
    fmt.Printf("Client created OK in %v\n", time.Since(start))

    vaults, _ := client.Vaults().List(ctx)
    for _, v := range vaults {
        fmt.Printf("Vault: %s (%s)\n", v.Title, v.ID)
    }
}

Build: CGO_ENABLED=0 GOOS=linux go build -o sdktest .

  • Runs on Arch host (kernel 7.1) → success in ~4 s
  • Same binary copied into an Alpine pod on this Debian 13 cluster (kernel 6.12), OP_SERVICE_ACCOUNT_TOKEN from a K8s Secret → context deadline exceeded at 30 s

Working alternatives from inside the same pod

  • curl -X POST https://my.1password.com/api/v2/auth/verify → 401 in ~400 ms
  • op vault list (using the same Service Account token) → lists the vault

Only the SDK's Extism / wazero HTTP path stalls.

Hypothesis (unverified)

Extism's HTTP host function (via tetratelabs/wazero) hits a runtime edge case in this specific pod environment (Debian 13, kernel 6.12) that does not appear on the host (Arch, kernel 7.1). Standard Go net/http (curl, op CLI) works fine on both. Only the WASM-plugin HTTP path hangs.

What would help

  • Anyone else reproducing this on Debian 13 / kernel 6.12 nodes?
  • A debug env var to make the SDK log wazero's HTTP fetcher progress (DNS lookup, TCP connect, TLS handshake, request write, response wait) — the current error is opaque.
  • If the WASM HTTP fetcher has timeouts independent of the outer context, exposing them would help.

Happy to run more tests / capture packet traces if pointed at what to look for.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions