Skip to content

fix(http task): add cookie jar to http.Client#74

Open
snehalahire-pattern wants to merge 1 commit into
mainfrom
fix/http-task-cookie-jar
Open

fix(http task): add cookie jar to http.Client#74
snehalahire-pattern wants to merge 1 commit into
mainfrom
fix/http-task-cookie-jar

Conversation

@snehalahire-pattern

Copy link
Copy Markdown

Summary

  • Add Jar: cookiejar.New(nil) to both http.Client constructions in internal/pkg/pipeline/task/http/http.go (the default getClient() and the proxy-override path)
  • Fixes Amazon Seller Central redirect-cookie auth flows that previously hit Go's 10-redirect cap because Set-Cookie headers issued mid-redirect were discarded
  • Jar is per-httpCore-instance (request-scoped), so no cross-task cookie leakage

Root cause

http.Client{} was built with Jar == nil. Amazon's auth flow returns a 302 with a Set-Cookie and expects the client to store and replay it on the next hop. With no jar, Go's client dropped every Set-Cookie and re-sent only the static Cookie header — Amazon kept redirecting to set the missing cookie → 10-redirect loop → failure.

Behavior delta

  • Before: any Set-Cookie from any HTTP task response was silently dropped.
  • After: cookies honored per RFC 6265. For non-redirect flows that happen to return Set-Cookie, the cookie will now be sent on subsequent requests to the same domain within the same task invocation. This is a correctness improvement (standard cookie semantics), not a regression — no grep'd consumer depends on the previous broken behavior.

Test plan

  • go build ./... — clean
  • go vet ./... — clean (pre-existing warnings in compress.go/archive.go/file.go unrelated)
  • go test ./... — passes (ok internal/pkg/pipeline)
  • Manual verification: run amazon_advertising_deals_data.yaml submit_download_request in staging against Seller Central and confirm no stopped after 10 redirects error

Refs: DATA-8676

🤖 Generated with Claude Code

The http task's client was built with Jar == nil, so endpoints that
authenticate via a "set a cookie during a redirect, then redirect back"
flow (e.g. Amazon Seller Central merchandising/deals APIs) failed with
"stopped after 10 redirects". Without a jar, Set-Cookie headers issued
mid-redirect were discarded and only the static Cookie header was
re-sent, so the required cookie was never present and Amazon kept
redirecting to set it — looping until Go's 10-redirect cap.

Attach a fresh cookiejar.New(nil) to both http.Client constructions
(default getClient and the proxy-override path). Per-task-instance
jars keep cookies request-scoped; no cross-task leakage.

Refs: DATA-8676

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@snehalahire-pattern snehalahire-pattern requested a review from a team as a code owner June 11, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant