fix(http task): add cookie jar to http.Client#74
Open
snehalahire-pattern wants to merge 1 commit into
Open
Conversation
The http task's client was built with Jar == nil, so endpoints that authenticate via a "set a cookie during a redirect, then redirect back" flow (e.g. Amazon Seller Central merchandising/deals APIs) failed with "stopped after 10 redirects". Without a jar, Set-Cookie headers issued mid-redirect were discarded and only the static Cookie header was re-sent, so the required cookie was never present and Amazon kept redirecting to set it — looping until Go's 10-redirect cap. Attach a fresh cookiejar.New(nil) to both http.Client constructions (default getClient and the proxy-override path). Per-task-instance jars keep cookies request-scoped; no cross-task leakage. Refs: DATA-8676 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Jar: cookiejar.New(nil)to bothhttp.Clientconstructions ininternal/pkg/pipeline/task/http/http.go(the defaultgetClient()and the proxy-override path)Set-Cookieheaders issued mid-redirect were discardedhttpCore-instance (request-scoped), so no cross-task cookie leakageRoot cause
http.Client{}was built withJar == nil. Amazon's auth flow returns a 302 with aSet-Cookieand expects the client to store and replay it on the next hop. With no jar, Go's client dropped everySet-Cookieand re-sent only the staticCookieheader — Amazon kept redirecting to set the missing cookie → 10-redirect loop → failure.Behavior delta
Set-Cookiefrom any HTTP task response was silently dropped.Set-Cookie, the cookie will now be sent on subsequent requests to the same domain within the same task invocation. This is a correctness improvement (standard cookie semantics), not a regression — no grep'd consumer depends on the previous broken behavior.Test plan
go build ./...— cleango vet ./...— clean (pre-existing warnings incompress.go/archive.go/file.gounrelated)go test ./...— passes (ok internal/pkg/pipeline)amazon_advertising_deals_data.yamlsubmit_download_requestin staging against Seller Central and confirm nostopped after 10 redirectserrorRefs: DATA-8676
🤖 Generated with Claude Code