feat(proxy): route outbound requests through single authenticated http_proxy

[mekarpeles]

Summary

Routes all outbound service calls (Amazon Creators API, reCAPTCHA) through a single http_proxy URL with embedded credentials — the model ops now supports.

Before: per-service http_proxies.{amazon,recaptcha} entries with separate url, user, password keys and manual URL construction in get_proxy_params().

After: flat http_proxy: http://user:pass@proxy:8080 in config, consumed by setup_requests() (sets HTTP_PROXY/HTTPS_PROXY env vars) and passed directly to the Amazon API client.

Changes

• openlibrary/core/vendors.py — AmazonCreatorsAPI takes proxy_url: str = "" and passes it to the mekarpeles/python-amazon-paapi@proxy-support fork's proxy= kwarg (covers both urllib3 REST leg and OAuth2 token refresh via requests)
• scripts/affiliate_server.py — load_config() reads flat config.get("http_proxy") instead of per-service http_proxies.amazon block
• openlibrary/plugins/recaptcha/recaptcha.py — drops explicit proxies=get_proxy_params("recaptcha"); relies on env vars set by setup_requests() which requests honours automatically
• openlibrary/plugins/upstream/utils.py — removes get_proxy_params() entirely
• conf/openlibrary.yml — documents single-URL proxy format
• requirements.txt — pins to mekarpeles/python-amazon-paapi@proxy-support fork

Testing

docker compose run --no-deps --rm home python -m pytest \
  openlibrary/plugins/upstream/tests/test_utils.py \
  openlibrary/plugins/recaptcha/ -x -v
# 18 passed

Pre-commit clean (mypy/generate-pot skip locally as expected).

Related

Closes #12715
See also: sergioteula/python-amazon-paapi#150

[Copilot]

Pull request overview

Adds per-service HTTP proxy configuration to support authenticated squid ACLs (fixing #12715) while keeping existing global http_proxy behavior as the default fallback.

Changes:

• Introduces get_proxy_params(service_tag) to build a requests-compatible proxies dict from http_proxies.<service> config.
• Routes reCAPTCHA verification through the per-service proxy configuration when present.
• Updates the affiliate server to prefer http_proxies.amazon while retaining backward compatibility with legacy proxy config keys.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
scripts/affiliate_server.py	Prefer http_proxies.amazon for Amazon proxy settings, with fallback to legacy flat keys.
openlibrary/plugins/upstream/utils.py	Add get_proxy_params() helper for service-specific proxy (including auth).
openlibrary/plugins/recaptcha/recaptcha.py	Pass proxies=get_proxy_params("recaptcha") to the verification request.
conf/openlibrary.yml	Document the new http_proxies config shape (commented examples).

[mekarpeles]

Investigation: Amazon 403 on worker restart

Root cause identified. The creatorsapi_python_sdk's OAuth2TokenManager.refresh_token() (in creatorsapi_python_sdk/auth/oauth2_token_manager.py) uses bare requests.post() to fetch OAuth tokens — no proxy configuration of its own, just whatever HTTP_PROXY/HTTPS_PROXY env vars are set.

The flow:

1. load_config() creates AmazonCreatorsAPI(proxy_url=..., proxy_creds=...), which injects a custom RESTClientObject with proxy+auth into self.api._api_client.rest_client — this covers API calls only.
2. setup_requests() sets HTTP_PROXY/HTTPS_PROXY to the bare proxy URL (no auth).
3. OAuth2TokenManager caches the access token in-process memory. On every worker restart, the cache is cold and refresh_token() is called.
4. refresh_token() does requests.post("https://api.amazon.com/auth/o2/token", ...) — this goes through the bare env-var proxy, which has no auth → 403 under the new squid ACL policy.

Why it worked initially: The token was valid in-process during the session. The failure only surfaces when the token expires (~1hr) or, more visibly, on every worker restart (cold cache → immediate refresh attempt).

Our PR does not fix this. Our change is backward-compatible in production (falls back to config.get("http_proxy") since http_proxies.amazon won't be set yet), but neither the old nor the new code routes the OAuth token fetch through an authenticated proxy.

Fix options (not in this PR — needs a follow-up):

1. Add token endpoints to no_proxy_addresses (simplest — config-only, no code change):

   no_proxy_addresses:
     - api.amazon.com          # LWA v3.1 token endpoint
     - api.amazon.co.uk        # LWA v3.2
     - creatorsapi.auth.us-east-1.amazoncognito.com  # Cognito v2.1

   This lets the token fetch go direct, bypassing the squid proxy entirely.

2. Set the auth-bearing proxy in env vars — change setup_requests() to use the authenticated URL for HTTP_PROXY/HTTPS_PROXY. This makes all requests calls (including OAuth refresh) go through the authenticated proxy. Riskier: affects everything globally.

3. Patch OAuth2TokenManager in our code to inject a proxy-aware requests.Session — surgical but couples us to the SDK's internals.

Recommended: Option 1 (add to no_proxy_addresses in olsystem config) as an immediate ops fix, independent of this PR. This PR can still merge — it doesn't worsen the situation.

[jimchamp]

I'm not certain that the is_lwa method exists on OAuth2Config objects. Other than that, this looks okay to me.

Configuring the recaptcha URL has caused a merge conflict on this branch.

[mekarpeles]

Hi — I'm Pierre, an AI-assisted PR health agent running on behalf of @mekarpeles.

Folded 1 bot commit: The [pre-commit.ci] auto fixes commit (70282ac) was folded into its parent commit fix(proxy): route Amazon OAuth token refresh through authenticated proxy.

The single unresolved Copilot thread (conf/openlibrary.yml: http_proxies url field) was a false positive — the url field is intentional to allow per-service proxy URL overrides. Replied and resolved.

Note: This PR still has a merge conflict (CONFLICTING). The conflict is in semantic code (not mechanical), so it's been left for @mekarpeles to resolve manually.

Safeguards: Confirmed all non-bot commits authored by @mekarpeles. Used git push --force-with-lease.

To reverse: git reset --hard 70282ac then git push --force-with-lease.

[mekarpeles]

Pushed an update (f0a2aff) that simplifies the proxy implementation significantly.

Root cause that was fixed upstream

urllib3.ProxyManager ignores credentials embedded in the proxy URL for HTTPS CONNECT tunneling — they must be passed via proxy_headers=make_headers(proxy_basic_auth=...) instead. The previous vendors.py worked around this with ~100 lines of post-construction injection. The fix is now in the SDK itself: mekarpeles/python-amazon-paapi@proxy-support, which has also been submitted upstream as sergioteula/python-amazon-paapi#150.

What changed in this commit

• requirements.txt: pins to the fork branch (git+https://github.com/mekarpeles/python-amazon-paapi@proxy-support) instead of the upstream ==6.2.0 release which has no proxy support
• vendors.py: deleted _ProxyAwareTokenManager subclass, manual RESTClientObject replacement, and OAuth2Config assembly (114 lines removed); replaced with a 21-line block that builds an authenticated proxy URL and passes proxy= directly to AmazonCreatorsApi()

Tested on ol-home0

End-to-end test via amz.py --from-config /olsystem/etc/openlibrary.yml returned 1 item for ASIN 0316769177 through the squid proxy — both the urllib3 HTTPS CONNECT leg and the OAuth2 token refresh succeeded.