Skip to content

Fix Retry-After HTTP-date parsing and honor Retry-After on 503 (#143) - #146

Merged
vedaant00 merged 2 commits into
mldsveda:mainfrom
HeaTTap:fix/retry-after-http-date-503-143
Aug 14, 2026
Merged

Fix Retry-After HTTP-date parsing and honor Retry-After on 503 (#143)#146
vedaant00 merged 2 commits into
mldsveda:mainfrom
HeaTTap:fix/retry-after-http-date-503-143

Conversation

@HeaTTap

@HeaTTap HeaTTap commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

fixes #143

added a shared parse_retry_after helper in http.py to handle both delay-seconds and HTTP-date formats for Retry-After headers

updated sync and async HTTP clients to use this helper for 429 rate limit retries and 503 service unavailable responses

added unit tests for parse_retry_after, HTTP-date 429 handling, and 503 Retry-After support across sync and async clients

@vedaant00
vedaant00 force-pushed the fix/retry-after-http-date-503-143 branch from 742d2fd to 8388054 Compare August 14, 2026 00:08
@vedaant00
vedaant00 requested a lite review from Copilot August 14, 2026 00:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes Retry-After handling in the core HTTP clients by correctly parsing both delay-seconds and HTTP-date header forms, and by honoring Retry-After on 503 responses (in addition to 429), with added unit/integration test coverage.

Changes:

  • Added a shared parse_retry_after helper to parse Retry-After as either delay-seconds or HTTP-date.
  • Updated sync + async HTTP clients to use the helper for 429 handling and to honor Retry-After on 503 retry backoffs.
  • Added tests covering the helper and the 429/503 behaviors across sync and async clients.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/pyscrappy/core/http.py Adds parse_retry_after and uses it for 429 retries and 503 backoff delay selection (sync).
src/pyscrappy/core/async_http.py Imports and applies parse_retry_after for 429 retries and 503 backoff delay selection (async).
tests/test_core/test_http.py Adds sync tests for HTTP-date Retry-After, 503 honoring, and parse_retry_after unit tests.
tests/test_core/test_async_http.py Adds async tests for HTTP-date Retry-After and 503 honoring.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/test_core/test_async_http.py
Comment thread tests/test_core/test_http.py Outdated
Comment thread tests/test_core/test_async_http.py Outdated
- ruff format tests/test_core/test_async_http.py: add the missing blank
  lines before the appended @pytest.mark.anyio test (the seam left after
  the earlier conflict resolution). This is what CI lint was failing on.
- Set retry_delay=0.0 in the sync and async HTTP-date 429 tests so a
  positive sleep can only come from actually parsing the date header, not
  from the fallback default (Copilot). The 503 tests already assert an
  exact numeric delay, so they're unaffected.
@vedaant00

Copy link
Copy Markdown
Collaborator

Fixed the lint: the format change from my earlier conflict-resolution push didn't make it into the commit, so the blank-line seam before the appended anyio test was unformatted. Applied ruff format for real this time. Also took Copilot's point and set retry_delay=0.0 on the two HTTP-date 429 tests so a positive sleep can only come from parsing the date, not the fallback default. Full suite 492 passed, lint clean. Good to merge.

@vedaant00
vedaant00 merged commit 08f6ec8 into mldsveda:main Aug 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry-After: crashes on an HTTP-date value, and 503 ignores the header (sync + async)

3 participants