Skip to content

Share URL patterns across rule sets, benchmark a rich user agent - #89

Merged
wRAR merged 2 commits into
scrapy:masterfrom
AdrianAtZyte:pattern-cache
Sep 16, 2026
Merged

wRAR merged 2 commits into
scrapy:masterfrom
AdrianAtZyte:pattern-cache

Conversation

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

No description provided.

@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.20%. Comparing base (ddf86e9) to head (5fa179c).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #89      +/-   ##
==========================================
+ Coverage   97.64%   98.20%   +0.56%     
==========================================
  Files           5        5              
  Lines         382      390       +8     
  Branches       78       79       +1     
==========================================
+ Hits          373      383      +10     
+ Misses          5        4       -1     
+ Partials        4        3       -1     
Files with missing lines Coverage Δ
src/protego/_protego.py 96.87% <100.00%> (+0.01%) ⬆️
src/protego/_ruleset.py 100.00% <100.00%> (+1.59%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Sep 16, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 13.81%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 1 (👁 1) regressed benchmark

Performance Changes

Benchmark BASE HEAD Efficiency
test_parse 117 ms 30.3 ms ×3.9
👁 test_match 660.1 µs 1,971.1 µs -66.51%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing AdrianAtZyte:pattern-cache (5fa179c) with master (ddf86e9)

Open in CodSpeed

@AdrianAtZyte

Copy link
Copy Markdown
Contributor Author

The test_match regression is the benchmark change, not the code: it now matches as ccbot, which walks 58 wildcard rules instead of the two plain rules of *. With the same user agent, matching is slightly faster on this branch than on master. I have acknowledged it on CodSpeed.

@AdrianAtZyte
AdrianAtZyte marked this pull request as ready for review September 16, 2026 04:27
@AdrianAtZyte
AdrianAtZyte marked this pull request as draft September 16, 2026 04:27
@AdrianAtZyte
AdrianAtZyte marked this pull request as ready for review September 16, 2026 05:02
@wRAR
wRAR merged commit 700ed93 into scrapy:master Sep 16, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants