Further changes - #5
Merged
Merged
Conversation
📝 WalkthroughWalkthroughAdds a complete ChangesGoogle style guide scraper
Cursor MCP configuration
Estimated code review effort: 4 (Complex) | ~45 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.cursor/mcp.json:
- Around line 3-9: Pin the package version in the context7 MCP configuration by
replacing `@latest` in the command arguments with an explicitly reviewed exact
`@upstash/context7-mcp` version. Keep the existing npx -y invocation and update
the pinned version deliberately when upgrading.
In @.cursor/plans/google_style_guide_scraper_plan.md:
- Around line 179-303: Remove the obsolete “Ready-to-Run Reference
Implementation Script” section from the plan, or clearly label the entire
document and script as historical and non-executable. Ensure agents are directed
to the authoritative namespaced-anchor design in
google-style-guide-to-markdown.md rather than this invalid implementation.
In @.cursor/plans/google-style-guide-to-markdown.md:
- Around line 170-175: Update the retry policy in the Fetch section to define a
finite maximum attempt count, cap exponential backoff delays, and honor valid
Retry-After response headers for 429 and 5xx responses. Ensure retries stop
after the configured limit so the style-guide command cannot run indefinitely.
- Around line 210-217: Update the “Assemble” output specification to remove the
runtime-generated generated_at frontmatter field, or require it as an explicit
stable input/source revision; ensure identical inputs produce identical Markdown
bytes and update the stated static frontmatter line count accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 199f8c6d-ea12-4f70-a074-65376d478d05
📒 Files selected for processing (4)
.cursor/mcp.json.cursor/plans/google-style-guide-to-markdown.md.cursor/plans/google_style_guide_scraper_plan.md.cursor/settings.json
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
web2md/google_style_guide.py (1)
546-564: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMake the size and term-count thresholds non-fatal or tunable.
SIZE_MIN/SIZE_MAXandWORD_LIST_TERM_EXPECTED ± 30encode a snapshot of the current upstream guide. If Google adds or removes content, thestyle-guideMake target fails and writes no output, even though the conversion succeeded. Structural checks (broken anchors, leftoverdevsite-, heading depth) deserve a hard failure; volume heuristics do not.♻️ Suggested split
- size = len(md.encode("utf-8")) - if not (SIZE_MIN <= size <= SIZE_MAX): - errors.append( + warnings_out: list[str] = [] + size = len(md.encode("utf-8")) + if not (SIZE_MIN <= size <= SIZE_MAX): + warnings_out.append( f"output size {size / 1024 / 1024:.2f} MB outside " f"{SIZE_MIN / 1024 / 1024:.1f}–{SIZE_MAX / 1024 / 1024:.1f} MB" ) @@ if not (low <= term_count <= high): - errors.append( + warnings_out.append( f"word-list term count {term_count} outside {low}–{high} " f"(expected ~{WORD_LIST_TERM_EXPECTED})" ) + for warn in warnings_out: + print(f"VALIDATION WARNING: {warn}", file=sys.stderr)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web2md/google_style_guide.py` around lines 546 - 564, Update the validation logic around the size and word-list term-count checks so these volume heuristics are configurable or reported non-fatally rather than added to the fatal errors that prevent output. Preserve hard failures for structural validation checks, and reuse the existing configuration or warning mechanism if available; specifically adjust the SIZE_MIN/SIZE_MAX and WORD_LIST_TERM_EXPECTED/WORD_LIST_TERM_TOLERANCE handling without changing the conversion output.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web2md/google_style_guide.py`:
- Around line 168-197: Update the retry loop around the 429/5xx handling to
assign an appropriate HTTP error for the response to last_error before
continuing. Ensure that when all attempts return retryable statuses, the final
SystemExit message from the fetch flow includes the HTTP failure instead of
None, while preserving the existing retry behavior.
In `@web2md/README.md`:
- Around line 14-16: Update the README installation instructions around “uv sync
--group web2md” to accurately reflect dependency installation: use the mode that
excludes the project’s marker-pdf dependencies, such as “uv sync --only-group
web2md”, or remove the claim that the setup is separate from the heavy
marker-pdf stack.
---
Nitpick comments:
In `@web2md/google_style_guide.py`:
- Around line 546-564: Update the validation logic around the size and word-list
term-count checks so these volume heuristics are configurable or reported
non-fatally rather than added to the fatal errors that prevent output. Preserve
hard failures for structural validation checks, and reuse the existing
configuration or warning mechanism if available; specifically adjust the
SIZE_MIN/SIZE_MAX and WORD_LIST_TERM_EXPECTED/WORD_LIST_TERM_TOLERANCE handling
without changing the conversion output.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1cec4aad-a153-4e3a-9c07-b09d4bb60419
⛔ Files ignored due to path filters (3)
md/.gitkeepis excluded by!md/**md/GoogleDeveloperDocumentationStyleGuide.mdis excluded by!md/**uv.lockis excluded by!**/*.lock,!uv.lock
📒 Files selected for processing (9)
.claude/plans/.gitkeep.cspell.json.cursor/plans/.gitkeep.cursor/plans/google-style-guide-to-markdown.md.gitignoreMakefilepyproject.tomlweb2md/README.mdweb2md/google_style_guide.py
🚧 Files skipped from review as they are similar to previous changes (1)
- .cursor/plans/google-style-guide-to-markdown.md
There was a problem hiding this comment.
♻️ Duplicate comments (1)
web2md/README.md (1)
25-34: 🚀 Performance & Scalability | 🟠 MajorKeep scraper commands separate from project dependencies.
--group web2mddoes not omit[project].dependencies;--only-group web2mdomits the project and its dependencies. The current commands can therefore installmarker-pdfandpsutil, contradicting the README's isolation claim. (docs.astral.sh)
web2md/README.md#L25-L34: changeuv sync --group web2mdand both direct scraper commands to--only-group web2md, or remove the isolation claim.Makefile#L71-L73: run the scraper withuv run --only-group web2md ....As per path instructions: “flag commands, paths, options, and examples that this change has made wrong or stale.” Context7 was not available in this review; the official uv documentation was checked.
Proposed command changes
-uv sync --group web2md +uv sync --only-group web2md -uv run --group web2md python web2md/src/web2md.py +uv run --only-group web2md python web2md/src/web2md.py -uv run --group web2md python web2md/src/web2md.py --refresh +uv run --only-group web2md python web2md/src/web2md.py --refresh - uv run --group web2md python web2md/src/web2md.py + uv run --only-group web2md python web2md/src/web2md.py#!/usr/bin/env bash set -euo pipefail echo "Dependencies selected by --group web2md:" uv tree --group web2md | grep -E 'marker-pdf|psutil' || true echo "Dependencies selected by --only-group web2md:" if uv tree --only-group web2md | grep -E 'marker-pdf|psutil'; then echo "Heavy project dependencies remain selected" >&2 exit 1 fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web2md/README.md` around lines 25 - 34, Update the scraper commands in web2md/README.md lines 25-34 to use --only-group web2md instead of --group web2md, including the sync and both direct uv run examples. Update the scraper invocation in Makefile lines 71-73 to use --only-group web2md as well, preserving the existing module and arguments.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@web2md/README.md`:
- Around line 25-34: Update the scraper commands in web2md/README.md lines 25-34
to use --only-group web2md instead of --group web2md, including the sync and
both direct uv run examples. Update the scraper invocation in Makefile lines
71-73 to use --only-group web2md as well, preserving the existing module and
arguments.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 45a26b49-2076-4cf6-b124-a23de7889a69
⛔ Files ignored due to path filters (2)
md/GoogleDeveloperDocumentationStyleGuide.mdis excluded by!md/**uv.lockis excluded by!**/*.lock,!uv.lock
📒 Files selected for processing (22)
.coderabbit.yaml.cspell.json.cursor/plans/google-style-guide-to-markdown.md.github/workflows/ci.yml.gitignoreAGENTS.mdMakefileREADME.mdpyproject.tomlweb2md/README.mdweb2md/src/web2md.pyweb2md/tests/conftest.pyweb2md/tests/test_assemble.pyweb2md/tests/test_cleaning.pyweb2md/tests/test_cli.pyweb2md/tests/test_converter.pyweb2md/tests/test_discovery.pyweb2md/tests/test_fetch.pyweb2md/tests/test_links.pyweb2md/tests/test_run.pyweb2md/tests/test_slugs.pyweb2md/tests/test_validate.py
🚧 Files skipped from review as they are similar to previous changes (2)
- .gitignore
- .cursor/plans/google-style-guide-to-markdown.md
Owner
Author
|
@coderabbitai summary |
✅ Action performedSummary regeneration triggered. |
Owner
Author
|
@coderabbitai full review |
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 39 minutes. |
Owner
Author
|
@mprokopov I will merge this PR since |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The PR adds a
md/GoogleStyleGuide.mdtest input file. It was generated with theweb2mdscript.Added
web2md/src/web2md.py).It walks a documentation site and writes one Markdown document into
md/,as an upstream source step alongside the existing
pdf2md/. No model isinvolved. Target site and output filename are two constants at the top of the
module (
SOURCE_URL,OUTPUT_FILE); everything else URL-shaped is derivedfrom
SOURCE_URL.--refresh(re-fetch instead of reusing thecache),
--cache-dir, and--output. Fetched HTML is cached underweb2md/cache/(gitignored) with bounded retries, so re-runs are cheap.md/GoogleStyleGuide.md: a scraped snapshot of theGoogle developer documentation style guide
(16,514 lines), now tracked as wiki input.
web2md/tests/(11 test modules plusconftest.py). HTTP is served throughhttpx.MockTransport, so no test opensa socket and writes stay in
tmp_path. Coverage spans the pure helpers(slugs, anchors, link rewriting, Markdown conversion, assembly), every
validate_outputerror branch, fetch retry and caching, and one end-to-endrun()over a synthetic 72-page site.make test(runs pytest via the overridablePYTESTvariable) andmake scrape(runs the scraper by path) targets.testjob that reusesmake test, installing only thetestandweb2mddependency groups with
--only-groupso it never pulls marker-pdf or torch.It checks out with
persist-credentials: false.test(pytest) andweb2md(httpx, beautifulsoup4, lxml,markdownify), keeping each CI job's install narrow;
uv.lockupdated.E,W,F,I,UP,B,SIM,C4,RET,PT,D), Google docstring convention,line-length = 100, anda per-file ignore so tests need no module/function docstrings. Pytest is
configured in
pyproject.tomlwithpythonpath = ["web2md/src"]in place of a[build-system].**/*.pyand**/tests/**/*.py, coveringmutable default arguments, blocking calls in async functions, leaked secrets,
missing context managers and timeouts, unsanitized shell/LLM input, fixture
isolation, and the no-real-network rule for tests.
.cursor/mcp.json(Context7 and GitHub MCPservers) and
.cursor/settings.json.web2md/README.md, a "Starting from a website" section inREADME.md, and aweb2md/section inAGENTS.md.Changed
md/is no longer gitignored, so scraped source documents are tracked..gitignorenow also covers__pycache__/,.pytest_cache/,.ruff_cache/and
web2md/cache/.make lintextended toweb2md/**andweb2md/README.md.finishing_touchesfor docstrings and unit tests are enabled nowthat first-party code exists, and
.cursor/mcp.jsonis excluded from reviewto silence its
@latestwarning.groups and the per-tool
web2md/src+web2md/testslayout instead of theprevious "adopt
src/md2okf/when real Python lands" placeholder..cspell.jsonextended with Python tooling vocabulary (httpx, conftest,monkeypatch, pydocstyle, and similar).
Removed
.claude/plans/compile-wiki-skill-container.md,observable-wiki-compilation.mdandobserve-pi-agent-sandbox.md; thedirectory is kept via
.gitkeep.Summary by CodeRabbit
New Features
Documentation
Tests