[ci/doc] Walk ray.data.llm as its own head module instead of allowlisting it - #65040
Draft
dstrodtman wants to merge 4 commits into
Draft
[ci/doc] Walk ray.data.llm as its own head module instead of allowlisting it#65040dstrodtman wants to merge 4 commits into
dstrodtman wants to merge 4 commits into
Conversation
The code<->docs API consistency check only sees APIs that its module walk reaches, and the walk only follows submodules a parent `__init__` actually imports. A public, `@PublicAPI`-annotated subpackage that its parent does not import is invisible to the check, so its APIs can silently rot undocumented. `ray.serve.llm` and `ray.data.llm` are both such surfaces today: they are documented and annotated, but no walk reaches them. - Add `ray.serve.llm` to the serve `head_modules` so its public surface is walked. It is import-safe in the docbuild image (transformers is a soft `try_import`, vLLM is only imported lazily). - Add a cross-team guard that enumerates one level of subpackages under every configured head module and fails when an annotated subpackage is neither reached by any walk nor knowingly excluded. A reviewed allowlist records the intentional exclusions and why. - Allowlist `ray.data.llm`: it eagerly imports the vLLM/SGLang engine processor configs, which do a hard `import transformers` at module load, and transformers is absent from the docbuild image, so it cannot be imported by this check. Its surface stays documented in `doc/source/data/api/llm.rst`. - Track reachable module names on `Module` so the guard can tell "walked" from "merely name-prefixed" (`ray.data.llm` is prefixed by `ray.data` yet never imported). - Unit-test the guard's decision core and helpers against the mock module. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
…ting it Alternative to allowlisting ray.data.llm in the coverage guard. The consistency check now runs its walks (and the coverage guard) inside _mock_uninstalled_backends, which mocks the backends absent from the docbuild image. The engine processor configs ray.data.llm imports eagerly (vLLM/SGLang -> a hard `import transformers`) therefore resolve, so ray.data.llm imports cleanly and can be a walk root like ray.serve.llm rather than an un-importable subpackage that must be excluded. - Add ray.data.llm to the data team head_modules. Its public surface is documented in doc/source/data/api/llm.rst, reachable from api.rst's toctree, so the walk checks the code surface against the real docs. - Run the cross-team guard inside _mock_uninstalled_backends so its coverage check and import probes see the same mocks as the walks. A promoted head like ray.data.llm is then counted as covered rather than flagged. - Empty UNWALKED_ANNOTATED_ALLOWLIST: both known escapees (ray.serve.llm and ray.data.llm) are now walked. Keep the list as the reviewed record for any future subpackage that genuinely cannot be a walk root. - Decouple the guard decision-core unit tests from real module names so they assert the function's behavior, not the current head/allowlist config. The decisive check is "doc: check API doc consistency" against the real docbuild image, which exercises the mocked ray.data.llm import path this approach depends on. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
…check Touch to tag this checker-only PR into the `doc` pipeline so doc_api_policy_check runs against the docbuild image. Revert before un-drafting. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
The "doc: check API doc consistency" job's first real run against the docbuild image surfaced the existing coverage gaps. Freeze them so the check passes against a known baseline. Each entry is tracked debt owed a document-or-deprecate decision by the owning library team, not a permanent exclusion; any new gap outside these lists still fails the build, so the debt can only shrink. - Guard: add the 13 unwalked annotated subpackages to UNWALKED_ANNOTATED_ALLOWLIST (10 annotated-not-walked, 3 unverifiable-import). - Data: add the 2 undocumented ray.data.llm APIs (HttpRequestStageConfig, ServeDeploymentProcessorConfig) to the data tracked_doc_debt. - Serve: add the 3 undocumented ray.serve.llm builder functions (build_dp_deployment, build_dp_openai_app, build_pd_openai_app) to the serve tracked_doc_debt. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This is an alternative to #64514. Same goal — no
@PublicAPIsubpackage silently escapes the code↔docs consistency walk — but it walksray.data.llminstead of allowlisting it.#64514 allowlists
ray.data.llmon the premise that its eagerimport transformers(via the vLLM/SGLang engine processor configs) can't be imported in the docbuild image. That premise predates the_mock_uninstalled_backendswrapper now onmaster, which mocks exactly the backends the docbuild image lacks. Under that mock,ray.data.llmimports cleanly and can be a walk root likeray.serve.llm— so its documented surface gets checked rather than merely excluded.What
ray.data.llmto the data teamhead_modules. Its surface is documented indoc/source/data/api/llm.rst, reachable fromapi.rst's toctree, so the walk checks the code surface against the real docs._mock_uninstalled_backendsso its coverage check and import probes see the same mocks as the walks; a promoted head likeray.data.llmis counted as covered, not flagged.UNWALKED_ANNOTATED_ALLOWLIST: both known escapees (ray.serve.llm,ray.data.llm) are now walked. The list stays as the reviewed record for any future subpackage that genuinely can't be a walk root.Relationship to #64514
Pick one, not both. This branch carries #64514's guard commit plus one delta commit; if this approach is preferred, the two squash into one before un-drafting.
Checks
The decisive check is doc: check API doc consistency — it exercises the mocked
ray.data.llmimport path this approach depends on, against the real docbuild image. Kept as a draft until that check is confirmed green.🤖 Generated with Claude Code