Skip to content

Assorted new renderer fixes - #185

Open
stephenfin wants to merge 15 commits into
masterfrom
new-renderer-fixes
Open

Assorted new renderer fixes#185
stephenfin wants to merge 15 commits into
masterfrom
new-renderer-fixes

Conversation

@stephenfin

@stephenfin stephenfin commented Oct 11, 2026

Copy link
Copy Markdown
Member

A assortment of mostly LLM-assisted fixes for the new renderer, stemming from an investigation comparing the two.

I'll likely come back and split this up into multiple PRs. This is mainly here as a reminder to do so.

Commits

  • Handle integer status codes
  • schema_utils: Fix example generation for arrays with oneOf/anyOf items
  • Fix programmatic option handling and renderer edge cases
  • Add backwards-compat options (format, request, examples)
  • Add endpoint filtering options (paths, include, exclude, methods)
  • Add tag grouping support (group)
  • Fix path metadata and parameter inheritance during normalization
  • Fix filtering with multiple exclusion patterns
  • Honor legacy request and example options in the new renderer
  • Preserve examples on composed array item schemas
  • Handle OpenAPI 3.1 schema shapes in rendering and examples
  • Preserve required constraints through composed schema descriptions
  • Preserve falsey examples and support schema examples arrays
  • Render callback and webhook operations in the HTTP domain
  • Make all in-tree example builds pass strict warning checks

@stephenfin
Handle integer status codes
8f5b273 
Signed-off-by: Stephen Finucane <stephen@that.guru>
@stephenfin
schema_utils: Fix example generation for arrays with oneOf/anyOf items
5cc7b4f 
Handle oneOf and anyOf in array items safely when elements are dicts,
sets, or primitive types.

This builds upon the partial fix in commit f038bf1.

Signed-off-by: Stephen Finucane <stephen@that.guru>
@stephenfin
Fix programmatic option handling and renderer edge cases
80ba1a6 
- Normalize programmatic option values while retaining the directive
  converters and empty-value validation already provided by master.
- Filter operations by _HTTP_METHODS so path-level vendor extensions and
  other non-operation keys are ignored.
- Copy schema_type list in _get_markers_from_object to prevent mutating
  cached specs in-place.
- Safely handle operations without a 'responses' key.
- Pass default=str to json.dumps() when formatting examples to handle
  datetime/date objects parsed from YAML.
- Preserve OpenAPI response code case, including the special 'default'
  value, and add a regression test for selecting its example.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Add backwards-compat options (format, request, examples)
d11e1c9 
- Support 'format' option, translating 'markdown' to 'commonmark' and
  'rst'/'restructuredtext' to 'restructuredtext'.
- Accept legacy 'request' and 'examples' flags in option_spec so
  existing directives do not fail with docutils warnings.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Add endpoint filtering options (paths, include, exclude, methods)
9c5b4ca 
- Support filtering paths using an explicit list of paths ('paths').
- Support filtering paths using include/exclude regex patterns
  ('include', 'exclude').
- Support filtering operations by HTTP methods ('methods').
- Copy operation definitions with deepcopy to avoid mutating cached
  specs.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Add tag grouping support (group)
ad3e7c7 
- Support 'group' option to group operations under section headers by
  their primary tag (or 'default').

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Fix path metadata and parameter inheritance during normalization
2d0ef76 
Only normalize HTTP operations, leaving Path Item metadata and vendor
extensions untouched. Merge inherited parameters by (name, in), allowing
operation-level definitions to override their path-level counterparts.

Retain inheritance before Swagger conversion so shared body parameters
continue to become request bodies. Add renderer entry-point regressions
for OpenAPI 2.0, 3.0 and 3.1, including grouped and repeated rendering.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Fix filtering with multiple exclusion patterns
fe0b23e 
Retain a path only when none of the exclusion patterns matches it.
Checking paths once prevents excluded endpoints from reappearing and
avoids duplicating retained endpoints. An empty exclusion list now
leaves the selected paths unchanged.

Add regressions for disjoint, overlapping, nonmatching and empty
exclusions, combined with explicit path selection, inclusion patterns
and tag grouping.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Honor legacy request and example options in the new renderer
6221897 
Make the examples flag enable schema-derived examples and select all
responses by default, while allowing explicit generation and response
selection options to take precedence. Disabling examples suppresses both
supplied and generated request and response examples.

Make the request flag control request schema hints independently of
response hints and examples. Honor the parsed boolean value of
no-json-schema-description, including an explicit false value.

Document these compatibility rules and retain the new renderer's
existing defaults when legacy flags are absent. Add entry-point tests
and HTML assertions for real OpenAPI 2.0 and 3.0 directives.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Preserve examples on composed array item schemas
c7ab773 
Generate array items from their complete schema instead of bypassing
item-level examples when oneOf or anyOf is present. Reuse the existing
composition handling and remove special cases for sets and strings,
which are not valid OpenAPI composition values.

Replace the invalid fixture with an ordered array of schemas. Add
coverage for explicit and falsey item examples, array-level precedence,
structured items, and generated request and response bodies. Validate
the new utility examples against their JSON schemas.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Handle OpenAPI 3.1 schema shapes in rendering and examples
a27db3f 
Support nullable type arrays, null values, boolean schemas and arrays
without items during example generation. Prefer non-null union members,
but fall back when a member cannot contribute an example. Omit forbidden
optional properties and handle prefixItems separately from array tails.

Signal known impossible examples with NoExampleError and let the
renderer skip that media type rather than invent a value or fail the
document. Handle boolean schemas and nullable structural types during
description traversal, and retain individual type markers with
applicable formats.

Add utility validation and renderer regressions, including generation
from the in-tree issue-112 reproducer. Quote its null type correctly and
document the best-effort nature of schema-derived examples.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Preserve required constraints through composed schema descriptions
b6a841e 
Accumulate required names when merging allOf schemas, including nested
objects, and retain nested conjunctions instead of replacing them.
Resolve nested allOf members without mutating the input schema or losing
false subschemas.

Carry a property's required marker through composition and array
traversal. Resolve composed root array items before deciding whether
their fields can be described.

Add regressions for merged requirements, optional counterparts, repeated
rendering and the in-tree expanded Petstore schemas. Update the two
Petstore snapshots to include composed array fields and inherited
required markers, and correct an allOf typo in an existing fixture.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Preserve falsey examples and support schema examples arrays
18ac1d6 
Select supplied examples by key presence rather than truthiness, and
support the first literal value in OpenAPI 3.1 schema examples arrays.
Retain media-level and legacy singular-example precedence, including
when examples are zero, false, null or empty.

Skip annotation-only Example Objects and fall back to schema examples.
Recognize plural annotations during nested generation and allOf example
selection, without treating an empty examples array as a supplied value.

Serialize generated and plural-schema JSON instances correctly, while
preserving legacy preformatted strings and fetched payloads. Keep
fetched content and rendering metadata out of the input specification.

Document the selection rules and add request, response and utility
regressions covering falsey values, precedence and JSON serialization.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Render callback and webhook operations in the HTTP domain
7d8636b 
Keep callbacks with their selected parent operation and render
webhook-only specifications using a receiver URL placeholder, not a
fabricated API path. Reuse operation rendering and Path Item parameter
inheritance for outbound operations, including references, payloads and
response descriptions.

Register dedicated outbound HTTP descriptions that retain typed fields
but do not acquire URL targets or incoming-route index entries. This
prevents repeated callback expressions and webhook destinations from
colliding.

Document filtering and grouping semantics, verify operation coverage in
all in-tree example smoke tests, and add strict callback/webhook builds.
Isolate docutils registrations between test builds.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@stephenfin
Make all in-tree example builds pass strict warning checks
6fd5c43 
Enable warnings-as-errors for every example smoke test now that docutils
registrations are isolated between builds and operation coverage is
checked.

Use plain response fields for OpenAPI default and uppercase response
ranges rather than passing them to HTTP-domain roles that require
numeric codes. Quote JSON examples when their schema explicitly declares
a string, retain verbatim fetched and legacy structured payloads, and
use default form encoding for object-valued
application/x-www-form-urlencoded examples.

Separate parameter fields from request body literal blocks so examples
remain valid reStructuredText when request-schema descriptions are
disabled or unavailable. Add regressions and refresh affected rendering
snapshots.

Signed-off-by: Stephen Finucane <stephen@that.guru>
Assisted-by: OpenAI GPT-6.1 (via pi)
@dstrodtman

dstrodtman commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Thanks for the release, and for the triage access.

I ran the Ray Jobs API spec against this branch, since it's the next thing in my way. On 0.10.0 with the new renderer, :generate-examples-from-schemas: no longer crashes, but -W builds still fail: a response whose schema is a bare string enum renders as PENDING under Content-Type: application/json, and the http lexer rejects it. On this branch the same build passes under -W and the enum renders as "PENDING". Bisecting through the stack, it's clean as of 18ac1d6 ("Preserve falsey examples and support schema examples arrays"), and 6fd5c43 extends the same quoting rule.

You mentioned splitting this up. If it'd help, I'm happy to do the split: carve it into separate PRs along the commit boundaries, keep your authorship and sign-offs, and rebase each onto master. I'd start with the example-serialization slice, since that's the one with a downstream consumer waiting on it. If you'd rather split it yourself, or land it whole, that works too. Tell me which and I'll stay out of the way.

This was referenced Oct 13, 2026
Open
Merged
dstrodtman added a commit to ray-project/ray that referenced this pull request Oct 13, 2026
@dstrodtman @claude
[doc] Bump sphinxcontrib-openapi to 0.10.0 (#66957)
226d267 
## Description

Bumps `sphinxcontrib-openapi`, which renders the [Ray Jobs REST API
specification](https://docs.ray.io/en/master/cluster/running-applications/job-submission/api.html)
page, from 0.9.0 to
[0.10.0](https://pypi.org/project/sphinxcontrib-openapi/0.10.0/), and
regenerates the docs deplock.

0.10.0 fixes five upstream bugs, two of which this page's source comment
named as blockers for request and response examples:

-
[sphinx-contrib/openapi#166](sphinx-contrib/openapi#166):
`:generate-examples-from-schemas:` crashed on an `allOf` that targets a
non-object schema, which is how `JobDetails` attaches descriptions to
the `JobType` and `JobStatus` enums.
-
[sphinx-contrib/openapi#165](sphinx-contrib/openapi#165):
`:response-examples-for:` and the other whitespace-delimited options
were rejected as unknown options.
- [#173](sphinx-contrib/openapi#173),
[#171](sphinx-contrib/openapi#171), and
[#178](sphinx-contrib/openapi#178): an
operation summary written as a YAML block scalar rendered literal `**`,
the old renderer's request body block was left unclosed, and the 3.0
renderer crashed on a non-object request body.

Examples still can't be enabled. With `:generate-examples-from-schemas:`
on 0.10.0, the stop endpoint's 200 response, a bare `JobStatus` enum,
renders as `PENDING` rather than `"PENDING"` under `Content-Type:
application/json`. The `http` lexer rejects it, and the warning fails
the build. That's
[sphinx-contrib/openapi#172](sphinx-contrib/openapi#172),
fixed for the `httpdomain` renderer on the open
[sphinx-contrib/openapi#185](sphinx-contrib/openapi#185).
This PR updates the source comment in `api.md` to name that bug instead
of the two that are now fixed. The rendered page doesn't change.

What changes:

- `doc/requirements-doc.txt`: `sphinxcontrib-openapi==0.9.0` → `0.10.0`.
- `python/deplocks/docs/docbuild_depset_py3.11.lock`: regenerated with
raydepsets. The diff is the one package block. 0.10.0 declares the same
dependencies as 0.9.0, so there's no transitive churn.
- `doc/source/cluster/running-applications/job-submission/api.md`:
source comment only.

0.10.0 raises `requires-python` from `>=3.10` to `>=3.11`. The only
consumer is the docbuild depset, which is Python 3.11 on both Read the
Docs and the `docbuild-py3.11` CI image.

## Related issues

Related to sphinx-contrib/openapi#172.

## Additional information

**Not a duplicate.** No open PR touches `sphinxcontrib-openapi` or bumps
this pin, and no open Dependabot PR targets `doc/requirements-doc.txt`.

**Tests run:**

- `bazelisk run //ci/raydepsets:raydepsets -- build
ci/raydepsets/configs/docs.depsets.yaml --name docbuild_depset_311`
regenerated the lock.
- `bazelisk run //ci/raydepsets:raydepsets -- build
ci/raydepsets/configs/docs.depsets.yaml --check` passed for all three
docs depsets. The doctest lock is unchanged.
- `make -C doc rtd-build` in a Python 3.11 venv installed from
`doc/requirements-doc.lock.txt`: `build succeeded.`, zero Sphinx
warnings under `-W`. The `rtd-doctor` preflight confirmed the
environment matches Read the Docs. The rendered Jobs API page has all 8
endpoints, including `/stop`.
- The Jobs spec rendered standalone with `openapi_default_renderer =
"httpdomain"` under `-W`: clean on 0.10.0 with no example options. With
`:generate-examples-from-schemas:` it fails on the unquoted `PENDING`,
which is the bug the updated comment describes.

AI assistance (Claude Code) was used to prepare this change. I reviewed
every changed line.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@stephenfin

stephenfin commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

You mentioned splitting this up. If it'd help, I'm happy to do the split: carve it into separate PRs along the commit boundaries, keep your authorship and sign-offs, and rebase each onto master. I'd start with the example-serialization slice, since that's the one with a downstream consumer waiting on it. If you'd rather split it yourself, or land it whole, that works too. Tell me which and I'll stay out of the way.

Works for me: as noted, this is mostly LLM-assisted so I'm happy for you to rework this as you see fit. I'm also happy for you to prioritise fixes that affect your projects first 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stephenfin @dstrodtman