Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions nemoguardrails/server/experimental/contracts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,3 +104,7 @@ choose deployment rails or enable runtime profile selection.
This is a NeMo Guardrails document using JSON Schema vocabulary, not an OpenAPI
document or an OpenAPI Overlay. Generic schema tools do not execute its guardrail
annotations.

## Provider boundaries

- [OpenAI Chat Completions](openai/README.md)
48 changes: 48 additions & 0 deletions nemoguardrails/server/experimental/contracts/openai/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# OpenAI Chat Completions contract

Operation `createChatCompletion` uses the `single_text.v1` capability profile
for its guarded text boundary. This page summarizes that boundary and its
provider source. Neither the summary nor the source pin enables runtime
behavior. See [Guard contracts](../README.md) for the document vocabulary.

## Guarded boundary

| Area | Single-text boundary |
| --- | --- |
| Request | One user message with non-empty string content at `messages[0].content`. |
| Buffered response | One assistant choice with non-empty string content at `choices[0].message.content`. |
| Constrained values | User/assistant roles, single-item arrays, and request `n` constrained to one. |
| Unsupported content | Tool, audio, multimodal, refusal, and separate reasoning content where explicitly disabled. |
| Opaque data | Reviewed provider-owned metadata and controls, not additional guarded subjects. |

This request boundary does not accept conversation histories, system/developer
messages alongside the user message, or multimodal content blocks. Disabled
fields are null-only: a non-null value does not become acceptable merely because
it requests plain text or no tools.

Request `logprobs` and `top_logprobs` settings are provider-owned, while non-null
buffered response `choices[0].logprobs` is unsupported. That asymmetry is a
boundary limitation, not something document validation resolves.

Text replacement eligibility is separate from endpoint support for replacement
outcomes. Non-empty response annotations block text replacement; their mere
presence does not. Unrelated provider data must remain intact.

The Python projections, bindings, and endpoint determine exact acceptance and
runtime behavior. Their machine-readable contract belongs with the integration,
not in a separately maintained handwritten policy here. Recognizing the request
Comment on lines +31 to +33

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Streaming contract is missing

This revision removes chat-completions.guard.yaml and says the machine-readable contract belongs with the integration, but PR #2435 modifies that file rather than adding it. Its streaming test reads the file directly, so stacking that PR on this head leaves the file missing and the test fails. The buffered export in PR #2414 does not provide the stream section the test needs.

Prompt To Fix With AI
This is a comment left during a code review.
Path: nemoguardrails/server/experimental/contracts/openai/README.md
Line: 31-33

Comment:
**Streaming contract is missing**

This revision removes `chat-completions.guard.yaml` and says the machine-readable contract belongs with the integration, but PR #2435 modifies that file rather than adding it. Its streaming test reads the file directly, so stacking that PR on this head leaves the file missing and the test fails. The buffered export in PR #2414 does not provide the stream section the test needs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

`stream` flag does not itself provide a streaming endpoint. Streaming needs its
own event classification, lifecycle handling, and implementation tests; the
buffered boundary is not a claim about accepted stream events.

## Provider provenance

[source.yaml](source.yaml) records the OpenAPI document's immutable revision,
document version, download location, and expected SHA-256 digest. The pin is a
review baseline, not a claim to cover every current OpenAI field or compatible
provider.

The offline metadata tests check pin structure and URL/revision consistency.
They do not fetch the document or verify its bytes against the digest. Reviewing
a provider update requires checking the source artifact and changes to guarded,
constrained, disabled, and opaque fields separately from format validation.
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
document_url: https://github.com/openai/openai-openapi/blob/df63773f69f542ef875b9f00c3837c25ba5f4f2a/openapi.yaml
download_url: https://raw.githubusercontent.com/openai/openai-openapi/df63773f69f542ef875b9f00c3837c25ba5f4f2a/openapi.yaml
revision: df63773f69f542ef875b9f00c3837c25ba5f4f2a
document_version: 2.3.0
sha256: f2dae1a9aced09b91310db89edda51bf1e36ecbfb05230c3a50b239c07708469
68 changes: 68 additions & 0 deletions tests/server/experimental/test_openai_chat_contract.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Validate pinned provider metadata without contacting the provider."""

import re
from pathlib import Path
from urllib.parse import urlparse

import pytest
import yaml

REPOSITORY_ROOT = Path(__file__).parents[3]
SOURCE_PATH = REPOSITORY_ROOT / "nemoguardrails/server/experimental/contracts/openai/source.yaml"


@pytest.fixture(scope="module")
def openai_source() -> dict[str, str]:
"""Load source provenance, not a separately authored operation policy."""
source = yaml.safe_load(SOURCE_PATH.read_text(encoding="utf-8"))
assert isinstance(source, dict)
return source


def test_openai_source_declares_revision_and_digest(openai_source: dict[str, str]) -> None:
"""Pin metadata has the expected shape; this does not verify upstream bytes."""
assert set(openai_source) == {
"document_url",
"download_url",
"revision",
"document_version",
"sha256",
}
assert re.fullmatch(r"[0-9a-f]{40}", openai_source["revision"])
assert re.fullmatch(r"[0-9a-f]{64}", openai_source["sha256"])
assert openai_source["document_version"].strip()


@pytest.mark.parametrize(
("key", "host", "prefix"),
[
("document_url", "github.com", "/openai/openai-openapi/blob"),
("download_url", "raw.githubusercontent.com", "/openai/openai-openapi"),
],
)
def test_openai_source_urls_use_the_declared_revision(
openai_source: dict[str, str], key: str, host: str, prefix: str
) -> None:
"""Both locations identify the same pinned file rather than a moving branch."""
url = urlparse(openai_source[key])

assert url.scheme == "https"
assert url.netloc == host
assert url.path == f"{prefix}/{openai_source['revision']}/openapi.yaml"
assert not url.query
assert not url.fragment
Loading