diff --git a/CHANGELOG.md b/CHANGELOG.md index f9a55ca..ef4cfad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ All notable changes to this project will be documented here. The project follows - A stateless-by-default OpenAI Responses API adapter with text, refusal, function-call, and tool-result normalization. - Content-free sync/async attempt observation and inspectable provider health snapshots. - Cross-request transient-failure cooldown that deprioritizes unhealthy routes without removing last-resort fallback. +- A canonical support-ticket triage reference consumer and deterministic public-API contract fixture. ### Changed diff --git a/README.md b/README.md index cd70d9f..830fde9 100644 --- a/README.md +++ b/README.md @@ -187,6 +187,8 @@ The built-in HTTP providers own API translation. `UnifiedLLM` owns policy and ha For new OpenAI integrations, use `OpenAIResponsesProvider`. It targets `/responses`, keeps server-side storage disabled by default, and participates in the same retry, fallback, request-size, and response-size boundaries. See [examples/responses_api.py](examples/responses_api.py). +For a production-shaped consumer, see [examples/support_triage.py](examples/support_triage.py) and its [consumer contract](docs/CONSUMER_CONTRACT.md). The example routes support tickets through a strict function schema and validates every model-supplied field before application use. + ## Security, privacy, reliability, and cost - Treat endpoint URLs, API keys, models, and custom provider adapters as trusted operator configuration. diff --git a/ROADMAP.md b/ROADMAP.md index 1b8d977..527db79 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,7 +6,7 @@ This roadmap separates four gates: merge, release, publication, and flagship ado Portfolio role: **reusable library or sdk**. Keep this as a small, independently versioned package. Samsarix Unified should consume it only through a public API adapter; private monorepo imports and copied implementations are out of scope. -Current disposition: Already integrated; roadmap changes should use an ordinary review branch. +Current disposition: The library is productized on its own default branch, but no external canonical consumer has adopted this reference contract yet. Roadmap changes should use an ordinary review branch. ## Stabilize the productized default @@ -17,7 +17,8 @@ Current disposition: Already integrated; roadmap changes should use an ordinary - Completed in the next hardening increment: account for the exact outbound body, cap inbound response bytes and normalized structures, and validate custom adapters. - Completed in the following API increment: add a stateless-by-default OpenAI Responses adapter without weakening the shared bounds or fallback contract. - Completed in the health increment: expose content-free attempt observation and deprioritize repeatedly failing routes during a bounded cooldown. -- Next: prove one canonical consumer and a capped live endpoint before publication. +- Completed as repository evidence: a canonical support-triage reference consumer with a public-API contract fixture, explicit limits, ownership, compatibility, and rollback guidance. +- Next: adopt the reference contract in one external canonical consumer and prove a capped live endpoint before publication. - Review priority: prove one consumer, green wheel CI, one capped live endpoint, and trusted publication. ## Release candidate diff --git a/docs/CONSUMER_CONTRACT.md b/docs/CONSUMER_CONTRACT.md new file mode 100644 index 0000000..1c7121d --- /dev/null +++ b/docs/CONSUMER_CONTRACT.md @@ -0,0 +1,64 @@ +# Support-triage consumer contract + +## Purpose and status + +`examples/support_triage.py` is the repository's canonical reference consumer. It demonstrates a production-shaped use case: classify an untrusted support ticket into a queue and urgency through one strict function call, then validate the model-supplied arguments before application use. + +This is deterministic contract evidence, not a claim of live deployment or Samsarix Unified adoption. A live endpoint remains an external release gate because it requires operator-owned credentials and can incur provider charges. + +## Supported public boundary + +The consumer imports only names exported by `unified_llm`: + +- `OpenAIResponsesProvider` +- `Route` +- `UnifiedLLM` + +It does not import implementation modules, private symbols, legacy services, persistence, or account logic. The fixture targets the current `0.1.0` public contract. Until a `0.2.0` release establishes a wider compatibility promise, consumers should pin the exact prerelease version or artifact digest they verify. + +## Contract + +| Boundary | Consumer requirement | Evidence | +| --- | --- | --- | +| Authentication | The API key is supplied only to the provider adapter and becomes an authorization header, never JSON payload data. | `test_support_triage_public_api_contract` | +| Privacy | Responses requests set `store=false`; the router has no persistence or content logging; the observer receives only sanitized attempt metadata. Model-written summary text is validated and discarded, then replaced with a local queue/urgency template. | Exact outbound payload assertion, adversarial summary fixture, and core architecture tests | +| Request safety | Ticket and strict tool schema are included in the exact request-byte calculation. The reference client caps requests at 64,000 bytes and output at 1,000 tokens. | Router request-bound tests and reference client configuration | +| Response safety | Raw responses are capped at 256,000 bytes, normalized content at 8,000 characters, and tool calls at four. | Reference client configuration and provider-bound tests | +| Decision integrity | Exactly one named function call is required. JSON must contain only `queue`, `urgency`, and `summary`; enum values and summary length are checked after parsing. The returned summary is generated locally from validated enums. | Happy-path, adversarial-summary, and malformed-decision tests | +| Failure behavior | Configuration, validation, provider, and fallback failures remain typed SDK exceptions. Invalid business output becomes a consumer-owned `ValueError`; no empty success value is returned. | Consumer negative tests and SDK error contract | +| Metadata | The decision records normalized provider, served model, and reported token total for audit/cost attribution. | Happy-path fixture | +| Lifecycle | The reference uses `async with UnifiedLLM(...)`, closing SDK-owned HTTP resources deterministically. | `main()` and lifecycle tests | + +## Verification + +Run from a clean checkout: + +```bash +python -m pytest tests/test_consumer_contract.py -q +python -m mypy unified_llm tests examples +python -m ruff check . +python -m ruff format --check . +``` + +The HTTP fixture uses `httpx.MockTransport`; it requires no credentials, sends no network traffic, and asserts the provider boundary directly. + +## Live conformance gate + +Before describing this consumer as live-compatible, a maintainer must run one capped request against the intended endpoint using a revocable, least-privilege key and record: + +1. exact package artifact digest and Python version; +2. provider, endpoint class, and exact model identifier without recording credentials or ticket content; +3. request/response limits and whether provider-side storage was disabled; +4. returned tool-call shape, usage metadata, and typed failure behavior; +5. timestamp, maintainer, cost, and credential revocation or rotation result. + +Live evidence must not contain prompts, responses, authorization headers, customer data, or secrets. + +## Ownership, compatibility, and rollback + +- Owner: Samsarix LLC +- Support: `support@samsarix.com` +- Commercial contact: `contact@samsarix.com` +- Compatibility: exact verified prerelease artifact until a published compatibility window exists +- Rollback: stop invoking `triage_ticket`, remove the reference-consumer integration, and pin the previously verified SDK artifact; the core router and provider adapters remain independent of this example +- Adoption signal: count successful consumer-owned contract runs and, only after live adoption, valid routed tickets versus rejected malformed decisions diff --git a/examples/__init__.py b/examples/__init__.py new file mode 100644 index 0000000..64b0e6b --- /dev/null +++ b/examples/__init__.py @@ -0,0 +1 @@ +"""Runnable public-API examples and reference consumers.""" diff --git a/examples/support_triage.py b/examples/support_triage.py new file mode 100644 index 0000000..be7b6fa --- /dev/null +++ b/examples/support_triage.py @@ -0,0 +1,126 @@ +"""Reference consumer: classify a support ticket through a bounded function call.""" + +from __future__ import annotations + +import asyncio +import json +import os +from dataclasses import dataclass +from typing import Any + +from unified_llm import ConfigurationError, OpenAIResponsesProvider, Route, UnifiedLLM + +TRIAGE_TOOL: dict[str, Any] = { + "type": "function", + "function": { + "name": "route_support_ticket", + "description": "Return the queue, urgency, and safe one-line summary for a support ticket.", + "parameters": { + "type": "object", + "properties": { + "queue": {"type": "string", "enum": ["billing", "bug", "security", "general"]}, + "urgency": {"type": "string", "enum": ["low", "normal", "high", "critical"]}, + "summary": {"type": "string", "minLength": 1, "maxLength": 240}, + }, + "required": ["queue", "urgency", "summary"], + "additionalProperties": False, + }, + "strict": True, + }, +} + +_QUEUES = frozenset({"billing", "bug", "security", "general"}) +_URGENCIES = frozenset({"low", "normal", "high", "critical"}) + + +@dataclass(frozen=True, slots=True) +class TriageDecision: + queue: str + urgency: str + summary: str + provider: str + model: str + total_tokens: int + + +def build_client(api_key: str) -> UnifiedLLM: + """Build the reference consumer's privacy-first production client.""" + + provider = OpenAIResponsesProvider(name="openai", api_key=api_key, store=False) + return UnifiedLLM( + [Route(provider, "gpt-5-mini")], + request_timeout=20, + max_attempts_per_route=2, + max_total_attempts=2, + max_request_bytes=64_000, + max_response_bytes=256_000, + max_response_chars=8_000, + max_tool_calls=4, + max_output_tokens=1_000, + ) + + +async def triage_ticket(client: UnifiedLLM, ticket: str) -> TriageDecision: + """Classify one ticket and reject malformed model tool arguments.""" + + if not isinstance(ticket, str) or not ticket.strip(): + raise ValueError("ticket must be a non-empty string") + response = await client.chat_with_tools( + [ + { + "role": "developer", + "content": ( + "Classify support tickets. Never include credentials, tokens, payment details, " + "or other secrets in the summary. Always call route_support_ticket exactly once." + ), + }, + {"role": "user", "content": ticket}, + ], + tools=[TRIAGE_TOOL], + max_tokens=300, + temperature=0, + ) + if len(response.tool_calls) != 1: + raise ValueError("provider must return exactly one support-routing tool call") + function = response.tool_calls[0].get("function") + if not isinstance(function, dict) or function.get("name") != "route_support_ticket": + raise ValueError("provider returned the wrong support-routing tool") + arguments = function.get("arguments") + if not isinstance(arguments, str): + raise ValueError("provider returned invalid support-routing arguments") + try: + decision = json.loads(arguments) + except (TypeError, ValueError) as exc: + raise ValueError("provider returned malformed support-routing JSON") from exc + if not isinstance(decision, dict) or set(decision) != {"queue", "urgency", "summary"}: + raise ValueError("provider returned an invalid support-routing object") + queue = decision["queue"] + urgency = decision["urgency"] + summary = decision["summary"] + if not isinstance(queue, str) or queue not in _QUEUES: + raise ValueError("provider returned an unsupported support-routing value") + if not isinstance(urgency, str) or urgency not in _URGENCIES: + raise ValueError("provider returned an unsupported support-routing value") + if not isinstance(summary, str) or not summary.strip() or len(summary) > 240: + raise ValueError("provider returned an invalid support-routing summary") + return TriageDecision( + queue=queue, + urgency=urgency, + summary=f"{urgency.capitalize()}-urgency ticket routed to {queue} support.", + provider=response.provider, + model=response.model, + total_tokens=response.total_tokens, + ) + + +async def main() -> None: + api_key = os.getenv("OPENAI_API_KEY") + if not api_key: + raise ConfigurationError("OPENAI_API_KEY is required.") + async with build_client(api_key) as client: + decision = await triage_ticket(client, input("Support ticket: ")) + print(decision) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/tests/test_consumer_contract.py b/tests/test_consumer_contract.py new file mode 100644 index 0000000..85a9382 --- /dev/null +++ b/tests/test_consumer_contract.py @@ -0,0 +1,149 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import httpx +import pytest + +from examples.support_triage import TRIAGE_TOOL, TriageDecision, main, triage_ticket +from unified_llm import ConfigurationError, OpenAIResponsesProvider, Route, UnifiedLLM, UnifiedLLMResponse + +ROOT = Path(__file__).parents[1] + + +async def test_support_triage_public_api_contract() -> None: + seen: dict[str, object] = {} + + def handler(request: httpx.Request) -> httpx.Response: + seen["method"] = request.method + seen["url"] = str(request.url) + seen["authorization"] = request.headers.get("Authorization") + seen["payload"] = json.loads(request.content) + return httpx.Response( + 200, + json={ + "model": "gpt-5-mini-2026-06-01", + "status": "completed", + "output": [ + { + "type": "function_call", + "call_id": "call-triage", + "name": "route_support_ticket", + "arguments": json.dumps( + {"queue": "security", "urgency": "critical", "summary": "Possible account takeover"} + ), + } + ], + "usage": {"input_tokens": 30, "output_tokens": 12, "total_tokens": 42}, + }, + ) + + http_client = httpx.AsyncClient(transport=httpx.MockTransport(handler)) + provider = OpenAIResponsesProvider(name="openai", api_key="test-secret", client=http_client, store=False) + client = UnifiedLLM( + [Route(provider, "gpt-5-mini")], + max_request_bytes=64_000, + max_response_bytes=256_000, + max_response_chars=8_000, + max_tool_calls=4, + ) + result = await triage_ticket(client, "Unknown login changed my recovery email and locked me out.") + + assert result == TriageDecision( + queue="security", + urgency="critical", + summary="Critical-urgency ticket routed to security support.", + provider="openai", + model="gpt-5-mini-2026-06-01", + total_tokens=42, + ) + assert seen["method"] == "POST" + assert seen["url"] == "https://api.openai.com/v1/responses" + assert seen["authorization"] == "Bearer test-secret" + payload = seen["payload"] + assert isinstance(payload, dict) + assert payload["store"] is False + assert payload["max_output_tokens"] == 300 + assert payload["temperature"] == 0 + assert payload["tools"] == [ + { + "type": "function", + "name": TRIAGE_TOOL["function"]["name"], + "description": TRIAGE_TOOL["function"]["description"], + "parameters": TRIAGE_TOOL["function"]["parameters"], + "strict": True, + } + ] + assert "test-secret" not in json.dumps(payload) + await client.aclose() + await http_client.aclose() + + +class DecisionProvider: + name = "decision" + + def __init__(self, *, name: str = "route_support_ticket", arguments: str, count: int = 1) -> None: + self.function_name = name + self.arguments = arguments + self.count = count + + async def complete(self, **_kwargs: object) -> UnifiedLLMResponse: + call = { + "id": "call-1", + "type": "function", + "function": {"name": self.function_name, "arguments": self.arguments}, + } + return UnifiedLLMResponse( + content="", + model="contract-model", + provider=self.name, + tool_calls=tuple(dict(call) for _ in range(self.count)), + ) + + +@pytest.mark.parametrize( + "provider", + [ + DecisionProvider(arguments="not-json"), + DecisionProvider(arguments='{"queue":"unknown","urgency":"normal","summary":"x"}'), + DecisionProvider(arguments='{"queue":[],"urgency":"normal","summary":"x"}'), + DecisionProvider(arguments='{"queue":"bug","urgency":{},"summary":"x"}'), + DecisionProvider(arguments='{"queue":"bug","urgency":"normal","summary":"x","extra":true}'), + DecisionProvider(name="wrong_tool", arguments='{"queue":"bug","urgency":"normal","summary":"x"}'), + DecisionProvider(arguments='{"queue":"bug","urgency":"normal","summary":"x"}', count=2), + ], +) +async def test_support_triage_rejects_malformed_model_decisions(provider: DecisionProvider) -> None: + client = UnifiedLLM([Route(provider, "contract-model")]) + with pytest.raises(ValueError, match="provider"): + await triage_ticket(client, "The app crashed") + + +async def test_support_triage_rejects_empty_ticket() -> None: + client = UnifiedLLM([Route(DecisionProvider(arguments="{}"), "contract-model")]) + with pytest.raises(ValueError, match="ticket"): + await triage_ticket(client, " ") + + +async def test_support_triage_never_returns_model_supplied_summary() -> None: + secret = "sk-secret-that-must-not-cross-the-boundary" + provider = DecisionProvider(arguments=json.dumps({"queue": "billing", "urgency": "high", "summary": secret})) + client = UnifiedLLM([Route(provider, "contract-model")]) + + decision = await triage_ticket(client, "My payment failed") + + assert decision.summary == "High-urgency ticket routed to billing support." + assert secret not in decision.summary + + +async def test_support_triage_main_requires_api_key(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + with pytest.raises(ConfigurationError, match="OPENAI_API_KEY"): + await main() + + +def test_reference_consumer_uses_only_public_package_imports() -> None: + source = (ROOT / "examples" / "support_triage.py").read_text(encoding="utf-8") + assert "unified_llm.unified_llm" not in source + assert "from unified_llm import" in source