-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add support triage consumer contract #6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Support-triage consumer contract | ||
|
|
||
| ## Purpose and status | ||
|
|
||
| `examples/support_triage.py` is the repository's canonical reference consumer. It demonstrates a production-shaped use case: classify an untrusted support ticket into a queue and urgency through one strict function call, then validate the model-supplied arguments before application use. | ||
|
|
||
| This is deterministic contract evidence, not a claim of live deployment or Samsarix Unified adoption. A live endpoint remains an external release gate because it requires operator-owned credentials and can incur provider charges. | ||
|
|
||
| ## Supported public boundary | ||
|
|
||
| The consumer imports only names exported by `unified_llm`: | ||
|
|
||
| - `OpenAIResponsesProvider` | ||
| - `Route` | ||
| - `UnifiedLLM` | ||
|
|
||
| It does not import implementation modules, private symbols, legacy services, persistence, or account logic. The fixture targets the current `0.1.0` public contract. Until a `0.2.0` release establishes a wider compatibility promise, consumers should pin the exact prerelease version or artifact digest they verify. | ||
|
|
||
| ## Contract | ||
|
|
||
| | Boundary | Consumer requirement | Evidence | | ||
| | --- | --- | --- | | ||
| | Authentication | The API key is supplied only to the provider adapter and becomes an authorization header, never JSON payload data. | `test_support_triage_public_api_contract` | | ||
| | Privacy | Responses requests set `store=false`; the router has no persistence or content logging; the observer receives only sanitized attempt metadata. Model-written summary text is validated and discarded, then replaced with a local queue/urgency template. | Exact outbound payload assertion, adversarial summary fixture, and core architecture tests | | ||
| | Request safety | Ticket and strict tool schema are included in the exact request-byte calculation. The reference client caps requests at 64,000 bytes and output at 1,000 tokens. | Router request-bound tests and reference client configuration | | ||
| | Response safety | Raw responses are capped at 256,000 bytes, normalized content at 8,000 characters, and tool calls at four. | Reference client configuration and provider-bound tests | | ||
| | Decision integrity | Exactly one named function call is required. JSON must contain only `queue`, `urgency`, and `summary`; enum values and summary length are checked after parsing. The returned summary is generated locally from validated enums. | Happy-path, adversarial-summary, and malformed-decision tests | | ||
| | Failure behavior | Configuration, validation, provider, and fallback failures remain typed SDK exceptions. Invalid business output becomes a consumer-owned `ValueError`; no empty success value is returned. | Consumer negative tests and SDK error contract | | ||
| | Metadata | The decision records normalized provider, served model, and reported token total for audit/cost attribution. | Happy-path fixture | | ||
| | Lifecycle | The reference uses `async with UnifiedLLM(...)`, closing SDK-owned HTTP resources deterministically. | `main()` and lifecycle tests | | ||
|
|
||
| ## Verification | ||
|
|
||
| Run from a clean checkout: | ||
|
|
||
| ```bash | ||
| python -m pytest tests/test_consumer_contract.py -q | ||
| python -m mypy unified_llm tests examples | ||
| python -m ruff check . | ||
| python -m ruff format --check . | ||
| ``` | ||
|
|
||
| The HTTP fixture uses `httpx.MockTransport`; it requires no credentials, sends no network traffic, and asserts the provider boundary directly. | ||
|
|
||
| ## Live conformance gate | ||
|
|
||
| Before describing this consumer as live-compatible, a maintainer must run one capped request against the intended endpoint using a revocable, least-privilege key and record: | ||
|
|
||
| 1. exact package artifact digest and Python version; | ||
| 2. provider, endpoint class, and exact model identifier without recording credentials or ticket content; | ||
| 3. request/response limits and whether provider-side storage was disabled; | ||
| 4. returned tool-call shape, usage metadata, and typed failure behavior; | ||
| 5. timestamp, maintainer, cost, and credential revocation or rotation result. | ||
|
|
||
| Live evidence must not contain prompts, responses, authorization headers, customer data, or secrets. | ||
|
|
||
| ## Ownership, compatibility, and rollback | ||
|
|
||
| - Owner: Samsarix LLC | ||
| - Support: `support@samsarix.com` | ||
| - Commercial contact: `contact@samsarix.com` | ||
| - Compatibility: exact verified prerelease artifact until a published compatibility window exists | ||
| - Rollback: stop invoking `triage_ticket`, remove the reference-consumer integration, and pin the previously verified SDK artifact; the core router and provider adapters remain independent of this example | ||
| - Adoption signal: count successful consumer-owned contract runs and, only after live adoption, valid routed tickets versus rejected malformed decisions |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| """Runnable public-API examples and reference consumers.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| """Reference consumer: classify a support ticket through a bounded function call.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
| import json | ||
| import os | ||
| from dataclasses import dataclass | ||
| from typing import Any | ||
|
|
||
| from unified_llm import ConfigurationError, OpenAIResponsesProvider, Route, UnifiedLLM | ||
|
|
||
| TRIAGE_TOOL: dict[str, Any] = { | ||
| "type": "function", | ||
| "function": { | ||
| "name": "route_support_ticket", | ||
| "description": "Return the queue, urgency, and safe one-line summary for a support ticket.", | ||
| "parameters": { | ||
| "type": "object", | ||
| "properties": { | ||
| "queue": {"type": "string", "enum": ["billing", "bug", "security", "general"]}, | ||
| "urgency": {"type": "string", "enum": ["low", "normal", "high", "critical"]}, | ||
| "summary": {"type": "string", "minLength": 1, "maxLength": 240}, | ||
| }, | ||
| "required": ["queue", "urgency", "summary"], | ||
| "additionalProperties": False, | ||
| }, | ||
| "strict": True, | ||
| }, | ||
| } | ||
|
|
||
| _QUEUES = frozenset({"billing", "bug", "security", "general"}) | ||
| _URGENCIES = frozenset({"low", "normal", "high", "critical"}) | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class TriageDecision: | ||
| queue: str | ||
| urgency: str | ||
| summary: str | ||
| provider: str | ||
| model: str | ||
| total_tokens: int | ||
|
|
||
|
|
||
| def build_client(api_key: str) -> UnifiedLLM: | ||
| """Build the reference consumer's privacy-first production client.""" | ||
|
|
||
| provider = OpenAIResponsesProvider(name="openai", api_key=api_key, store=False) | ||
| return UnifiedLLM( | ||
| [Route(provider, "gpt-5-mini")], | ||
| request_timeout=20, | ||
| max_attempts_per_route=2, | ||
| max_total_attempts=2, | ||
| max_request_bytes=64_000, | ||
| max_response_bytes=256_000, | ||
| max_response_chars=8_000, | ||
| max_tool_calls=4, | ||
| max_output_tokens=1_000, | ||
| ) | ||
|
|
||
|
|
||
| async def triage_ticket(client: UnifiedLLM, ticket: str) -> TriageDecision: | ||
| """Classify one ticket and reject malformed model tool arguments.""" | ||
|
|
||
| if not isinstance(ticket, str) or not ticket.strip(): | ||
| raise ValueError("ticket must be a non-empty string") | ||
| response = await client.chat_with_tools( | ||
| [ | ||
| { | ||
| "role": "developer", | ||
| "content": ( | ||
| "Classify support tickets. Never include credentials, tokens, payment details, " | ||
| "or other secrets in the summary. Always call route_support_ticket exactly once." | ||
| ), | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| }, | ||
| {"role": "user", "content": ticket}, | ||
| ], | ||
| tools=[TRIAGE_TOOL], | ||
| max_tokens=300, | ||
| temperature=0, | ||
| ) | ||
| if len(response.tool_calls) != 1: | ||
| raise ValueError("provider must return exactly one support-routing tool call") | ||
| function = response.tool_calls[0].get("function") | ||
| if not isinstance(function, dict) or function.get("name") != "route_support_ticket": | ||
| raise ValueError("provider returned the wrong support-routing tool") | ||
| arguments = function.get("arguments") | ||
| if not isinstance(arguments, str): | ||
| raise ValueError("provider returned invalid support-routing arguments") | ||
| try: | ||
| decision = json.loads(arguments) | ||
| except (TypeError, ValueError) as exc: | ||
| raise ValueError("provider returned malformed support-routing JSON") from exc | ||
| if not isinstance(decision, dict) or set(decision) != {"queue", "urgency", "summary"}: | ||
| raise ValueError("provider returned an invalid support-routing object") | ||
| queue = decision["queue"] | ||
| urgency = decision["urgency"] | ||
| summary = decision["summary"] | ||
| if not isinstance(queue, str) or queue not in _QUEUES: | ||
| raise ValueError("provider returned an unsupported support-routing value") | ||
| if not isinstance(urgency, str) or urgency not in _URGENCIES: | ||
| raise ValueError("provider returned an unsupported support-routing value") | ||
| if not isinstance(summary, str) or not summary.strip() or len(summary) > 240: | ||
| raise ValueError("provider returned an invalid support-routing summary") | ||
| return TriageDecision( | ||
| queue=queue, | ||
| urgency=urgency, | ||
| summary=f"{urgency.capitalize()}-urgency ticket routed to {queue} support.", | ||
| provider=response.provider, | ||
| model=response.model, | ||
| total_tokens=response.total_tokens, | ||
| ) | ||
|
|
||
|
|
||
| async def main() -> None: | ||
| api_key = os.getenv("OPENAI_API_KEY") | ||
| if not api_key: | ||
| raise ConfigurationError("OPENAI_API_KEY is required.") | ||
| async with build_client(api_key) as client: | ||
| decision = await triage_ticket(client, input("Support ticket: ")) | ||
| print(decision) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| asyncio.run(main()) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| from pathlib import Path | ||
|
|
||
| import httpx | ||
| import pytest | ||
|
|
||
| from examples.support_triage import TRIAGE_TOOL, TriageDecision, main, triage_ticket | ||
| from unified_llm import ConfigurationError, OpenAIResponsesProvider, Route, UnifiedLLM, UnifiedLLMResponse | ||
|
|
||
| ROOT = Path(__file__).parents[1] | ||
|
|
||
|
|
||
| async def test_support_triage_public_api_contract() -> None: | ||
| seen: dict[str, object] = {} | ||
|
|
||
| def handler(request: httpx.Request) -> httpx.Response: | ||
| seen["method"] = request.method | ||
| seen["url"] = str(request.url) | ||
| seen["authorization"] = request.headers.get("Authorization") | ||
| seen["payload"] = json.loads(request.content) | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| return httpx.Response( | ||
| 200, | ||
| json={ | ||
| "model": "gpt-5-mini-2026-06-01", | ||
| "status": "completed", | ||
| "output": [ | ||
| { | ||
| "type": "function_call", | ||
| "call_id": "call-triage", | ||
| "name": "route_support_ticket", | ||
| "arguments": json.dumps( | ||
| {"queue": "security", "urgency": "critical", "summary": "Possible account takeover"} | ||
| ), | ||
| } | ||
| ], | ||
| "usage": {"input_tokens": 30, "output_tokens": 12, "total_tokens": 42}, | ||
| }, | ||
| ) | ||
|
|
||
| http_client = httpx.AsyncClient(transport=httpx.MockTransport(handler)) | ||
| provider = OpenAIResponsesProvider(name="openai", api_key="test-secret", client=http_client, store=False) | ||
| client = UnifiedLLM( | ||
| [Route(provider, "gpt-5-mini")], | ||
| max_request_bytes=64_000, | ||
| max_response_bytes=256_000, | ||
| max_response_chars=8_000, | ||
| max_tool_calls=4, | ||
| ) | ||
| result = await triage_ticket(client, "Unknown login changed my recovery email and locked me out.") | ||
|
|
||
| assert result == TriageDecision( | ||
| queue="security", | ||
| urgency="critical", | ||
| summary="Critical-urgency ticket routed to security support.", | ||
| provider="openai", | ||
| model="gpt-5-mini-2026-06-01", | ||
| total_tokens=42, | ||
| ) | ||
| assert seen["method"] == "POST" | ||
| assert seen["url"] == "https://api.openai.com/v1/responses" | ||
| assert seen["authorization"] == "Bearer test-secret" | ||
| payload = seen["payload"] | ||
| assert isinstance(payload, dict) | ||
| assert payload["store"] is False | ||
| assert payload["max_output_tokens"] == 300 | ||
| assert payload["temperature"] == 0 | ||
| assert payload["tools"] == [ | ||
| { | ||
| "type": "function", | ||
| "name": TRIAGE_TOOL["function"]["name"], | ||
| "description": TRIAGE_TOOL["function"]["description"], | ||
| "parameters": TRIAGE_TOOL["function"]["parameters"], | ||
| "strict": True, | ||
| } | ||
| ] | ||
| assert "test-secret" not in json.dumps(payload) | ||
| await client.aclose() | ||
| await http_client.aclose() | ||
|
|
||
|
|
||
| class DecisionProvider: | ||
| name = "decision" | ||
|
|
||
| def __init__(self, *, name: str = "route_support_ticket", arguments: str, count: int = 1) -> None: | ||
| self.function_name = name | ||
| self.arguments = arguments | ||
| self.count = count | ||
|
|
||
| async def complete(self, **_kwargs: object) -> UnifiedLLMResponse: | ||
| call = { | ||
| "id": "call-1", | ||
| "type": "function", | ||
| "function": {"name": self.function_name, "arguments": self.arguments}, | ||
| } | ||
| return UnifiedLLMResponse( | ||
| content="", | ||
| model="contract-model", | ||
| provider=self.name, | ||
| tool_calls=tuple(dict(call) for _ in range(self.count)), | ||
| ) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize( | ||
| "provider", | ||
| [ | ||
| DecisionProvider(arguments="not-json"), | ||
| DecisionProvider(arguments='{"queue":"unknown","urgency":"normal","summary":"x"}'), | ||
| DecisionProvider(arguments='{"queue":[],"urgency":"normal","summary":"x"}'), | ||
| DecisionProvider(arguments='{"queue":"bug","urgency":{},"summary":"x"}'), | ||
| DecisionProvider(arguments='{"queue":"bug","urgency":"normal","summary":"x","extra":true}'), | ||
| DecisionProvider(name="wrong_tool", arguments='{"queue":"bug","urgency":"normal","summary":"x"}'), | ||
| DecisionProvider(arguments='{"queue":"bug","urgency":"normal","summary":"x"}', count=2), | ||
| ], | ||
| ) | ||
| async def test_support_triage_rejects_malformed_model_decisions(provider: DecisionProvider) -> None: | ||
| client = UnifiedLLM([Route(provider, "contract-model")]) | ||
| with pytest.raises(ValueError, match="provider"): | ||
| await triage_ticket(client, "The app crashed") | ||
|
|
||
|
|
||
| async def test_support_triage_rejects_empty_ticket() -> None: | ||
| client = UnifiedLLM([Route(DecisionProvider(arguments="{}"), "contract-model")]) | ||
| with pytest.raises(ValueError, match="ticket"): | ||
| await triage_ticket(client, " ") | ||
|
|
||
|
|
||
| async def test_support_triage_never_returns_model_supplied_summary() -> None: | ||
| secret = "sk-secret-that-must-not-cross-the-boundary" | ||
| provider = DecisionProvider(arguments=json.dumps({"queue": "billing", "urgency": "high", "summary": secret})) | ||
| client = UnifiedLLM([Route(provider, "contract-model")]) | ||
|
|
||
| decision = await triage_ticket(client, "My payment failed") | ||
|
|
||
| assert decision.summary == "High-urgency ticket routed to billing support." | ||
| assert secret not in decision.summary | ||
|
|
||
|
|
||
| async def test_support_triage_main_requires_api_key(monkeypatch: pytest.MonkeyPatch) -> None: | ||
| monkeypatch.delenv("OPENAI_API_KEY", raising=False) | ||
| with pytest.raises(ConfigurationError, match="OPENAI_API_KEY"): | ||
| await main() | ||
|
|
||
|
|
||
| def test_reference_consumer_uses_only_public_package_imports() -> None: | ||
| source = (ROOT / "examples" / "support_triage.py").read_text(encoding="utf-8") | ||
| assert "unified_llm.unified_llm" not in source | ||
| assert "from unified_llm import" in source | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.