diff --git a/cases/domux-clarify-commit/README.md b/cases/domux-clarify-commit/README.md new file mode 100644 index 0000000..a6cc49f --- /dev/null +++ b/cases/domux-clarify-commit/README.md @@ -0,0 +1,208 @@ +--- +title: Domux 有歧义指令先澄清、再绑定、后提交 +author: MittaPei +date: 2026-08-27 +category: smart-home-command +testedRevision: 6c71a32f4d624cadfd9fce9d10240d8068e53456 +runtime: transformers-5.15.0 +hardware: NVIDIA A800-SXM4-80GB on Linux 5.15 +downloadSource: huggingface +channels: + - https://huggingface.co/iFlytekOpenSource/Domux/discussions/5 +--- + +# Clarify, Bind, Commit：有歧义的 Domux 指令先确认，再执行 + +> 在 48 组清晰／歧义配对指令上，Domux 的 clear specificity 为 48/48， +> ambiguity sensitivity 为 47/48；原始 v1 clarify-and-commit 质量门为 +> **FAIL（42/48）**。最终参考实现没有覆盖这个失败结论，而是把 4 条已记录的 +> Domux 输出接入真实 Home Assistant：3/3 精确状态变更，1/1 状态漂移在调用前拒绝。 + +![Domux 消歧确认后提交的评测与 Home Assistant 验收证据](preview.png) + +## 真实任务 / Task + +“关掉那盏灯”“把卧室空调调到 22 度”可能对应多个同名设备。若集成层默认选择 +列表第一项，模型的不确定性会变成真实设备误操作。 + +本案例实现一条窄而完整的边界：解析 Domux 七槽位输出 → 查找候选 → 有歧义时 +展示最多 3 个选项 → 将确认绑定到用户、会话、候选、状态、TTL 和一次性 nonce → +提交前重新校验 → 调用 Home Assistant → 回读并核对精确后置状态。 + +## Hugging Face 下载证据 / Hugging Face download + +- Model: `iFlytekOpenSource/Domux` +- Revision: `6c71a32f4d624cadfd9fce9d10240d8068e53456` +- Full BF16 snapshot: 13 files，`10,279,032,574` bytes +- Snapshot manifest file SHA-256: + `85ffc0d612758c2dfb1cde6b42623a8f793ad5f4e42e6734694b0c2107389200` +- Canonical 13-file entry digest: + `5a13462b24fc9b00d132c42718e037bc42fc51a3c6752041998e085579f01416` + +为避免继承 shell 中的自定义 HF endpoint： + +```bash +env -u HF_ENDPOINT -u HUGGINGFACE_HUB_BASE_URL \ + hf download iFlytekOpenSource/Domux \ + --revision 6c71a32f4d624cadfd9fce9d10240d8068e53456 \ + --local-dir ./domux-6c71a32 +``` + +`run_model.py` 在加载前核验 revision、文件大小、hash 和 Hub 元数据；仓库不包含权重。 + +## 环境 / Setup + +- Python 3.12.12；`transformers.AutoModelForMultimodalLM` +- transformers 5.15.0；torch 2.10.0+cu128；huggingface_hub 1.27.0； + accelerate 1.13.0 +- 单卡 NVIDIA A800-SXM4-80GB；BF16 +- greedy；`temperature=0`；`max_new_tokens=128`；seed `20260826` +- 2 条独立 warm-up，不计入开发或评测数据 +- 48 base × clear/ambiguous = 96 probes；0 failure；0 selective rerun +- model load 4.564 s；peak allocated 10,244,988,416 bytes +- Home Assistant 官方镜像 `2026.8.3`，固定 digest，loopback 随机端口， + 1.5 CPU / 2 GiB / 512 PIDs / `restart=no` + +## 实际过程 / What happened + +同一 base 的实际输入和原始输出： + +```text +Turn off the Light in the Living Room on the Ground Floor. +turnOff|Light|*|*|*|Living Room|Ground Floor + +Turn off the light. +turnOff|Light|*|*|*|*|* +``` + +另两条代表记录： + +```text +Set the hall curtain to 20 percent. +set|Curtain|position|20|Percent|Hall|* + +Set the bedroom AC to 22 degrees. +set|AC|temperature|22|Celsius|Bedroom|* +``` + +全部 96 条输入、raw output、token 数、延迟和 hash 保存在 +[`evidence/v1/domux_raw.jsonl`](evidence/v1/domux_raw.jsonl)，没有删样本或选择性重跑。 + +真实 HA 日志绑定了 4 组固定模型输入／输出、4 条冻结场景和执行时的 policy/runner +hash。精简摘录： + +```json +{ + "schema_version": 4, + "status": "passed", + "successful_transitions": 3, + "rejected_before_dispatch": 1, + "sut_dispatches": 3, + "service_calls": "5 setup + 3 SUT + 1 fault injection = 9", + "artifact_sha256": "aa5a70e5d19a0cd90fd673e3f19224231da086dff766ee05aaead8141a6017f0" +} +``` + +三条成功路径覆盖灯、窗帘位置与空调温度；每条只产生一次 SUT 调用，nonce 重放 +新增调用为 0。第四条在 prepare 后通过 HA REST 注入真实状态漂移，提交返回 +`state_changed / INVALIDATED`，在 dispatch 前拒绝。 + +## 结果 / Results + +统计单元是 48 个配对 base，不把 96 条 probe 当成独立样本。二元比例使用 Wilson +双侧 95% 区间；预注册比较使用 exact McNemar + Holm 校正；延迟先在 base 内配对， +再汇总 48 个 base。 + +| 指标 | 结果 | 方法 | +|---|---:|---| +| Clear specificity | 48/48 | fixed v1 paired suite | +| Ambiguity sensitivity / paired discrimination | 47/48 | fixed v1 paired suite | +| Candidate coverage | 44/48 | intended entity in bounded candidates | +| Guarded B2 exact delta | 42/48 | original v1; gate **FAIL** | +| Safe-abstain baseline | 47/48 | dispatch 1/48；wrong target 0/48 | +| Pair latency | median 638.807 ms；p95 922.7055 ms | 48 within-base medians | +| Real HA commits | 3/3 | exact before/after state + postcondition | +| Real HA drift rejection | 1/1 | out-of-band change；0 SUT dispatch | + +v1 要求所有 eligible B2 clean/guard trial 通过，因此 42/48 必须写 FAIL。当前顶层 +`clarify_commit.py` 是查看失败后的最终参考实现，不用于改写原始 v1 数字。 + +## 失败语义 / Failure cases + +| 场景 | 结果 | 设备调用 | +|---|---|---:| +| malformed `turnOn\|Light` | `ParseError` before prepare | 0 | +| expired confirmation | `expired` | 0 | +| reused nonce | `replayed_nonce` | 0 additional | +| bound state changed | `state_changed / INVALIDATED` | 0 | +| postcondition mismatch | `FAILED_POSTCONDITION`; nonce consumed | 1 already occurred | + +最后一项不能假装“未执行”：调用已经发生而状态不可确认时，系统保留失败事实并禁止 +相同 nonce 重试。 + +## 复现 / Reproduction + +核心代码测试不需要 GPU 或 Home Assistant： + +```bash +cd cases/domux-clarify-commit +python -m unittest discover -s tests -q +``` + +预期：`Ran 159 tests`，`OK`。 + +有 Docker 时可重跑真实 HA（仅创建带任务标签的容器/volume，结束时清理）： + +```bash +python ha_acceptance.py --output /tmp/domux-ha-acceptance.json +``` + +完整模型重跑： + +```bash +CUDA_VISIBLE_DEVICES=0 python run_model.py \ + --snapshot ./domux-6c71a32 \ + --dataset data/scenarios.jsonl \ + --output /tmp/domux-raw.jsonl \ + --metadata-output /tmp/domux-metadata.json \ + --mode formal --split eval --precision bf16 + +python evaluate.py \ + --from-frozen-evidence /tmp/domux-raw.jsonl \ + --evidence-metadata /tmp/domux-metadata.json \ + --output-dir /tmp/domux-eval +``` + +评测质量门不通过时 evaluator 以 exit code 1 结束，同时仍完整写出 report/trials； +这是结果，不是基础设施故障。 + +## 价值 / Why it mattered + +- 模型 raw output 与设备执行分开计数； +- 有歧义先澄清，确认前 0 device call； +- 确认绑定具体状态和一次性 nonce，漂移与重放在执行前失效； +- 调用后回读精确状态，不把 HTTP 200 等同于任务成功； +- 不覆盖失败指标，也不把 4 条 HA 子集包装成生产结论。 + +## 公开的 Hugging Face Discussion / Published Hugging Face Discussion + +- https://huggingface.co/iFlytekOpenSource/Domux/discussions/5 + +## 安全、隐私与许可 / Safety, privacy, and licensing + +- 64 个原创合成 base，只含 Light、Curtain、AC；排除门锁、燃气、报警、摄像头、 + 医疗设备和真实硬件。 +- 无 token、cookie、私有 endpoint、个人缓存路径、真实家庭或业务数据。 +- 合成数据/data card 以 CC BY 4.0 提供；案例代码沿用仓库 Apache-2.0；模型、 + Gemma、Home Assistant 与 runtime 仍按各自上游许可。 +- 没有模型权重；只提交脚本、输入、原始文本输出、元数据和已脱敏日志。 + +## 踩坑记录 / Notes and gotchas + +1. 96 probes 只有 48 个独立 base，不能扩大统计分母。 +2. setup、fault injection 和 SUT dispatch 必须分账。 +3. HTTP 200 后仍要回读并核对目标状态。 +4. HA 测试使用 4 个语义映射实体，不是完整场景 inventory。 +5. 澄清答案来自冻结的合成 scenario gold，确认后没有再次调用模型；因此这不是 + 不间断 live model-to-HA，也不认证 production safety。 +6. `preview.png` 是结果信息图；真实运行证据是 raw JSONL 和 HA JSON 日志。 diff --git a/cases/domux-clarify-commit/clarify_commit.py b/cases/domux-clarify-commit/clarify_commit.py new file mode 100644 index 0000000..536bc0a --- /dev/null +++ b/cases/domux-clarify-commit/clarify_commit.py @@ -0,0 +1,5500 @@ +#!/usr/bin/env python3 +"""Deterministic grounding and one-time commit for Domux seven-slot outputs. + +The model remains a parser. This module deliberately keeps inventory lookup, +clarification, authorization lifetime, state binding, dispatch and postcondition +checks outside the model so every boundary can be audited and replayed. +""" + +from __future__ import annotations + +import copy +import hashlib +import json +import math +import re +import secrets +import threading +import time +import urllib.error +import urllib.parse +import urllib.request +from contextlib import ExitStack +from dataclasses import dataclass, replace +from functools import lru_cache +from typing import Any, Callable, Iterable, Mapping, Sequence + + +SLOTS = ("action", "device", "attribute", "value", "unit", "room", "floor") +SUPPORTED_DOMAINS = frozenset({"light", "cover", "climate"}) +MAX_UTTERANCE_CHARS = 2048 +ENTITY_ID_RE = re.compile(r"^[a-z][a-z0-9_]*\.[a-z0-9_]+$") +GENERIC_DEVICE_ALIASES = { + "light": "light", + "lights": "light", + "lamp": "light", + "lamps": "light", + "lighting": "light", + "curtain": "cover", + "curtains": "cover", + "blind": "cover", + "blinds": "cover", + "shade": "cover", + "shades": "cover", + "cover": "cover", + "covers": "cover", + "ac": "climate", + "acs": "climate", + "a c": "climate", + "a/c": "climate", + "air conditioner": "climate", + "air conditioners": "climate", + "air conditioning": "climate", +} +GENERIC_DEVICE_NOUN_RE = ( + "(?:" + + "|".join( + re.escape(alias) + for alias in sorted(GENERIC_DEVICE_ALIASES, key=len, reverse=True) + ) + + ")" +) +GENERIC_REFERENCE_NOUN_RE = ( + rf"(?:one|ones|device|devices|{GENERIC_DEVICE_NOUN_RE}|" + r"room|rooms|floor|floors|level|levels)" +) +CORRECTION_SELECTOR_ACTION_RE = r"(?:use|select|choose|mean)" +SELECTOR_ACTION_RE = rf"(?:{CORRECTION_SELECTOR_ACTION_RE}|touch|act\s+on)" +STATE_CHANGE_ACTION_RE = ( + r"(?:turn(?:\s+(?:on|off))?|switch(?:\s+(?:on|off))?|open|close|set|" + r"change|make|move|adjust|raise|lower|increase|decrease|execute|proceed|" + r"dispatch)" +) +EXECUTION_CONTROL_ACTION_RE = ( + rf"(?:{STATE_CHANGE_ACTION_RE}|confirm|authorize|approve|go\s+ahead|do\s+it)" +) +STATE_COMMAND_RE = ( + rf"(?:(?:(?:please|just)\s+)*{STATE_CHANGE_ACTION_RE}|" + rf"(?:can|could|would|will)\s+you\s+(?:please\s+)?(?:just\s+)?" + rf"{STATE_CHANGE_ACTION_RE}|" + rf"(?:i|we)\s+(?:just\s+)?(?:want|need)\s+(?:you\s+)?to\s+" + rf"(?:please\s+)?(?:just\s+)?" + rf"{STATE_CHANGE_ACTION_RE})" +) +RESTART_COMMAND_RE = ( + rf"(?:{STATE_COMMAND_RE}|(?:(?:please|just)\s+)*" + rf"(?:(?:i|we)\s+(?:just\s+)?{SELECTOR_ACTION_RE}|{SELECTOR_ACTION_RE}))" +) +OTHER_REFERENCE_RE = ( + rf"\b(?:the\s+other(?!\s+than\b)(?:\s+(?:[a-z0-9]+\s+){{0,3}}" + rf"{GENERIC_REFERENCE_NOUN_RE})?|other(?!\s+than\b)" + rf"(?:\s+[a-z0-9]+){{0,3}}\s+{GENERIC_REFERENCE_NOUN_RE})\b" +) +POSTPOSED_NO_MARKER_RE = r"\bno\b\s*(?:[,，:：—–.!?;。！？；-])" +COLOR_RGB = { + "blue": [0, 0, 255], + "cyan": [0, 255, 255], + "cool white": [201, 226, 255], + "green": [0, 128, 0], + "lavender": [230, 230, 250], + "magenta": [255, 0, 255], + "orange": [255, 165, 0], + "pink": [255, 192, 203], + "purple": [128, 0, 128], + "red": [255, 0, 0], + "sky blue": [135, 206, 235], + "warm white": [255, 244, 229], + "white": [255, 255, 255], + "yellow": [255, 255, 0], +} + + +class ParseError(ValueError): + """The raw Domux text is not a bounded sequence of seven-slot records.""" + + +class GroundingError(ValueError): + """A structurally valid model output cannot be mapped to an executable plan.""" + + +class AdapterError(RuntimeError): + """The Home Assistant adapter could not read or change controlled state.""" + + +class ServiceCallError(AdapterError): + """A dispatch failure with action-local, non-heuristic outcome metadata.""" + + def __init__( + self, + message: str, + *, + attempted: bool, + acknowledged: bool, + outcome_unknown: bool, + ) -> None: + super().__init__(message) + self.attempted = attempted + self.acknowledged = acknowledged + self.outcome_unknown = outcome_unknown + + +@dataclass(frozen=True) +class ServiceCallResult: + after: Mapping[str, object] + attempted: bool + acknowledged: bool + outcome_unknown: bool = False + + +def normalize_text(value: object) -> str: + """Normalize only for comparison; never use this to overwrite raw evidence.""" + + return " ".join( + str(value) + .replace("_", " ") + .replace("-", " ") + .replace("’", "'") + .replace("‘", "'") + .split() + ).casefold() + + +def canonical_json(value: object) -> str: + return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + + +def digest_json(value: object) -> str: + return hashlib.sha256(canonical_json(value).encode("utf-8")).hexdigest() + + +@dataclass(frozen=True) +class DomuxInstruction: + action: str + device: str + attribute: str + value: str + unit: str + room: str + floor: str + + @classmethod + def from_fields(cls, fields: Sequence[str]) -> "DomuxInstruction": + if len(fields) != len(SLOTS): + raise ParseError(f"expected seven fields, got {len(fields)}") + cleaned = tuple(field.strip() for field in fields) + if any(not field for field in cleaned): + raise ParseError("empty fields must be represented by '*'") + if not cleaned[0] or cleaned[0] == "*": + raise ParseError("action cannot be omitted") + return cls(*cleaned) + + def to_pipe(self) -> str: + return "|".join(getattr(self, slot) for slot in SLOTS) + + def canonical_slots(self) -> dict[str, str]: + return {slot: normalize_text(getattr(self, slot)) for slot in SLOTS} + + +def parse_domux_output(raw_output: str, *, max_instructions: int = 8) -> tuple[DomuxInstruction, ...]: + """Parse raw model text without silently dropping malformed segments.""" + + if not isinstance(raw_output, str) or not raw_output.strip(): + raise ParseError("model output is empty") + if any(ord(char) < 32 and char not in "\r\n\t" for char in raw_output): + raise ParseError("model output contains control characters") + segments = [segment.strip() for segment in raw_output.replace("&", "\n").splitlines()] + if any(not segment for segment in segments): + segments = [segment for segment in segments if segment] + if not segments: + raise ParseError("model output has no non-empty instruction") + if len(segments) > max_instructions: + raise ParseError(f"model output has more than {max_instructions} instructions") + parsed: list[DomuxInstruction] = [] + for index, segment in enumerate(segments, start=1): + fields = segment.split("|") + if len(fields) != len(SLOTS): + raise ParseError(f"instruction {index} has {len(fields)} fields, expected seven") + parsed.append(DomuxInstruction.from_fields(fields)) + return tuple(parsed) + + +@dataclass(frozen=True) +class EntitySpec: + entity_id: str + domain: str + device: str + room: str + floor: str + aliases: tuple[str, ...] = () + + def __post_init__(self) -> None: + if self.domain not in SUPPORTED_DOMAINS: + raise ValueError(f"unsupported entity domain: {self.domain}") + if not self.entity_id.startswith(f"{self.domain}."): + raise ValueError(f"entity_id/domain mismatch: {self.entity_id}") + if not ENTITY_ID_RE.fullmatch(self.entity_id): + raise ValueError(f"invalid Home Assistant entity_id: {self.entity_id}") + + def stable_metadata(self) -> dict[str, object]: + return { + "entity_id": self.entity_id, + "domain": self.domain, + "device": self.device, + "room": self.room, + "floor": self.floor, + "aliases": sorted(self.aliases, key=lambda value: (normalize_text(value), value)), + } + + +@dataclass(frozen=True) +class SessionContext: + recent_entity_ids: tuple[str, ...] = () + + +class EntityRegistry: + """A deterministic, immutable view of the allowed Home Assistant entities.""" + + def __init__(self, entities: Iterable[EntitySpec]): + by_id: dict[str, EntitySpec] = {} + for entity in entities: + if entity.entity_id in by_id: + raise ValueError(f"duplicate entity_id: {entity.entity_id}") + by_id[entity.entity_id] = entity + if not by_id: + raise ValueError("registry cannot be empty") + self._by_id = by_id + + def get(self, entity_id: str) -> EntitySpec: + try: + return self._by_id[entity_id] + except KeyError as exc: + raise GroundingError(f"entity is not in the allowed registry: {entity_id}") from exc + + @property + def entities(self) -> tuple[EntitySpec, ...]: + return tuple(sorted(self._by_id.values(), key=self._sort_key)) + + @staticmethod + def _sort_key(entity: EntitySpec) -> tuple[str, str, str, str]: + return tuple(normalize_text(part) for part in ( + entity.floor, entity.room, entity.device, entity.entity_id, + )) + + @staticmethod + def _domain_hint(instruction: DomuxInstruction) -> str | None: + device = normalize_text(instruction.device) + if device in GENERIC_DEVICE_ALIASES: + return GENERIC_DEVICE_ALIASES[device] + attribute = normalize_text(instruction.attribute) + if attribute in {"brightness", "color", "color temperature", "colortemperature"}: + return "light" + if attribute in {"position", "openness"}: + return "cover" + if attribute in {"temperature", "mode", "wind speed", "windspeed", "fan speed"}: + return "climate" + return None + + @staticmethod + def _device_matches(entity: EntitySpec, requested: str) -> bool: + requested_norm = normalize_text(requested) + if requested_norm in {"*", "it", "that", "that one", "this", "this one"}: + return True + names = {normalize_text(entity.device), *(normalize_text(alias) for alias in entity.aliases)} + if requested_norm in names: + return True + domain = GENERIC_DEVICE_ALIASES.get(requested_norm) + return domain == entity.domain + + def candidates( + self, + instruction: DomuxInstruction, + context: SessionContext | None = None, + ) -> tuple[EntitySpec, ...]: + domain_hint = self._domain_hint(instruction) + room = normalize_text(instruction.room) + floor = normalize_text(instruction.floor) + candidates = [ + entity + for entity in self._by_id.values() + if (domain_hint is None or entity.domain == domain_hint) + and self._device_matches(entity, instruction.device) + and (room == "*" or normalize_text(entity.room) == room) + and (floor == "*" or normalize_text(entity.floor) == floor) + ] + + requested = normalize_text(instruction.device) + if context and requested in {"*", "it", "that", "that one", "this", "this one"}: + recent = set(context.recent_entity_ids) + contextual = [entity for entity in candidates if entity.entity_id in recent] + if contextual: + candidates = contextual + return tuple(sorted(candidates, key=self._sort_key)) + + def metadata_digest(self, entity_ids: Iterable[str]) -> str: + ordered = [self.get(entity_id).stable_metadata() for entity_id in sorted(set(entity_ids))] + return digest_json(ordered) + + def with_replacement(self, entity: EntitySpec) -> "EntityRegistry": + if entity.entity_id not in self._by_id: + raise GroundingError(f"cannot replace unknown entity: {entity.entity_id}") + updated = dict(self._by_id) + updated[entity.entity_id] = entity + return EntityRegistry(updated.values()) + + +@dataclass(frozen=True) +class Clarification: + required: bool + reason: str + candidates: tuple[EntitySpec, ...] + prompt: str | None + reasons: tuple[str, ...] = () + unresolved_slots: tuple[str, ...] = () + + +def _candidate_option(index: int, candidate: EntitySpec) -> str: + """Render a visibly unique, stable authorization choice.""" + + aliases = sorted( + {alias for alias in candidate.aliases if normalize_text(alias)}, + key=lambda value: (normalize_text(value), value), + ) + alias_text = f" / alias: {', '.join(aliases)}" if aliases else "" + return ( + f"{index}. {candidate.floor} / {candidate.room} / {candidate.device}" + f"{alias_text} / id: {candidate.entity_id}" + ) + + +def clarification_for(candidates: Sequence[EntitySpec], *, max_display: int = 3) -> Clarification: + if not candidates: + return Clarification(True, "no_registry_match", (), None, ("no_registry_match",)) + if len(candidates) == 1: + return Clarification(False, "unique_registry_match", tuple(candidates), None) + displayed = tuple(candidates[:max_display]) + if len(candidates) > max_display: + prompt = "Which room or floor should I use? More than three devices match." + return Clarification(True, "too_many_candidates", displayed, prompt) + options = "; ".join( + _candidate_option(index, candidate) + for index, candidate in enumerate(displayed, start=1) + ) + return Clarification( + True, + "multiple_registry_matches", + displayed, + f"Which device: {options}?", + ("multiple_registry_matches",), + ) + + +def _phrase_in(normalized_text: str, phrase: object) -> bool: + normalized_phrase = normalize_text(phrase) + if not normalized_phrase or normalized_phrase == "*": + return False + return re.search( + rf"(? tuple[float, ...]: + return tuple(float(value) for value in re.findall(r"(? str: + return str(int(value)) if value.is_integer() else format(value, ".15g") + + +def _operation_value_token(value: str) -> str | None: + if value == "*": + return None + try: + return f"number:{_canonical_number_token(float(value))}" + except ValueError: + return f"value:{normalize_text(value)}" + + +def _value_is_explicitly_excluded(text: str, value: str) -> bool: + """Bind a negative choice to the exact operation value it excludes.""" + + token = normalize_text(value) + if not token or token == "*": + return False + prefix = ( + r"(?:\b(?:anything|something|everything|any(?:\s+[a-z]+)?|a\s+value)\s+" + r"(?:but|other\s+than|besides|apart\s+from)\s+(?:the\s+)?|" + r"\b(?:other\s+than|except|besides|apart\s+from|avoid|without)\s+" + r"(?:using\s+)?(?:the\s+)?|" + r"\b(?:do\s+not|don't|dont|never)\s+(?:use|choose|select)\s+(?:the\s+)?|" + r"\bnot\s+(?:the\s+)?|" + r"\bno\s+(?:the\s+)?)" + ) + return re.search( + rf"{prefix}{re.escape(token)}(?![a-z0-9])", + normalize_text(text), + ) is not None + + +def _has_operation_value_exclusion(text: str) -> bool: + """Detect requests that permit a set of values but do not choose one.""" + + normalized = normalize_text(text) + if re.search( + r"\b(?:anything|something|everything|any(?:\s+[a-z]+)?|a\s+value)\s+" + r"(?:but|other\s+than|besides|apart\s+from)\b|" + r"\b(?:do\s+not|don't|dont|never)\s+(?:use|choose|select)\b|" + r"\b(?:set|change|make|use)\b.{0,40}\b" + r"(?:avoid|without|other\s+than|besides|apart\s+from)\b", + normalized, + ): + return True + known_values = ( + *COLOR_RGB, + "fan only", "heat cool", "cool", "heat", "dry", "fan", "auto", + "low", "medium", "high", + ) + return any(_value_is_explicitly_excluded(normalized, value) for value in known_values) + + +def _excluded_operation_value_tokens( + text: str, + source_instructions: Sequence[DomuxInstruction] = (), +) -> tuple[str, ...]: + """Return audit metadata for values explicitly ruled out by the user.""" + + known_values = ( + *COLOR_RGB, + "fan only", "heat cool", "cool", "heat", "dry", "fan", "auto", + "low", "medium", "high", + ) + tokens = { + token + for value in known_values + if _value_is_explicitly_excluded(text, value) + if (token := _operation_value_token(value)) is not None + } + for number in _numbers_in(normalize_text(text)): + rendered = _canonical_number_token(number) + if _value_is_explicitly_excluded(text, rendered): + tokens.add(f"number:{rendered}") + for instruction in source_instructions: + if _value_is_explicitly_excluded(text, instruction.value): + token = _operation_value_token(instruction.value) + if token is not None: + tokens.add(token) + return tuple(sorted(tokens)) + + +def _value_supported(text: str, value: str) -> bool: + if value == "*": + return True + normalized = normalize_text(text) + try: + wanted = float(value) + except ValueError: + return _phrase_in(normalized, value) + if any(math.isclose(wanted, number, rel_tol=0, abs_tol=0.001) for number in _numbers_in(normalized)): + return True + return math.isclose(wanted, 50.0, rel_tol=0, abs_tol=0.001) and _phrase_in(normalized, "halfway") + + +def _attribute_supported(text: str, instruction: DomuxInstruction) -> bool: + attribute = normalize_text(instruction.attribute) + if attribute == "*": + return True + normalized = normalize_text(text) + if attribute == "brightness": + return any(_phrase_in(normalized, term) for term in ("brightness", "bright", "brighter", "dimmer")) or ( + _phrase_in(normalized, "percent") + and any(_phrase_in(normalized, term) for term in ("light", "lamp")) + ) + if attribute in {"position", "openness"}: + return _phrase_in(normalized, "position") or _phrase_in(normalized, "openness") or ( + any(_phrase_in(normalized, term) for term in ("percent", "halfway", "open", "close", "move", "adjust")) + and any(_phrase_in(normalized, term) for term in ("curtain", "blind", "shade")) + ) or any(_phrase_in(normalized, term) for term in ("open", "close")) + if attribute == "temperature": + return any(_phrase_in(normalized, term) for term in ( + "temperature", "degree", "degrees", "celsius", "warmer", "cooler" + )) or ( + bool(_numbers_in(normalized)) + and any(_phrase_in(normalized, term) for term in ("ac", "air conditioner", "air conditioning")) + ) + if attribute == "colortemperature": + return "color temperature" in normalized or _phrase_in(normalized, "kelvin") or bool( + re.search(r"\d+(?:\.\d+)?\s*k\b", normalized) + ) + if attribute == "color": + return _phrase_in(normalized, "color") or any( + _phrase_in(normalized, color) for color in COLOR_RGB + ) + if attribute == "mode": + return _phrase_in(normalized, "mode") or ( + any(_phrase_in(normalized, mode) for mode in ("cool", "heat", "dry", "fan", "auto")) + and any(_phrase_in(normalized, term) for term in ("ac", "air conditioner", "air conditioning")) + ) + if attribute in {"windspeed", "wind speed", "fan speed"}: + return any(_phrase_in(normalized, term) for term in ("wind", "fan", "low", "medium", "high")) + return False + + +def _attribute_supported_for_entity( + text: str, + instruction: DomuxInstruction, + entity: EntitySpec, +) -> bool: + """Allow registry-domain inference only when the operation value is explicit.""" + + if _attribute_supported(text, instruction): + return True + attribute = normalize_text(instruction.attribute) + if attribute == "*": + return True + value_supported = _value_supported(text, instruction.value) + unit_supported = _unit_supported(text, instruction) + normalized = normalize_text(text) + if entity.domain == "cover" and attribute in {"position", "openness"}: + return value_supported and unit_supported and normalize_text(instruction.unit) == "percent" + if entity.domain == "light" and attribute == "brightness": + return value_supported and unit_supported and normalize_text(instruction.unit) == "percent" + if entity.domain == "light" and attribute == "colortemperature": + return value_supported and unit_supported and normalize_text(instruction.unit) == "kelvin" + if entity.domain == "light" and attribute == "color": + return value_supported and any(_phrase_in(normalized, color) for color in COLOR_RGB) + if entity.domain == "climate" and attribute == "temperature": + return value_supported and unit_supported and normalize_text(instruction.unit) == "celsius" + if entity.domain == "climate" and attribute == "mode": + return value_supported and any( + _phrase_in(normalized, mode) + for mode in ("cool", "heat", "dry", "fan only", "fan", "auto") + ) + if entity.domain == "climate" and attribute in {"windspeed", "wind speed", "fan speed"}: + return value_supported and unit_supported and normalize_text(instruction.unit) == "level" + return False + + +def _relative_adjust_directions(text: str) -> frozenset[str]: + """Return only relative directions explicitly authored by the user. + + A bare ``adjust`` carries no direction. Treating it as evidence for either + ``adjustUp`` or ``adjustDown`` would let a model choose a state transition + that the user never authorized. + """ + + normalized = normalize_text(text) + if _has_negated_direction(normalized): + return frozenset() + directions: set[str] = set() + if re.search( + r"\b(?:raise|increase)\b|" + r"\b(?:brighter|warmer|higher)\b(?!\s+(?:floor|level|mode)\b)", + normalized, + ): + directions.add("adjustup") + if re.search( + r"\blower\b(?!\s+(?:floor|level|mode)\b)|\bdecrease\b|" + r"\b(?:dimmer|cooler)\b(?!\s+(?:floor|level|mode)\b)", + normalized, + ): + directions.add("adjustdown") + return frozenset(directions) + + +def _action_supported(text: str, instruction: DomuxInstruction) -> bool: + normalized = normalize_text(text) + if _has_negative_action_authorization(normalized) or _has_negated_direction(normalized): + return False + action = normalize_text(instruction.action) + directional = _directional_actions(normalized) + # A proposed ``set`` must not bypass opposing open/close or on/off clauses. + # Such clauses describe more than one state transition and need an explicit + # clarification even when the model happens to emit a value-bearing action. + if len(directional) > 1: + return False + if action == "turnon": + return directional == {"turnon"} + if action == "turnoff": + return directional == {"turnoff"} + if action == "set": + return any(_phrase_in(normalized, term) for term in ( + "set", "change", "make", "move", "adjust", "open", "raise", "lower", "use", "confirm" + )) + if action in {"adjustup", "adjustdown"}: + return _relative_adjust_directions(normalized) == {action} + return False + + +def _directional_actions(normalized_text: str) -> set[str]: + """Extract action words without treating prepositions such as ``on Floor`` as actions.""" + + result: set[str] = set() + if re.search(r"\bon\s+(?:or|and\s*/?\s*or)\s+off\b|\boff\s+(?:or|and\s*/?\s*or)\s+on\b", normalized_text): + result.update(("turnon", "turnoff")) + if re.search(r"\bopen\s+(?:or|and\s*/?\s*or)\s+close\b|\bclose\s+(?:or|and\s*/?\s*or)\s+open\b", normalized_text): + result.update(("turnon", "turnoff")) + immediate = re.findall(r"\b(?:turn|switch)\s+(on|off)\b", normalized_text) + result.update("turnon" if value == "on" else "turnoff" for value in immediate) + # Support "switch that device off", but only when the direction ends the + # clause. This deliberately does not match "on the Ground Floor". It is + # collected even when another immediate action exists, so opposing clauses + # cannot be silently collapsed into the first action. + trailing = re.findall( + r"\b(?:turn|switch)\s+(?:the\s+)?(?:that\s+)?(?:[a-z0-9]+\s+){0,8}" + r"(on|off)(?=\s*(?:(?:right\s+)?now|please)?\s*(?:[,.!?;]|$))", + normalized_text, + ) + result.update("turnon" if value == "on" else "turnoff" for value in trailing) + if _phrase_in(normalized_text, "open") or _phrase_in(normalized_text, "start"): + result.add("turnon") + if _phrase_in(normalized_text, "close") or _phrase_in(normalized_text, "shut"): + result.add("turnoff") + return result + + +def _unit_supported(text: str, instruction: DomuxInstruction) -> bool: + unit = normalize_text(instruction.unit) + if unit == "*": + return True + normalized = normalize_text(text) + fahrenheit = _phrase_in(normalized, "fahrenheit") or bool( + re.search(r"-?\d+(?:\.\d+)?\s*(?:°\s*)?f\b", normalized) + ) + kelvin = _phrase_in(normalized, "kelvin") or bool( + re.search(r"-?\d+(?:\.\d+)?\s*k\b", normalized) + ) + celsius = _phrase_in(normalized, "celsius") or bool( + re.search(r"-?\d+(?:\.\d+)?\s*(?:°\s*)?c\b", normalized) + ) + if unit == "percent": + return _phrase_in(normalized, "percent") or "%" in text or _phrase_in(normalized, "halfway") + if unit == "celsius": + if fahrenheit or kelvin: + return False + # In this English-only case, an otherwise unqualified "degree(s)" is + # interpreted using the registered HA entity's Celsius capability. + return celsius or any(_phrase_in(normalized, term) for term in ("degree", "degrees")) + if unit == "kelvin": + return kelvin and not (fahrenheit or celsius) + if unit == "level": + return any(_phrase_in(normalized, term) for term in ("level", "low", "medium", "high")) + return False + + +def _distinct_named_matches( + text: str, + values: Iterable[str], +) -> tuple[tuple[int, int, str], ...]: + """Return longest label matches while preserving occurrence provenance.""" + + normalized = normalize_text(text) + matches: list[tuple[int, int, str]] = [] + for value in {normalize_text(item) for item in values if normalize_text(item)}: + for match in re.finditer( + rf"(?= item[1] + for other in matches + ) + ] + return tuple(sorted(kept)) + + +def _distinct_named_values(text: str, values: Iterable[str]) -> frozenset[str]: + """Return values with shorter matches removed only at overlapping spans.""" + + return frozenset( + value for _start, _end, value in _distinct_named_matches(text, values) + ) + + +def _negative_clause_is_selector_only(text: str) -> bool: + """Return whether a negative clause only excludes a target selector.""" + + normalized = normalize_text(text) + if re.search( + rf"\b(?:{EXECUTION_CONTROL_ACTION_RE}|want|need|have)\b", + normalized, + ): + return False + return bool( + re.search( + r"\b(?:do\s+not|don't|dont|never|cannot|can't|cant|won't|wont|" + r"wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|mustn't|" + r"mustnt|not)\s+(?:please\s+)?" + rf"{CORRECTION_SELECTOR_ACTION_RE}\b", + normalized, + ) + or re.match( + r"\s*(?:not|no|except|besides|other\s+than|anything\s+but|" + r"everything\s+but|all\s+but|apart\s+from|instead\s+of|avoid|without)\b", + normalized, + ) + ) + + +def _negative_clause_has_target_predicate(text: str) -> bool: + """Require a negated target to be attached to a bounded predicate.""" + + normalized = normalize_text(text) + if re.search( + rf"\b(?:{SELECTOR_ACTION_RE}|{EXECUTION_CONTROL_ACTION_RE}|want|need|have)\b", + normalized, + ): + return True + return bool(re.match( + r"\s*(?:not|no|except|besides|other\s+than|anything\s+but|" + r"everything\s+but|all\s+but|apart\s+from|instead\s+of|avoid|without)" + r"\s+(?:(?:a|an|the|this|that|any|all|every|each|either)\s+)?" + r"[a-z0-9]", + normalized, + )) + + +def _negative_clause_is_control_withdrawal(text: str) -> bool: + """Return whether a complete targetless execution-control was withdrawn.""" + + normalized = normalize_text(text).rstrip(" ,;:—.!?") + negative_lead = ( + r"(?:do\s+not|don't|dont|never(?:\s+ever)?|cannot|can't|cant|won't|" + r"wont|wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|mustn't|" + r"mustnt|(?:must|may|shall|should|can|could|will|would)\s+not)" + ) + control = r"(?:proceed|execute|confirm|authorize|approve|dispatch|go\s+ahead|do\s+it)" + return bool(re.fullmatch( + rf"\s*{negative_lead}\s+(?:please\s+)?{control}\s*", + normalized, + )) + + +def _selector_span_is_negative_base( + normalized_text: str, + start: int, + end: int | None = None, + selector_spans: Sequence[tuple[int, int]] = (), +) -> bool: + """Return whether a selector occurrence is inside the current negative clause. + + Inventory labels may be quoted or parenthesized, and a clarifying noun + phrase may be more than a fixed number of words after ``not``. Character + distance is therefore not a safe scope boundary. Keep negation active + within the current clause and reset it only at an explicit hard boundary, + positive contrast, or new imperative. + """ + + # Keep the complete prefix. Punctuation is interpreted below only after + # checking what appeared between the negator and the apparent restart. + # That prevents emphasis such as ``not!!! Study`` and an entity ID such as + # ``climate.study`` from silently terminating negative scope. + prefix = normalized_text[:start] + prefix_offset = 0 + quantified_exclusion = False + command_quantified_exclusion = False + anaphoric_exclusion = False + contrastive_boundaries = tuple(re.finditer(r"\b(?:but|rather)\b", prefix)) + if contrastive_boundaries: + boundary = contrastive_boundaries[-1] + before = prefix[:boundary.start()] + after = prefix[boundary.end():] + command_quantified_exclusion = bool( + re.search(r"\b(?:anything|everything)\b[\s,([{'\"]*$", before) + ) + quantified_exclusion = bool( + command_quantified_exclusion + or re.search( + r"\b(?:all|any|each|either|every)\b[^.!?;—]*$", + before, + ) + ) + # ``but by that I mean X`` can resolve an anaphor inside the + # preceding negative clause; it is not a positive correction. The + # shorter ``but I mean X`` is also carried when the negative clause + # ended in a generic/deictic target. Stable-target corrections such + # as ``do not use Living, but I mean Study`` remain positive. + negative_before = re.search( + r"\b(?:do\s+not|don't|dont|never|cannot|can't|cant|won't|wont|" + r"wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|mustn't|" + r"mustnt|not|no|except|besides|avoid|without)\b", + before, + ) + anaphoric_exclusion = bool( + negative_before + and ( + re.match( + r"\s*by\s+(?:that|it|this)[\s,;:—-]*(?:i|we)\s+mean\b", + after, + ) + or ( + re.match(r"\s*(?:i|we)\s+mean\b", after) + and re.search( + rf"\b(?:it|that|this|{GENERIC_REFERENCE_NOUN_RE})\b" + r"[\s,()'\"—-]*$", + before, + ) + ) + ) + ) + prefix = prefix[boundary.end():] + prefix_offset += boundary.end() + negative_action = rf"(?:{SELECTOR_ACTION_RE}|{STATE_CHANGE_ACTION_RE})" + strong_negative = ( + rf"(?:do\s+not|don't|dont|not)(?:\s+to)?\s+{negative_action}|" + r"(?:do\s+not|don't|dont|never|cannot|can't|cant|won't|wont|" + r"wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|mustn't|mustnt)|" + r"(?:except|besides|other\s+than|anything\s+but|apart\s+from|" + r"everything\s+but|all\s+but|instead\s+of|avoid|without)|no" + ) + carried_exclusion = quantified_exclusion or anaphoric_exclusion + last_negative_start: int | None = None + if carried_exclusion: + tail = prefix + tail_offset = prefix_offset + else: + negative_matches = tuple(re.finditer( + rf"\b(?:{strong_negative})\b|\bnot\b", + prefix, + )) + if not negative_matches: + return False + last_negative = negative_matches[-1] + last_negative_start = prefix_offset + last_negative.start() + tail = prefix[last_negative.end():] + tail_offset = prefix_offset + last_negative.end() + if last_negative.group(0) == "no" and re.match( + r"\s*,\s*(?:actually|perhaps|rather|instead|sorry|" + r"(?:i\s+)?(?:mean|want|choose|select|use)|just\s+use)\b", + tail, + ): + return False + # A shared negator can govern a coordinated verb list: ``do not use, + # choose, or mean Study Lamp``. In that construction the commas and + # conjunctions introduce no positive imperative because no target occurs + # between the verbs. Keep the whole list negative; a later imperative + # remains expressible with ``then``/``and then``, hard punctuation, or a + # comma after an already named target. + coordinated_delimiter = r"(?:,\s*(?:(?:and|or)\s+)?|(?:and|or)\s+)" + coordinated_negative_actions = ( + None + if carried_exclusion + else re.match( + rf"\s*(?:{coordinated_delimiter}(?:please\s+)?" + rf"{negative_action}\b\s*)+", + tail, + ) + ) + restart_tail = ( + tail[coordinated_negative_actions.end():] + if coordinated_negative_actions + else tail + ) + restart_tail_offset = tail_offset + ( + coordinated_negative_actions.end() + if coordinated_negative_actions + else 0 + ) + restart_command = RESTART_COMMAND_RE + if quantified_exclusion and not command_quantified_exclusion and re.match( + rf"\s*{restart_command}\b", + tail, + ): + return False + def has_prior_stable_target(relative_end: int) -> bool: + absolute_end = restart_tail_offset + relative_end + return any( + tail_offset <= selector_start + and selector_end <= absolute_end + for selector_start, selector_end in selector_spans + ) + + def has_prior_target(relative_end: int) -> bool: + absolute_end = restart_tail_offset + relative_end + generic_target = re.search( + rf"\b(?:anything|everything|it|that|this|{GENERIC_REFERENCE_NOUN_RE})\b", + normalized_text[tail_offset:absolute_end], + ) + return has_prior_stable_target(relative_end) or generic_target is not None + + state_restart = rf"{STATE_COMMAND_RE}\b" + + def positive_restart( + relative_end: int, + command_text: str, + boundary_text: str, + ) -> bool: + complete_state_command = re.match(state_restart, command_text) is not None + absolute_end = restart_tail_offset + relative_end + negative_clause_start = ( + tail_offset if last_negative_start is None else last_negative_start + ) + negative_clause = ( + "" + if not carried_exclusion and last_negative_start is None + else normalized_text[negative_clause_start:absolute_end] + ) + stable_target = has_prior_stable_target(relative_end) + # A colon can introduce the content being rejected ("I don't want + # this: turn off Study"). Without a stable target on the negative + # side it is not an authorization boundary. + if boundary_text == ":" and not stable_target: + return False + explicit_sequence = bool( + boundary_text == ";" or re.fullmatch(r"(?:and\s+then|then)", boundary_text) + ) + if ( + complete_state_command + and explicit_sequence + and _negative_clause_is_control_withdrawal(negative_clause) + ): + return True + if not _negative_clause_has_target_predicate(negative_clause): + if carried_exclusion and stable_target and complete_state_command: + return explicit_sequence + return False + if stable_target: + if complete_state_command: + return True + return _negative_clause_is_selector_only(negative_clause) + if not has_prior_target(relative_end): + return False + return complete_state_command + + def selector_only_restart(relative_end: int) -> bool: + if ( + last_negative_start is None + or not has_prior_stable_target(relative_end) + ): + return False + absolute_end = restart_tail_offset + relative_end + negative_clause = normalized_text[last_negative_start:absolute_end] + return ( + _negative_clause_has_target_predicate(negative_clause) + and _negative_clause_is_selector_only(negative_clause) + ) + + punctuation_restarts = re.finditer( + rf"(?P[!?;,:—]|\.(?![a-z0-9]))\s*" + rf"(?P{restart_command})\b", + restart_tail, + ) + for restart in punctuation_restarts: + if positive_restart( + restart.start(), + restart.group("command"), + restart.group("delimiter"), + ): + return False + for restart in re.finditer( + rf"\b(?Pand\s+then|then)\s+" + rf"(?P{restart_command})\b", + restart_tail, + ): + if positive_restart( + restart.start(), + restart.group("command"), + restart.group("delimiter"), + ): + return False + for boundary in re.finditer(r"[!?;:—]|\.(?![a-z0-9])", restart_tail): + suffix = restart_tail[boundary.end():] + if selector_only_restart(boundary.start()) and re.fullmatch( + r"[\s([{\"']*(?:(?:the|this|that|my|your|our)\s+)?", + suffix, + ): + return False + effective_end = end if end is not None else start + effective_end = max( + ( + selector_end + for selector_start, selector_end in selector_spans + if selector_start <= start + and effective_end <= selector_end + ), + default=effective_end, + ) + if ( + not carried_exclusion + and selector_only_restart(len(restart_tail)) + and re.search(r"[,;:—]\s*(?:(?:the|this|that|my|your|our)\s+)?$", restart_tail) + and re.match(r"\s+instead\b", normalized_text[effective_end:]) + ): + return False + return True + + +def _overlapping_selector_component( + seed: tuple[int, int], + selector_spans: Sequence[tuple[int, int]], +) -> frozenset[tuple[int, int]]: + """Return the connected overlap component containing one selector span.""" + + component = {seed} + remaining = set(selector_spans) - component + changed = True + while changed: + changed = False + component_start = min(start for start, _end in component) + component_end = max(end for _start, end in component) + overlapping = { + span + for span in remaining + if span[0] < component_end and component_start < span[1] + } + if overlapping: + component.update(overlapping) + remaining.difference_update(overlapping) + changed = True + return frozenset(component) + + +@lru_cache(maxsize=512) +def _postposed_no_correction_analysis( + normalized_text: str, + selector_spans: tuple[tuple[int, int], ...], +) -> tuple[tuple[tuple[tuple[int, int], str], ...], bool]: + """Analyze every ``old target, no, replacement`` component once. + + A postposed ``no`` withdraws the preceding positive selector. The next + selector becomes positive only when the text between ``no,`` and that + selector is a bounded correction introducer. Unknown or negative + replacement language therefore withdraws the old target without silently + authorizing a new one. + """ + + spans = tuple(sorted(set(selector_spans))) + # ASCII hyphens have already become spaces in ``normalize_text``. Match + # from the correction word itself so transcripts using ``-``, en/em dash, + # ASCII/CJK punctuation, or ``no:`` all retain the same safe semantics. + marker_pattern = POSTPOSED_NO_MARKER_RE + correction_gap = ( + r"\s*(?:(?:actually|perhaps|rather|instead|sorry|please|just|the|this|that|" + r"my|your|our)\s+|(?:(?:i|we)\s+)?(?:mean|want|choose|select|use)\s+)*" + ) + state_command_gap = rf"\s*(?:{STATE_COMMAND_RE})\s+(?:(?:the|this|that|my|your|our)\s+)?" + polite_selector_gap = ( + rf"\s*(?:can|could|would|will)\s+you\s+(?:please\s+)?(?:just\s+)?" + rf"{CORRECTION_SELECTOR_ACTION_RE}\s+" + r"(?:(?:the|this|that|my|your|our)\s+)?" + ) + withdrawn: set[tuple[int, int]] = set() + replacements: set[tuple[int, int]] = set() + unsafe_replacement = False + for marker in re.finditer(marker_pattern, normalized_text): + before = tuple(span for span in spans if span[1] <= marker.start()) + if not before: + continue + old_seed = max(before, key=lambda span: (span[1], span[0])) + old_component = _overlapping_selector_component(old_seed, spans) + old_was_positive_replacement = bool(old_component & replacements) + old_was_withdrawn = bool(old_component & withdrawn) + old_was_base_negative = ( + False + if old_was_positive_replacement + else _selector_span_is_negative_base( + normalized_text, + old_seed[0], + old_seed[1], + spans, + ) + ) + if old_was_withdrawn or ( + old_was_base_negative and not old_was_positive_replacement + ): + # ``do not use A—no, B`` is not a positive A-to-B correction. + continue + after = tuple(span for span in spans if span[0] >= marker.end()) + if not after: + # ``no`` may be correcting a value or mode rather than the target + # (for example, ``Study Lamp to 30—no, perhaps 60``). Without a + # later selector there is no evidence of a target correction. + continue + replacement_seed = min(after, key=lambda span: (span[0], -span[1])) + gap = normalized_text[marker.end():replacement_seed[0]] + positive_replacement = ( + re.fullmatch(correction_gap, gap) is not None + or re.fullmatch(state_command_gap, gap) is not None + or re.fullmatch(polite_selector_gap, gap) is not None + ) + withdrawn.update(old_component) + replacements.difference_update(old_component) + if not positive_replacement: + # Once both sides name stable selectors, an unknown or negative + # bridge cannot keep the old authorization alive or authorize the + # replacement. The request-level gate below also prevents a + # clarification from expanding to some third, unmentioned target. + unsafe_replacement = True + continue + replacement_component = _overlapping_selector_component( + replacement_seed, + spans, + ) + replacements.update(replacement_component) + roles = tuple(sorted( + (*((span, "withdrawn") for span in withdrawn), + *((span, "replacement") for span in replacements)), + )) + return roles, unsafe_replacement + + +def _postposed_no_correction_role( + normalized_text: str, + start: int, + end: int, + selector_spans: Sequence[tuple[int, int]], +) -> str | None: + """Return the cached postposed-correction role of one selector span.""" + + roles, _unsafe = _postposed_no_correction_analysis( + normalized_text, + tuple(sorted(set(selector_spans))), + ) + current = (start, end) + for span, role in roles: + if span == current: + return role + return None + + +def _selector_span_is_negative( + normalized_text: str, + start: int, + end: int | None = None, + selector_spans: Sequence[tuple[int, int]] = (), +) -> bool: + """Apply postposed corrections before the general negative-scope rules.""" + + effective_end = start if end is None else end + role = _postposed_no_correction_role( + normalized_text, + start, + effective_end, + selector_spans, + ) + if role == "withdrawn": + return True + if role == "replacement": + return False + return _selector_span_is_negative_base( + normalized_text, + start, + end, + selector_spans, + ) + + +def _positive_named_matches( + text: str, + values: Iterable[str], + *, + selector_spans: Sequence[tuple[int, int]] = (), +) -> tuple[tuple[int, int, str], ...]: + normalized = normalize_text(text) + matches = _distinct_named_matches(normalized, values) + known_spans = tuple(selector_spans) or tuple( + (start, end) for start, end, _value in matches + ) + return tuple( + match + for match in matches + if not _selector_span_is_negative( + normalized, + match[0], + match[1], + known_spans, + ) + and not ( + re.search(r"\bleave\s+(?:the\s+)?$", normalized[:match[0]]) + and re.match( + r"(?:\s+[a-z0-9]+){0,4}\s+(?:unchanged|as\s+is)\b", + normalized[match[1]:], + ) + ) + ) + + +def _label_is_grammar_collision(value: str) -> bool: + """Return whether a metadata label is also ordinary command language.""" + + words = re.findall(r"[a-z0-9]+", normalize_text(value)) + if not words or all(word.isdigit() for word in words): + return True + command_words = { + "a", "about", "adjust", "an", "and", "any", "around", "at", "auto", + "before", "between", "brightness", "bright", "brighter", "by", "change", + "choose", "close", "color", "confirm", "cool", "cooler", "decrease", + "degree", "degrees", "device", "dimmer", "do", "dry", "fan", "fahrenheit", + "first", "for", "from", "halfway", "heat", "high", "in", "increase", + "it", "kelvin", "last", "left", "level", "light", "lighting", "low", + "lower", "make", "medium", "middle", "mode", "move", "next", "no", "not", + "now", "of", "off", "on", "one", "open", "openness", "or", "other", + "percent", "please", "position", "proceed", "raise", "right", "second", + "select", "set", "side", "speed", "switch", "temperature", "that", "the", + "third", "this", "to", "turn", "use", "value", "wait", "warmer", "white", + "wind", "with", "without", "yes", "you", + *{ + word + for color in COLOR_RGB + for word in re.findall(r"[a-z0-9]+", normalize_text(color)) + }, + } + return all(word in command_words for word in words) + + +def _selector_match_is_anchored( + normalized_text: str, + start: int, + end: int, + value: str, + category: str, + registry: EntityRegistry, + *, + allow_topic: bool = True, +) -> bool: + """Require inventory words to occur in a target-selector position.""" + + if category in {"domain", "entity"}: + return True + prefix = normalized_text[:start] + suffix = normalized_text[end:] + device_labels = { + *GENERIC_DEVICE_ALIASES, + *(normalize_text(entity.device) for entity in registry.entities), + } + device_pattern = "|".join( + re.escape(label) for label in sorted(device_labels, key=len, reverse=True) + ) + locative_before = bool(re.search( + r"\b(?:in|inside|at|on|from)\s+(?:the\s+)?$", + prefix, + )) + device_after = bool(re.match( + rf"\s+(?:(?:or|and)\s+(?:the\s+)?(?:[a-z0-9]+\s+){{0,3}})?" + rf"(?:{device_pattern})(?![a-z0-9])", + suffix, + )) + explicit_slot = bool(re.search( + r"\b(?:room|floor|level|device)\s+(?:named\s+|called\s+)?$|" + r"\b(?:in|on)\s+the\s+(?:room|floor|level)\s+$", + prefix, + )) or bool(re.match(r"\s+(?:room|floor|level|device)\b", suffix)) + topic_before_command = not re.search( + r"\bfor\s+(?:the\s+)?$", + prefix, + ) and bool(re.match( + r"\s*[,;:]\s*(?:please\s+)?(?:turn|switch|open|close|set|change|" + r"make|adjust|raise|lower|increase|decrease|move|use|select|choose)\b", + suffix, + )) + if category != "domain" and _label_is_grammar_collision(value): + return locative_before or explicit_slot + value_words = set(re.findall(r"[a-z0-9]+", value)) + if category == "room": + if value_words.intersection({"room"}): + return True + if value in { + "any", "first", "second", "third", "last", "middle", "next", "one", + "right", "now", "please", "light", "that", "this", "other", "it", + }: + return locative_before or explicit_slot + return ( + locative_before or device_after or explicit_slot + or (allow_topic and topic_before_command) + ) + if category == "floor": + if value_words.intersection({"floor", "level", "storey", "story"}): + return True + return ( + locative_before or device_after or explicit_slot + or (allow_topic and topic_before_command) + ) + + # Specific device names containing a device noun are self-anchoring. + if any(_phrase_in(value, label) for label in GENERIC_DEVICE_ALIASES): + return True + command_before = bool(re.search( + r"\b(?:turn|switch|open|close|set|change|make|move|adjust|raise|lower|" + r"increase|decrease|use|select|choose|mean)\b(?:\s+(?:on|off))?" + r"(?:\s+(?:the|that|this|my))?\s+$|" + r"\b(?:the|that|this|my)\s+$", + prefix, + )) + operation_after = bool(re.match( + r"\s+(?:on|off|open|closed|to|by)(?:\b|$)", + suffix, + )) + return command_before or operation_after or locative_before or explicit_slot + + +def _registry_selector_spans( + text: str, + registry: EntityRegistry, +) -> tuple[tuple[int, int], ...]: + """Return actual stable inventory-selector spans for scope transitions.""" + + normalized = normalize_text(text) + labels = ( + ("room", tuple(entity.room for entity in registry.entities)), + ("floor", tuple(entity.floor for entity in registry.entities)), + ("device", tuple( + entity.device + for entity in registry.entities + if normalize_text(entity.device) not in GENERIC_DEVICE_ALIASES + )), + ("alias", tuple(alias for entity in registry.entities for alias in entity.aliases)), + ("entity", tuple(entity.entity_id for entity in registry.entities)), + ) + spans: set[tuple[int, int]] = set() + for category, values in labels: + for start, end, value in _distinct_named_matches(normalized, values): + if not _label_is_grammar_collision(value) or _selector_match_is_anchored( + normalized, + start, + end, + value, + category, + registry, + ): + spans.add((start, end)) + return tuple(sorted(spans)) + + +def _final_authorization_clause(text: str, registry: EntityRegistry) -> str: + """Return the clause after a complete, explicit positive restart. + + A negative command may name an old target before a new immediate command, + for example ``do not turn off Living; turn off Study``. The latter clause + is independent authorization only when the negative clause already named + a stable or generic target and the restart begins with a supported command. + This target-before-boundary rule keeps emphasis such as ``do not! turn off + Study`` inside negative scope. + """ + + normalized = normalize_text(text) + negative_anchor = ( + r"\b(?:do\s+not|don't|dont|never(?:\s+ever)?|cannot|can't|cant|" + r"won't|wont|wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|" + r"mustn't|mustnt|not|no|avoid|without|except|besides|other\s+than|" + r"anything\s+but|everything\s+but|all\s+but|apart\s+from|instead\s+of)\b" + ) + negatives = tuple(re.finditer(negative_anchor, normalized)) + if not negatives: + return normalized + last_negative = negatives[-1] + selector_spans = _registry_selector_spans(normalized, registry) + generic_target = re.compile( + rf"\b(?:anything|everything|it|that|this|{GENERIC_REFERENCE_NOUN_RE})\b" + ) + + def has_prior_target(boundary_start: int) -> bool: + if any( + last_negative.end() <= selector_start + and selector_end <= boundary_start + for selector_start, selector_end in selector_spans + ): + return True + return generic_target.search( + normalized, + last_negative.end(), + boundary_start, + ) is not None + + command = RESTART_COMMAND_RE + state_command = rf"{STATE_COMMAND_RE}\b" + + def valid_restart(restart: re.Match[str]) -> bool: + boundary_start = restart.start("boundary") + if boundary_start < last_negative.end(): + return False + complete_state_command = ( + re.match(state_command, restart.group("command")) is not None + ) + negative_clause = normalized[last_negative.start():boundary_start] + if not _negative_clause_has_target_predicate(negative_clause): + return False + stable_target = any( + last_negative.end() <= selector_start + and selector_end <= boundary_start + for selector_start, selector_end in selector_spans + ) + boundary_text = restart.group("boundary") + if boundary_text == ":" and not stable_target: + return False + explicit_sequence = bool( + boundary_text == ";" + or re.fullmatch(r"(?:and\s+then|then)", boundary_text) + ) + if ( + complete_state_command + and explicit_sequence + and _negative_clause_is_control_withdrawal(negative_clause) + ): + return True + if stable_target: + if complete_state_command: + return True + return _negative_clause_is_selector_only(negative_clause) + if not has_prior_target(boundary_start): + return False + # A generic withdrawal can only be superseded by a complete new + # state-changing command. ``I mean Study`` merely identifies which + # generic target was unwanted and cannot revive an earlier action. + return complete_state_command + + restart_points: list[int] = [] + for restart in re.finditer( + rf"(?P[!?;,:—]|\.(?![a-z0-9]))\s*" + rf"(?:(?:but|rather)\s+)?(?P{command})\b", + normalized, + ): + if valid_restart(restart): + restart_points.append(restart.start("command")) + for restart in re.finditer( + rf"(?P\b(?:and\s+then|then|but|rather)\b)\s+" + rf"(?P{command})\b", + normalized, + ): + if valid_restart(restart): + restart_points.append(restart.start("command")) + if not restart_points: + return normalized + return normalized[min(restart_points):] + + +def _selector_label_evidence( + text: str, + label: str, + category: str, + registry: EntityRegistry, +) -> bool: + """Recognize a label in a clarification without protocol-word collisions.""" + + normalized = normalize_text(text) + matches = _positive_named_matches( + normalized, + (label,), + selector_spans=_registry_selector_spans(normalized, registry), + ) + if not matches: + return False + if not _label_is_grammar_collision(label): + return True + return any( + _selector_match_is_anchored( + normalized, start, end, value, category, registry, + ) + for start, end, value in matches + ) + + +def _targeted_named_values(text: str, values: Iterable[str]) -> frozenset[str]: + """Extract enum values that immediately follow an explicit target cue.""" + + normalized = normalize_text(text) + targets: set[str] = set() + for value in _distinct_named_values(normalized, values): + if re.search( + rf"\b(?:to|into)\b(?:\s+(?:the|a|an|target|new)){{0,3}}\s+{re.escape(value)}\b", + normalized, + ): + targets.add(value) + return frozenset(targets) + + +def _instruction_numeric_value(instruction: DomuxInstruction) -> float | None: + try: + return float(instruction.value) + except ValueError: + return None + + +def _operational_conflicts(text: str, instruction: DomuxInstruction) -> frozenset[str]: + """Find explicit semantics that one proposed seven-slot tuple cannot bind. + + Presence checks alone are unsafe for corrections such as ``from 50 to 20``: + both numbers occur, but only 20 is the requested target. This routine is a + deliberately small fail-closed binder for directional actions, target + numbers, colors, HVAC modes, and incompatible unit families. + """ + + normalized = normalize_text(text) + conflicts: set[str] = set() + if _has_operation_value_exclusion(text): + # The seven-slot contract cannot encode "anything except X". A + # concrete, non-excluded replacement must be supplied in clarification. + conflicts.add("value") + directional_actions = _directional_actions(normalized) + proposed_action = normalize_text(instruction.action) + if len(directional_actions) > 1: + conflicts.add("action") + if "turnoff" in directional_actions and proposed_action in {"set", "adjustup", "adjustdown"}: + # HA set/adjust services for these domains can turn a light/cover/AC on; + # they cannot stand in for an explicit off/close operation. + conflicts.add("action") + + numbers = set(_numbers_in(normalized)) + if _phrase_in(normalized, "halfway"): + numbers.add(50.0) + target_numbers = { + float(value) + for value in re.findall( + r"\b(?:to|at|around|about)\s+(?:around\s+|about\s+)?(-?\d+(?:\.\d+)?)\b", + normalized, + ) + } + by_numbers = { + float(value) + for value in re.findall(r"\bby\s+(-?\d+(?:\.\d+)?)\b", normalized) + } + relative_numbers = { + float(value) + for value in re.findall( + r"(?])" + r"\s*-?\d+(?:\.\d+)?\b", + normalized, + )) + numeric_from_to = bool(re.search( + r"\bfrom\s+-?\d+(?:\.\d+)?\s+to\s+-?\d+(?:\.\d+)?\b", + normalized, + )) + proposed_number = _instruction_numeric_value(instruction) + if "turnon" in directional_actions and proposed_action in {"set", "adjustup", "adjustdown"}: + attribute = normalize_text(instruction.attribute) + compatible_turn_on = ( + attribute in {"brightness", "color", "colortemperature"} + or (proposed_action == "set" and attribute == "mode") + or ( + attribute in {"position", "openness"} + and ( + (proposed_action == "set" and proposed_number is not None and proposed_number > 0) + or proposed_action == "adjustup" + ) + ) + ) + if not compatible_turn_on: + conflicts.add("action") + absolute_numeric_cue = bool(target_numbers - by_numbers - relative_numbers) or bool( + re.search(r"\b(?:to|at)\s+halfway\b", normalized) + ) + if proposed_action in {"adjustup", "adjustdown"} and absolute_numeric_cue: + conflicts.update(("action", "value")) + if proposed_action == "set" and by_numbers: + conflicts.update(("action", "value")) + if ( + proposed_action in {"adjustup", "adjustdown"} + and numbers + and not (by_numbers or relative_numbers) + ): + conflicts.add("value") + if ( + range_expression + or inequality_expression + or len(target_numbers) > 1 + or (len(numbers) > 1 and not numeric_from_to) + ): + conflicts.add("value") + elif len(target_numbers) == 1: + target = next(iter(target_numbers)) + if proposed_number is None or not math.isclose(proposed_number, target, rel_tol=0, abs_tol=0.001): + conflicts.add("value") + elif len(numbers) == 1: + only = next(iter(numbers)) + if proposed_number is None or not math.isclose(proposed_number, only, rel_tol=0, abs_tol=0.001): + conflicts.add("value") + + colors = _distinct_named_values(normalized, COLOR_RGB) + color_targets = _targeted_named_values(normalized, COLOR_RGB) + proposed_value = normalize_text(instruction.value) + color_from_to = bool(re.search( + r"\bfrom\b(?:\s+[a-z0-9]+){0,3}\s+(?:" + "|".join( + re.escape(color) for color in sorted(COLOR_RGB, key=len, reverse=True) + ) + r")\s+to\s+(?:" + "|".join( + re.escape(color) for color in sorted(COLOR_RGB, key=len, reverse=True) + ) + r")\b", + normalized, + )) + if len(color_targets) > 1 or (len(colors) > 1 and not color_from_to): + conflicts.add("value") + elif len(color_targets) == 1 and proposed_value != next(iter(color_targets)): + conflicts.add("value") + + hvac_values = ("fan only", "heat cool", "cool", "heat", "dry", "fan", "auto") + mode_text = re.sub(r"\b(?:fan|wind)\s+speed\b", " ", normalized) + modes = _distinct_named_values(mode_text, hvac_values) + mode_targets = _targeted_named_values(mode_text, hvac_values) + + def canonical_mode(value: str) -> str: + return "fan only" if normalize_text(value) == "fan" else normalize_text(value) + + proposed_mode = canonical_mode(instruction.value) + canonical_targets = {canonical_mode(value) for value in mode_targets} + mode_pattern = r"(?:fan\s+only|heat\s+cool|cool|heat|dry|fan|auto)" + mode_from_to = bool(re.search( + rf"\bfrom\s+{mode_pattern}\s+to\s+{mode_pattern}\b", + mode_text, + )) + if len(canonical_targets) > 1 or (len(modes) > 1 and not mode_from_to): + conflicts.add("value") + elif len(canonical_targets) == 1 and proposed_mode != next(iter(canonical_targets)): + conflicts.add("value") + + # A mode/color plus a numeric target is more than one operation for this + # seven-slot contract; do not let either proposed tuple silently drop half. + if numbers and modes and any(_phrase_in(normalized, cue) for cue in ("ac", "air conditioner", "mode")): + conflicts.update(("attribute", "value")) + if numbers and colors and any(_phrase_in(normalized, cue) for cue in ("light", "lamp", "color")): + conflicts.update(("attribute", "value")) + + families: set[str] = set() + color_temperature_text = re.sub(r"\bcolor\s+temperature\b", " ", normalized) + light_cue = any(_phrase_in(normalized, cue) for cue in ("light", "lamp")) + cover_cue = any(_phrase_in(normalized, cue) for cue in ("curtain", "blind", "shade")) + climate_cue = any(_phrase_in(normalized, cue) for cue in ("ac", "air conditioner", "air conditioning")) + if any(_phrase_in(normalized, cue) for cue in ("brightness", "brighter", "dimmer")) or ( + light_cue and ("%" in text or _phrase_in(normalized, "percent")) + ): + families.add("brightness") + if any(_phrase_in(normalized, cue) for cue in ("position", "openness")) or ( + cover_cue and (numbers or _phrase_in(normalized, "halfway")) + ): + families.add("position") + if _phrase_in(normalized, "color temperature") or _phrase_in(normalized, "kelvin") or bool( + re.search(r"-?\d+(?:\.\d+)?\s*k\b", normalized) + ): + families.add("color_temperature") + if colors or _phrase_in(color_temperature_text, "color"): + families.add("color") + temperature_text = re.sub(r"\bcolor\s+temperature\b", " ", normalized) + if any(_phrase_in(temperature_text, cue) for cue in ( + "temperature", "celsius", "fahrenheit", "degree", "degrees", + )) or ( + climate_cue and (numbers or any(_phrase_in(normalized, cue) for cue in ("warmer", "cooler"))) + ): + families.add("temperature") + if _phrase_in(normalized, "mode") or (climate_cue and modes): + families.add("mode") + if any(_phrase_in(normalized, cue) for cue in ("wind speed", "fan speed")): + families.add("fan_speed") + if len(families) > 1: + conflicts.add("attribute") + setter_cue = any(_phrase_in(normalized, cue) for cue in ( + "set", "change", "make", "adjust", "raise", "lower", "use", + )) + if proposed_action in {"turnon", "turnoff"} and setter_cue and families: + conflicts.update(("action", "attribute")) + + fahrenheit = _phrase_in(normalized, "fahrenheit") or bool( + re.search(r"-?\d+(?:\.\d+)?\s*(?:°\s*)?f\b", normalized) + ) + kelvin = _phrase_in(normalized, "kelvin") or bool( + re.search(r"-?\d+(?:\.\d+)?\s*k\b", normalized) + ) + celsius = _phrase_in(normalized, "celsius") or bool( + re.search(r"-?\d+(?:\.\d+)?\s*(?:°\s*)?c\b", normalized) + ) + proposed_unit = normalize_text(instruction.unit) + if ( + (proposed_unit == "celsius" and (fahrenheit or kelvin)) + or (proposed_unit == "kelvin" and (fahrenheit or celsius)) + or (proposed_unit not in {"*", "celsius", "kelvin"} and fahrenheit) + ): + conflicts.add("unit") + return frozenset(conflicts) + + +def _source_selector( + utterance: str, + instruction: DomuxInstruction, + registry: EntityRegistry, +) -> DomuxInstruction: + """Erase model-proposed grounding fields that the user's words do not support.""" + + normalized = normalize_text(utterance) + fields = {slot: getattr(instruction, slot) for slot in SLOTS} + + device_is_registered = any( + EntityRegistry._device_matches(entity, instruction.device) + for entity in registry.entities + ) + room_is_registered = instruction.room == "*" or any( + normalize_text(entity.room) == normalize_text(instruction.room) + for entity in registry.entities + ) + floor_is_registered = instruction.floor == "*" or any( + normalize_text(entity.floor) == normalize_text(instruction.floor) + for entity in registry.entities + ) + if not _phrase_in(normalized, instruction.device) or not device_is_registered: + fields["device"] = "*" + if not _phrase_in(normalized, instruction.room) or not room_is_registered: + fields["room"] = "*" + if not _phrase_in(normalized, instruction.floor) or not floor_is_registered: + fields["floor"] = "*" + if not _attribute_supported(utterance, instruction): + fields["attribute"] = "*" + return DomuxInstruction(**fields) + + +def _partial_registry_selector_slots( + utterance: str, + registry: EntityRegistry, +) -> tuple[str, ...]: + """Find incomplete multi-token registry labels in the user's own text. + + Token overlap is useful for recognizing that a selector is present, but it + is not enough to authorize a canonical room, floor, or device. For + example, ``dining light`` must not silently become ``Dining Room``. The + slot remains repairable through clarification instead of being guessed. + """ + + normalized = normalize_text(utterance) + ignored = { + "ac", "air", "conditioner", "conditioning", "device", "room", "floor", + "level", "light", "lamp", "curtain", "blind", "shade", + "auto", "cool", "dry", "fan", "heat", "high", "low", "medium", + *COLOR_RGB, + } + labels: dict[str, set[str]] = { + "device": {entity.device for entity in registry.entities}, + "room": {entity.room for entity in registry.entities}, + "floor": {entity.floor for entity in registry.entities}, + } + full_spans: list[tuple[int, int]] = [] + for value in {value for values in labels.values() for value in values}: + value_norm = normalize_text(value) + full_spans.extend( + (match.start(), match.end()) + for match in re.finditer( + rf"(? tuple[str, ...]: + """Keep recognized-but-unparsed inventory words from being discarded. + + A registered label in ``in Study`` or ``Study light`` is a positive + selector; one in a bounded negative scope is an exclusion. Any remaining + occurrence (for example ``for Reading`` or a room named ``Right`` in + ``right now``) is semantically ambiguous and must force clarification + instead of falling back to a globally unique device. + """ + + labels: tuple[tuple[str, str, tuple[str, ...]], ...] = ( + ("room", "room", tuple(entity.room for entity in registry.entities)), + ("floor", "floor", tuple(entity.floor for entity in registry.entities)), + ("device", "device", tuple( + entity.device + for entity in registry.entities + if normalize_text(entity.device) not in GENERIC_DEVICE_ALIASES + )), + ("alias", "entity", tuple( + alias for entity in registry.entities for alias in entity.aliases + )), + ("entity", "entity", tuple(entity.entity_id for entity in registry.entities)), + ) + unresolved: list[str] = [] + normalized = normalize_text(utterance) + selector_spans = _registry_selector_spans(normalized, registry) + occurrences = tuple( + (start, end, value, category, slot) + for category, slot, values in labels + for start, end, value in _positive_named_matches( + normalized, + values, + selector_spans=selector_spans, + ) + ) + anchored_spans = tuple( + (start, end) + for start, end, value, category, _slot in occurrences + if _selector_match_is_anchored( + normalized, start, end, value, category, registry, + ) + ) + for start, end, value, category, slot in occurrences: + if _selector_match_is_anchored( + normalized, start, end, value, category, registry, + ): + continue + if any( + outer_start <= start and end <= outer_end + and (outer_start, outer_end) != (start, end) + for outer_start, outer_end in anchored_spans + ): + continue + unresolved.append(slot) + return tuple(dict.fromkeys(unresolved)) + + +def _positional_unknown_selector_words( + utterance: str, + registry: EntityRegistry, +) -> frozenset[str]: + """Allow one user-authored noncanonical selector token in a target position. + + The token is still marked unresolved by the caller, so it can lead only to + clarification. This preserves inputs such as ``upstairs AC`` and + ``downstairs temperature`` without letting arbitrary model slot text + expand the executable grammar. + """ + + normalized = normalize_text(utterance) + known_words: set[str] = set() + for entity in registry.entities: + for label in ( + entity.entity_id, entity.room, entity.floor, entity.device, *entity.aliases, + ): + known_words.update(re.findall(r"[a-z0-9]+", normalize_text(label))) + for alias in GENERIC_DEVICE_ALIASES: + known_words.update(re.findall(r"[a-z0-9]+", normalize_text(alias))) + reserved = { + "a", "an", "and", "at", "change", "close", "confirm", "decrease", "do", + "increase", "it", "lower", "make", "move", "not", "open", "other", + "please", "raise", "set", "switch", "that", "the", "this", "to", "turn", + # Operation values, modifiers, and units can naturally precede an + # attribute noun (``Cool mode``, ``40 percent brightness``). They are + # not positional selectors. + "auto", "bright", "brighter", "celsius", "color", "cool", "cooler", "degree", + "degrees", "dimmer", "dry", "fan", "fahrenheit", "halfway", "heat", + "high", "kelvin", "low", "medium", "percent", "warmer", "white", + *{ + word + for color in COLOR_RGB + for word in re.findall(r"[a-z0-9]+", normalize_text(color)) + }, + } + target_nouns = sorted( + { + *(normalize_text(alias) for alias in GENERIC_DEVICE_ALIASES), + "brightness", + "color", + "mode", + "openness", + "position", + "temperature", + }, + key=len, + reverse=True, + ) + unknown: set[str] = set() + for noun in target_nouns: + for match in re.finditer( + rf"(? frozenset[str]: + """Return unknown positional words immediately modifying an operation slot.""" + + normalized = normalize_text(utterance) + unknown = _positional_unknown_selector_words(utterance, registry) + attribute_noun = r"(?:brightness|color|mode|openness|position|temperature)" + return frozenset( + word for word in unknown + if re.search( + rf"(? tuple[str, ...]: + """Detect relational location phrases that do not name registry metadata.""" + + normalized = normalize_text(utterance) + qualifiers = ( + r"(?:my|your|our|their|his|her|this|that|one|other|right|left|current|" + r"same|nearby|any|some|another|a|an)" + ) + unresolved: list[str] = [] + if re.search( + rf"\b(?:in|inside|at|from|for)\s+(?:the\s+)?{qualifiers}\s+room\b", + normalized, + ): + unresolved.append("room") + if re.search( + rf"\b(?:in|inside|at|on|from|for)\s+(?:the\s+)?{qualifiers}\s+" + rf"(?:floor|level|storey|story)\b", + normalized, + ): + unresolved.append("floor") + if re.search(r"\bin\s+the\s+middle\b", normalized): + unresolved.extend(("room", "context")) + return tuple(unresolved) + + +def _missing_required_slots(instruction: DomuxInstruction) -> tuple[str, ...]: + action = normalize_text(instruction.action) + missing: list[str] = [] + if action in {"set", "adjustup", "adjustdown"} and instruction.attribute == "*": + missing.append("attribute") + if action == "set" and instruction.value == "*": + missing.append("value") + if action == "set" and instruction.attribute not in {"*", "color", "mode"} and instruction.unit == "*": + missing.append("unit") + return tuple(missing) + + +def _has_uncertainty_or_conflict(utterance: str) -> bool: + normalized = normalize_text(utterance) + return "—" in utterance or any(_phrase_in(normalized, cue) for cue in ( + "perhaps", "maybe", "confirm", "which", "not sure", "not decided", + "have not decided", "ask me", "do not choose", "do not guess", "or did", + )) or _phrase_in(normalized, "or") or _phrase_in(normalized, "either") or bool( + re.search(r"\b(?:not|except|instead\s+of)\s+(?:the\s+)?[a-z0-9]", normalized) + ) + + +def _has_negative_action_authorization(text: str) -> bool: + """Detect negation scoped over execution, even with intervening pronouns.""" + + normalized = normalize_text(text) + action = ( + rf"(?:{STATE_CHANGE_ACTION_RE}|act|go\s+ahead|do\s+it|touch|confirm|" + r"authorize|approve)" + ) + bridge = r"(?:(?:you|me|us|i|we|to|need|want|let|allow|please|just)\s+){0,8}" + return bool(re.search( + rf"\b(?:do\s+not|don't|dont|never(?:\s+ever)?|rather\s+not)\s+{bridge}{action}\b|" + rf"\b(?:must|may|shall|should|can|could|will|would)\s+not\s+{bridge}{action}\b|" + rf"\bnot\s+(?:to\s+)?{action}\b|" + rf"\b(?:refuse|forbid)\b.{{0,64}}\b{action}\b|" + rf"\b(?:withdraw|revoke|deny)\b.{{0,40}}\b(?:permission|authorization|consent)\b" + rf"(?:.{{0,64}}\b{action}\b)?|" + rf"\bunder\s+no\s+circumstances\b.{{0,64}}\b{action}\b", + normalized, + )) + + +def _has_negated_direction(text: str) -> bool: + """Reject a relative direction appearing under an explicit negative scope.""" + + normalized = normalize_text(text) + direction = ( + r"(?:raise|increase|raising|increasing|brighter|warmer|higher|" + r"lower|decrease|lowering|decreasing|dimmer|cooler)" + ) + bridge = r"(?:(?:make|move|adjust|set|turn)\s+)?(?:(?:it|that|this|the\s+\w+)\s+)?" + qualifier = r"(?:(?:any|even|more|less|at\s+all)\s+){0,3}" + return bool(re.search( + rf"\b(?:do\s+not|don't|dont|never(?:\s+ever)?|no(?:\s+more)?|not|" + rf"anything\s+but|other\s+than|besides|apart\s+from)\s+" + rf"{bridge}{qualifier}{direction}\b|" + rf"\b(?:avoid|without)\s+(?:making\s+(?:it|that|this)\s+)?{direction}\b|" + rf"\bno\s+need\s+(?:to|for)\b.{{0,48}}\b{direction}\b", + normalized, + )) + + +def _generic_withdrawal_pattern(generic_target: str) -> str: + """Build the shared generic-target withdrawal grammar.""" + + selection_action = r"(?:use|select|choose|touch|act\s+on)" + intent_verb = ( + rf"(?:{selection_action}|(?:want|need|have)" + rf"(?:\s+(?:you|me|us|them|him|her))?" + rf"(?:\s+to\s+{selection_action})?)" + ) + negative_lead = ( + r"(?:do\s+not|don't|dont|never(?:\s+ever)?|cannot|can't|cant|won't|" + r"wont|wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|mustn't|" + r"mustnt|(?:must|may|shall|should|can|could|will|would)\s+not)" + ) + speaker = r"(?:(?:i|we|you)\s+)?" + no_relation = rf"(?:for|of|to\s+{selection_action})" + return ( + rf"\b(?:{negative_lead}\s+{intent_verb}|" + rf"{speaker}(?:have|want|need)\s+no" + rf"(?:\s+(?:need|use)\s+{no_relation})?|" + rf"no\s+(?:need|use)\s+{no_relation}|" + rf"{speaker}(?:need|want)\s+not\s+(?:to\s+)?{selection_action}|" + rf"{speaker}(?:would|should|could)\s+rather\s+not\s+{selection_action})" + rf"\s+{generic_target}\b" + ) + + +def _has_generic_withdrawal(text: str) -> bool: + """Detect withdrawal/negative possession of a generic device target.""" + + normalized = normalize_text(text) + generic_target = ( + rf"(?:anything|everything|it|that|this|(?:(?:any|a|an|the|this|that|one|all|every|" + rf"each|either)\s+)?{GENERIC_REFERENCE_NOUN_RE})" + ) + return bool(re.search(_generic_withdrawal_pattern(generic_target), normalized)) + + +def _has_generic_domain_withdrawal( + text: str, + noun_pattern: str, + *, + selector_spans: Sequence[tuple[int, int]] = (), +) -> bool: + """Return whether a withdrawal directly governs one generic domain noun.""" + + normalized = normalize_text(text) + generic_target = ( + rf"(?:(?:any|a|an|the|this|that|one|all|every|each|either)\s+)?" + rf"(?:{noun_pattern})" + ) + pattern = _generic_withdrawal_pattern(generic_target) + for match in re.finditer(pattern, normalized): + suffix = normalized[match.end():] + qualified = re.match( + r"\s+(?:but|that|which|in|on|at|from|called|named|i\s+mean|" + r"we\s+mean|namely)\b|" + r"\s*,\s*(?:namely|called|named|i\s+mean|we\s+mean)\b|" + r"\s*[—(]\s*[a-z0-9]", + suffix, + ) + if qualified and any( + selector_start >= match.end() + and not re.search( + r"[.!?;]", + normalized[match.end():selector_start], + ) + for selector_start, _selector_end in selector_spans + ): + continue + return True + return False + + +def _has_negative_selector_correction( + utterance: str, + registry: EntityRegistry, +) -> bool: + """Detect a named-selector correction whose bridge is not safely positive.""" + + normalized = normalize_text(utterance) + spans = tuple(_registry_selector_spans(normalized, registry)) + _roles, unsafe = _postposed_no_correction_analysis(normalized, spans) + return unsafe + + +def _has_negative_or_cancelled_intent( + utterance: str, + registry: EntityRegistry | None = None, +) -> bool: + normalized = normalize_text(utterance) + ambiguous_negative_correction = ( + registry is not None + and _has_negative_selector_correction(normalized, registry) + ) + negative_imperative = ( + _has_negative_action_authorization(normalized) + or _has_negated_direction(normalized) + or bool(re.search( + r"\b(?:do\s+not|don't|dont|never(?:\s+ever)?)\s+" + rf"{STATE_CHANGE_ACTION_RE}\b", + normalized, + )) + ) + withdrawn_request = bool(re.search( + r"\b(?:i\s+)?(?:do\s+not|don't|dont)\s+(?:want|need)\b.{0,48}\b" + r"(?:turn|switch|open|close|set|change|make|move|adjust|raise|lower|increase|" + r"decrease|brighter|dimmer|warmer|cooler|higher)\b|" + r"\bno\s+need\s+to\s+(?:turn|switch|open|close|set|change|make|move|adjust|raise|lower|" + r"increase|decrease)\b|" + r"\b(?:refrain\s+from|avoid)\s+(?:turning|switching|opening|closing|setting|" + r"changing|making|moving|adjusting|raising|lowering)\b|" + r"\bwithout\s+(?:turning|switching|opening|closing|setting|changing|making|" + r"moving|adjusting|raising|lowering)\b|" + r"\bjust\s+kidding\b|" + r"\b(?:scratch|nix|disregard|ignore)\s+(?:it|that|this)\b|" + r"\b(?:i\s+)?changed?\s+my\s+mind\b|\bhold\s+on\b|" + r"\b(?:actually\s+)?(?:never\s+mind|cancel(?:\s+(?:it|that|this|the\s+request))?)\b", + normalized, + )) + negative_preference = bool(re.search( + r"\b(?:(?:i|we)\s+)?(?:do\s+not|don't|dont|cannot|can't|cant|" + r"won't|wont|wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|" + r"mustn't|mustnt|(?:can|could|would|should|will|must|may)\s+not)\s+" + r"(?:want|need|have)\b", + normalized, + )) + terminal_cancel = bool(re.fullmatch( + r"(?:cancel|never mind|do nothing|stop|wait|forget it|" + r"(?:actually\s+)?(?:do not|don't|dont))[.!]?", + normalized, + )) + trailing_cancel = bool(re.search( + r"(?:^|[,;:—-]\s*)(?:(?:(?:please|i\s+mean)\s+)?no(?:\s+please)?|" + r"i\s+want\s+no|not now|forget it|wait|stop|don't bother|dont bother|" + r"(?:i\s+)?(?:do not|don't|dont)\s+(?:want|need)\s+(?:it|that|this)|" + r"(?:actually\s+|please\s+)?(?:do not|don't|dont))[.!]?\s*$", + normalized, + )) + suspensive_wait = bool( + re.search(r"\bwait\b", normalized) + and not re.search( + r"\bwait\b.{0,64}\b(?:actually|perhaps|rather|instead|confirm)\b", + normalized, + ) + ) + return ( + negative_imperative + or _has_generic_withdrawal(normalized) + or withdrawn_request + or negative_preference + or terminal_cancel + or trailing_cancel + or suspensive_wait + or ambiguous_negative_correction + ) + + +def _has_unsupported_condition_or_time(utterance: str) -> bool: + """Reject modifiers this immediate, single-action executor cannot honor.""" + + normalized = normalize_text(utterance) + # "Confirm ... before acting" describes this safety protocol itself, not + # a delayed execution condition. Keep it available to clarification. + normalized = re.sub( + r"\bbefore\s+(?:acting|executing|proceeding|you\s+act|you\s+execute)\b", + " ", + normalized, + ) + number_word = ( + r"(?:an?|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|" + r"few|couple(?:\s+of)?|\d+(?:\.\d+)?)" + ) + return bool(re.search( + r"^\s*wait\b|" + r"\b(?:if|unless|when|whenever|once|while|after|before|until|provided|providing|assuming)\b|" + r"\b(?:as\s+long\s+as|so\s+long\s+as|in\s+case|depending\s+on|subject\s+to)\b|" + r"\bas\s+soon\s+as\b|" + r"\b(?:during|upon\s+arrival|at\s+dusk|at\s+dawn|as\s+needed|momentarily)\b|" + r"\b(?:tomorrow|tonight|later|today|noon|midnight|sunrise|sunset|" + r"morning|afternoon|evening)\b|" + r"\b(?:for\s+now|from\s+now\s+on|now\s+and\s+then)\b|" + r"\b(?:this|next)\s+(?:morning|afternoon|evening|night|week|month|year)\b|" + r"\bon\s+(?:monday|tuesday|wednesday|thursday|friday|saturday|sunday|" + r"weekday|weekend)\b|" + r"\b(?:every|each)\s+(?:morning|afternoon|evening|night|day|week|month|" + r"weekday|weekend)\b|" + r"\b(?:daily|nightly|weekly|monthly|briefly|temporarily|momentarily|schedule|scheduled)\b|" + rf"\b(?:in|for)\s+{number_word}\s+(?:seconds?|minutes?|hours?|days?|weeks?)\b|" + r"\b(?:in|for)\s+(?:half|a\s+half|quarter|a\s+quarter)\s+" + r"(?:of\s+)?(?:an?\s+)?(?:hour|day)\b|" + r"\bat\s+(?:noon|midnight|sunrise|sunset)\b|" + r"\bat\s+\d{1,2}:\d{2}(?:\s*(?:am|pm))?\b|" + r"\bat\s+\d{1,2}\s*(?:am|pm)\b|" + r"\bat\s+(?:one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve)" + r"(?:\s+o'?clock)?\b", + normalized, + )) + + +def _has_informational_request(utterance: str) -> bool: + """Distinguish questions about an action from an authorization to execute it.""" + + normalized = re.sub(r"^[^a-z0-9]+", "", normalize_text(utterance)) + return bool(re.match( + r"^(?:(?:should|would|could|can|may)\s+i\b|" + r"(?:i|we|you)\s+(?:do|did)\s+(?:please\s+)?" + r"(?:turn|switch|open|close|set|change|make|move|adjust|raise|lower|increase|decrease)\b|" + r"(?:do|did|have|has)\s+(?:you|we)\s+(?:want|need|have|mean|intend|plan|" + r"expect|already|ever|turn|turned|switch|switched|open|opened|close|closed|" + r"set|change|changed|make|made|adjust|adjusted)\b|" + r"(?:should|would|could|can|may|will)\s+we\b|" + r"(?:should|would|could|can|may|will)\s+you\s+" + r"(?:want|need|have|mean|intend|plan|expect)\b|" + r"do\s+i\s+(?:turn|switch|open|close|set|change|make|move|adjust)\b|" + r"(?:would|could)\s+it\s+be\s+(?:safe|okay|ok|wise|advisable)\b|" + r"is\s+it\s+(?:safe|okay|ok|wise|advisable)\b|" + r"do\s+you\s+(?:recommend|suggest|think)\b|" + r"do\s+i\s+(?:need|have)\s+to\b|" + r"are\s+you\s+going\s+to\b|" + r"why\s+(?:should|would|could|can|do)\s+i\b|" + r"(?:please\s+)?explain\s+why\b|" + r"what\s+if\b|" + r"what\s+(?:happens|would\s+happen)\b|" + r"(?:tell|show|explain)\s+me\s+(?:how|whether|what)\b|" + r"how\s+(?:do|can|should|would)\s+i\b)", + normalized, + )) + + +def _has_supported_request_grammar( + utterance: str, + registry: EntityRegistry, + source_instructions: Sequence[DomuxInstruction] = (), +) -> bool: + """Accept only the documented immediate, single-operation language. + + Negative and temporal detectors provide useful reason codes, but they can + never enumerate every English paraphrase. This bounded vocabulary is the + fail-closed backstop: registry labels, numeric values, and the small Domux + operation language are accepted; an unconsumed word requires a new request + rather than being silently discarded from an executable plan. + """ + + selector_words: set[str] = set() + for entity in registry.entities: + for label in ( + entity.entity_id, entity.room, entity.floor, entity.device, *entity.aliases, + ): + selector_words.update(re.findall(r"[a-z0-9]+", normalize_text(label))) + for alias in GENERIC_DEVICE_ALIASES: + selector_words.update(re.findall(r"[a-z0-9]+", normalize_text(alias))) + # Permit an unknown single-token mode only when the user's own syntax binds + # it as ``to mode`` and the model puts that exact token in value. + # Attribute/unit text never extends the grammar: otherwise an untrusted raw + # output could launder an ignored condition into a replaceable source slot. + normalized_utterance = normalize_text(utterance) + source_mode_words = { + value + for instruction in source_instructions + if normalize_text(instruction.attribute) == "mode" + if re.fullmatch(r"[a-z0-9]+", (value := normalize_text(instruction.value))) + if re.search( + rf"\bto\s+{re.escape(value)}\s+mode\b", + normalized_utterance, + ) + } + positional_selector_words = _positional_unknown_selector_words(utterance, registry) + + request_words = { + # Immediate command and clarification-protocol framing. + "adjust", "ask", "acting", "avoid", "before", "change", "check", "choose", + "close", "confirm", "decided", "decrease", "did", "do", "execute", + "guess", "increase", "lower", "make", "mean", "move", "need", "open", + "proceed", "raise", "set", "switch", "then", "turn", "value", "wait", "want", + # Operation attributes, values, and units. + "auto", "blue", "bright", "brighter", "brightness", "celsius", "color", + "cool", "cooler", "degree", "degrees", "dimmer", "dry", "fan", "green", + "fahrenheit", "halfway", "heat", "high", "kelvin", "level", "low", "medium", "mode", + "off", "on", "openness", "percent", "position", "red", "speed", + "temperature", "warm", "warmer", "white", "wind", "yellow", + # Selector and polite-command glue. These words carry no operation by + # themselves; the source-to-slot checks still require positive evidence. + "about", "ac", "air", "am", "and", "around", "at", + "any", "anything", "apart", "besides", "but", "by", "can", "cannot", + "conditioner", "could", "curtain", "device", + "floor", "for", "from", "have", "i", "in", "instead", "it", "its", "just", + "light", "me", "my", "no", "not", "now", "of", + "may", "middle", "must", "never", "one", "or", "other", "perhaps", + "please", "right", "room", "shall", "should", "side", + "something", "than", + "sure", "talked", "that", "the", "this", "to", + "use", "we", "which", "will", "would", "you", + *{word for color in COLOR_RGB for word in normalize_text(color).split()}, + } + allowed = selector_words | request_words | source_mode_words | positional_selector_words + normalized = normalized_utterance.replace("don't", "do not").replace("dont", "do not") + words = re.findall(r"[a-z0-9]+", normalized) + return all( + word in allowed or word.isdigit() or re.fullmatch(r"\d+(?:\d+)?k", word) + for word in words + ) + + +def _has_direct_generic_exclusion( + text: str, + noun_pattern: str, + *, + selector_spans: Sequence[tuple[int, int]] = (), +) -> bool: + """Return whether an exclusion directly governs a generic noun phrase.""" + + normalized = normalize_text(text) + direct_marker = ( + r"(?:not(?:\s+in)?|no|except(?:\s+for)?|besides|other\s+than|" + r"anything\s+but|everything\s+but|all\s+but|apart\s+from|" + r"instead\s+of|avoid|without)" + r"(?:\s+(?:using|selecting|choosing|touching))?" + ) + negative_command = ( + r"(?:(?:do\s+not|don't|dont|never|cannot|can't|cant|won't|wont|" + r"wouldn't|wouldnt|shouldn't|shouldnt|couldn't|couldnt|mustn't|mustnt|" + r"(?:must|may|shall|should|can|could|will|would)\s+not)" + r"(?:\s+(?:want|need)(?:\s+(?:you|me|us|them|him|her))?\s+to)?\s+" + r"(?:use|select|choose|touch|act\s+on))" + ) + pattern = ( + rf"\b(?:{direct_marker}|{negative_command})\s+" + rf"(?:(?:a|an|the|this|that|any|all|every|each|either)\s+)?" + rf"(?:{noun_pattern})\b" + ) + for match in re.finditer(pattern, normalized): + suffix = normalized[match.end():] + qualified = re.match( + r"\s+(?:but|that|which|in|on|at|from|called|named|i\s+mean|" + r"we\s+mean|namely)\b|" + r"\s*,\s*(?:namely|called|named|i\s+mean|we\s+mean)\b|" + r"\s*[—(]\s*[a-z0-9]", + suffix, + ) + if qualified and any( + selector_start >= match.end() + and not re.search( + r"[.!?;]", + normalized[match.end():selector_start], + ) + for selector_start, _selector_end in selector_spans + ): + continue + return True + return False + + +def _has_unresolved_generic_exclusion( + text: str, + registry: EntityRegistry | None = None, +) -> bool: + """Reject generic exclusions that do not identify a stable entity ID.""" + + normalized = normalize_text(text) + selector_spans = ( + _registry_selector_spans(normalized, registry) + if registry is not None + else () + ) + return _has_direct_generic_exclusion( + normalized, + rf"(?:anything|everything|{GENERIC_REFERENCE_NOUN_RE})", + selector_spans=selector_spans, + ) or bool(re.search( + r"\bnot\s+(?:this|that)\b|" + rf"\b(?:leave|keep)\s+(?:(?:this|that|the)\s+)?{GENERIC_REFERENCE_NOUN_RE}\s+" + r"(?:unchanged|as\s+is)\b|" + rf"\b(?:use|select|choose|mean)\s+{OTHER_REFERENCE_RE}", + normalized, + )) + + +def _excluded_generic_domains( + text: str, + registry: EntityRegistry, +) -> frozenset[str]: + labels = { + domain: tuple( + alias for alias, alias_domain in GENERIC_DEVICE_ALIASES.items() + if alias_domain == domain + ) + for domain in SUPPORTED_DOMAINS + } + selector_spans = _registry_selector_spans(text, registry) + return frozenset( + domain for domain, names in labels.items() + if any( + _has_direct_generic_exclusion( + text, + re.escape(normalize_text(name)), + selector_spans=selector_spans, + ) + or _has_generic_domain_withdrawal( + text, + re.escape(normalize_text(name)), + selector_spans=selector_spans, + ) + for name in names + ) + ) + + +def _has_universal_generic_exclusion(text: str, registry: EntityRegistry) -> bool: + """Return whether a withdrawal excludes every supported device domain.""" + + normalized = normalize_text(text) + selector_spans = _registry_selector_spans(normalized, registry) + generic_target = ( + r"(?:anything|everything|(?:(?:any|a|an|the|this|that|one|all|every|" + r"each|either)\s+)?devices?)" + ) + return bool(re.search( + _generic_withdrawal_pattern(generic_target), + normalized, + )) or _has_direct_generic_exclusion( + normalized, + r"(?:anything|everything|devices?)", + selector_spans=selector_spans, + ) + + +def _explicit_operational_requirements(utterance: str) -> frozenset[str]: + """Return operational slots explicitly present in the user's own words. + + This is a fail-closed preservation check, not a replacement semantic + parser. Its purpose is to stop a model proposal from silently dropping a + number, unit, or attribute that would materially change the operation. + """ + + normalized = normalize_text(utterance) + requirements: set[str] = set() + has_operational_number = bool(re.search( + r"\b(?:to|by|at|around|about)\s+-?\d+(?:\.\d+)?\b|" + r"\b(?:brightness|position|openness|temperature|degrees?|celsius|kelvin)\b" + r"(?:\s+[a-z]+){0,3}\s+-?\d+(?:\.\d+)?\b", + normalized, + )) or _phrase_in(normalized, "halfway") + percent = "%" in utterance or _phrase_in(normalized, "percent") or _phrase_in(normalized, "halfway") + celsius = any(_phrase_in(normalized, term) for term in ("celsius", "degree", "degrees")) + kelvin = _phrase_in(normalized, "kelvin") or bool(re.search(r"\d+(?:\.\d+)?\s*k\b", normalized)) + if has_operational_number: + requirements.add("value") + if percent or celsius or kelvin: + requirements.update(("attribute", "unit", "value")) + if any(_phrase_in(normalized, term) for term in ( + "brightness", "bright", "dimmer", "position", "openness", "temperature", + "color temperature", "wind speed", "fan speed", + )): + requirements.add("attribute") + named_color = any(_phrase_in(normalized, color) for color in COLOR_RGB) + light_cue = any(_phrase_in(normalized, term) for term in ("light", "lamp")) + color_operation = any(_phrase_in(normalized, term) for term in ("make", "color")) or bool( + re.search(r"\bto\b(?:\s+[a-z0-9]+){0,4}\s+(?:" + "|".join( + re.escape(normalize_text(color)) for color in COLOR_RGB + ) + r")\b", normalized) + ) + if named_color and light_cue and color_operation: + requirements.update(("attribute", "value")) + mode_cue = _phrase_in(normalized, "mode") + named_mode = any(_phrase_in(normalized, mode) for mode in ("cool", "heat", "dry", "fan only", "auto")) + climate_cue = any(_phrase_in(normalized, term) for term in ("ac", "air conditioner", "air conditioning")) + mode_operation = mode_cue or bool(re.search( + r"\b(?:to|use)\b(?:\s+[a-z0-9]+){0,4}\s+(?:cool|heat|dry|fan\s+only|auto)\b", + normalized, + )) + if mode_cue or (named_mode and climate_cue and mode_operation): + requirements.add("attribute") + if named_mode and climate_cue and mode_operation: + requirements.add("value") + return frozenset(requirements) + + +def _label_is_excluded( + text: str, + label: str, + *, + selector_spans: Sequence[tuple[int, int]] = (), +) -> bool: + normalized_label = normalize_text(label) + if not normalized_label: + return False + normalized = normalize_text(text) + matches = _distinct_named_matches(normalized, (normalized_label,)) + known_spans = tuple(selector_spans) or tuple( + (start, end) for start, end, _value in matches + ) + correction_occurrences = tuple( + (start, end, role) + for start, end, _value in matches + if (role := _postposed_no_correction_role( + normalized, + start, + end, + known_spans, + )) is not None + ) + if correction_occurrences: + latest_start, _latest_end, latest_role = max( + correction_occurrences, + key=lambda item: (item[1], item[0]), + ) + explicitly_scoped = latest_role == "withdrawn" or any( + start > latest_start + and _selector_span_is_negative_base( + normalized, + start, + end, + known_spans, + ) + for start, end, _value in matches + ) + else: + explicitly_scoped = any( + _selector_span_is_negative( + normalized, + start, + end, + known_spans, + ) + for start, end, _value in matches + ) + return explicitly_scoped or bool(re.search( + rf"\b(?:not(?:\s+in)?|except|besides|other\s+than|anything\s+but|" + rf"apart\s+from|instead\s+of)\s+(?:the\s+)?{re.escape(normalized_label)}\b|" + rf"\bleave\s+(?:the\s+)?{re.escape(normalized_label)}\b.*\bunchanged\b", + normalized, + )) + + +def _positive_target_domains(text: str, registry: EntityRegistry) -> frozenset[str]: + """Return domains named by positive domain/entity/alias/device selectors. + + Room and floor labels deliberately do not count: they can be shared across + domains and therefore cannot by themselves reverse a domain-wide exclusion. + """ + + normalized = normalize_text(text) + selector_spans = _registry_selector_spans(normalized, registry) + state_command = re.match(rf"{STATE_COMMAND_RE}\b", normalized) + spatial_spans = tuple( + (start, end) + for values in ( + tuple(entity.room for entity in registry.entities), + tuple(entity.floor for entity in registry.entities), + ) + for start, end, _value in _distinct_named_matches(normalized, values) + ) + + target_modifiers = tuple({ + normalize_text(value) + for entity in registry.entities + for value in (entity.room, entity.floor) + if normalize_text(value) + }) + target_fillers = { + "a", "all", "an", "any", "each", "either", "every", "my", "one", + "our", "that", "the", "this", "your", + } + target_nouns = { + "ac", "acs", "air", "blind", "blinds", "conditioner", "conditioners", + "conditioning", "cover", "covers", "curtain", + "curtains", "device", "devices", "lamp", "lamps", "light", "lighting", + "lights", "shade", "shades", + } + operation_bridge_words = { + "about", "around", "at", "auto", "blue", "bright", "brighter", + "brightness", "by", "celsius", "close", "closed", "color", "cool", + "cooler", "decrease", "degree", "degrees", "dimmer", "dry", "fan", + "fahrenheit", "for", "from", "green", "halfway", "heat", "high", "in", + "increase", "kelvin", "level", "low", "lower", "medium", "mode", "of", + "on", "open", "openness", "percent", "position", "raise", "red", "set", + "speed", "temperature", "to", "value", "warm", "warmer", "white", "wind", + "yellow", + *{word for color in COLOR_RGB for word in normalize_text(color).split()}, + } + operation_cues = { + "brightness", "celsius", "color", "degree", "degrees", "fahrenheit", + "halfway", "kelvin", "level", "mode", "openness", "percent", "position", + "speed", "temperature", "value", "wind", + *{word for color in COLOR_RGB for word in normalize_text(color).split()}, + } + + def selector_is_target_anchored(start: int) -> bool: + if state_command is None or start < state_command.end(): + return False + actions = tuple(re.finditer(STATE_CHANGE_ACTION_RE, normalized[:start])) + bridge_start = actions[-1].end() if actions else state_command.end() + bridge = normalized[bridge_start:start] + for modifier in sorted(target_modifiers, key=len, reverse=True): + bridge = re.sub( + rf"(? frozenset[str]: + """Resolve the bounded noun phrase containing one shared selector label.""" + + boundary_pattern = ( + r"[!?;,:—]|\.(?![a-z0-9])|" + r"\b(?:and\s+then|then|but|rather|and|or)\b" + ) + boundaries = tuple( + match + for match in re.finditer(boundary_pattern, text) + if not any( + selector_start < match.start() < selector_end + or selector_start < match.end() < selector_end + for selector_start, selector_end in selector_spans + ) + ) + phrase_start = max( + (match.end() for match in boundaries if match.end() <= start), + default=0, + ) + phrase_end = min( + (match.start() for match in boundaries if match.start() >= end), + default=len(text), + ) + phrase = text[phrase_start:phrase_end] + relative_start = start - phrase_start + predicate = ( + rf"\b(?:{SELECTOR_ACTION_RE}|{EXECUTION_CONTROL_ACTION_RE}|want|need|have|" + r"avoid|without|except|besides|not|no|using|selecting|choosing|touching)\b" + ) + predicates = tuple(re.finditer(predicate, phrase[:relative_start])) + if predicates: + phrase = phrase[predicates[-1].end():] + selector = _explicit_selector_match( + registry, + phrase, + allow_bare_meaningful=True, + ) + if not selector.present or selector.conflicting_slots or not selector.candidates: + return frozenset() + return frozenset(entity.entity_id for entity in selector.candidates) + + +def _negated_entity_ids(registry: EntityRegistry, utterance: str) -> tuple[str, ...]: + normalized = normalize_text(utterance) + selector_spans = _registry_selector_spans(normalized, registry) + # A later state command reverses a domain-wide exclusion only when that + # command itself positively names the domain or a domain-specific stable + # selector. A bare action, ``a device``, room, or floor cannot silently + # make an excluded domain eligible for clarification again. + authorization_clause = _final_authorization_clause(normalized, registry) + excluded_domains = set(_excluded_generic_domains(normalized, registry)) + if _has_universal_generic_exclusion(normalized, registry): + excluded_domains.update(SUPPORTED_DOMAINS) + if ( + authorization_clause != normalized + and re.match(rf"{STATE_COMMAND_RE}\b", authorization_clause) + ): + excluded_domains.difference_update( + _positive_target_domains(authorization_clause, registry) + ) + + label_owners: dict[str, set[str]] = {} + for entity in registry.entities: + labels = (entity.entity_id, entity.room, entity.floor, entity.device, *entity.aliases) + for label in labels: + normalized_label = normalize_text(label) + if normalized_label: + label_owners.setdefault(normalized_label, set()).add(entity.entity_id) + + def label_excludes(entity_id: str, label: str) -> bool: + normalized_label = normalize_text(label) + if not _label_is_excluded( + normalized, + normalized_label, + selector_spans=selector_spans, + ): + return False + if len(label_owners.get(normalized_label, ())) <= 1: + return True + scoped_matches = tuple( + (start, end) + for start, end, _value in _distinct_named_matches( + normalized, + (normalized_label,), + ) + if _selector_span_is_negative( + normalized, + start, + end, + selector_spans, + ) + ) + if not scoped_matches: + # Preserve the previous fail-closed behavior for exclusion forms + # recognized by the fallback grammar (for example ``leave X + # unchanged``) but not by the general negative-scope parser. + return True + for start, end in scoped_matches: + phrase_ids = _selector_phrase_candidate_ids( + normalized, + start, + end, + registry, + selector_spans, + ) + if not phrase_ids or entity_id in phrase_ids: + return True + return False + + excluded: list[str] = [] + for entity in registry.entities: + labels = ( + entity.entity_id, + entity.room, + entity.floor, + entity.device, + *entity.aliases, + ) + if entity.domain in excluded_domains or any( + label_excludes(entity.entity_id, label) + for label in labels + if normalize_text(label) not in GENERIC_DEVICE_ALIASES + ): + excluded.append(entity.entity_id) + return tuple(sorted(set(excluded))) + + +def _operational_text(text: str, entities: Sequence[EntitySpec]) -> str: + """Remove registry selector spans before interpreting values or attributes.""" + + result = normalize_text(text) + labels: dict[str, bool] = {} + for entity in entities: + for value in (entity.room, entity.floor): + normalized_value = normalize_text(value) + if normalized_value: + labels.setdefault(normalized_value, True) + values = [*entity.aliases] + if normalize_text(entity.device) not in GENERIC_DEVICE_ALIASES: + values.append(entity.device) + for value in values: + normalized_value = normalize_text(value) + if normalized_value: + labels[normalized_value] = False + for label in sorted(labels, key=len, reverse=True): + spatial_prefix = ( + r"(?:(? str: + """Return only operation evidence that remains authorized after a restart. + + A pure selector correction (``do not use Living, I mean Study``) keeps the + operation stated before that correction. A new state-command clause after + a withdrawal replaces the earlier operation instead: any action, attribute, + value, or unit must then be present in that clause or in the clarification + answer. This prevents an incomplete restart such as ``adjust Study`` from + borrowing a withdrawn ``turn off`` action from an earlier clause. + """ + + normalized = normalize_text(text) + authorization_clause = _final_authorization_clause(normalized, registry) + if ( + authorization_clause != normalized + and re.match(rf"{STATE_COMMAND_RE}\b", authorization_clause) + ): + return _operational_text(authorization_clause, entities) + return _operational_text(normalized, entities) + + +def _clarification_operational_text(answer: str, chosen: EntitySpec) -> str: + """Remove candidate-selection evidence before interpreting operation slots.""" + + result = normalize_text(answer) + generic_device_labels = tuple( + alias for alias, domain in GENERIC_DEVICE_ALIASES.items() + if domain == chosen.domain + ) + labels: dict[str, bool] = {} + for value in (chosen.room, chosen.floor): + normalized_value = normalize_text(value) + if normalized_value: + labels.setdefault(normalized_value, True) + for value in ( + chosen.entity_id, + chosen.device, + *chosen.aliases, + *generic_device_labels, + ): + normalized_value = normalize_text(value) + if normalized_value: + labels[normalized_value] = False + for label in sorted(labels, key=len, reverse=True): + spatial_prefix = ( + r"(?:(? frozenset[str]: + operation_only = _clarification_operational_text(answer, chosen) + requirements = set(_explicit_operational_requirements(operation_only)) + if chosen.domain == "light" and any(_phrase_in(operation_only, color) for color in COLOR_RGB): + requirements.update(("attribute", "value")) + if chosen.domain == "climate" and any( + _phrase_in(operation_only, mode) for mode in ("cool", "heat", "dry", "fan only", "auto") + ): + requirements.update(("attribute", "value")) + return frozenset(requirements) + + +def _clarification_has_positive_authorization( + answer: str, + chosen: EntitySpec, + candidates: Sequence[EntitySpec] = (), +) -> bool: + """Recognize a bounded fail-closed clarification-answer grammar. + + Candidate metadata and a small positive operation vocabulary are allowed; + any unparsed residual word rejects the answer. Merely finding an action + token somewhere in arbitrary prose is never sufficient authorization. + """ + + answer_normalized = normalize_text(answer) + if _has_negative_action_authorization(answer_normalized): + return False + if answer_normalized.isdigit(): + return True + operation_only = _clarification_operational_text(answer, chosen) + ordered_words = re.findall(r"[a-z0-9]+", operation_only) + words = set(ordered_words) + selector_fillers = { + "the", "one", "option", "number", "device", "please", "room", "floor", + "in", "at", "on", "located", "by", "other", "i", "mean", "and", "for", + "to", "use", "not", "leave", "unchanged", + } + if not words or words.issubset(selector_fillers): + return True + selector_words: set[str] = set() + for entity in (*candidates, chosen): + for label in ( + entity.entity_id, entity.room, entity.floor, entity.device, *entity.aliases, + ): + selector_words.update(re.findall(r"[a-z0-9]+", normalize_text(label))) + for alias in GENERIC_DEVICE_ALIASES: + selector_words.update(re.findall(r"[a-z0-9]+", normalize_text(alias))) + operation_words = { + "yes", "confirm", "confirmed", "proceed", "go", "ahead", "do", "execute", + "it", "that", "this", "turn", "switch", "open", "close", "shut", "start", + "set", "change", "make", "move", "adjust", "raise", "lower", "increase", + "decrease", "brighter", "dimmer", "warmer", "cooler", "temperature", + "brightness", "position", "openness", "color", "mode", "fan", "wind", + "speed", "celsius", "kelvin", "percent", "degree", "degrees", "halfway", + "instead", "its", "right", "now", "off", "low", "medium", "high", "level", + "cool", "heat", "dry", "auto", "only", "from", "into", "target", "new", + *{word for color in COLOR_RGB for word in normalize_text(color).split()}, + } + allowed = selector_fillers | selector_words | operation_words + if any(word not in allowed and not word.isdigit() for word in ordered_words): + return False + affirmative = bool(re.fullmatch( + r"(?:please\s+)?(?:yes(?:\s+please)?|confirm(?:ed)?|proceed|go\s+ahead|" + r"do\s+it(?:\s+(?:right\s+)?now)?|execute\s+it)[.!]?", + operation_only.strip(), + )) + identifies_candidate = any( + _phrase_in(answer_normalized, label) + for label in (chosen.entity_id, chosen.room, chosen.floor, chosen.device, *chosen.aliases) + ) + requirements = _answer_operational_requirements(answer, chosen) + directions = _directional_actions(operation_only) + return affirmative or bool(requirements) or bool(directions) or identifies_candidate + + +def _mentioned_entities( + registry: EntityRegistry, + utterance: str, + source_instructions: Sequence[DomuxInstruction], +) -> tuple[EntitySpec, ...]: + normalized = normalize_text(utterance) + hinted_domains = { + domain for instruction in source_instructions + if (domain := EntityRegistry._domain_hint(instruction)) is not None + } + mentioned: list[EntitySpec] = [] + for entity in registry.entities: + if hinted_domains and entity.domain not in hinted_domains: + continue + discriminators = (entity.room, *entity.aliases) + if any(_phrase_in(normalized, value) for value in discriminators): + mentioned.append(entity) + return tuple(mentioned) + + +@dataclass(frozen=True) +class _ExplicitSelectorMatch: + present: bool + discriminating: bool + candidates: tuple[EntitySpec, ...] + slots: tuple[str, ...] + conflicting_slots: tuple[str, ...] + + +def _explicit_selector_match( + registry: EntityRegistry, + utterance: str, + *, + allow_bare_meaningful: bool = False, +) -> _ExplicitSelectorMatch: + """Bind user-authored inventory selectors independently of model output. + + Values are ORed within a slot (``Study or Living Room``) and ANDed across + slots (``Kitchen light``). A present selector with zero matches remains an + explicit contradiction; callers must never fall back to model/global + candidates in that case. + """ + + normalized = normalize_text(utterance) + labels: dict[str, tuple[str, ...]] = { + "domain": tuple(GENERIC_DEVICE_ALIASES), + "room": tuple(entity.room for entity in registry.entities), + "floor": tuple(entity.floor for entity in registry.entities), + "device": tuple( + entity.device + for entity in registry.entities + if normalize_text(entity.device) not in GENERIC_DEVICE_ALIASES + ), + "alias": tuple(alias for entity in registry.entities for alias in entity.aliases), + "entity": tuple(entity.entity_id for entity in registry.entities), + } + selector_spans = _registry_selector_spans(normalized, registry) + matches = { + category: tuple( + match + for match in _positive_named_matches( + normalized, + values, + selector_spans=selector_spans, + ) + if _selector_match_is_anchored( + normalized, + match[0], + match[1], + match[2], + category, + registry, + ) + or (allow_bare_meaningful and not _label_is_grammar_collision(match[2])) + ) + for category, values in labels.items() + } + # A generic device word can be part of a registered spatial label (for + # example ``AC Room`` or ``Light Room``). Once the complete room/floor + # label is anchored, that contained token is spatial metadata, not a + # second device-domain selector. A separate target token outside the + # spatial span remains available and is still intersected below. + spatial_spans = tuple( + (start, end) + for category in ("room", "floor") + for start, end, _value in matches[category] + ) + matches["domain"] = tuple( + match + for match in matches["domain"] + if not any( + spatial_start <= match[0] and match[1] <= spatial_end + for spatial_start, spatial_end in spatial_spans + ) + ) + longer_specific_spans = tuple( + (start, end) + for category in ("device", "alias", "entity") + for start, end, _value in matches[category] + ) + matches["domain"] = tuple( + match + for match in matches["domain"] + if not any( + start <= match[0] and match[1] <= end and (start, end) != match[:2] + for start, end in longer_specific_spans + ) + ) + slot_for = { + "domain": "device", + "device": "device", + "room": "room", + "floor": "floor", + "alias": "entity", + "entity": "entity", + } + slots = list(dict.fromkeys( + slot_for[category] + for category, category_matches in matches.items() + if category_matches + )) + present = bool(slots) + if not present: + return _ExplicitSelectorMatch(False, False, (), (), ()) + discriminating = any(matches[category] for category in matches if category != "domain") + + values = { + category: {value for _start, _end, value in category_matches} + for category, category_matches in matches.items() + } + + conflicting_slots: list[str] = [] + mentioned_domains = { + GENERIC_DEVICE_ALIASES[value] for value in values["domain"] + } + if len(mentioned_domains) > 1 or len(values["device"]) > 1: + conflicting_slots.append("device") + if len(values["room"]) > 1: + conflicting_slots.append("room") + if len(values["floor"]) > 1: + conflicting_slots.append("floor") + if len(values["alias"]) > 1 or len(values["entity"]) > 1: + conflicting_slots.append("entity") + + def matching_ids(category: str, value: str) -> set[str]: + if category == "domain": + domain = GENERIC_DEVICE_ALIASES[value] + return {entity.entity_id for entity in registry.entities if entity.domain == domain} + if category == "room": + return { + entity.entity_id for entity in registry.entities + if normalize_text(entity.room) == value + } + if category == "floor": + return { + entity.entity_id for entity in registry.entities + if normalize_text(entity.floor) == value + } + if category == "device": + return { + entity.entity_id for entity in registry.entities + if normalize_text(entity.device) == value + } + if category == "alias": + return { + entity.entity_id for entity in registry.entities + if any(normalize_text(alias) == value for alias in entity.aliases) + } + return { + entity.entity_id for entity in registry.entities + if normalize_text(entity.entity_id) == value + } + + # Overlapping text spans can have several inventory meanings (for example + # a room ``Study`` and a device ``Study Light``). A connected overlap + # component is ORed; treating its interpretations as independent AND + # constraints could silently pick one target from an ambiguous phrase. + entries: list[tuple[int, int, str, str]] = [] + for category, category_matches in matches.items(): + for start, end, value in category_matches: + entries.append((start, end, category, value)) + components: list[list[tuple[int, int, str, str]]] = [] + remaining = set(entries) + while remaining: + component = [remaining.pop()] + changed = True + while changed: + changed = False + component_start = min(item[0] for item in component) + component_end = max(item[1] for item in component) + overlapping = { + item for item in remaining + if item[0] < component_end and component_start < item[1] + } + if overlapping: + component.extend(overlapping) + remaining.difference_update(overlapping) + changed = True + components.append(component) + ambiguous_entries = { + entry + for component in components + if len({entry[2] for entry in component}) > 1 + for entry in component + } + constraints: list[set[str]] = [] + for component in components: + if not any(entry in ambiguous_entries for entry in component): + continue + ids: set[str] = set() + for _start, _end, category, value in component: + ids.update(matching_ids(category, value)) + constraints.append(ids) + for category, category_matches in matches.items(): + unambiguous = { + value + for start, end, value in category_matches + if (start, end, category, value) not in ambiguous_entries + } + if unambiguous: + ids: set[str] = set() + for value in unambiguous: + ids.update(matching_ids(category, value)) + constraints.append(ids) + + candidate_ids = {entity.entity_id for entity in registry.entities} + for constraint in constraints: + candidate_ids.intersection_update(constraint) + candidates = tuple( + entity for entity in registry.entities if entity.entity_id in candidate_ids + ) + return _ExplicitSelectorMatch( + True, + discriminating, + candidates, + tuple(slots), + tuple(conflicting_slots), + ) + + +def _explicit_mentioned_entities( + registry: EntityRegistry, + utterance: str, +) -> tuple[EntitySpec, ...]: + """Compatibility wrapper for callers that need only explicit candidates.""" + + return _explicit_selector_match(registry, utterance).candidates + + +def _request_candidates( + utterance: str, + source_instructions: Sequence[DomuxInstruction], + registry: EntityRegistry, + context: SessionContext, +) -> tuple[tuple[DomuxInstruction, ...], tuple[EntitySpec, ...]]: + selectors = tuple( + _source_selector(utterance, instruction, registry) + for instruction in source_instructions + ) + + explicit = _explicit_selector_match(registry, utterance) + deictic = _has_deictic_reference(utterance) + if _has_other_reference(utterance) and not context.recent_entity_ids: + return selectors, () + if deictic and context.recent_entity_ids: + contextual = tuple( + registry._by_id[entity_id] + for entity_id in dict.fromkeys(context.recent_entity_ids) + if entity_id in registry._by_id + ) + if explicit.present: + explicitly_mentioned = {entity.entity_id for entity in explicit.candidates} + contextual = tuple( + entity for entity in contextual if entity.entity_id in explicitly_mentioned + ) + if _has_other_reference(utterance) and len(contextual) < 2: + return selectors, () + return selectors, tuple(sorted(contextual, key=registry._sort_key)) + + if explicit.present: + return selectors, explicit.candidates + + by_id: dict[str, EntitySpec] = {} + for selector in selectors: + for entity in registry.candidates(selector, context): + by_id[entity.entity_id] = entity + if _has_uncertainty_or_conflict(utterance): + for entity in _mentioned_entities(registry, utterance, source_instructions): + by_id[entity.entity_id] = entity + return selectors, tuple(sorted(by_id.values(), key=registry._sort_key)) + + +def _has_other_reference(text: str) -> bool: + return bool(re.search(OTHER_REFERENCE_RE, normalize_text(text))) + + +def _has_deictic_reference(text: str) -> bool: + normalized = normalize_text(text) + return _has_other_reference(normalized) or any( + _phrase_in(normalized, phrase) + for phrase in ("it", "that", "that one", "this", "this one") + ) + + +@dataclass(frozen=True) +class GroundedRequest: + utterance: str + raw_output: str + source_instructions: tuple[DomuxInstruction, ...] + selector_instructions: tuple[DomuxInstruction, ...] + context_entity_ids: tuple[str, ...] + negated_entity_ids: tuple[str, ...] + excluded_operation_value_tokens: tuple[str, ...] + candidates: tuple[EntitySpec, ...] + clarification: Clarification + request_digest: str + + +@dataclass(frozen=True) +class ResolvedRequest: + grounded: GroundedRequest + chosen: EntitySpec + confirmed_instruction: DomuxInstruction + clarification_digest: str + + +def ground_domux_request( + utterance: str, + raw_output: str, + registry: EntityRegistry, + context: SessionContext | None = None, +) -> GroundedRequest: + if not isinstance(utterance, str) or not utterance.strip(): + raise GroundingError("user utterance is empty") + if len(utterance) > MAX_UTTERANCE_CHARS: + raise GroundingError("user utterance exceeds the supported length") + context = context or SessionContext() + source = parse_domux_output(raw_output) + selectors, candidates = _request_candidates(utterance, source, registry, context) + authorization_utterance = _final_authorization_clause(utterance, registry) + operational_utterance = _authorized_operational_text( + utterance, + registry, + candidates or registry.entities, + ) + negated_entity_ids = _negated_entity_ids(registry, utterance) + excluded_value_tokens = _excluded_operation_value_tokens(utterance, source) + reasons: list[str] = [] + unresolved: list[str] = [] + explicit_selector = _explicit_selector_match(registry, utterance) + valid_context_ids = tuple( + entity_id + for entity_id in dict.fromkeys(context.recent_entity_ids) + if entity_id in registry._by_id + ) + other_reference = _has_other_reference(utterance) + if not candidates: + reasons.append("no_registry_match") + elif len(candidates) > 1: + reasons.append("multiple_registry_matches") + if len(source) != 1: + reasons.append("multiple_model_instructions") + if _has_uncertainty_or_conflict(utterance): + reasons.append("uncertainty_or_conflict") + if explicit_selector.conflicting_slots: + reasons.append("multiple_explicit_selectors") + unresolved.extend(explicit_selector.conflicting_slots) + if _has_negative_or_cancelled_intent(authorization_utterance, registry): + reasons.append("negative_or_cancelled_intent") + if not _has_supported_request_grammar(utterance, registry, source): + reasons.append("unsupported_request_grammar") + unresolved.append("authorization") + if ( + _has_deictic_reference(utterance) + and not context.recent_entity_ids + and not explicit_selector.discriminating + ): + reasons.append("unsupported_request_grammar") + unresolved.append("authorization") + if other_reference: + reasons.append("other_reference_requires_selection") + unresolved.append("context") + if len(valid_context_ids) < 2: + reasons.append("unsupported_request_grammar") + unresolved.append("authorization") + if _has_deictic_reference(utterance) and any( + entity_id not in registry._by_id + for entity_id in context.recent_entity_ids + ): + reasons.append("stale_context_reference") + unresolved.append("context") + if ( + _has_unresolved_generic_exclusion(authorization_utterance, registry) + and not _excluded_generic_domains(authorization_utterance, registry) + and not (other_reference and len(valid_context_ids) >= 2) + ): + reasons.append("unsupported_request_grammar") + unresolved.append("authorization") + informational_request = _has_informational_request(utterance) + if informational_request: + reasons.append("informational_request") + unresolved.append("authorization") + elif _has_unsupported_condition_or_time(utterance): + reasons.append("unsupported_condition_or_time") + unresolved.append("condition_or_time") + if negated_entity_ids: + reasons.append("negated_selector") + if excluded_value_tokens: + reasons.append("excluded_operation_value") + unresolved.append("value") + explicit_requirements = _explicit_operational_requirements(operational_utterance) + unresolved.extend(_partial_registry_selector_slots(utterance, registry)) + unresolved.extend(_unanchored_registry_selector_slots(utterance, registry)) + unresolved.extend(_unresolved_selector_phrase_slots(utterance)) + if _positional_unknown_selector_words(utterance, registry): + unresolved.append("device") + if _positional_unknown_attribute_words(utterance, registry): + unresolved.append("attribute") + reasons.append("unknown_operation_modifier") + for instruction, selector in zip(source, selectors): + for slot in ("device", "attribute", "room", "floor"): + if getattr(instruction, slot) != getattr(selector, slot): + unresolved.append(slot) + unresolved.extend(_missing_required_slots(instruction)) + if not _action_supported(operational_utterance, instruction): + unresolved.append("action") + if not _attribute_supported(operational_utterance, instruction) and not ( + len(candidates) == 1 + and _attribute_supported_for_entity(operational_utterance, instruction, candidates[0]) + ): + unresolved.append("attribute") + if not _value_supported(operational_utterance, instruction.value): + unresolved.append("value") + if not _unit_supported(operational_utterance, instruction): + unresolved.append("unit") + unresolved.extend(sorted(_operational_conflicts(operational_utterance, instruction))) + if normalize_text(instruction.action) == "turnon" and any( + candidate.domain == "climate" for candidate in candidates + ): + reasons.append("climate_mode_confirmation_required") + unresolved.extend(("attribute", "value")) + for slot in sorted(explicit_requirements): + if getattr(instruction, slot) == "*": + unresolved.append(slot) + if unresolved: + reasons.append("ungrounded_or_missing_slots") + reasons = list(dict.fromkeys(reasons)) + unresolved_slots = tuple(dict.fromkeys(unresolved)) + + base = clarification_for(candidates) + required = bool(reasons) + if not required: + clarification = base + elif not candidates or any(reason in reasons for reason in ( + "negative_or_cancelled_intent", "informational_request", "unsupported_condition_or_time", + "unsupported_request_grammar", + )): + clarification = Clarification( + True, + next((reason for reason in ( + "negative_or_cancelled_intent", "informational_request", "unsupported_condition_or_time", + "unsupported_request_grammar", + ) if reason in reasons), reasons[0]), + tuple(candidates[:3]), + None, + tuple(reasons), + unresolved_slots, + ) + elif len(candidates) > 3: + displayed = tuple(candidates[:3]) + clarification = Clarification( + True, + "too_many_candidates", + displayed, + "More than three devices match. Please provide a room or floor before choosing.", + tuple(dict.fromkeys((*reasons, "too_many_candidates"))), + unresolved_slots, + ) + else: + displayed = tuple(candidates[:3]) + options = "; ".join( + _candidate_option(index, candidate) + for index, candidate in enumerate(displayed, start=1) + ) + detail = ", ".join(unresolved_slots) if unresolved_slots else "device or value" + clarification = Clarification( + True, + reasons[0], + displayed, + f"Please confirm {detail}. Candidates: {options}.", + tuple(reasons), + unresolved_slots, + ) + context_ids = tuple(dict.fromkeys(context.recent_entity_ids)) + payload = { + "utterance_sha256": hashlib.sha256(utterance.encode("utf-8")).hexdigest(), + "raw_output_sha256": hashlib.sha256(raw_output.encode("utf-8")).hexdigest(), + "source": [instruction.to_pipe() for instruction in source], + "selectors": [instruction.to_pipe() for instruction in selectors], + "context_entity_ids": context_ids, + "negated_entity_ids": negated_entity_ids, + "excluded_operation_value_tokens": excluded_value_tokens, + "candidate_ids": [candidate.entity_id for candidate in candidates], + "reasons": clarification.reasons, + "unresolved_slots": clarification.unresolved_slots, + } + return GroundedRequest( + utterance=utterance, + raw_output=raw_output, + source_instructions=source, + selector_instructions=selectors, + context_entity_ids=context_ids, + negated_entity_ids=negated_entity_ids, + excluded_operation_value_tokens=excluded_value_tokens, + candidates=candidates, + clarification=clarification, + request_digest=digest_json(payload), + ) + + +def resolve_clarification( + answer: str, + candidates: Sequence[EntitySpec], + *, + registry: EntityRegistry | None = None, +) -> EntitySpec: + if not isinstance(answer, str) or not answer.strip(): + raise GroundingError("clarification answer is empty") + if not candidates: + raise GroundingError("clarification has no candidates") + answer_norm = normalize_text(answer) + answer_registry = EntityRegistry(candidates) + scope_registry = registry or answer_registry + answer_selector_spans = _registry_selector_spans( + answer_norm, + scope_registry, + ) + answer_selector = _explicit_selector_match( + answer_registry, + answer, + allow_bare_meaningful=True, + ) + if answer_selector.present and ( + answer_selector.conflicting_slots or not answer_selector.candidates + ): + raise GroundingError("clarification answer has inconsistent target selectors") + if _has_unresolved_generic_exclusion(answer_norm, scope_registry): + selected = ( + answer_selector.candidates[0] + if len(answer_selector.candidates) == 1 + else None + ) + excluded_domains = _excluded_generic_domains(answer, scope_registry) + if not ( + answer_selector.discriminating + and selected is not None + and not answer_selector.conflicting_slots + and excluded_domains + and selected.domain not in excluded_domains + ): + raise GroundingError("clarification answer uses an unresolved generic exclusion") + other = re.search(OTHER_REFERENCE_RE, answer_norm) + if other is not None: + suffix = answer_norm[other.end():] + explicit_after = any( + _phrase_in(suffix, label) + for entity in candidates + for label in (entity.entity_id, entity.room, entity.floor, *entity.aliases) + ) + if not explicit_after: + raise GroundingError("clarification answer does not identify which other candidate") + if answer_norm.isdigit(): + index = int(answer_norm) - 1 + if 0 <= index < len(candidates): + return candidates[index] + raise GroundingError("clarification index is outside the displayed candidates") + exact_id = [entity for entity in candidates if normalize_text(entity.entity_id) == answer_norm] + if len(exact_id) == 1: + return exact_id[0] + if len(candidates) == 1: + candidate = candidates[0] + exclusion_labels = [candidate.room, candidate.floor, *candidate.aliases] + if normalize_text(candidate.device) not in GENERIC_DEVICE_ALIASES: + exclusion_labels.append(candidate.device) + for label in exclusion_labels: + if _label_is_excluded( + answer_norm, + label, + selector_spans=answer_selector_spans, + ): + raise GroundingError("clarification answer excludes the only candidate") + if _answer_is_noncommittal(answer_norm): + raise GroundingError("clarification answer is noncommittal") + identifies_candidate = any( + _selector_label_evidence(answer, label, category, scope_registry) + for category, label in ( + ("entity", candidate.entity_id), + ("room", candidate.room), + ("floor", candidate.floor), + *(("alias", alias) for alias in candidate.aliases), + ) + ) + supplies_operation = bool(_explicit_operational_requirements(answer)) or bool( + _directional_actions(answer_norm) + ) or bool(_numbers_in(answer_norm)) or any( + _phrase_in(answer_norm, value) + for value in (*COLOR_RGB, "cool", "heat", "dry", "fan only", "auto", "low", "medium", "high") + ) + explicitly_affirms = bool(re.match( + r"^(?:yes\b|confirm\b|confirmed\b|proceed\b|do\s+it\b|that\s+one\b)", + answer_norm, + )) + if not (identifies_candidate or supplies_operation or explicitly_affirms): + raise GroundingError("clarification answer provides no confirmation evidence") + return candidate + feature_counts: dict[tuple[str, str], int] = {} + for entity in candidates: + for kind, value in (("room", entity.room), ("floor", entity.floor), ("device", entity.device)): + key = (kind, normalize_text(value)) + feature_counts[key] = feature_counts.get(key, 0) + 1 + scored: list[tuple[int, EntitySpec]] = [] + selector_candidate_ids = { + entity.entity_id for entity in answer_selector.candidates + } if answer_selector.present else {entity.entity_id for entity in candidates} + for entity in candidates: + if entity.entity_id not in selector_candidate_ids: + continue + negative = False + exclusion_labels = [entity.room, entity.floor, *entity.aliases] + if normalize_text(entity.device) not in GENERIC_DEVICE_ALIASES: + exclusion_labels.append(entity.device) + for label in exclusion_labels: + if _label_is_excluded( + answer_norm, + label, + selector_spans=answer_selector_spans, + ): + negative = True + if negative: + continue + score = 0 + combined = normalize_text(f"{entity.floor} {entity.room} {entity.device}") + if _phrase_in(answer_norm, combined): + score += 20 + for kind, value, weight in ( + ("room", entity.room, 8), ("floor", entity.floor, 5), ("device", entity.device, 4) + ): + key = (kind, normalize_text(value)) + selector_category = ( + "domain" if normalize_text(value) in GENERIC_DEVICE_ALIASES else kind + ) + if feature_counts[key] < len(candidates) and _selector_label_evidence( + answer, value, selector_category, scope_registry, + ): + score += weight + score += 10 * sum( + _selector_label_evidence(answer, alias, "alias", scope_registry) + for alias in entity.aliases + ) + if score: + scored.append((score, entity)) + if not scored: + raise GroundingError("clarification answer does not select a candidate") + best = max(score for score, _ in scored) + matches = [entity for score, entity in scored if score == best] + if len(matches) != 1: + raise GroundingError(f"clarification answer selects {len(matches)} candidates") + return matches[0] + + +def _answer_repairs_target_slots( + answer: str, + chosen: EntitySpec, + candidates: Sequence[EntitySpec], + unresolved_slots: Sequence[str], + registry: EntityRegistry, +) -> bool: + answer_normalized = normalize_text(answer) + exact_global = answer_normalized.isdigit() or _phrase_in(answer_normalized, chosen.entity_id) + if exact_global: + return True + alias_evidence = any( + _selector_label_evidence(answer, alias, "alias", registry) + and sum( + normalize_text(alias) in {normalize_text(item) for item in candidate.aliases} + for candidate in candidates + ) == 1 + for alias in chosen.aliases + ) + if alias_evidence: + return True + + answer_selector = _explicit_selector_match(registry, answer) + strong_target_evidence = ( + answer_selector.present + and tuple(entity.entity_id for entity in answer_selector.candidates) == (chosen.entity_id,) + ) + + room_evidence = _selector_label_evidence(answer, chosen.room, "room", registry) + floor_evidence = _selector_label_evidence(answer, chosen.floor, "floor", registry) + specific_device_evidence = ( + normalize_text(chosen.device) not in GENERIC_DEVICE_ALIASES + and _selector_label_evidence(answer, chosen.device, "device", registry) + ) + room_unique = room_evidence and sum( + normalize_text(candidate.room) == normalize_text(chosen.room) + for candidate in candidates + ) == 1 + floor_unique = floor_evidence and sum( + normalize_text(candidate.floor) == normalize_text(chosen.floor) + for candidate in candidates + ) == 1 + device_unique = specific_device_evidence and sum( + normalize_text(candidate.device) == normalize_text(chosen.device) + for candidate in candidates + ) == 1 + for slot in set(unresolved_slots).intersection({ + "device", "room", "floor", "entity", "context", + }): + if slot == "room" and not room_evidence: + return False + if slot == "floor" and not (floor_evidence or strong_target_evidence): + return False + if slot == "device" and not ( + specific_device_evidence or strong_target_evidence + or room_unique or (room_evidence and floor_evidence) + ): + return False + if slot == "entity" and not ( + strong_target_evidence or room_unique or floor_unique or device_unique + ): + return False + if slot == "context" and not ( + strong_target_evidence or room_unique or floor_unique or device_unique + ): + return False + return True + + +def _initial_target_is_fully_bound( + grounded: GroundedRequest, + chosen: EntitySpec, + registry: EntityRegistry, +) -> bool: + """Return whether the user's original selector already fixes the target. + + A clarification may then repair only operation slots. Model-proposed + room/floor fields, context-only references, partial labels, and unknown or + relational selectors never qualify as an already-bound target. + """ + + explicit = _explicit_selector_match(registry, grounded.utterance) + return ( + explicit.discriminating + and tuple(entity.entity_id for entity in explicit.candidates) == (chosen.entity_id,) + and not explicit.conflicting_slots + and not _partial_registry_selector_slots(grounded.utterance, registry) + and not _unanchored_registry_selector_slots(grounded.utterance, registry) + and not _unresolved_selector_phrase_slots(grounded.utterance) + and not _positional_unknown_selector_words(grounded.utterance, registry) + and not _has_other_reference(grounded.utterance) + and "stale_context_reference" not in grounded.clarification.reasons + and chosen.entity_id not in grounded.negated_entity_ids + ) + + +def _validate_confirmed_instruction( + grounded: GroundedRequest, + answer: str, + confirmed: DomuxInstruction, + chosen: EntitySpec, + registry: EntityRegistry, +) -> None: + answer_normalized = normalize_text(answer) + answer_selector_spans = _registry_selector_spans(answer_normalized, registry) + if _answer_cancels(answer_normalized) or _has_negative_or_cancelled_intent(answer) or any( + _phrase_in(answer_normalized, phrase) + for phrase in ("do not act", "do not execute", "don't act", "don't execute") + ): + raise GroundingError("clarification answer cancels the request") + if _has_unsupported_condition_or_time(answer): + raise GroundingError("clarification answer contains an unsupported condition or time") + if _has_informational_request(answer): + raise GroundingError("clarification answer is informational, not an authorization") + if chosen.entity_id in grounded.negated_entity_ids: + raise GroundingError("clarification selected an entity explicitly excluded by the user") + chosen_exclusion_labels = [ + chosen.entity_id, + chosen.room, + chosen.floor, + *chosen.aliases, + ] + if normalize_text(chosen.device) not in GENERIC_DEVICE_ALIASES: + chosen_exclusion_labels.append(chosen.device) + if any( + _label_is_excluded( + answer_normalized, + label, + selector_spans=answer_selector_spans, + ) + for label in chosen_exclusion_labels + ): + raise GroundingError("clarification answer explicitly excludes the selected entity") + confirmed_value_token = _operation_value_token(confirmed.value) + if ( + confirmed_value_token in grounded.excluded_operation_value_tokens + or _value_is_explicitly_excluded(grounded.utterance, confirmed.value) + or _value_is_explicitly_excluded(answer, confirmed.value) + ): + raise GroundingError("confirmed value was explicitly excluded by the user") + answer_is_candidate_index = answer_normalized.isdigit() + operational_slots = ("action", "attribute", "value", "unit") + if answer_is_candidate_index and any( + slot in grounded.clarification.unresolved_slots for slot in operational_slots + ): + raise GroundingError( + "a candidate index cannot supply a missing or conflicting operation slot" + ) + # A displayed index is selector UI, not an operational number. Keeping it + # out of value parsing prevents option 2 from becoming 2 percent/degrees. + operational_answer = ( + "" if answer_is_candidate_index + else _clarification_operational_text(answer, chosen) + ) + if _has_informational_request(operational_answer): + raise GroundingError("clarification answer is informational, not an authorization") + if _has_unsupported_condition_or_time(operational_answer): + raise GroundingError("clarification answer contains an unsupported condition or time") + if _has_negative_or_cancelled_intent(operational_answer): + raise GroundingError("clarification answer cancels the request") + if not _clarification_has_positive_authorization(answer, chosen, registry.entities): + raise GroundingError("clarification answer has no positive authorization evidence") + target_was_bound = _initial_target_is_fully_bound(grounded, chosen, registry) + if not target_was_bound and not _answer_repairs_target_slots( + answer, chosen, grounded.candidates, + grounded.clarification.unresolved_slots, registry, + ): + raise GroundingError( + "clarification answer must explicitly identify the repaired target" + ) + answer_requirements = _answer_operational_requirements(answer, chosen) + for slot in answer_requirements: + if getattr(confirmed, slot) == "*": + raise GroundingError(f"confirmed plan drops explicit {slot} from the clarification answer") + answer_directions = _directional_actions(answer_normalized) + confirmed_action = normalize_text(confirmed.action) + if len(answer_directions) > 1: + raise GroundingError("clarification answer contains opposing actions") + if answer_directions and confirmed_action not in answer_directions: + # "Open it to 35 percent" is a set-position answer, not a full-open + # authorization. Other cross-action patches fail closed. + if not (confirmed_action == "set" and "value" in answer_requirements): + raise GroundingError("confirmed action conflicts with the clarification answer") + if answer_requirements and any( + _phrase_in(answer_normalized, verb) for verb in ("set", "change", "make", "use") + ) and confirmed_action not in {"set", "adjustup", "adjustdown"}: + raise GroundingError("clarification answer introduces an unbound set operation") + if not EntityRegistry._device_matches(chosen, confirmed.device): + raise GroundingError("confirmed device does not match the selected entity") + if normalize_text(confirmed.room) != normalize_text(chosen.room): + raise GroundingError("confirmed room does not match the selected entity") + if normalize_text(confirmed.floor) != normalize_text(chosen.floor): + raise GroundingError("confirmed floor does not match the selected entity") + confirmed_candidates = registry.candidates(confirmed) + if tuple(entity.entity_id for entity in confirmed_candidates) != (chosen.entity_id,): + raise GroundingError("confirmed instruction does not resolve uniquely to the selected entity") + operational_utterance = _authorized_operational_text( + grounded.utterance, + registry, + grounded.candidates or registry.entities, + ) + answer_conflicts = _operational_conflicts(operational_answer, confirmed) + if answer_conflicts: + detail = ", ".join(sorted(answer_conflicts)) + raise GroundingError(f"clarification answer has unresolved operational conflicts: {detail}") + evidence = f"{operational_utterance}\n{operational_answer}" + original_conflicts = set(_operational_conflicts(operational_utterance, confirmed)) + if not ( + _action_supported(evidence, confirmed) + or _action_supported(operational_answer, confirmed) + ): + raise GroundingError("confirmed action is not supported by the user text") + if not _attribute_supported_for_entity(evidence, confirmed, chosen): + raise GroundingError("confirmed attribute is not supported by the user text") + if not _value_supported(evidence, confirmed.value) and not ( + "value" in original_conflicts and _value_supported(operational_answer, confirmed.value) + ): + raise GroundingError("confirmed value is not supported by the user text") + if not _unit_supported(evidence, confirmed) and not ( + "unit" in original_conflicts and _unit_supported(operational_answer, confirmed) + ): + raise GroundingError("confirmed unit is not supported by the user text") + answer_replaces_conflicted_action = ( + "action" in original_conflicts + and _action_supported(operational_answer, confirmed) + and normalize_text(confirmed.action) in {"turnon", "turnoff"} + ) + for slot in _explicit_operational_requirements(operational_utterance): + if getattr(confirmed, slot) == "*": + if answer_replaces_conflicted_action and slot in {"attribute", "value", "unit"}: + continue + raise GroundingError(f"confirmed plan drops explicit {slot} from the user request") + + def answer_supports(slot: str) -> bool: + if answer_is_candidate_index and slot in operational_slots: + return False + value = getattr(confirmed, slot) + if value == "*": + return True + normalized_value = normalize_text(value) + if re.search( + rf"\b(?:not|instead\s+of)\b(?:\s+[a-z0-9]+){{0,3}}\s+{re.escape(normalized_value)}\b", + answer_normalized, + ): + return False + if slot == "action": + if re.search(r"\bdo\s+not\s+(?:turn|switch|open|close|set|change|make)\b", answer_normalized): + return False + return _action_supported(operational_answer, confirmed) + if slot == "attribute": + return _attribute_supported_for_entity(operational_answer, confirmed, chosen) + if slot == "value": + return _value_supported(operational_answer, confirmed.value) + return _unit_supported(operational_answer, confirmed) + + if ( + "unknown_operation_modifier" in grounded.clarification.reasons + and not answer_supports("attribute") + ): + raise GroundingError("unknown operation modifier is not repaired by the answer") + + # A conflicted value/action must be selected in the answer itself. It may + # not be assembled from unrelated clauses in the original request. + original_normalized = normalize_text(operational_utterance) + numbers = set(_numbers_in(original_normalized)) + if _phrase_in(original_normalized, "halfway"): + numbers.add(50.0) + matched_named_values = { + value for value in (*COLOR_RGB, "cool", "heat", "dry", "fan", "auto", "low", "medium", "high") + if _phrase_in(original_normalized, value) + } + named_values = { + value for value in matched_named_values + if not any( + value != longer and _phrase_in(normalize_text(longer), value) + for longer in matched_named_values + ) + } + conflicted_slots: set[str] = set(original_conflicts) + if len(numbers) > 1 or len(named_values) > 1: + conflicted_slots.add("value") + has_on = any(_phrase_in(original_normalized, term) for term in ("turn on", "switch on", "open")) + has_off = any(_phrase_in(original_normalized, term) for term in ("turn off", "switch off", "close")) + if has_on and has_off: + conflicted_slots.add("action") + for slot in conflicted_slots: + if not answer_supports(slot): + raise GroundingError(f"conflicting {slot} is not independently confirmed by the answer") + + missing_slots = { + slot + for source in grounded.source_instructions + for slot in _missing_required_slots(source) + } + for slot in missing_slots: + if slot in operational_slots and not answer_supports(slot): + raise GroundingError(f"missing {slot} is not supplied by the answer") + + # The confirmed operational tuple must be one source tuple plus only + # answer-supported patches. This prevents cross-clause Frankenstein plans. + context = SessionContext(grounded.context_entity_ids) + patchable = False + for source in grounded.source_instructions: + changed = [ + slot for slot in operational_slots + if normalize_text(getattr(source, slot)) != normalize_text(getattr(confirmed, slot)) + ] + if not all(answer_supports(slot) for slot in changed): + continue + source_candidates = registry.candidates( + _source_selector(grounded.utterance, source, registry), + context, + ) + source_targets_chosen = any(entity.entity_id == chosen.entity_id for entity in source_candidates) + if source_targets_chosen or all(answer_supports(slot) for slot in conflicted_slots): + patchable = True + break + if not patchable: + raise GroundingError("confirmed plan is not a valid answer-supported patch of one source instruction") + + +def _answer_cancels(answer_normalized: str) -> bool: + return _has_negative_action_authorization(answer_normalized) or bool( + re.match(r"^(?:actually\s+)?no(?:\s+thanks?)?(?:\b|[,.!])", answer_normalized) + ) or bool( + re.search(r"\b(?:do\s+not|don't|dont)\s*[.!]?\s*$", answer_normalized) + ) or bool( + re.search( + r"\b(?:cancel(?:\s+(?:it|that|this|the\s+request))?|never\s+mind|do\s+nothing|" + r"stop(?:\s+(?:it|that|this))?|not\s+now|wait|hold(?:\s+on)?|" + r"(?:do\s+not|don't)\s+(?:do|proceed|act|execute)\b(?:.{0,20}\byet\b)?|" + r"(?:do\s+not|don't|rather\s+not)\s+(?:go\s+ahead|proceed|do|act|execute|" + r"turn|switch|open|close|set|change|make|move|adjust|touch)\b|" + r"(?:i\s+)?(?:do\s+not|don't)\s+want\b|" + r"forget\s+(?:it|that|this)|(?:i\s+)?changed?\s+my\s+mind|" + r"not\s+(?:anymore|any\s+longer)|skip\s+(?:it|that|this)|" + r"refrain\s+from|" + r"(?:leave|keep)\s+(?:(?:the\s+)?[a-z0-9_.-]+\s+|it\s+|that\s+|this\s+)?" + r"(?:on|off|open|closed|unchanged|as\s+is))\b", + answer_normalized, + ) + ) + + +def _answer_is_noncommittal(answer_normalized: str) -> bool: + return bool(re.search( + r"\b(?:do\s+not\s+know|don't\s+know|not\s+sure|still\s+not\s+sure|unsure|" + r"whatever|maybe|perhaps|ask\s+me\s+later|later)\b", + answer_normalized, + )) + + +def resolve_clarification_submission( + grounded: GroundedRequest, + *, + answer: str, + confirmed_instruction: DomuxInstruction, + registry: EntityRegistry, +) -> ResolvedRequest: + if not grounded.clarification.required: + raise GroundingError("request is already unique; use resolve_unique_request") + if "negative_or_cancelled_intent" in grounded.clarification.reasons: + raise GroundingError("cancelled or negated requests cannot be confirmed from this turn") + if "informational_request" in grounded.clarification.reasons: + raise GroundingError("informational questions cannot authorize execution from this turn") + if "unsupported_condition_or_time" in grounded.clarification.reasons: + raise GroundingError("conditional or timed requests require a new immediate command") + if "unsupported_request_grammar" in grounded.clarification.reasons: + raise GroundingError("unsupported request language requires a new immediate command") + confirmed_value_token = _operation_value_token(confirmed_instruction.value) + if confirmed_value_token in grounded.excluded_operation_value_tokens: + raise GroundingError("confirmed value was explicitly excluded by the user") + if not answer.strip(): + raise GroundingError("clarification answer is empty") + if _answer_is_noncommittal(normalize_text(answer)): + raise GroundingError("clarification answer is noncommittal") + if len(grounded.candidates) > len(grounded.clarification.candidates): + raise GroundingError("candidate set is not narrow enough to present safely") + answer_selector = _explicit_selector_match( + registry, + answer, + allow_bare_meaningful=True, + ) + if answer_selector.candidates and all( + entity.entity_id in grounded.negated_entity_ids + for entity in answer_selector.candidates + ): + raise GroundingError("clarification selects an explicitly excluded entity") + chosen = resolve_clarification( + answer, + grounded.candidates, + registry=registry, + ) + _validate_confirmed_instruction(grounded, answer, confirmed_instruction, chosen, registry) + clarification_digest = digest_json({ + "request_digest": grounded.request_digest, + "answer_sha256": hashlib.sha256(answer.encode("utf-8")).hexdigest(), + "chosen_entity_id": chosen.entity_id, + "confirmed_instruction": confirmed_instruction.to_pipe(), + }) + return ResolvedRequest(grounded, chosen, confirmed_instruction, clarification_digest) + + +def resolve_unique_request(grounded: GroundedRequest, registry: EntityRegistry) -> ResolvedRequest: + if grounded.clarification.required: + raise GroundingError("request still requires clarification") + if len(grounded.source_instructions) != 1 or len(grounded.candidates) != 1: + raise GroundingError("unique request invariant failed") + chosen = grounded.candidates[0] + confirmed = grounded.source_instructions[0] + # A unique source may retain '*' room/floor; candidate uniqueness, not a + # client-provided selector, binds the target in this path. + if not EntityRegistry._device_matches(chosen, confirmed.device): + raise GroundingError("unique request device does not match its candidate") + if confirmed.room != "*" and normalize_text(confirmed.room) != normalize_text(chosen.room): + raise GroundingError("unique request room does not match its candidate") + if confirmed.floor != "*" and normalize_text(confirmed.floor) != normalize_text(chosen.floor): + raise GroundingError("unique request floor does not match its candidate") + clarification_digest = digest_json({ + "request_digest": grounded.request_digest, + "answer": "unique_without_clarification", + "chosen_entity_id": chosen.entity_id, + "confirmed_instruction": confirmed.to_pipe(), + }) + return ResolvedRequest(grounded, registry.get(chosen.entity_id), confirmed, clarification_digest) + + +@dataclass(frozen=True, init=False) +class CanonicalPlan: + _source_slots_json: str + entity_id: str + domain: str + service: str + _service_data_json: str + _expected_projection_json: str + + def __init__( + self, + *, + source_slots: Mapping[str, str], + entity_id: str, + domain: str, + service: str, + service_data: Mapping[str, object], + expected_projection: Mapping[str, object], + ) -> None: + object.__setattr__(self, "_source_slots_json", canonical_json(dict(source_slots))) + object.__setattr__(self, "entity_id", entity_id) + object.__setattr__(self, "domain", domain) + object.__setattr__(self, "service", service) + object.__setattr__(self, "_service_data_json", canonical_json(dict(service_data))) + object.__setattr__(self, "_expected_projection_json", canonical_json(dict(expected_projection))) + + @property + def source_slots(self) -> Mapping[str, str]: + return json.loads(self._source_slots_json) + + @property + def service_data(self) -> Mapping[str, object]: + return json.loads(self._service_data_json) + + @property + def expected_projection(self) -> Mapping[str, object]: + return json.loads(self._expected_projection_json) + + def stable_dict(self) -> dict[str, object]: + return { + "source_slots": dict(self.source_slots), + "entity_id": self.entity_id, + "domain": self.domain, + "service": self.service, + "service_data": dict(self.service_data), + "expected_projection": dict(self.expected_projection), + } + + @property + def digest(self) -> str: + return digest_json(self.stable_dict()) + + +def _numeric(value: str, *, minimum: float, maximum: float) -> float: + try: + parsed = float(value) + except ValueError as exc: + raise GroundingError(f"expected numeric value, got {value!r}") from exc + if not math.isfinite(parsed) or not minimum <= parsed <= maximum: + raise GroundingError(f"numeric value {parsed} is outside [{minimum}, {maximum}]") + return parsed + + +def _require_unit(instruction: DomuxInstruction, expected: str) -> None: + if normalize_text(instruction.unit) != normalize_text(expected): + raise GroundingError( + f"{instruction.attribute} requires unit {expected!r}, got {instruction.unit!r}" + ) + + +def _require_placeholders(instruction: DomuxInstruction, *slots: str) -> None: + invalid = [slot for slot in slots if getattr(instruction, slot) != "*"] + if invalid: + raise GroundingError( + f"{instruction.action} requires '*' for unused slots: {', '.join(invalid)}" + ) + + +def _require_adjust_unit(instruction: DomuxInstruction, numeric_unit: str) -> None: + expected = "*" if instruction.value == "*" else numeric_unit + _require_unit(instruction, expected) + + +def _require_temperature_alignment( + attributes: Mapping[str, object], + temperature: float, + minimum: float, +) -> None: + if "target_temp_step" not in attributes: + return + step = float(attributes["target_temp_step"]) + if not math.isfinite(step) or step <= 0: + raise GroundingError("climate target_temp_step must be a positive number") + offset_steps = (temperature - minimum) / step + if not math.isclose(offset_steps, round(offset_steps), rel_tol=0, abs_tol=1e-8): + raise GroundingError( + f"temperature {temperature} does not align with the advertised {step} degree step" + ) + + +def controlled_projection(raw_state: Mapping[str, object], domain: str) -> dict[str, object]: + attributes = raw_state.get("attributes") + attrs = attributes if isinstance(attributes, Mapping) else {} + projected: dict[str, object] = { + "entity_id": raw_state.get("entity_id"), + "state": raw_state.get("state"), + } + keys = { + "light": ("brightness", "color_temp_kelvin", "rgb_color"), + "cover": ("current_position",), + "climate": ("temperature", "fan_mode"), + }[domain] + for key in keys: + if key in attrs: + value = attrs[key] + # Home Assistant's LightEntity state surface reports active color + # attributes as null while off, even when an integration retains + # the last value internally for the next turn-on. + if domain == "light" and raw_state.get("state") == "off": + value = None + projected[key] = list(value) if isinstance(value, tuple) else value + return projected + + +def planning_projection(raw_state: Mapping[str, object], domain: str) -> dict[str, object]: + """Bind every state field that can affect ``build_plan``. + + The smaller controlled projection is used for outcome assertions. This + projection is deliberately separate and includes advertised capabilities, + units, and ranges so a previously approved plan cannot outlive a capability + change that leaves the visible value untouched. + """ + + projected = controlled_projection(raw_state, domain) + attributes = raw_state.get("attributes") + attrs = attributes if isinstance(attributes, Mapping) else {} + keys = { + "light": ("supported_color_modes", "min_color_temp_kelvin", "max_color_temp_kelvin"), + "cover": ("supported_features",), + "climate": ( + "hvac_modes", "fan_modes", "supported_features", "temperature_unit", "min_temp", "max_temp", + "target_temp_step", + ), + }[domain] + for key in keys: + if key in attrs: + value = attrs[key] + projected[key] = list(value) if isinstance(value, tuple) else value + return projected + + +def validate_state_shape(raw_state: Mapping[str, object], expected_entity_id: str | None = None) -> None: + entity_id = raw_state.get("entity_id") + if not isinstance(entity_id, str) or not ENTITY_ID_RE.fullmatch(entity_id): + raise AdapterError("Home Assistant state has an invalid entity_id") + if expected_entity_id is not None and entity_id != expected_entity_id: + raise AdapterError("Home Assistant returned state for a different entity") + if not isinstance(raw_state.get("state"), str): + raise AdapterError("Home Assistant state value must be a string") + if not isinstance(raw_state.get("attributes"), Mapping): + raise AdapterError("Home Assistant state attributes must be an object") + + +def build_plan( + instruction: DomuxInstruction, + entity: EntitySpec, + current_state: Mapping[str, object], +) -> CanonicalPlan: + action = normalize_text(instruction.action) + attribute = normalize_text(instruction.attribute) + value = instruction.value.strip() + service: str + service_data: dict[str, object] = {"entity_id": entity.entity_id} + expected = controlled_projection(current_state, entity.domain) + + if entity.domain == "light": + light_attributes = current_state.get("attributes", {}) + if not isinstance(light_attributes, Mapping): + raise GroundingError("light attributes must be an object") + color_modes = { + normalize_text(mode) for mode in light_attributes.get("supported_color_modes", ()) + if isinstance(mode, str) + } + if action == "turnon": + _require_placeholders(instruction, "attribute", "value", "unit") + service, expected["state"] = "turn_on", "on" + # HA does not expose the retained brightness/color values while a + # light is off, so a bare turn-on can only promise the on state. + for key in ("brightness", "color_temp_kelvin", "rgb_color"): + expected.pop(key, None) + elif action == "turnoff": + _require_placeholders(instruction, "attribute", "value", "unit") + service, expected["state"] = "turn_off", "off" + for key in ("brightness", "color_temp_kelvin", "rgb_color"): + if key in expected: + expected[key] = None + elif action == "set" and attribute == "brightness": + _require_unit(instruction, "Percent") + if not color_modes.intersection({"brightness", "white", "color temp", "color_temp", "rgb", "rgbw", "rgbww", "hs", "xy"}): + raise GroundingError("light entity does not advertise brightness support") + percent = _numeric(value, minimum=0, maximum=100) + service_data["brightness_pct"] = percent + service = "turn_on" + expected["state"] = "off" if percent == 0 else "on" + expected["brightness"] = None if percent == 0 else round(percent * 255 / 100) + elif action == "set" and attribute == "color": + _require_unit(instruction, "*") + if not color_modes.intersection({"rgb", "rgbw", "rgbww", "hs", "xy"}): + raise GroundingError("light entity does not advertise color support") + color = normalize_text(value) + if color not in COLOR_RGB: + raise GroundingError(f"unsupported light color: {value!r}") + service_data["rgb_color"] = COLOR_RGB[color] + service, expected["state"] = "turn_on", "on" + expected["rgb_color"] = COLOR_RGB[color] + elif action == "set" and attribute == "colortemperature": + _require_unit(instruction, "Kelvin") + if not color_modes.intersection({"color temp", "color_temp"}): + raise GroundingError("light entity does not advertise color-temperature support") + minimum = float(light_attributes.get("min_color_temp_kelvin", 3000)) + maximum = float(light_attributes.get("max_color_temp_kelvin", 6500)) + kelvin = _numeric(value, minimum=minimum, maximum=maximum) + if not kelvin.is_integer(): + raise GroundingError("light color temperature must be an integer Kelvin value") + kelvin = int(kelvin) + service_data["color_temp_kelvin"] = kelvin + service, expected["state"] = "turn_on", "on" + expected["color_temp_kelvin"] = round(kelvin) + elif action in {"adjustup", "adjustdown"} and attribute == "brightness": + _require_adjust_unit(instruction, "Percent") + if not color_modes.intersection({"brightness", "white", "color temp", "color_temp", "rgb", "rgbw", "rgbww", "hs", "xy"}): + raise GroundingError("light entity does not advertise brightness support") + before = current_state.get("attributes", {}) + current = float(before.get("brightness", 0)) * 100 / 255 + step = 10.0 if value == "*" else _numeric(value, minimum=0, maximum=100) + target = min(100.0, current + step) if action == "adjustup" else max(0.0, current - step) + # Bind the postcondition to the value actually sent to Home + # Assistant. Rounding only the service payload can otherwise + # create a one-level mismatch at half-step boundaries (for + # example, 128/255 minus 10 percent). + dispatched_target = round(target, 2) + service_data["brightness_pct"] = dispatched_target + service = "turn_on" + expected["state"] = "off" if dispatched_target == 0 else "on" + expected["brightness"] = ( + None + if dispatched_target == 0 + else round(dispatched_target * 255 / 100) + ) + else: + raise GroundingError(f"unsupported light operation: {instruction.to_pipe()}") + elif entity.domain == "cover": + cover_attributes = current_state.get("attributes", {}) + if not isinstance(cover_attributes, Mapping): + raise GroundingError("cover attributes must be an object") + supported_features = int(cover_attributes.get("supported_features", 0)) + if action == "turnon": + _require_placeholders(instruction, "attribute", "value", "unit") + if not (supported_features & 1): + raise GroundingError("cover entity does not advertise open support") + service, expected["state"] = "open_cover", "open" + if "current_position" in expected: + expected["current_position"] = 100 + elif action == "turnoff": + _require_placeholders(instruction, "attribute", "value", "unit") + if not (supported_features & 2): + raise GroundingError("cover entity does not advertise close support") + service, expected["state"] = "close_cover", "closed" + if "current_position" in expected: + expected["current_position"] = 0 + elif action == "set" and attribute in {"position", "openness"}: + _require_unit(instruction, "Percent") + if not (supported_features & 4): + raise GroundingError("cover entity does not advertise position support") + position = _numeric(value, minimum=0, maximum=100) + if not position.is_integer(): + raise GroundingError("cover position must be an integer percent") + position = int(position) + service_data["position"] = position + service = "set_cover_position" + expected["state"] = "closed" if position == 0 else "open" + expected["current_position"] = round(position) + elif action in {"adjustup", "adjustdown"} and attribute in {"position", "openness"}: + _require_adjust_unit(instruction, "Percent") + if not (supported_features & 4): + raise GroundingError("cover entity does not advertise position support") + if "current_position" not in cover_attributes: + raise GroundingError("cover adjustment requires an observed current_position") + current = float(cover_attributes["current_position"]) + if not math.isfinite(current) or not current.is_integer(): + raise GroundingError("cover current_position must be an integer percent") + current = int(current) + step = 10.0 if value == "*" else _numeric(value, minimum=0, maximum=100) + if not step.is_integer(): + raise GroundingError("cover position adjustment must be an integer percent") + step = int(step) + target = min(100.0, current + step) if action == "adjustup" else max(0.0, current - step) + target = int(target) + service_data["position"] = target + service = "set_cover_position" + expected["state"] = "closed" if target == 0 else "open" + expected["current_position"] = round(target) + else: + raise GroundingError(f"unsupported cover operation: {instruction.to_pipe()}") + else: + attributes = current_state.get("attributes", {}) + if not isinstance(attributes, Mapping): + raise GroundingError("climate attributes must be an object") + hvac_modes = { + normalize_text(mode): mode for mode in attributes.get("hvac_modes", ()) + if isinstance(mode, str) + } + fan_modes = { + normalize_text(mode): mode for mode in attributes.get("fan_modes", ()) + if isinstance(mode, str) + } + supported_features = int(attributes.get("supported_features", 0)) + temperature_unit = str(attributes.get("temperature_unit", "")) + if not hvac_modes: + raise GroundingError("climate entity does not advertise hvac_modes") + if action == "turnon": + _require_placeholders(instruction, "attribute", "value", "unit") + active_modes = [value for key, value in hvac_modes.items() if key != "off"] + if len(active_modes) != 1: + raise GroundingError( + "climate turnOn requires exactly one advertised active mode; confirm a mode explicitly" + ) + service_data["hvac_mode"] = active_modes[0] + service, expected["state"] = "set_hvac_mode", active_modes[0] + elif action == "turnoff": + _require_placeholders(instruction, "attribute", "value", "unit") + if "off" in hvac_modes: + service = "set_hvac_mode" + service_data["hvac_mode"] = hvac_modes["off"] + elif supported_features & 128: + # Some integrations expose TURN_OFF as a feature instead of + # listing off in hvac_modes. + service = "turn_off" + else: + raise GroundingError("climate entity does not advertise turn-off support") + expected["state"] = "off" + elif action == "set" and attribute == "temperature": + _require_unit(instruction, "Celsius") + if temperature_unit not in {"°C", "C", "Celsius"}: + raise GroundingError("only Celsius climate entities are supported") + if not (supported_features & 1): + raise GroundingError("climate entity does not support target temperature") + minimum = float(attributes.get("min_temp", 16)) + maximum = float(attributes.get("max_temp", 30)) + temperature = _numeric(value, minimum=minimum, maximum=maximum) + _require_temperature_alignment(attributes, temperature, minimum) + service_data["temperature"] = temperature + service = "set_temperature" + expected["temperature"] = temperature + elif action == "set" and attribute == "mode": + _require_unit(instruction, "*") + mode = normalize_text(value) + mode = "fan only" if mode == "fan" else mode + if mode not in hvac_modes or mode == "off": + raise GroundingError(f"unsupported climate mode: {value!r}") + advertised_mode = hvac_modes[mode] + service_data["hvac_mode"] = advertised_mode + service = "set_hvac_mode" + expected["state"] = advertised_mode + elif action == "set" and attribute in {"windspeed", "wind speed", "fan speed"}: + _require_unit(instruction, "Level") + fan_mode = normalize_text(value) + if not (supported_features & 8) or fan_mode not in fan_modes: + raise GroundingError(f"unsupported fan mode: {value!r}") + advertised_fan_mode = fan_modes[fan_mode] + service_data["fan_mode"] = advertised_fan_mode + service = "set_fan_mode" + expected["fan_mode"] = advertised_fan_mode + elif action in {"adjustup", "adjustdown"} and attribute == "temperature": + _require_adjust_unit(instruction, "Celsius") + if temperature_unit not in {"°C", "C", "Celsius"} or not (supported_features & 1): + raise GroundingError("climate entity does not support Celsius target temperature") + current = float(current_state.get("attributes", {}).get("temperature", 24)) + if not math.isfinite(current): + raise GroundingError("current climate temperature must be finite") + step = 1.0 if value == "*" else _numeric(value, minimum=0, maximum=14) + minimum = float(attributes.get("min_temp", 16)) + maximum = float(attributes.get("max_temp", 30)) + target = min(maximum, current + step) if action == "adjustup" else max(minimum, current - step) + _require_temperature_alignment(attributes, target, minimum) + service_data["temperature"] = target + service = "set_temperature" + expected["temperature"] = target + else: + raise GroundingError(f"unsupported climate operation: {instruction.to_pipe()}") + + return CanonicalPlan( + source_slots=instruction.canonical_slots(), + entity_id=entity.entity_id, + domain=entity.domain, + service=service, + service_data=service_data, + expected_projection=expected, + ) + + +def projection_matches(actual: Mapping[str, object], expected: Mapping[str, object]) -> bool: + """Match all operation-controlled fields while allowing extra HA state.""" + + if not set(expected).issubset(actual): + return False + for key, expected_value in expected.items(): + actual_value = actual.get(key) + if isinstance(expected_value, float) and isinstance(actual_value, (int, float)): + if not math.isclose(float(actual_value), expected_value, rel_tol=0, abs_tol=0.01): + return False + elif actual_value != expected_value: + return False + return True + + +class InMemoryHAAdapter: + """Deterministic adapter used for policy evaluation and offline replay.""" + + def __init__(self, states: Mapping[str, Mapping[str, object]]): + self._states = json.loads(json.dumps(states)) + for entity_id, state in self._states.items(): + validate_state_shape(state, entity_id) + self.sut_calls: list[dict[str, object]] = [] + self.setup_calls: list[dict[str, object]] = [] + self.force_postcondition_mismatch = False + + def get_state(self, entity_id: str) -> dict[str, object]: + try: + state = json.loads(json.dumps(self._states[entity_id])) + except KeyError as exc: + raise AdapterError(f"state not found for allowed entity: {entity_id}") from exc + validate_state_shape(state, entity_id) + return state + + def set_state_for_setup(self, entity_id: str, state: Mapping[str, object]) -> None: + validate_state_shape(state, entity_id) + self._states[entity_id] = json.loads(json.dumps(state)) + self.setup_calls.append({"kind": "setup", "entity_id": entity_id}) + + def mutate_state_for_setup(self, entity_id: str) -> None: + state = self.get_state(entity_id) + domain = entity_id.split(".", 1)[0] + if domain == "light": + state["state"] = "off" if state.get("state") == "on" else "on" + elif domain == "cover": + current = int(state.get("attributes", {}).get("current_position", 0)) + updated = 100 - current + state.setdefault("attributes", {})["current_position"] = updated + state["state"] = "closed" if updated == 0 else "open" + else: + current = float(state.get("attributes", {}).get("temperature", 24)) + state.setdefault("attributes", {})["temperature"] = 25 if current != 25 else 24 + self.set_state_for_setup(entity_id, state) + + def call_service(self, domain: str, service: str, data: Mapping[str, object]) -> ServiceCallResult: + entity_id = str(data["entity_id"]) + if not entity_id.startswith(f"{domain}."): + raise ServiceCallError( + "service domain does not match entity", + attempted=False, + acknowledged=False, + outcome_unknown=False, + ) + try: + before = self.get_state(entity_id) + except Exception as exc: + raise ServiceCallError( + "state read failed before in-memory dispatch", + attempted=False, + acknowledged=False, + outcome_unknown=False, + ) from exc + after = self.get_state(entity_id) + attrs = after.setdefault("attributes", {}) + event = { + "kind": "sut", + "domain": domain, + "service": service, + "data": dict(data), + "before": controlled_projection(before, domain), + "after": None, + "acknowledged": False, + "outcome": "attempted", + } + self.sut_calls.append(event) + try: + if domain == "light": + if service == "turn_on": + if "brightness_pct" in data: + brightness_pct = float(data["brightness_pct"]) + if brightness_pct == 0: + after["state"] = "off" + else: + after["state"] = "on" + attrs["brightness"] = round(brightness_pct * 255 / 100) + else: + after["state"] = "on" + if "rgb_color" in data: + attrs["rgb_color"] = list(data["rgb_color"]) + if "color_temp_kelvin" in data: + attrs["color_temp_kelvin"] = round(float(data["color_temp_kelvin"])) + elif service == "turn_off": + after["state"] = "off" + else: + raise AdapterError(f"unsupported light service: {service}") + elif domain == "cover": + if service == "open_cover": + after["state"], attrs["current_position"] = "open", 100 + elif service == "close_cover": + after["state"], attrs["current_position"] = "closed", 0 + elif service == "set_cover_position": + position = round(float(data["position"])) + after["state"] = "closed" if position == 0 else "open" + attrs["current_position"] = position + else: + raise AdapterError(f"unsupported cover service: {service}") + elif domain == "climate": + if service == "turn_on": + after["state"] = "cool" + elif service == "turn_off": + after["state"] = "off" + elif service == "set_temperature": + attrs["temperature"] = float(data["temperature"]) + elif service == "set_hvac_mode": + after["state"] = data["hvac_mode"] + elif service == "set_fan_mode": + attrs["fan_mode"] = data["fan_mode"] + else: + raise AdapterError(f"unsupported climate service: {service}") + else: + raise AdapterError(f"unsupported service domain: {domain}") + except Exception as exc: + event["outcome"] = "rejected_before_acknowledgement" + raise ServiceCallError( + "in-memory service rejected the request", + attempted=True, + acknowledged=False, + outcome_unknown=False, + ) from exc + if not self.force_postcondition_mismatch: + self._states[entity_id] = after + observed = self.get_state(entity_id) + event["acknowledged"] = True + event["outcome"] = "observed" + event["after"] = controlled_projection(observed, domain) + return ServiceCallResult( + after=observed, + attempted=True, + acknowledged=True, + outcome_unknown=False, + ) + + +class _NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, req: object, fp: object, code: int, msg: str, headers: object, newurl: str) -> None: + del req, fp, code, msg, headers, newurl + return None + + +class _RequestFailure(AdapterError): + def __init__( + self, + message: str, + *, + response_received: bool, + status_code: int | None = None, + ) -> None: + super().__init__(message) + self.response_received = response_received + self.status_code = status_code + + +class HomeAssistantRESTAdapter: + """Minimal client for Home Assistant's official REST API. + + The default loopback restriction prevents an example token from being used + against a public or production Home Assistant instance by accident. + """ + + def __init__( + self, + base_url: str, + token: str, + *, + timeout_seconds: float = 10.0, + poll_seconds: float = 5.0, + allow_non_loopback: bool = False, + ): + parsed = urllib.parse.urlparse(base_url) + if parsed.scheme not in {"http", "https"} or not parsed.hostname: + raise ValueError("base_url must be an absolute HTTP(S) URL") + if not allow_non_loopback and parsed.hostname not in {"127.0.0.1", "localhost", "::1"}: + raise ValueError("only loopback Home Assistant URLs are allowed by default") + if allow_non_loopback and parsed.hostname not in {"127.0.0.1", "localhost", "::1"} and parsed.scheme != "https": + raise ValueError("non-loopback Home Assistant URLs require HTTPS") + if not token: + raise ValueError("Home Assistant token is required") + self._base_url = base_url.rstrip("/") + self._token = token + self._timeout = timeout_seconds + self._poll_seconds = poll_seconds + self._opener = urllib.request.build_opener(_NoRedirect()) + self.sut_calls: list[dict[str, object]] = [] + + def _request(self, method: str, path: str, payload: Mapping[str, object] | None = None) -> object: + body = None if payload is None else canonical_json(payload).encode("utf-8") + request = urllib.request.Request( + f"{self._base_url}{path}", + data=body, + method=method, + headers={ + "Authorization": f"Bearer {self._token}", + "Content-Type": "application/json", + }, + ) + try: + with self._opener.open(request, timeout=self._timeout) as response: + raw = response.read() + except urllib.error.HTTPError as exc: + status_code = exc.code + exc.close() + raise _RequestFailure( + f"Home Assistant REST request failed for {method} {path}", + response_received=True, + status_code=status_code, + ) from exc + except (urllib.error.URLError, TimeoutError) as exc: + raise _RequestFailure( + f"Home Assistant REST request failed for {method} {path}", + response_received=False, + ) from exc + return json.loads(raw) if raw else None + + def get_state(self, entity_id: str) -> dict[str, object]: + encoded = urllib.parse.quote(entity_id, safe=".") + result = self._request("GET", f"/api/states/{encoded}") + if not isinstance(result, dict): + raise AdapterError("Home Assistant returned a non-object state") + validate_state_shape(result, entity_id) + if entity_id.startswith("climate.") and "temperature_unit" not in result["attributes"]: + config = self._request("GET", "/api/config") + if not isinstance(config, dict): + raise AdapterError("Home Assistant returned a non-object config") + unit_system = config.get("unit_system") + if not isinstance(unit_system, Mapping): + raise AdapterError("Home Assistant config has no unit_system object") + temperature_unit = unit_system.get("temperature") + if not isinstance(temperature_unit, str) or not temperature_unit.strip(): + raise AdapterError("Home Assistant config has no temperature unit") + result = copy.deepcopy(result) + attributes = result["attributes"] + if not isinstance(attributes, dict): + attributes = dict(attributes) + result["attributes"] = attributes + attributes["temperature_unit"] = temperature_unit + return result + + def call_service(self, domain: str, service: str, data: Mapping[str, object]) -> ServiceCallResult: + entity_id = str(data["entity_id"]) + if not entity_id.startswith(f"{domain}."): + raise ServiceCallError( + "service domain does not match entity", + attempted=False, + acknowledged=False, + outcome_unknown=False, + ) + try: + before = self.get_state(entity_id) + except Exception as exc: + raise ServiceCallError( + "Home Assistant state read failed before dispatch", + attempted=False, + acknowledged=False, + outcome_unknown=False, + ) from exc + event = { + "kind": "sut", + "domain": domain, + "service": service, + "data": dict(data), + "before": controlled_projection(before, domain), + "after": None, + "acknowledged": False, + "outcome": "attempted", + } + self.sut_calls.append(event) + try: + self._request("POST", f"/api/services/{domain}/{service}", data) + except _RequestFailure as exc: + # A 4xx is an explicit rejection. A 5xx can arrive after HA (or an + # integration it called) has already applied a side effect, so the + # exact outcome remains unknown even though an HTTP response exists. + unknown = not exc.response_received or ( + exc.status_code is not None and 500 <= exc.status_code < 600 + ) + event["outcome"] = "request_error_outcome_unknown" if unknown else "request_rejected" + raise ServiceCallError( + "Home Assistant service request failed", + attempted=True, + acknowledged=False, + outcome_unknown=unknown, + ) from exc + event["acknowledged"] = True + event["outcome"] = "acknowledged_state_pending" + deadline = time.monotonic() + self._poll_seconds + try: + after = self.get_state(entity_id) + while time.monotonic() < deadline and after == before: + time.sleep(0.1) + after = self.get_state(entity_id) + except Exception as exc: + event["outcome"] = "acknowledged_state_unknown" + raise ServiceCallError( + "Home Assistant acknowledged the service but state observation failed", + attempted=True, + acknowledged=True, + outcome_unknown=True, + ) from exc + event["after"] = controlled_projection(after, domain) + event["outcome"] = "observed" + return ServiceCallResult( + after=after, + attempted=True, + acknowledged=True, + outcome_unknown=False, + ) + + def wait_for_projection( + self, + entity_id: str, + domain: str, + expected: Mapping[str, object], + ) -> dict[str, object]: + """Poll through transient HA states until the approved projection is visible.""" + + deadline = time.monotonic() + self._poll_seconds + latest = self.get_state(entity_id) + while not projection_matches(controlled_projection(latest, domain), expected): + if time.monotonic() >= deadline: + break + time.sleep(0.1) + latest = self.get_state(entity_id) + return latest + + +def state_binding( + adapter: Any, + registry: EntityRegistry, + entity_ids: Iterable[str], + *, + for_planning: bool = False, +) -> dict[str, object]: + result: dict[str, object] = {} + for entity_id in sorted(set(entity_ids)): + entity = registry.get(entity_id) + raw = adapter.get_state(entity_id) + result[entity_id] = ( + planning_projection(raw, entity.domain) + if for_planning + else controlled_projection(raw, entity.domain) + ) + return result + + +@dataclass(frozen=True) +class Confirmation: + actor_id: str + session_id: str + nonce: str + request_digest: str + clarification_digest: str + plan_digest: str + candidate_digest: str + + +@dataclass(frozen=True) +class PreparedAction: + actor_id: str + session_id: str + nonce: str + request_digest: str + clarification_digest: str + plan_digest: str + candidate_digest: str + entity_id: str + created_at: float + expires_at: float + + def confirmation(self) -> Confirmation: + return Confirmation( + actor_id=self.actor_id, + session_id=self.session_id, + nonce=self.nonce, + request_digest=self.request_digest, + clarification_digest=self.clarification_digest, + plan_digest=self.plan_digest, + candidate_digest=self.candidate_digest, + ) + + +@dataclass +class _StoredAction: + actor_id: str + session_id: str + nonce: str + utterance: str + raw_output: str + context_entity_ids: tuple[str, ...] + request_digest: str + clarification_digest: str + plan: CanonicalPlan + plan_digest: str + candidate_ids: tuple[str, ...] + state_entity_ids: tuple[str, ...] + candidate_digest: str + state_digest: str + created_at: float + expires_at: float + status: str = "PREPARED" + consumed: bool = False + + +@dataclass(frozen=True) +class _ActionTombstone: + nonce: str + plan_digest: str + candidate_digest: str + created_at: float + expires_at: float + status: str + consumed: bool + + +@dataclass(frozen=True) +class CommitResult: + accepted: bool + dispatched: bool + reason: str + status: str + nonce: str + plan_digest: str | None = None + before: Mapping[str, object] | None = None + after: Mapping[str, object] | None = None + acknowledged: bool = False + outcome_unknown: bool = False + before_registry_digest: str | None = None + after_registry_digest: str | None = None + + +class PreparedActionStore: + """Thread-safe one-time authorization store for a single process.""" + + def __init__( + self, + *, + ttl_seconds: float = 30.0, + clock: Callable[[], float] = time.monotonic, + nonce_factory: Callable[[], str] = lambda: secrets.token_urlsafe(24), + max_items: int = 10_000, + ): + if ttl_seconds <= 0: + raise ValueError("ttl_seconds must be positive") + if max_items <= 0: + raise ValueError("max_items must be positive") + self._ttl = ttl_seconds + self._clock = clock + self._nonce_factory = nonce_factory + self._max_items = max_items + self._items: dict[str, _StoredAction | _ActionTombstone] = {} + self._entity_locks: dict[str, threading.Lock] = {} + self._lock = threading.RLock() + + def prepare( + self, + *, + actor_id: str, + session_id: str, + grounded: GroundedRequest, + registry: EntityRegistry, + adapter: Any, + clarification_answer: str | None = None, + confirmed_instruction: DomuxInstruction | None = None, + state_dependencies: Sequence[str] = (), + ) -> PreparedAction: + if not actor_id or not session_id: + raise ValueError("actor_id and session_id are required") + context = SessionContext(grounded.context_entity_ids) + server_grounded = ground_domux_request( + grounded.utterance, + grounded.raw_output, + registry, + context, + ) + if server_grounded.request_digest != grounded.request_digest: + raise GroundingError("grounded request does not match the server-side reconstruction") + if server_grounded.clarification.required: + if clarification_answer is None or confirmed_instruction is None: + raise GroundingError("clarification answer and complete confirmed instruction are required") + resolved = resolve_clarification_submission( + server_grounded, + answer=clarification_answer, + confirmed_instruction=confirmed_instruction, + registry=registry, + ) + else: + if clarification_answer is not None or confirmed_instruction is not None: + raise GroundingError("a unique request cannot be replaced by client confirmation fields") + resolved = resolve_unique_request(server_grounded, registry) + + candidate_ids = tuple(sorted(entity.entity_id for entity in server_grounded.candidates)) + chosen = registry.get(resolved.chosen.entity_id) + context_entity_ids = tuple(server_grounded.context_entity_ids) + state_entity_ids = tuple(sorted({chosen.entity_id, *context_entity_ids, *state_dependencies})) + for entity_id in state_entity_ids: + registry.get(entity_id) + raw_states: dict[str, Mapping[str, object]] = {} + for entity_id in state_entity_ids: + raw_states[entity_id] = adapter.get_state(entity_id) + plan = build_plan(resolved.confirmed_instruction, chosen, raw_states[chosen.entity_id]) + plan_digest = plan.digest + candidate_digest = registry.metadata_digest(candidate_ids) + bound_projection = { + entity_id: planning_projection(raw_states[entity_id], registry.get(entity_id).domain) + for entity_id in state_entity_ids + } + state_digest = digest_json(bound_projection) + created = self._clock() + nonce = self._nonce_factory() + if not nonce: + raise ValueError("nonce factory returned an empty value") + action = _StoredAction( + actor_id=actor_id, + session_id=session_id, + nonce=nonce, + utterance=server_grounded.utterance, + raw_output=server_grounded.raw_output, + context_entity_ids=context_entity_ids, + request_digest=server_grounded.request_digest, + clarification_digest=resolved.clarification_digest, + plan=plan, + plan_digest=plan_digest, + candidate_ids=candidate_ids, + state_entity_ids=state_entity_ids, + candidate_digest=candidate_digest, + state_digest=state_digest, + created_at=created, + expires_at=created + self._ttl, + ) + with self._lock: + self._purge_expired_locked(created) + if len(self._items) >= self._max_items: + now = self._clock() + removable = [ + key for key, item in self._items.items() + if item.consumed or item.status != "PREPARED" or now > item.expires_at + ] + for key in removable: + self._items.pop(key, None) + if len(self._items) >= self._max_items: + raise RuntimeError("prepared-action store capacity exceeded") + if nonce in self._items: + raise ValueError("nonce collision") + self._items[nonce] = action + return PreparedAction( + actor_id=actor_id, + session_id=session_id, + nonce=nonce, + request_digest=action.request_digest, + clarification_digest=action.clarification_digest, + plan_digest=action.plan_digest, + candidate_digest=action.candidate_digest, + entity_id=action.plan.entity_id, + created_at=action.created_at, + expires_at=action.expires_at, + ) + + def _reject( + self, + action: _StoredAction | None, + nonce: str, + reason: str, + status: str, + *, + mutate: bool = True, + ) -> CommitResult: + if action is not None and mutate: + action.status = status + result = CommitResult( + accepted=False, + dispatched=False, + reason=reason, + status=status, + nonce=nonce, + plan_digest=None if action is None else action.plan_digest, + ) + if action is not None and mutate and status != "PREPARED": + with self._lock: + self._redact_locked(action) + return result + + def _redact_locked(self, action: _StoredAction) -> None: + """Replace terminal actions with a digest-only replay tombstone.""" + + if self._items.get(action.nonce) is not action: + return + self._items[action.nonce] = _ActionTombstone( + nonce=action.nonce, + plan_digest=action.plan_digest, + candidate_digest=action.candidate_digest, + created_at=action.created_at, + expires_at=action.expires_at, + status=action.status, + consumed=action.consumed, + ) + + def _purge_expired_locked(self, now: float) -> int: + expired = [ + item for item in self._items.values() + if isinstance(item, _StoredAction) + and item.status == "PREPARED" + and now > item.expires_at + ] + for item in expired: + item.status = "EXPIRED" + self._redact_locked(item) + return len(expired) + + def purge_expired(self) -> int: + """Redact abandoned prepared requests; applications may call this from a timer.""" + + with self._lock: + return self._purge_expired_locked(self._clock()) + + @staticmethod + def _tombstone_result(action: _ActionTombstone) -> CommitResult: + return CommitResult( + accepted=False, + dispatched=False, + reason="replayed_nonce" if action.consumed else "action_not_prepared", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + ) + + def _confirmation_error( + self, + action: _StoredAction, + confirmation: Confirmation, + *, + enforce_lifecycle: bool, + ) -> str | None: + if enforce_lifecycle and action.consumed: + return "replayed_nonce" + if enforce_lifecycle and action.status != "PREPARED": + return "action_not_prepared" + checks = ( + (confirmation.actor_id, action.actor_id, "actor_mismatch"), + (confirmation.session_id, action.session_id, "session_mismatch"), + (confirmation.request_digest, action.request_digest, "request_mismatch"), + (confirmation.clarification_digest, action.clarification_digest, "clarification_mismatch"), + (confirmation.plan_digest, action.plan_digest, "plan_mismatch"), + (confirmation.candidate_digest, action.candidate_digest, "confirmation_candidate_mismatch"), + ) + return next((reason for actual, expected, reason in checks if actual != expected), None) + + def _execute( + self, + action: _StoredAction, + *, + registry: EntityRegistry, + adapter: Any, + before_all: Mapping[str, Mapping[str, object]], + ) -> CommitResult: + before = dict(before_all[action.plan.entity_id]) + before_digest = digest_json(before_all) + try: + receipt = adapter.call_service( + action.plan.domain, + action.plan.service, + action.plan.service_data, + ) + if not isinstance(receipt, ServiceCallResult): + raise ServiceCallError( + "adapter violated the ServiceCallResult contract", + attempted=True, + acknowledged=False, + outcome_unknown=True, + ) + after_raw = receipt.after + except ServiceCallError as exc: + action.status = "FAILED_DISPATCH" + return CommitResult( + accepted=True, + dispatched=exc.attempted, + acknowledged=exc.acknowledged, + outcome_unknown=exc.outcome_unknown, + reason="dispatch_failed", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + before=before, + after=None, + before_registry_digest=before_digest, + ) + except Exception: + action.status = "FAILED_DISPATCH" + return CommitResult( + accepted=True, + dispatched=True, + acknowledged=False, + outcome_unknown=True, + reason="dispatch_failed", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + before=before, + after=None, + before_registry_digest=before_digest, + ) + wait_for_projection = getattr(adapter, "wait_for_projection", None) + if callable(wait_for_projection): + try: + after_raw = wait_for_projection( + action.plan.entity_id, + action.plan.domain, + action.plan.expected_projection, + ) + except Exception: + action.status = "FAILED_POSTCONDITION" + return CommitResult( + accepted=True, + dispatched=receipt.attempted, + acknowledged=receipt.acknowledged, + outcome_unknown=True, + reason="postcondition_state_unknown", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + before=before, + after=controlled_projection(after_raw, action.plan.domain), + before_registry_digest=before_digest, + ) + try: + after_all = state_binding(adapter, registry, (entity.entity_id for entity in registry.entities)) + except Exception: + action.status = "FAILED_POSTCONDITION" + return CommitResult( + accepted=True, + dispatched=receipt.attempted, + acknowledged=receipt.acknowledged, + outcome_unknown=True, + reason="postcondition_state_unknown", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + before=before, + after=controlled_projection(after_raw, action.plan.domain), + before_registry_digest=before_digest, + ) + after = after_all[action.plan.entity_id] + exact = set(after_all) == set(before_all) and all( + ( + projection_matches(after_all[entity_id], action.plan.expected_projection) + if entity_id == action.plan.entity_id + else after_all[entity_id] == before_all[entity_id] + ) + for entity_id in before_all + ) + after_digest = digest_json(after_all) + if not exact: + action.status = "FAILED_POSTCONDITION" + return CommitResult( + accepted=True, + dispatched=receipt.attempted, + acknowledged=receipt.acknowledged, + outcome_unknown=receipt.outcome_unknown, + reason="postcondition_mismatch", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + before=before, + after=after, + before_registry_digest=before_digest, + after_registry_digest=after_digest, + ) + action.status = "COMMITTED" + return CommitResult( + accepted=True, + dispatched=receipt.attempted, + acknowledged=receipt.acknowledged, + outcome_unknown=receipt.outcome_unknown, + reason="committed", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + before=before, + after=after, + before_registry_digest=before_digest, + after_registry_digest=after_digest, + ) + + def commit( + self, + confirmation: Confirmation, + *, + registry: EntityRegistry, + adapter: Any, + ) -> CommitResult: + with self._lock: + action = self._items.get(confirmation.nonce) + if action is None: + return self._reject(None, confirmation.nonce, "unknown_nonce", "INVALIDATED") + if isinstance(action, _ActionTombstone): + return self._tombstone_result(action) + error = self._confirmation_error(action, confirmation, enforce_lifecycle=True) + if error: + return self._reject(action, action.nonce, error, action.status, mutate=False) + if self._clock() > action.expires_at: + return self._reject(action, action.nonce, "expired", "EXPIRED") + # The postcondition asserts that no registered entity changed as a + # side effect. Lock that same in-process scope, in sorted order, + # so concurrent commits cannot invalidate one another's evidence. + lock_entity_ids = tuple(sorted(entity.entity_id for entity in registry.entities)) + entity_locks = tuple( + self._entity_locks.setdefault(entity_id, threading.Lock()) + for entity_id in lock_entity_ids + ) + + with ExitStack() as lock_stack: + for entity_lock in entity_locks: + lock_stack.enter_context(entity_lock) + with self._lock: + action = self._items.get(confirmation.nonce) + if action is None: + return self._reject(None, confirmation.nonce, "unknown_nonce", "INVALIDATED") + if isinstance(action, _ActionTombstone): + return self._tombstone_result(action) + error = self._confirmation_error(action, confirmation, enforce_lifecycle=True) + if error: + return self._reject(action, action.nonce, error, action.status, mutate=False) + if self._clock() > action.expires_at: + return self._reject(action, action.nonce, "expired", "EXPIRED") + + context = SessionContext(action.context_entity_ids) + regrounded = ground_domux_request(action.utterance, action.raw_output, registry, context) + current_candidate_ids = tuple(sorted(entity.entity_id for entity in regrounded.candidates)) + current_candidate_digest = registry.metadata_digest(current_candidate_ids) + if ( + current_candidate_ids != action.candidate_ids + or current_candidate_digest != action.candidate_digest + ): + with self._lock: + return self._reject(action, action.nonce, "candidate_set_changed", "INVALIDATED") + try: + before_all = state_binding( + adapter, + registry, + (entity.entity_id for entity in registry.entities), + ) + except Exception: + return CommitResult( + accepted=False, + dispatched=False, + reason="predispatch_state_read_failed", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + ) + try: + current_bound = state_binding( + adapter, + registry, + action.state_entity_ids, + for_planning=True, + ) + except Exception: + return CommitResult( + accepted=False, + dispatched=False, + reason="predispatch_state_read_failed", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + ) + current_state_digest = digest_json(current_bound) + if current_state_digest != action.state_digest: + with self._lock: + return self._reject(action, action.nonce, "state_changed", "INVALIDATED") + with self._lock: + error = self._confirmation_error(action, confirmation, enforce_lifecycle=True) + if error: + return self._reject(action, action.nonce, error, action.status, mutate=False) + if self._clock() > action.expires_at: + return self._reject(action, action.nonce, "expired", "EXPIRED") + action.consumed = True + action.status = "DISPATCHING" + result = self._execute(action, registry=registry, adapter=adapter, before_all=before_all) + with self._lock: + self._redact_locked(action) + return result + + def snapshot(self, nonce: str) -> dict[str, object]: + with self._lock: + action = self._items[nonce] + if isinstance(action, _ActionTombstone): + return { + "nonce": action.nonce, + "plan_digest": action.plan_digest, + "candidate_digest": action.candidate_digest, + "created_at": action.created_at, + "expires_at": action.expires_at, + "status": action.status, + "consumed": action.consumed, + "redacted": True, + } + return { + "actor_id": action.actor_id, + "session_id": action.session_id, + "nonce": action.nonce, + "request_digest": action.request_digest, + "clarification_digest": action.clarification_digest, + "plan_digest": action.plan_digest, + "candidate_ids": list(action.candidate_ids), + "state_entity_ids": list(action.state_entity_ids), + "candidate_digest": action.candidate_digest, + "state_digest": action.state_digest, + "created_at": action.created_at, + "expires_at": action.expires_at, + "status": action.status, + "consumed": action.consumed, + "plan": action.plan.stable_dict(), + "redacted": False, + } + + +class ClarifyPrepareStore(PreparedActionStore): + """Credible baseline: server plan/session binding without temporal guards. + + It deliberately does not revalidate candidate metadata, relevant state, TTL, + or one-time use. Those are the only independent variables added by the + Clarify-and-Commit store above. + """ + + def commit( + self, + confirmation: Confirmation, + *, + registry: EntityRegistry, + adapter: Any, + ) -> CommitResult: + with self._lock: + action = self._items.get(confirmation.nonce) + if action is None: + return self._reject(None, confirmation.nonce, "unknown_nonce", "INVALIDATED") + error = self._confirmation_error(action, confirmation, enforce_lifecycle=False) + if error: + return self._reject(action, action.nonce, error, action.status, mutate=False) + action.status = "DISPATCHING" + try: + before_all = state_binding( + adapter, + registry, + (entity.entity_id for entity in registry.entities), + ) + except Exception: + return CommitResult( + accepted=False, + dispatched=False, + reason="predispatch_state_read_failed", + status=action.status, + nonce=action.nonce, + plan_digest=action.plan_digest, + ) + return self._execute(action, registry=registry, adapter=adapter, before_all=before_all) + + +def altered_confirmation(confirmation: Confirmation, **changes: object) -> Confirmation: + """Small explicit helper used by mutation tests and the frozen evaluator.""" + + return replace(confirmation, **changes) diff --git a/cases/domux-clarify-commit/data/DATA_CARD.md b/cases/domux-clarify-commit/data/DATA_CARD.md new file mode 100644 index 0000000..603dd3f --- /dev/null +++ b/cases/domux-clarify-commit/data/DATA_CARD.md @@ -0,0 +1,129 @@ +# Clarify-and-Commit synthetic scenario data card + +## Summary + +This case uses 64 synthetic smart-home base scenarios. Sixteen records are a +development split for implementation and smoke tests. Forty-eight records form +a fixed, policy-informed conformance split, balanced across four ambiguity +strata: + +| Stratum | Development | Evaluation | +|---|---:|---:| +| Duplicate entity names | 4 | 12 | +| Missing entity or required slot | 4 | 12 | +| Context-dependent reference | 4 | 12 | +| Unresolved negation or correction | 4 | 12 | +| **Total** | **16** | **48** | + +Every base contains a matched pair: one explicit command and one ambiguous +command that points to the same intended state change. The pair is one +statistical unit; the two commands are not treated as independent samples. + +## Creation and freeze history + +The scenario candidate was drafted before the case implementation or any +formal Domux output was inspected. Its original SHA-256 was +`e81b7879760e6c15c5dcde0a516946e8baac68a15d3aeef3aa119b63e8eb5578`. +Before any model run, a semantic review made eight declared changes only: + +1. canonicalized entity domains and deterministic Home Assistant state fields; +2. added one unrelated control entity per base for over-binding tests; +3. normalized synthetic support entity IDs to Home Assistant-safe identifiers; +4. aligned climate fixtures with Home Assistant's state-as-HVAC-mode contract; +5. added operation-relevant light and cover capability metadata; +6. replaced one unsafe implicit multi-mode climate turn-on with an explicitly + confirmed HVAC mode; +7. rewrote twelve correction probes whose final wording had already removed the + ambiguity; and +8. made confirmation, context dependencies and controlled deltas explicit. + +The resulting evaluation bytes were then frozen at SHA-256 +`dfdf7a0f40ec7df111ca56e5443b90aa6958f5a0bb7eca0eea0a63fa5f8d4d50`. +`freeze.json` also records the development and combined-file hashes. Any change +to the evaluation bytes invalidates the formal run; failed or inconvenient rows +remain in the denominator. + +The downstream grounding policy, evaluator, fixtures and regression tests were +subsequently developed with the 48 scenario commands, structures and expected +labels visible. This split is therefore **not an unseen holdout for the runtime +policy**. The one-shot property applies only to the model raw-output evidence: +after the code checkpoint is frozen, all 96 clear/ambiguous probes are generated +once, without inspecting or selectively rerunning their outputs. + +The registered model artifact is independently pinned in +`snapshot_manifest.json`: all thirteen root files, byte sizes, SHA-256 hashes, +Hub etags and the exact Hub commit are recorded. `freeze.json` binds that +manifest file, while the runner verifies the per-file Hub download metadata and +bytes before loading a model. + +The records were created for this case with AI-assisted drafting and deterministic +validation. They are not derived from the official 4,057-row Domux test set, a +private conversation, or real household telemetry. The submitter reviews the +final data and evidence before publication. + +## Schema + +Each JSONL record contains: + +- synthetic entity inventory and a controlled initial-state projection; +- session context with only synthetic prior turns and entity IDs; +- paired clear and ambiguous commands; +- displayed clarification answer and candidate IDs (maximum three); +- the confirmed seven-slot instruction and intended entity; +- exact before/after controlled projection; +- context state dependencies and one unrelated control entity; and +- split, category, schema version and license metadata. + +Volatile Home Assistant fields such as timestamps and context IDs are excluded. +Light projections use `state`, `brightness`, `rgb_color` and +`color_temp_kelvin`; cover projections use `state` and `current_position`; +climate projections use Home Assistant's HVAC mode in `state`, plus +`temperature` and `fan_mode` from attributes. + +## Intended use + +The data measures whether a downstream integration: + +- distinguishes the ambiguous probe from its explicit matched control; +- keeps the intended entity inside a deterministic set of at most three choices; +- performs no service call before clarification and confirmation; +- executes the confirmed state delta; and +- rejects a pre-declared replay, expiry, state or candidate change without + over-binding unrelated state. + +It is not a general Domux accuracy benchmark and it does not estimate ambiguity +prevalence in household traffic. It is a deterministic conformance suite for +the recorded formal v1 run. Its raw model outputs and report are never edited, +omitted, or selectively rerun. The final top-level integration was improved +after inspecting the v1 failures, so it is a reference implementation rather +than a basis for relabelling the original v1 result. The retained scripts can +run a fresh end-to-end observation; they do not claim byte-identical +reproduction of the historical v1 policy execution. + +## Safety and privacy + +Only synthetic Light, Curtain and AC entities are present. Locks, alarms, gas, +doors, cameras, health devices and real hardware are excluded. The data contains +no token, endpoint, private address, personal filesystem path, real user name, +real home layout or business prompt. + +## License + +To the extent copyright applies, the original synthetic records and this data +card are offered under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). +This does not change the licenses of Domux, Gemma, Home Assistant or any runtime. + +## Limitations + +- Forty-eight evaluation bases give wide confidence intervals, even for 48/48. +- Wilson intervals and exact McNemar results describe outcomes on this fixed, + policy-informed suite only; they do not establish population generalization. +- The 1:1 clear/ambiguous construction does not represent production prevalence, + so measured precision is not a production positive predictive value. +- A passing B2 quality gate means only that the predeclared clean and guard + checks passed on this suite; it is not a whole-model or production-safety + certification. +- Clarification answers are controlled selections, not a user study or an ASR + evaluation. +- Synthetic entity names and state transitions cannot establish production + safety, accessibility, exactly-once delivery or real-hardware reliability. diff --git a/cases/domux-clarify-commit/data/freeze.json b/cases/domux-clarify-commit/data/freeze.json new file mode 100644 index 0000000..66ab743 --- /dev/null +++ b/cases/domux-clarify-commit/data/freeze.json @@ -0,0 +1,23 @@ +{ + "schema_version": 1, + "data_version": "2026-08-26.7", + "full_file": "scenarios.jsonl", + "full_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "development_count": 16, + "development_sha256": "b6e4155835dfe430320282316bd107b8bbec7d2b6e95db9cd7af069c7cb4bace", + "evaluation_count": 48, + "evaluation_sha256": "dfdf7a0f40ec7df111ca56e5443b90aa6958f5a0bb7eca0eea0a63fa5f8d4d50", + "independent_candidate_sha256": "e81b7879760e6c15c5dcde0a516946e8baac68a15d3aeef3aa119b63e8eb5578", + "pre_model_reviewed_eval_sha256": "dfdf7a0f40ec7df111ca56e5443b90aa6958f5a0bb7eca0eea0a63fa5f8d4d50", + "protocol_file": "protocol.json", + "protocol_sha256": "e0b1c0d1db4237311707401f469f5cb69a581a711fce6d6a314172a9449641e0", + "snapshot_manifest_file": "snapshot_manifest.json", + "snapshot_manifest_sha256": "85ffc0d612758c2dfb1cde6b42623a8f793ad5f4e42e6734694b0c2107389200", + "evaluation_category_counts": { + "context_reference": 12, + "duplicate_entity": 12, + "missing_slot": 12, + "negation_correction": 12 + }, + "status": "pre_frozen_before_any_model_output" +} diff --git a/cases/domux-clarify-commit/data/protocol.json b/cases/domux-clarify-commit/data/protocol.json new file mode 100644 index 0000000..18f95c8 --- /dev/null +++ b/cases/domux-clarify-commit/data/protocol.json @@ -0,0 +1,127 @@ +{ + "schema_version": 1, + "protocol_id": "clarify-commit-v1", + "model": { + "id": "iFlytekOpenSource/Domux", + "revision": "6c71a32f4d624cadfd9fce9d10240d8068e53456", + "snapshot_manifest_file": "snapshot_manifest.json", + "snapshot_files_sha256": "5a13462b24fc9b00d132c42718e037bc42fc51a3c6752041998e085579f01416", + "primary_precision": "bf16", + "do_sample": false, + "temperature": 0, + "max_new_tokens": 128, + "seed": 20260826, + "warmup": { + "count": 2, + "source": "independent commands excluded from both data splits" + } + }, + "analysis_population": { + "unit": "base_id", + "development_bases": 16, + "evaluation_bases": 48, + "paired_probes_per_base": ["clear", "ambiguous"], + "rule": "No evaluation base may be removed, relabelled or selectively rerun after formal inference starts." + }, + "arms": { + "B0_unique_or_abstain": { + "grounding": true, + "clarification": false, + "behavior_on_multiple_candidates": "safe_abstain", + "executor_and_postcondition": "shared" + }, + "B1_clarify_and_prepare": { + "grounding": true, + "clarification": true, + "server_plan_binding": true, + "session_binding": true, + "candidate_revalidation": false, + "state_revalidation": false, + "ttl": false, + "one_time_nonce": false, + "executor_and_postcondition": "shared" + }, + "B2_clarify_and_commit": { + "grounding": true, + "clarification": true, + "server_plan_binding": true, + "session_binding": true, + "candidate_revalidation": true, + "state_revalidation": true, + "ttl_seconds": 30, + "one_time_nonce": true, + "executor_and_postcondition": "shared" + } + }, + "trial_reset": [ + "create a fresh adapter from the frozen fixture", + "compare its controlled projection hash with the frozen initial projection", + "create a fresh prepared action", + "apply exactly one declared mutation", + "commit and record SUT calls separately from setup calls", + "read all controlled entities and compare the expected delta" + ], + "mutations": { + "clean": {"eligible_bases": 48, "desired": "one dispatch and exact delta"}, + "replay": {"eligible_bases": 48, "desired": "second submission rejected with zero additional calls"}, + "expiry": {"eligible_bases": 48, "desired": "reject after expires_at plus one logical tick"}, + "target_drift": {"eligible_bases": 48, "desired": "reject after one controlled target field changes"}, + "session_swap": {"eligible_bases": 48, "desired": "reject alternate pre-declared session"}, + "plan_swap": {"eligible_bases": 48, "desired": "reject one valid but different plan digest"}, + "candidate_change": {"eligible_bases": 48, "desired": "reject one candidate metadata change"}, + "context_state_change": {"eligible_category": "context_reference", "eligible_bases": 12, "desired": "reject a bound context-source state change"}, + "unrelated_state_change": {"eligible_bases": 48, "desired": "allow and complete despite an unrelated controlled state change"} + }, + "language_metrics": { + "sensitivity": "ambiguous probes correctly sent to clarification / 48", + "specificity": "clear controls correctly treated as unique / 48", + "false_clarification_rate": "clear controls incorrectly sent to clarification / 48", + "paired_discrimination": "bases where ambiguous is clarified and clear is unique / 48", + "candidate_coverage": "bases where expected target is present and candidate count is one to three / 48", + "clarification_completion": "bases whose confirmed canonical plan reaches the intended entity and delta / 48", + "warning": "The deliberately balanced 1:1 probes do not estimate production PPV." + }, + "execution_metrics": { + "dispatch_coverage": "bases with the expected SUT dispatch / 48", + "exact_delta_success": "bases with exact target delta and no other controlled entity change / 48", + "wrong_target_transition": "bases with any unapproved controlled entity transition / 48", + "zero_preconfirm_calls": "bases with zero SUT calls before explicit confirmation / 48", + "universal_guard_rate": "per-base conjunction of replay, expiry, target drift, session, plan, candidate and unrelated-state outcomes / 48", + "context_guard_rate": "context-state mutation outcome / 12" + }, + "statistics": { + "binary_intervals": "Wilson two-sided 95%, z=1.959963984540054", + "paired_test": "two-sided exact McNemar using discordant base counts", + "primary_comparisons": [ + "B1 versus B0 ambiguous clean exact-delta success", + "B2 versus B1 universal guard pass" + ], + "multiplicity": "Holm correction across the two primary comparisons", + "latency_unit": "median of the clear and ambiguous latency within each base, then median and p95 across 48 bases", + "pseudo_replication_guards": [ + "two probes do not become n=96 independent bases", + "three arms reuse the same model output", + "mutations do not become n=384 independent bases", + "multi-line instructions, service calls and state events are not samples" + ] + }, + "pilot_gate": { + "base_ids": [ + "dev-duplicate_entity-01", + "dev-duplicate_entity-02", + "dev-duplicate_entity-03", + "dev-missing_slot-01", + "dev-missing_slot-02", + "dev-missing_slot-03", + "dev-context_reference-01", + "dev-context_reference-02", + "dev-context_reference-03", + "dev-negation_correction-01", + "dev-negation_correction-02", + "dev-negation_correction-03" + ], + "full_4057": "Only required if a quantized or trained model variant changes model outputs.", + "quantization": "Only proceed after the complete core E2E passes and a same-hardware matched run can finish.", + "training": "Only proceed for a repeated action/domain parsing failure, never for missing runtime inventory or state grounding." + } +} diff --git a/cases/domux-clarify-commit/data/scenarios.jsonl b/cases/domux-clarify-commit/data/scenarios.jsonl new file mode 100644 index 0000000..c5e1e16 --- /dev/null +++ b/cases/domux-clarify-commit/data/scenarios.jsonl @@ -0,0 +1,64 @@ +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Switch off the light.","base_id":"dev-duplicate_entity-01","candidate_entity_ids":["light.dev_duplicate_entity_01_1","light.dev_duplicate_entity_01_2"],"category":"duplicate_entity","clarification_answer":"The Workshop light.","clear_command":"Turn off the Workshop light on the Ground Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"Ground Floor","room":"Workshop","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_01_2","state":"off"},"before":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_01_2","state":"on"},"entity_id":"light.dev_duplicate_entity_01_2"},"expected_target_entity":"light.dev_duplicate_entity_01_2","initial_states":{"cover.support_dev_duplicate_entity_01":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.support_dev_duplicate_entity_01","state":"open"},"light.dev_duplicate_entity_01_1":{"attributes":{"brightness":119,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_01_1","state":"on"},"light.dev_duplicate_entity_01_2":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_01_2","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_duplicate_entity_01_1","floor":"Ground Floor","room":"Atrium"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_duplicate_entity_01_2","floor":"Ground Floor","room":"Workshop"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_dev_duplicate_entity_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"cover.support_dev_duplicate_entity_01"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the curtain to 45 percent.","base_id":"dev-duplicate_entity-02","candidate_entity_ids":["cover.dev_duplicate_entity_02_1","cover.dev_duplicate_entity_02_2"],"category":"duplicate_entity","clarification_answer":"The Music Room curtain.","clear_command":"Set the Music Room curtain on the First Floor to 45 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"First Floor","room":"Music Room","unit":"Percent","value":"45"},"expected_delta":{"after":{"attributes":{"current_position":45,"supported_features":7},"entity_id":"cover.dev_duplicate_entity_02_1","state":"open"},"before":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.dev_duplicate_entity_02_1","state":"open"},"entity_id":"cover.dev_duplicate_entity_02_1"},"expected_target_entity":"cover.dev_duplicate_entity_02_1","initial_states":{"cover.dev_duplicate_entity_02_1":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.dev_duplicate_entity_02_1","state":"open"},"cover.dev_duplicate_entity_02_2":{"attributes":{"current_position":65,"supported_features":7},"entity_id":"cover.dev_duplicate_entity_02_2","state":"open"},"light.support_dev_duplicate_entity_02":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_duplicate_entity_02","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.dev_duplicate_entity_02_1","floor":"First Floor","room":"Music Room"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.dev_duplicate_entity_02_2","floor":"First Floor","room":"Hobby Room"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_duplicate_entity_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_duplicate_entity_02"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the lab AC to 26 degrees.","base_id":"dev-duplicate_entity-03","candidate_entity_ids":["climate.dev_duplicate_entity_03_1","climate.dev_duplicate_entity_03_2"],"category":"duplicate_entity","clarification_answer":"The South Lab AC, Celsius.","clear_command":"Set the South Lab AC on the Second Floor to 26 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"Second Floor","room":"South Lab","unit":"Celsius","value":"26"},"expected_delta":{"after":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":26.0,"temperature_unit":"°C"},"entity_id":"climate.dev_duplicate_entity_03_2","state":"cool"},"before":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_duplicate_entity_03_2","state":"cool"},"entity_id":"climate.dev_duplicate_entity_03_2"},"expected_target_entity":"climate.dev_duplicate_entity_03_2","initial_states":{"climate.dev_duplicate_entity_03_1":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.dev_duplicate_entity_03_1","state":"cool"},"climate.dev_duplicate_entity_03_2":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_duplicate_entity_03_2","state":"cool"},"light.support_dev_duplicate_entity_03":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_duplicate_entity_03","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_duplicate_entity_03_1","floor":"Second Floor","room":"North Lab"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_duplicate_entity_03_2","floor":"Second Floor","room":"South Lab"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_duplicate_entity_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_duplicate_entity_03"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the light to 25 percent.","base_id":"dev-duplicate_entity-04","candidate_entity_ids":["light.dev_duplicate_entity_04_1","light.dev_duplicate_entity_04_2"],"category":"duplicate_entity","clarification_answer":"The Loft light.","clear_command":"Set the Loft light on the Third Floor to 25 percent brightness.","confirmed_instruction":{"action":"set","attribute":"brightness","device":"Light","floor":"Third Floor","room":"Loft","unit":"Percent","value":"25"},"expected_delta":{"after":{"attributes":{"brightness":64,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_04_1","state":"on"},"before":{"attributes":{"brightness":170,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_04_1","state":"off"},"entity_id":"light.dev_duplicate_entity_04_1"},"expected_target_entity":"light.dev_duplicate_entity_04_1","initial_states":{"cover.support_dev_duplicate_entity_04":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.support_dev_duplicate_entity_04","state":"open"},"light.dev_duplicate_entity_04_1":{"attributes":{"brightness":170,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_04_1","state":"off"},"light.dev_duplicate_entity_04_2":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_duplicate_entity_04_2","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_duplicate_entity_04_1","floor":"Third Floor","room":"Loft"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_duplicate_entity_04_2","floor":"Ground Floor","room":"Pantry"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_dev_duplicate_entity_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"cover.support_dev_duplicate_entity_04"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set the Test Lab AC.","base_id":"dev-missing_slot-01","candidate_entity_ids":["climate.dev_missing_slot_01_1"],"category":"missing_slot","clarification_answer":"Temperature 22 Celsius.","clear_command":"Set the Test Lab AC on the Ground Floor to 22 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"Ground Floor","room":"Test Lab","unit":"Celsius","value":"22"},"expected_delta":{"after":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_missing_slot_01_1","state":"cool"},"before":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.dev_missing_slot_01_1","state":"cool"},"entity_id":"climate.dev_missing_slot_01_1"},"expected_target_entity":"climate.dev_missing_slot_01_1","initial_states":{"climate.dev_missing_slot_01_1":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.dev_missing_slot_01_1","state":"cool"},"light.support_dev_missing_slot_01":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_missing_slot_01","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_missing_slot_01_1","floor":"Ground Floor","room":"Test Lab"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_missing_slot_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_missing_slot_01"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Move the Sunroom curtain.","base_id":"dev-missing_slot-02","candidate_entity_ids":["cover.dev_missing_slot_02_1"],"category":"missing_slot","clarification_answer":"Open it to 35 percent.","clear_command":"Set the Sunroom curtain to 35 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"Sunroom","unit":"Percent","value":"35"},"expected_delta":{"after":{"attributes":{"current_position":35,"supported_features":7},"entity_id":"cover.dev_missing_slot_02_1","state":"open"},"before":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.dev_missing_slot_02_1","state":"open"},"entity_id":"cover.dev_missing_slot_02_1"},"expected_target_entity":"cover.dev_missing_slot_02_1","initial_states":{"cover.dev_missing_slot_02_1":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.dev_missing_slot_02_1","state":"open"},"light.support_dev_missing_slot_02":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_missing_slot_02","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.dev_missing_slot_02_1","floor":"Ground Floor","room":"Sunroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_missing_slot_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_missing_slot_02"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set brightness to 40 percent.","base_id":"dev-missing_slot-03","candidate_entity_ids":["light.dev_missing_slot_03_1","light.dev_missing_slot_03_2"],"category":"missing_slot","clarification_answer":"The Gallery light.","clear_command":"Set the Gallery light to 40 percent brightness.","confirmed_instruction":{"action":"set","attribute":"brightness","device":"Light","floor":"Ground Floor","room":"Gallery","unit":"Percent","value":"40"},"expected_delta":{"after":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_missing_slot_03_2","state":"on"},"before":{"attributes":{"brightness":170,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_missing_slot_03_2","state":"off"},"entity_id":"light.dev_missing_slot_03_2"},"expected_target_entity":"light.dev_missing_slot_03_2","initial_states":{"cover.support_dev_missing_slot_03":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.support_dev_missing_slot_03","state":"open"},"light.dev_missing_slot_03_1":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_missing_slot_03_1","state":"on"},"light.dev_missing_slot_03_2":{"attributes":{"brightness":170,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_missing_slot_03_2","state":"off"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_missing_slot_03_1","floor":"Basement","room":"Archive"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_missing_slot_03_2","floor":"Ground Floor","room":"Gallery"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_dev_missing_slot_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"cover.support_dev_missing_slot_03"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Change the Server Room AC mode.","base_id":"dev-missing_slot-04","candidate_entity_ids":["climate.dev_missing_slot_04_1"],"category":"missing_slot","clarification_answer":"Use Heat mode.","clear_command":"Set the Server Room AC to Heat mode.","confirmed_instruction":{"action":"set","attribute":"mode","device":"AC","floor":"Basement","room":"Server Room","unit":"*","value":"Heat"},"expected_delta":{"after":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_missing_slot_04_1","state":"heat"},"before":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_missing_slot_04_1","state":"cool"},"entity_id":"climate.dev_missing_slot_04_1"},"expected_target_entity":"climate.dev_missing_slot_04_1","initial_states":{"climate.dev_missing_slot_04_1":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_missing_slot_04_1","state":"cool"},"light.support_dev_missing_slot_04":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_missing_slot_04","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_missing_slot_04_1","floor":"Basement","room":"Server Room"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_missing_slot_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_missing_slot_04"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Please switch that device off.","base_id":"dev-context_reference-01","candidate_entity_ids":["light.dev_context_reference_01_1","light.dev_context_reference_01_2"],"category":"context_reference","clarification_answer":"The Library light.","clear_command":"Turn off the Library light.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"First Floor","room":"Library","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_01_2","state":"off"},"before":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_01_2","state":"on"},"entity_id":"light.dev_context_reference_01_2"},"expected_target_entity":"light.dev_context_reference_01_2","initial_states":{"cover.support_dev_context_reference_01":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.support_dev_context_reference_01","state":"open"},"light.dev_context_reference_01_1":{"attributes":{"brightness":119,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_01_1","state":"on"},"light.dev_context_reference_01_2":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_01_2","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_context_reference_01_1","floor":"First Floor","room":"Map Room"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_context_reference_01_2","floor":"First Floor","room":"Library"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_dev_context_reference_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.dev_context_reference_01_1","light.dev_context_reference_01_2"],"prior_turns":[]},"split":"dev","state_dependencies":["light.dev_context_reference_01_1","light.dev_context_reference_01_2"],"unrelated_entity_id":"cover.support_dev_context_reference_01"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Set that one to 55 percent.","base_id":"dev-context_reference-02","candidate_entity_ids":["cover.dev_context_reference_02_1","cover.dev_context_reference_02_2"],"category":"context_reference","clarification_answer":"The East Hall curtain.","clear_command":"Set the East Hall curtain to 55 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"East Hall","unit":"Percent","value":"55"},"expected_delta":{"after":{"attributes":{"current_position":55,"supported_features":7},"entity_id":"cover.dev_context_reference_02_1","state":"open"},"before":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.dev_context_reference_02_1","state":"open"},"entity_id":"cover.dev_context_reference_02_1"},"expected_target_entity":"cover.dev_context_reference_02_1","initial_states":{"cover.dev_context_reference_02_1":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.dev_context_reference_02_1","state":"open"},"cover.dev_context_reference_02_2":{"attributes":{"current_position":65,"supported_features":7},"entity_id":"cover.dev_context_reference_02_2","state":"open"},"light.support_dev_context_reference_02":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_context_reference_02","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.dev_context_reference_02_1","floor":"Ground Floor","room":"East Hall"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.dev_context_reference_02_2","floor":"Ground Floor","room":"West Hall"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_context_reference_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["cover.dev_context_reference_02_1","cover.dev_context_reference_02_2"],"prior_turns":[]},"split":"dev","state_dependencies":["cover.dev_context_reference_02_1","cover.dev_context_reference_02_2"],"unrelated_entity_id":"light.support_dev_context_reference_02"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Set the other one to 23.","base_id":"dev-context_reference-03","candidate_entity_ids":["climate.dev_context_reference_03_1","climate.dev_context_reference_03_2"],"category":"context_reference","clarification_answer":"The Suite B AC, Celsius.","clear_command":"Set the Suite B AC to 23 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"Second Floor","room":"Suite B","unit":"Celsius","value":"23"},"expected_delta":{"after":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.dev_context_reference_03_2","state":"cool"},"before":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_context_reference_03_2","state":"cool"},"entity_id":"climate.dev_context_reference_03_2"},"expected_target_entity":"climate.dev_context_reference_03_2","initial_states":{"climate.dev_context_reference_03_1":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.dev_context_reference_03_1","state":"cool"},"climate.dev_context_reference_03_2":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_context_reference_03_2","state":"cool"},"light.support_dev_context_reference_03":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_context_reference_03","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_context_reference_03_1","floor":"Second Floor","room":"Suite A"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_context_reference_03_2","floor":"Second Floor","room":"Suite B"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_context_reference_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["climate.dev_context_reference_03_1","climate.dev_context_reference_03_2"],"prior_turns":[]},"split":"dev","state_dependencies":["climate.dev_context_reference_03_1","climate.dev_context_reference_03_2"],"unrelated_entity_id":"light.support_dev_context_reference_03"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Turn that on.","base_id":"dev-context_reference-04","candidate_entity_ids":["light.dev_context_reference_04_1","light.dev_context_reference_04_2"],"category":"context_reference","clarification_answer":"The Studio East light.","clear_command":"Turn on the Studio East light.","confirmed_instruction":{"action":"turnOn","attribute":"*","device":"Light","floor":"Ground Floor","room":"Studio East","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":170,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_04_1","state":"on"},"before":{"attributes":{"brightness":170,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_04_1","state":"off"},"entity_id":"light.dev_context_reference_04_1"},"expected_target_entity":"light.dev_context_reference_04_1","initial_states":{"cover.support_dev_context_reference_04":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.support_dev_context_reference_04","state":"open"},"light.dev_context_reference_04_1":{"attributes":{"brightness":170,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_04_1","state":"off"},"light.dev_context_reference_04_2":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_context_reference_04_2","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_context_reference_04_1","floor":"Ground Floor","room":"Studio East"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_context_reference_04_2","floor":"Ground Floor","room":"Studio West"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_dev_context_reference_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.dev_context_reference_04_1","light.dev_context_reference_04_2"],"prior_turns":[]},"split":"dev","state_dependencies":["light.dev_context_reference_04_1","light.dev_context_reference_04_2"],"unrelated_entity_id":"cover.support_dev_context_reference_04"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Turn off the Breakfast Room light—or the Dining Room light; confirm which.","base_id":"dev-negation_correction-01","candidate_entity_ids":["light.dev_negation_correction_01_1","light.dev_negation_correction_01_2"],"category":"negation_correction","clarification_answer":"The Dining Room light.","clear_command":"Turn off the Dining Room light.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"Ground Floor","room":"Dining Room","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_negation_correction_01_2","state":"off"},"before":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_negation_correction_01_2","state":"on"},"entity_id":"light.dev_negation_correction_01_2"},"expected_target_entity":"light.dev_negation_correction_01_2","initial_states":{"cover.support_dev_negation_correction_01":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.support_dev_negation_correction_01","state":"open"},"light.dev_negation_correction_01_1":{"attributes":{"brightness":119,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_negation_correction_01_1","state":"on"},"light.dev_negation_correction_01_2":{"attributes":{"brightness":136,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.dev_negation_correction_01_2","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_negation_correction_01_1","floor":"Ground Floor","room":"Breakfast Room"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.dev_negation_correction_01_2","floor":"Ground Floor","room":"Dining Room"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_dev_negation_correction_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"cover.support_dev_negation_correction_01"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the Meeting Room AC to 22—or perhaps 24; confirm the value.","base_id":"dev-negation_correction-02","candidate_entity_ids":["climate.dev_negation_correction_02_1"],"category":"negation_correction","clarification_answer":"Use 24 Celsius.","clear_command":"Set the Meeting Room AC to 24 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"First Floor","room":"Meeting Room","unit":"Celsius","value":"24"},"expected_delta":{"after":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.dev_negation_correction_02_1","state":"cool"},"before":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.dev_negation_correction_02_1","state":"cool"},"entity_id":"climate.dev_negation_correction_02_1"},"expected_target_entity":"climate.dev_negation_correction_02_1","initial_states":{"climate.dev_negation_correction_02_1":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.dev_negation_correction_02_1","state":"cool"},"light.support_dev_negation_correction_02":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_negation_correction_02","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_negation_correction_02_1","floor":"First Floor","room":"Meeting Room"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_negation_correction_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_negation_correction_02"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the Courtyard curtain to 30—or perhaps 70 percent; confirm.","base_id":"dev-negation_correction-03","candidate_entity_ids":["cover.dev_negation_correction_03_1"],"category":"negation_correction","clarification_answer":"Use 70 percent.","clear_command":"Set the Courtyard curtain to 70 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"Courtyard","unit":"Percent","value":"70"},"expected_delta":{"after":{"attributes":{"current_position":70,"supported_features":7},"entity_id":"cover.dev_negation_correction_03_1","state":"open"},"before":{"attributes":{"current_position":65,"supported_features":7},"entity_id":"cover.dev_negation_correction_03_1","state":"open"},"entity_id":"cover.dev_negation_correction_03_1"},"expected_target_entity":"cover.dev_negation_correction_03_1","initial_states":{"cover.dev_negation_correction_03_1":{"attributes":{"current_position":65,"supported_features":7},"entity_id":"cover.dev_negation_correction_03_1","state":"open"},"light.support_dev_negation_correction_03":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_negation_correction_03","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.dev_negation_correction_03_1","floor":"Ground Floor","room":"Courtyard"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_negation_correction_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_negation_correction_03"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the Workshop AC to Cool—or perhaps Heat; confirm the mode.","base_id":"dev-negation_correction-04","candidate_entity_ids":["climate.dev_negation_correction_04_1"],"category":"negation_correction","clarification_answer":"Use Cool mode.","clear_command":"Set the Workshop AC to Cool mode.","confirmed_instruction":{"action":"set","attribute":"mode","device":"AC","floor":"Basement","room":"Workshop","unit":"*","value":"Cool"},"expected_delta":{"after":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_negation_correction_04_1","state":"cool"},"before":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_negation_correction_04_1","state":"heat"},"entity_id":"climate.dev_negation_correction_04_1"},"expected_target_entity":"climate.dev_negation_correction_04_1","initial_states":{"climate.dev_negation_correction_04_1":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.dev_negation_correction_04_1","state":"heat"},"light.support_dev_negation_correction_04":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_dev_negation_correction_04","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.dev_negation_correction_04_1","floor":"Basement","room":"Workshop"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_dev_negation_correction_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"dev","state_dependencies":[],"unrelated_entity_id":"light.support_dev_negation_correction_04"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Turn off the light.","base_id":"eval-duplicate_entity-01","candidate_entity_ids":["light.eval_de_01_living","light.eval_de_01_bedroom","light.eval_de_01_kitchen"],"category":"duplicate_entity","clarification_answer":"The Living Room light on the Ground Floor.","clear_command":"Turn off the Light in the Living Room on the Ground Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"Ground Floor","room":"Living Room","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_01_living","state":"off"},"before":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_01_living","state":"on"},"entity_id":"light.eval_de_01_living"},"expected_target_entity":"light.eval_de_01_living","initial_states":{"cover.support_duplicate_entity_01":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_duplicate_entity_01","state":"closed"},"light.eval_de_01_bedroom":{"attributes":{"brightness":115,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_01_bedroom","state":"on"},"light.eval_de_01_kitchen":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_01_kitchen","state":"off"},"light.eval_de_01_living":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_01_living","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_01_living","floor":"Ground Floor","room":"Living Room"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_01_bedroom","floor":"Ground Floor","room":"Bedroom"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_01_kitchen","floor":"Ground Floor","room":"Kitchen"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_duplicate_entity_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_duplicate_entity_01"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the hall curtain to 20 percent.","base_id":"eval-duplicate_entity-02","candidate_entity_ids":["cover.eval_de_02_upstairs_hall","cover.eval_de_02_downstairs_hall"],"category":"duplicate_entity","clarification_answer":"The Hall curtain on the First Floor.","clear_command":"Set the Curtain in the Hall on the First Floor to 20 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"First Floor","room":"Hall","unit":"Percent","value":"20"},"expected_delta":{"after":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.eval_de_02_upstairs_hall","state":"open"},"before":{"attributes":{"current_position":80,"supported_features":7},"entity_id":"cover.eval_de_02_upstairs_hall","state":"open"},"entity_id":"cover.eval_de_02_upstairs_hall"},"expected_target_entity":"cover.eval_de_02_upstairs_hall","initial_states":{"cover.eval_de_02_downstairs_hall":{"attributes":{"current_position":55,"supported_features":7},"entity_id":"cover.eval_de_02_downstairs_hall","state":"open"},"cover.eval_de_02_upstairs_hall":{"attributes":{"current_position":80,"supported_features":7},"entity_id":"cover.eval_de_02_upstairs_hall","state":"open"},"light.support_duplicate_entity_02":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_02","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_02_upstairs_hall","floor":"First Floor","room":"Hall"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_02_downstairs_hall","floor":"Ground Floor","room":"Hall"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_02"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the bedroom AC to 22 degrees.","base_id":"eval-duplicate_entity-03","candidate_entity_ids":["climate.eval_de_03_bedroom_first","climate.eval_de_03_bedroom_second"],"category":"duplicate_entity","clarification_answer":"The Bedroom AC on the Second Floor, in Celsius.","clear_command":"Set the AC in the Bedroom on the Second Floor to 22 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"Second Floor","room":"Bedroom","unit":"Celsius","value":"22"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_03_bedroom_second","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_03_bedroom_second","state":"cool"},"entity_id":"climate.eval_de_03_bedroom_second"},"expected_target_entity":"climate.eval_de_03_bedroom_second","initial_states":{"climate.eval_de_03_bedroom_first":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_03_bedroom_first","state":"cool"},"climate.eval_de_03_bedroom_second":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_03_bedroom_second","state":"cool"},"light.support_duplicate_entity_03":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_03","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_03_bedroom_first","floor":"First Floor","room":"Bedroom"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_03_bedroom_second","floor":"Second Floor","room":"Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_03"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the reading light to 35 percent brightness.","base_id":"eval-duplicate_entity-04","candidate_entity_ids":["light.eval_de_04_office","light.eval_de_04_study","light.eval_de_04_den"],"category":"duplicate_entity","clarification_answer":"Use the Light in the Study on the Ground Floor.","clear_command":"Set the Light in the Study on the Ground Floor to 35 percent brightness.","confirmed_instruction":{"action":"set","attribute":"brightness","device":"Light","floor":"Ground Floor","room":"Study","unit":"Percent","value":"35"},"expected_delta":{"after":{"attributes":{"brightness":89,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_04_study","state":"on"},"before":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_04_study","state":"on"},"entity_id":"light.eval_de_04_study"},"expected_target_entity":"light.eval_de_04_study","initial_states":{"cover.support_duplicate_entity_04":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_duplicate_entity_04","state":"closed"},"light.eval_de_04_den":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_04_den","state":"off"},"light.eval_de_04_office":{"attributes":{"brightness":204,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_04_office","state":"on"},"light.eval_de_04_study":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_04_study","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_04_office","floor":"Ground Floor","room":"Office"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_04_study","floor":"Ground Floor","room":"Study"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_04_den","floor":"Ground Floor","room":"Den"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_duplicate_entity_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_duplicate_entity_04"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Open the balcony curtain to 70 percent.","base_id":"eval-duplicate_entity-05","candidate_entity_ids":["cover.eval_de_05_north_balcony","cover.eval_de_05_south_balcony"],"category":"duplicate_entity","clarification_answer":"The North Balcony curtain on the First Floor.","clear_command":"Set the Curtain in the North Balcony on the First Floor to 70 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"First Floor","room":"North Balcony","unit":"Percent","value":"70"},"expected_delta":{"after":{"attributes":{"current_position":70,"supported_features":7},"entity_id":"cover.eval_de_05_north_balcony","state":"open"},"before":{"attributes":{"current_position":10,"supported_features":7},"entity_id":"cover.eval_de_05_north_balcony","state":"open"},"entity_id":"cover.eval_de_05_north_balcony"},"expected_target_entity":"cover.eval_de_05_north_balcony","initial_states":{"cover.eval_de_05_north_balcony":{"attributes":{"current_position":10,"supported_features":7},"entity_id":"cover.eval_de_05_north_balcony","state":"open"},"cover.eval_de_05_south_balcony":{"attributes":{"current_position":35,"supported_features":7},"entity_id":"cover.eval_de_05_south_balcony","state":"open"},"light.support_duplicate_entity_05":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_05","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_05_north_balcony","floor":"First Floor","room":"North Balcony"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_05_south_balcony","floor":"First Floor","room":"South Balcony"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_05","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_05"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the guest bedroom AC to Dry mode.","base_id":"eval-duplicate_entity-06","candidate_entity_ids":["climate.eval_de_06_guest_ground","climate.eval_de_06_guest_first"],"category":"duplicate_entity","clarification_answer":"The Guest Bedroom AC on the Ground Floor.","clear_command":"Set the AC in the Guest Bedroom on the Ground Floor to Dry mode.","confirmed_instruction":{"action":"set","attribute":"mode","device":"AC","floor":"Ground Floor","room":"Guest Bedroom","unit":"*","value":"Dry"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_06_guest_ground","state":"dry"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_06_guest_ground","state":"cool"},"entity_id":"climate.eval_de_06_guest_ground"},"expected_target_entity":"climate.eval_de_06_guest_ground","initial_states":{"climate.eval_de_06_guest_first":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_06_guest_first","state":"heat"},"climate.eval_de_06_guest_ground":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_06_guest_ground","state":"cool"},"light.support_duplicate_entity_06":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_06","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_06_guest_ground","floor":"Ground Floor","room":"Guest Bedroom"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_06_guest_first","floor":"First Floor","room":"Guest Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_06","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_06"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Turn on the staircase light.","base_id":"eval-duplicate_entity-07","candidate_entity_ids":["light.eval_de_07_stair_ground","light.eval_de_07_stair_first","light.eval_de_07_stair_second"],"category":"duplicate_entity","clarification_answer":"The Staircase light on the First Floor.","clear_command":"Turn on the Light in the Staircase on the First Floor.","confirmed_instruction":{"action":"turnOn","attribute":"*","device":"Light","floor":"First Floor","room":"Staircase","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":140,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_07_stair_first","state":"on"},"before":{"attributes":{"brightness":140,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_07_stair_first","state":"off"},"entity_id":"light.eval_de_07_stair_first"},"expected_target_entity":"light.eval_de_07_stair_first","initial_states":{"cover.support_duplicate_entity_07":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_duplicate_entity_07","state":"closed"},"light.eval_de_07_stair_first":{"attributes":{"brightness":140,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_07_stair_first","state":"off"},"light.eval_de_07_stair_ground":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_07_stair_ground","state":"off"},"light.eval_de_07_stair_second":{"attributes":{"brightness":115,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_07_stair_second","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_07_stair_ground","floor":"Ground Floor","room":"Staircase"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_07_stair_first","floor":"First Floor","room":"Staircase"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_07_stair_second","floor":"Second Floor","room":"Staircase"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_duplicate_entity_07","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_duplicate_entity_07"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the main curtain to 75 percent.","base_id":"eval-duplicate_entity-08","candidate_entity_ids":["cover.eval_de_08_lounge","cover.eval_de_08_family","cover.eval_de_08_media"],"category":"duplicate_entity","clarification_answer":"The Curtain in the Media Room on the Ground Floor.","clear_command":"Set the Curtain in the Media Room on the Ground Floor to 75 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"Media Room","unit":"Percent","value":"75"},"expected_delta":{"after":{"attributes":{"current_position":75,"supported_features":7},"entity_id":"cover.eval_de_08_media","state":"open"},"before":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.eval_de_08_media","state":"open"},"entity_id":"cover.eval_de_08_media"},"expected_target_entity":"cover.eval_de_08_media","initial_states":{"cover.eval_de_08_family":{"attributes":{"current_position":45,"supported_features":7},"entity_id":"cover.eval_de_08_family","state":"open"},"cover.eval_de_08_lounge":{"attributes":{"current_position":30,"supported_features":7},"entity_id":"cover.eval_de_08_lounge","state":"open"},"cover.eval_de_08_media":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.eval_de_08_media","state":"open"},"light.support_duplicate_entity_08":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_08","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_08_lounge","floor":"Ground Floor","room":"Lounge"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_08_family","floor":"Ground Floor","room":"Family Room"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_08_media","floor":"Ground Floor","room":"Media Room"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_08","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_08"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Turn off the studio AC.","base_id":"eval-duplicate_entity-09","candidate_entity_ids":["climate.eval_de_09_studio_a","climate.eval_de_09_studio_b","climate.eval_de_09_studio_c"],"category":"duplicate_entity","clarification_answer":"The AC in Studio B on the Second Floor.","clear_command":"Turn off the AC in Studio B on the Second Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"AC","floor":"Second Floor","room":"Studio B","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_09_studio_b","state":"off"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_09_studio_b","state":"cool"},"entity_id":"climate.eval_de_09_studio_b"},"expected_target_entity":"climate.eval_de_09_studio_b","initial_states":{"climate.eval_de_09_studio_a":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_09_studio_a","state":"cool"},"climate.eval_de_09_studio_b":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_09_studio_b","state":"cool"},"climate.eval_de_09_studio_c":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_09_studio_c","state":"off"},"light.support_duplicate_entity_09":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_09","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_09_studio_a","floor":"Second Floor","room":"Studio A"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_09_studio_b","floor":"Second Floor","room":"Studio B"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_09_studio_c","floor":"Second Floor","room":"Studio C"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_09","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_09"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Make the light Warm White.","base_id":"eval-duplicate_entity-10","candidate_entity_ids":["light.eval_de_10_pantry","light.eval_de_10_kitchen","light.eval_de_10_dining"],"category":"duplicate_entity","clarification_answer":"The Dining Room light on the Ground Floor.","clear_command":"Set the Light in the Dining Room on the Ground Floor to Warm White.","confirmed_instruction":{"action":"set","attribute":"color","device":"Light","floor":"Ground Floor","room":"Dining Room","unit":"*","value":"Warm White"},"expected_delta":{"after":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[255,244,229],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_10_dining","state":"on"},"before":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[0,0,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_10_dining","state":"on"},"entity_id":"light.eval_de_10_dining"},"expected_target_entity":"light.eval_de_10_dining","initial_states":{"cover.support_duplicate_entity_10":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_duplicate_entity_10","state":"closed"},"light.eval_de_10_dining":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[0,0,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_10_dining","state":"on"},"light.eval_de_10_kitchen":{"attributes":{"brightness":191,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[201,226,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_10_kitchen","state":"on"},"light.eval_de_10_pantry":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[255,255,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_de_10_pantry","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_10_pantry","floor":"Ground Floor","room":"Pantry"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_10_kitchen","floor":"Ground Floor","room":"Kitchen"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_de_10_dining","floor":"Ground Floor","room":"Dining Room"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_duplicate_entity_10","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_duplicate_entity_10"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Close the bedroom curtain.","base_id":"eval-duplicate_entity-11","candidate_entity_ids":["cover.eval_de_11_bedroom_1","cover.eval_de_11_bedroom_2","cover.eval_de_11_bedroom_3"],"category":"duplicate_entity","clarification_answer":"The Curtain in Bedroom 2 on the First Floor.","clear_command":"Close the Curtain in Bedroom 2 on the First Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Curtain","floor":"First Floor","room":"Bedroom 2","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.eval_de_11_bedroom_2","state":"closed"},"before":{"attributes":{"current_position":90,"supported_features":7},"entity_id":"cover.eval_de_11_bedroom_2","state":"open"},"entity_id":"cover.eval_de_11_bedroom_2"},"expected_target_entity":"cover.eval_de_11_bedroom_2","initial_states":{"cover.eval_de_11_bedroom_1":{"attributes":{"current_position":65,"supported_features":7},"entity_id":"cover.eval_de_11_bedroom_1","state":"open"},"cover.eval_de_11_bedroom_2":{"attributes":{"current_position":90,"supported_features":7},"entity_id":"cover.eval_de_11_bedroom_2","state":"open"},"cover.eval_de_11_bedroom_3":{"attributes":{"current_position":40,"supported_features":7},"entity_id":"cover.eval_de_11_bedroom_3","state":"open"},"light.support_duplicate_entity_11":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_11","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_11_bedroom_1","floor":"First Floor","room":"Bedroom 1"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_11_bedroom_2","floor":"First Floor","room":"Bedroom 2"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_de_11_bedroom_3","floor":"First Floor","room":"Bedroom 3"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_11","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_11"} +{"ambiguity_dimensions":["entity"],"ambiguity_expected":true,"ambiguous_command":"Set the office AC to 25 degrees.","base_id":"eval-duplicate_entity-12","candidate_entity_ids":["climate.eval_de_12_east_office","climate.eval_de_12_west_office","climate.eval_de_12_open_office"],"category":"duplicate_entity","clarification_answer":"The West Office AC on the First Floor, in Celsius.","clear_command":"Set the AC in the West Office on the First Floor to 25 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"First Floor","room":"West Office","unit":"Celsius","value":"25"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_12_west_office","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":21.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_12_west_office","state":"cool"},"entity_id":"climate.eval_de_12_west_office"},"expected_target_entity":"climate.eval_de_12_west_office","initial_states":{"climate.eval_de_12_east_office":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_12_east_office","state":"cool"},"climate.eval_de_12_open_office":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_12_open_office","state":"fan_only"},"climate.eval_de_12_west_office":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":21.0,"temperature_unit":"°C"},"entity_id":"climate.eval_de_12_west_office","state":"cool"},"light.support_duplicate_entity_12":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_duplicate_entity_12","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_12_east_office","floor":"First Floor","room":"East Office"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_12_west_office","floor":"First Floor","room":"West Office"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_de_12_open_office","floor":"First Floor","room":"Open Office"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_duplicate_entity_12","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_duplicate_entity_12"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set the light to 40 percent.","base_id":"eval-missing_slot-01","candidate_entity_ids":["light.eval_ms_01_kitchen","light.eval_ms_01_bedroom"],"category":"missing_slot","clarification_answer":"Set the Kitchen light on the Ground Floor to 40 percent brightness.","clear_command":"Set the Light in the Kitchen on the Ground Floor to 40 percent brightness.","confirmed_instruction":{"action":"set","attribute":"brightness","device":"Light","floor":"Ground Floor","room":"Kitchen","unit":"Percent","value":"40"},"expected_delta":{"after":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_01_kitchen","state":"on"},"before":{"attributes":{"brightness":191,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_01_kitchen","state":"on"},"entity_id":"light.eval_ms_01_kitchen"},"expected_target_entity":"light.eval_ms_01_kitchen","initial_states":{"cover.support_missing_slot_01":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_missing_slot_01","state":"closed"},"light.eval_ms_01_bedroom":{"attributes":{"brightness":140,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_01_bedroom","state":"on"},"light.eval_ms_01_kitchen":{"attributes":{"brightness":191,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_01_kitchen","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_01_kitchen","floor":"Ground Floor","room":"Kitchen"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_01_bedroom","floor":"Ground Floor","room":"Bedroom"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_missing_slot_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_missing_slot_01"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set the office AC.","base_id":"eval-missing_slot-02","candidate_entity_ids":["climate.eval_ms_02_office"],"category":"missing_slot","clarification_answer":"Set its temperature to 23 Celsius.","clear_command":"Set the AC in the Office on the First Floor to 23 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"First Floor","room":"Office","unit":"Celsius","value":"23"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_02_office","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":26.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_02_office","state":"cool"},"entity_id":"climate.eval_ms_02_office"},"expected_target_entity":"climate.eval_ms_02_office","initial_states":{"climate.eval_ms_02_office":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":26.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_02_office","state":"cool"},"cover.support_missing_slot_02":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_missing_slot_02","state":"closed"},"light.eval_ms_02_office":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_02_office","state":"on"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_ms_02_office","floor":"First Floor","room":"Office"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_02_office","floor":"First Floor","room":"Office"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_missing_slot_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_missing_slot_02"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Adjust the bedroom curtain.","base_id":"eval-missing_slot-03","candidate_entity_ids":["cover.eval_ms_03_bedroom"],"category":"missing_slot","clarification_answer":"Open it to 65 percent; use the Bedroom curtain on the Second Floor.","clear_command":"Set the Curtain in the Bedroom on the Second Floor to 65 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Second Floor","room":"Bedroom","unit":"Percent","value":"65"},"expected_delta":{"after":{"attributes":{"current_position":65,"supported_features":7},"entity_id":"cover.eval_ms_03_bedroom","state":"open"},"before":{"attributes":{"current_position":25,"supported_features":7},"entity_id":"cover.eval_ms_03_bedroom","state":"open"},"entity_id":"cover.eval_ms_03_bedroom"},"expected_target_entity":"cover.eval_ms_03_bedroom","initial_states":{"climate.support_missing_slot_03":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_missing_slot_03","state":"cool"},"cover.eval_ms_03_bedroom":{"attributes":{"current_position":25,"supported_features":7},"entity_id":"cover.eval_ms_03_bedroom","state":"open"},"light.eval_ms_03_bedroom":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_03_bedroom","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_ms_03_bedroom","floor":"Second Floor","room":"Bedroom"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_03_bedroom","floor":"Second Floor","room":"Bedroom"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_missing_slot_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"climate.support_missing_slot_03"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Change the living room light color.","base_id":"eval-missing_slot-04","candidate_entity_ids":["light.eval_ms_04_living"],"category":"missing_slot","clarification_answer":"Change it to Blue.","clear_command":"Set the Light in the Living Room on the Ground Floor to Blue.","confirmed_instruction":{"action":"set","attribute":"color","device":"Light","floor":"Ground Floor","room":"Living Room","unit":"*","value":"Blue"},"expected_delta":{"after":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[0,0,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_04_living","state":"on"},"before":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[255,255,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_04_living","state":"on"},"entity_id":"light.eval_ms_04_living"},"expected_target_entity":"light.eval_ms_04_living","initial_states":{"climate.support_missing_slot_04":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_missing_slot_04","state":"cool"},"cover.eval_ms_04_living":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.eval_ms_04_living","state":"open"},"light.eval_ms_04_living":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[255,255,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_04_living","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_04_living","floor":"Ground Floor","room":"Living Room"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_ms_04_living","floor":"Ground Floor","room":"Living Room"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_missing_slot_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"climate.support_missing_slot_04"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set the upstairs AC to 21 degrees.","base_id":"eval-missing_slot-05","candidate_entity_ids":["climate.eval_ms_05_upstairs"],"category":"missing_slot","clarification_answer":"Use 21 Celsius for the Landing AC on the First Floor.","clear_command":"Set the AC in the Landing on the First Floor to 21 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"First Floor","room":"Landing","unit":"Celsius","value":"21"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":21.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_05_upstairs","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_05_upstairs","state":"cool"},"entity_id":"climate.eval_ms_05_upstairs"},"expected_target_entity":"climate.eval_ms_05_upstairs","initial_states":{"climate.eval_ms_05_upstairs":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_05_upstairs","state":"cool"},"cover.support_missing_slot_05":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_missing_slot_05","state":"closed"},"light.eval_ms_05_upstairs":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_05_upstairs","state":"on"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_ms_05_upstairs","floor":"First Floor","room":"Landing"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_05_upstairs","floor":"First Floor","room":"Landing"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_missing_slot_05","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_missing_slot_05"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Turn off the device in the study.","base_id":"eval-missing_slot-06","candidate_entity_ids":["light.eval_ms_06_study","climate.eval_ms_06_study"],"category":"missing_slot","clarification_answer":"Turn off the Study AC on the Ground Floor, not the light.","clear_command":"Turn off the AC in the Study on the Ground Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"AC","floor":"Ground Floor","room":"Study","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_06_study","state":"off"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_06_study","state":"cool"},"entity_id":"climate.eval_ms_06_study"},"expected_target_entity":"climate.eval_ms_06_study","initial_states":{"climate.eval_ms_06_study":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_06_study","state":"cool"},"cover.support_missing_slot_06":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_missing_slot_06","state":"closed"},"light.eval_ms_06_study":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_06_study","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_06_study","floor":"Ground Floor","room":"Study"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_ms_06_study","floor":"Ground Floor","room":"Study"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_missing_slot_06","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_missing_slot_06"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set the curtain to 30 percent.","base_id":"eval-missing_slot-07","candidate_entity_ids":["cover.eval_ms_07_patio","cover.eval_ms_07_lounge"],"category":"missing_slot","clarification_answer":"The Patio curtain on the Ground Floor.","clear_command":"Set the Curtain in the Patio on the Ground Floor to 30 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"Patio","unit":"Percent","value":"30"},"expected_delta":{"after":{"attributes":{"current_position":30,"supported_features":7},"entity_id":"cover.eval_ms_07_patio","state":"open"},"before":{"attributes":{"current_position":85,"supported_features":7},"entity_id":"cover.eval_ms_07_patio","state":"open"},"entity_id":"cover.eval_ms_07_patio"},"expected_target_entity":"cover.eval_ms_07_patio","initial_states":{"cover.eval_ms_07_lounge":{"attributes":{"current_position":60,"supported_features":7},"entity_id":"cover.eval_ms_07_lounge","state":"open"},"cover.eval_ms_07_patio":{"attributes":{"current_position":85,"supported_features":7},"entity_id":"cover.eval_ms_07_patio","state":"open"},"light.support_missing_slot_07":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_missing_slot_07","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_ms_07_patio","floor":"Ground Floor","room":"Patio"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_ms_07_lounge","floor":"Ground Floor","room":"Lounge"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_missing_slot_07","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_missing_slot_07"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set brightness to 55 percent.","base_id":"eval-missing_slot-08","candidate_entity_ids":["light.eval_ms_08_hall","light.eval_ms_08_entry","light.eval_ms_08_lounge"],"category":"missing_slot","clarification_answer":"Use the Hall light on the Ground Floor.","clear_command":"Set the Light in the Hall on the Ground Floor to 55 percent brightness.","confirmed_instruction":{"action":"set","attribute":"brightness","device":"Light","floor":"Ground Floor","room":"Hall","unit":"Percent","value":"55"},"expected_delta":{"after":{"attributes":{"brightness":140,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_08_hall","state":"on"},"before":{"attributes":{"brightness":204,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_08_hall","state":"on"},"entity_id":"light.eval_ms_08_hall"},"expected_target_entity":"light.eval_ms_08_hall","initial_states":{"cover.support_missing_slot_08":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_missing_slot_08","state":"closed"},"light.eval_ms_08_entry":{"attributes":{"brightness":115,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_08_entry","state":"on"},"light.eval_ms_08_hall":{"attributes":{"brightness":204,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_08_hall","state":"on"},"light.eval_ms_08_lounge":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_08_lounge","state":"off"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_08_hall","floor":"Ground Floor","room":"Hall"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_08_entry","floor":"Ground Floor","room":"Entry"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_08_lounge","floor":"Ground Floor","room":"Lounge"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_missing_slot_08","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_missing_slot_08"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Change the bedroom AC mode.","base_id":"eval-missing_slot-09","candidate_entity_ids":["climate.eval_ms_09_bedroom"],"category":"missing_slot","clarification_answer":"Set it to Heat mode.","clear_command":"Set the AC in the Bedroom on the First Floor to Heat mode.","confirmed_instruction":{"action":"set","attribute":"mode","device":"AC","floor":"First Floor","room":"Bedroom","unit":"*","value":"Heat"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":20.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_09_bedroom","state":"heat"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":20.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_09_bedroom","state":"cool"},"entity_id":"climate.eval_ms_09_bedroom"},"expected_target_entity":"climate.eval_ms_09_bedroom","initial_states":{"climate.eval_ms_09_bedroom":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":20.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_09_bedroom","state":"cool"},"cover.eval_ms_09_bedroom":{"attributes":{"current_position":40,"supported_features":7},"entity_id":"cover.eval_ms_09_bedroom","state":"open"},"light.support_missing_slot_09":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_missing_slot_09","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_ms_09_bedroom","floor":"First Floor","room":"Bedroom"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_ms_09_bedroom","floor":"First Floor","room":"Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_missing_slot_09","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_missing_slot_09"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Move the nursery curtain.","base_id":"eval-missing_slot-10","candidate_entity_ids":["cover.eval_ms_10_nursery"],"category":"missing_slot","clarification_answer":"Open the Nursery curtain to 80 percent.","clear_command":"Set the Curtain in the Nursery on the First Floor to 80 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"First Floor","room":"Nursery","unit":"Percent","value":"80"},"expected_delta":{"after":{"attributes":{"current_position":80,"supported_features":7},"entity_id":"cover.eval_ms_10_nursery","state":"open"},"before":{"attributes":{"current_position":35,"supported_features":7},"entity_id":"cover.eval_ms_10_nursery","state":"open"},"entity_id":"cover.eval_ms_10_nursery"},"expected_target_entity":"cover.eval_ms_10_nursery","initial_states":{"climate.support_missing_slot_10":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_missing_slot_10","state":"cool"},"cover.eval_ms_10_nursery":{"attributes":{"current_position":35,"supported_features":7},"entity_id":"cover.eval_ms_10_nursery","state":"open"},"light.eval_ms_10_nursery":{"attributes":{"brightness":76,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_10_nursery","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_ms_10_nursery","floor":"First Floor","room":"Nursery"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_10_nursery","floor":"First Floor","room":"Nursery"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_missing_slot_10","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"climate.support_missing_slot_10"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set the light color temperature to 3500K.","base_id":"eval-missing_slot-11","candidate_entity_ids":["light.eval_ms_11_entry","light.eval_ms_11_hall"],"category":"missing_slot","clarification_answer":"The Entry light on the Ground Floor.","clear_command":"Set the Light in the Entry on the Ground Floor to 3500 Kelvin.","confirmed_instruction":{"action":"set","attribute":"colorTemperature","device":"Light","floor":"Ground Floor","room":"Entry","unit":"Kelvin","value":"3500"},"expected_delta":{"after":{"attributes":{"brightness":153,"color_temp_kelvin":3500,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_11_entry","state":"on"},"before":{"attributes":{"brightness":153,"color_temp_kelvin":4500,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_11_entry","state":"on"},"entity_id":"light.eval_ms_11_entry"},"expected_target_entity":"light.eval_ms_11_entry","initial_states":{"cover.support_missing_slot_11":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_missing_slot_11","state":"closed"},"light.eval_ms_11_entry":{"attributes":{"brightness":153,"color_temp_kelvin":4500,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_11_entry","state":"on"},"light.eval_ms_11_hall":{"attributes":{"brightness":178,"color_temp_kelvin":4000,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_ms_11_hall","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_11_entry","floor":"Ground Floor","room":"Entry"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_ms_11_hall","floor":"Ground Floor","room":"Hall"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_missing_slot_11","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_missing_slot_11"} +{"ambiguity_dimensions":["entity_or_required_slot"],"ambiguity_expected":true,"ambiguous_command":"Set the downstairs temperature to 24.","base_id":"eval-missing_slot-12","candidate_entity_ids":["climate.eval_ms_12_dining","climate.eval_ms_12_lounge"],"category":"missing_slot","clarification_answer":"Set the Dining Room AC on the Ground Floor to 24 Celsius.","clear_command":"Set the AC in the Dining Room on the Ground Floor to 24 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"Ground Floor","room":"Dining Room","unit":"Celsius","value":"24"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_12_dining","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":26.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_12_dining","state":"cool"},"entity_id":"climate.eval_ms_12_dining"},"expected_target_entity":"climate.eval_ms_12_dining","initial_states":{"climate.eval_ms_12_dining":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":26.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_12_dining","state":"cool"},"climate.eval_ms_12_lounge":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.eval_ms_12_lounge","state":"cool"},"light.support_missing_slot_12":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_missing_slot_12","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_ms_12_dining","floor":"Ground Floor","room":"Dining Room"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_ms_12_lounge","floor":"Ground Floor","room":"Lounge"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_missing_slot_12","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_missing_slot_12"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Turn it off.","base_id":"eval-context_reference-01","candidate_entity_ids":["light.eval_cr_01_living","cover.eval_cr_01_living"],"category":"context_reference","clarification_answer":"Turn off the Living Room light, not the curtain.","clear_command":"Switch off the Ground Floor Light in the Living Room.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"Ground Floor","room":"Living Room","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_01_living","state":"off"},"before":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_01_living","state":"on"},"entity_id":"light.eval_cr_01_living"},"expected_target_entity":"light.eval_cr_01_living","initial_states":{"climate.support_context_reference_01":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_context_reference_01","state":"cool"},"cover.eval_cr_01_living":{"attributes":{"current_position":100,"supported_features":7},"entity_id":"cover.eval_cr_01_living","state":"open"},"light.eval_cr_01_living":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_01_living","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_01_living","floor":"Ground Floor","room":"Living Room"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_01_living","floor":"Ground Floor","room":"Living Room"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_context_reference_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.eval_cr_01_living","cover.eval_cr_01_living"],"prior_turns":["The Living Room light is on, and the Living Room curtain is fully open."]},"split":"eval","state_dependencies":["light.eval_cr_01_living","cover.eval_cr_01_living"],"unrelated_entity_id":"climate.support_context_reference_01"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Set that one to 22.","base_id":"eval-context_reference-02","candidate_entity_ids":["climate.eval_cr_02_office","climate.eval_cr_02_guest"],"category":"context_reference","clarification_answer":"The Office AC on the First Floor, and use Celsius.","clear_command":"Set the AC in the Office on the First Floor to 22 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"First Floor","room":"Office","unit":"Celsius","value":"22"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":22.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_02_office","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_02_office","state":"cool"},"entity_id":"climate.eval_cr_02_office"},"expected_target_entity":"climate.eval_cr_02_office","initial_states":{"climate.eval_cr_02_guest":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_02_guest","state":"cool"},"climate.eval_cr_02_office":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_02_office","state":"cool"},"light.support_context_reference_02":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_context_reference_02","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_cr_02_office","floor":"First Floor","room":"Office"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_cr_02_guest","floor":"First Floor","room":"Guest Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_context_reference_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["climate.eval_cr_02_office","climate.eval_cr_02_guest"],"prior_turns":["Compare the Office AC with the Guest Bedroom AC; both are on."]},"split":"eval","state_dependencies":["climate.eval_cr_02_office","climate.eval_cr_02_guest"],"unrelated_entity_id":"light.support_context_reference_02"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Open the other one halfway.","base_id":"eval-context_reference-03","candidate_entity_ids":["cover.eval_cr_03_east","cover.eval_cr_03_west"],"category":"context_reference","clarification_answer":"By the other one, I mean the East Bedroom curtain on the Second Floor.","clear_command":"Set the Curtain in the East Bedroom on the Second Floor to 50 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Second Floor","room":"East Bedroom","unit":"Percent","value":"50"},"expected_delta":{"after":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.eval_cr_03_east","state":"open"},"before":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.eval_cr_03_east","state":"open"},"entity_id":"cover.eval_cr_03_east"},"expected_target_entity":"cover.eval_cr_03_east","initial_states":{"cover.eval_cr_03_east":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.eval_cr_03_east","state":"open"},"cover.eval_cr_03_west":{"attributes":{"current_position":80,"supported_features":7},"entity_id":"cover.eval_cr_03_west","state":"open"},"light.support_context_reference_03":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_context_reference_03","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_03_east","floor":"Second Floor","room":"East Bedroom"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_03_west","floor":"Second Floor","room":"West Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_context_reference_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["cover.eval_cr_03_east","cover.eval_cr_03_west"],"prior_turns":["The East Bedroom curtain is at 20 percent and the West Bedroom curtain is at 80 percent."]},"split":"eval","state_dependencies":["cover.eval_cr_03_east","cover.eval_cr_03_west"],"unrelated_entity_id":"light.support_context_reference_03"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Set it to 50 percent.","base_id":"eval-context_reference-04","candidate_entity_ids":["light.eval_cr_04_office","cover.eval_cr_04_office"],"category":"context_reference","clarification_answer":"Set the Office light brightness to 50 percent.","clear_command":"Set the Light in the Office on the Ground Floor to 50 percent brightness.","confirmed_instruction":{"action":"set","attribute":"brightness","device":"Light","floor":"Ground Floor","room":"Office","unit":"Percent","value":"50"},"expected_delta":{"after":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_04_office","state":"on"},"before":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_04_office","state":"on"},"entity_id":"light.eval_cr_04_office"},"expected_target_entity":"light.eval_cr_04_office","initial_states":{"climate.support_context_reference_04":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_context_reference_04","state":"cool"},"cover.eval_cr_04_office":{"attributes":{"current_position":10,"supported_features":7},"entity_id":"cover.eval_cr_04_office","state":"open"},"light.eval_cr_04_office":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_04_office","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_04_office","floor":"Ground Floor","room":"Office"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_04_office","floor":"Ground Floor","room":"Office"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_context_reference_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.eval_cr_04_office","cover.eval_cr_04_office"],"prior_turns":["Show the Office light brightness and the Office curtain position."]},"split":"eval","state_dependencies":["light.eval_cr_04_office","cover.eval_cr_04_office"],"unrelated_entity_id":"climate.support_context_reference_04"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Turn the one in the middle off.","base_id":"eval-context_reference-05","candidate_entity_ids":["light.eval_cr_05_hall","light.eval_cr_05_kitchen","light.eval_cr_05_bedroom"],"category":"context_reference","clarification_answer":"I mean the Bedroom light on the First Floor.","clear_command":"Turn off the Light in the Bedroom on the First Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"First Floor","room":"Bedroom","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_05_bedroom","state":"off"},"before":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_05_bedroom","state":"on"},"entity_id":"light.eval_cr_05_bedroom"},"expected_target_entity":"light.eval_cr_05_bedroom","initial_states":{"cover.support_context_reference_05":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_context_reference_05","state":"closed"},"light.eval_cr_05_bedroom":{"attributes":{"brightness":102,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_05_bedroom","state":"on"},"light.eval_cr_05_hall":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_05_hall","state":"on"},"light.eval_cr_05_kitchen":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_05_kitchen","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_05_hall","floor":"Ground Floor","room":"Hall"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_05_kitchen","floor":"Ground Floor","room":"Kitchen"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_05_bedroom","floor":"First Floor","room":"Bedroom"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_context_reference_05","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.eval_cr_05_hall","light.eval_cr_05_kitchen","light.eval_cr_05_bedroom"],"prior_turns":["We discussed the Hall, Kitchen, and Bedroom lights in different orders."]},"split":"eval","state_dependencies":["light.eval_cr_05_hall","light.eval_cr_05_kitchen","light.eval_cr_05_bedroom"],"unrelated_entity_id":"cover.support_context_reference_05"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Set that one to Cool.","base_id":"eval-context_reference-06","candidate_entity_ids":["climate.eval_cr_06_north","climate.eval_cr_06_south"],"category":"context_reference","clarification_answer":"The South Office AC on the First Floor.","clear_command":"Set the AC in the South Office on the First Floor to Cool mode.","confirmed_instruction":{"action":"set","attribute":"mode","device":"AC","floor":"First Floor","room":"South Office","unit":"*","value":"Cool"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_06_south","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_06_south","state":"dry"},"entity_id":"climate.eval_cr_06_south"},"expected_target_entity":"climate.eval_cr_06_south","initial_states":{"climate.eval_cr_06_north":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_06_north","state":"heat"},"climate.eval_cr_06_south":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_06_south","state":"dry"},"light.support_context_reference_06":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_context_reference_06","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_cr_06_north","floor":"First Floor","room":"North Office"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_cr_06_south","floor":"First Floor","room":"South Office"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_context_reference_06","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["climate.eval_cr_06_north","climate.eval_cr_06_south"],"prior_turns":["The North Office AC and South Office AC are both active."]},"split":"eval","state_dependencies":["climate.eval_cr_06_north","climate.eval_cr_06_south"],"unrelated_entity_id":"light.support_context_reference_06"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Close the one we just talked about.","base_id":"eval-context_reference-07","candidate_entity_ids":["cover.eval_cr_07_office","cover.eval_cr_07_guest"],"category":"context_reference","clarification_answer":"Close the Office curtain on the Ground Floor.","clear_command":"Close the Curtain in the Office on the Ground Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Curtain","floor":"Ground Floor","room":"Office","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.eval_cr_07_office","state":"closed"},"before":{"attributes":{"current_position":90,"supported_features":7},"entity_id":"cover.eval_cr_07_office","state":"open"},"entity_id":"cover.eval_cr_07_office"},"expected_target_entity":"cover.eval_cr_07_office","initial_states":{"cover.eval_cr_07_guest":{"attributes":{"current_position":75,"supported_features":7},"entity_id":"cover.eval_cr_07_guest","state":"open"},"cover.eval_cr_07_office":{"attributes":{"current_position":90,"supported_features":7},"entity_id":"cover.eval_cr_07_office","state":"open"},"light.support_context_reference_07":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_context_reference_07","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_07_office","floor":"Ground Floor","room":"Office"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_07_guest","floor":"Ground Floor","room":"Guest Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_context_reference_07","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["cover.eval_cr_07_office","cover.eval_cr_07_guest"],"prior_turns":["Check both the Office curtain and Guest Bedroom curtain before we continue."]},"split":"eval","state_dependencies":["cover.eval_cr_07_office","cover.eval_cr_07_guest"],"unrelated_entity_id":"light.support_context_reference_07"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Open that one to 60 percent.","base_id":"eval-context_reference-08","candidate_entity_ids":["cover.eval_cr_08_east","cover.eval_cr_08_west","cover.eval_cr_08_north"],"category":"context_reference","clarification_answer":"The West Lounge curtain on the Ground Floor.","clear_command":"Set the Curtain in the West Lounge on the Ground Floor to 60 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"West Lounge","unit":"Percent","value":"60"},"expected_delta":{"after":{"attributes":{"current_position":60,"supported_features":7},"entity_id":"cover.eval_cr_08_west","state":"open"},"before":{"attributes":{"current_position":40,"supported_features":7},"entity_id":"cover.eval_cr_08_west","state":"open"},"entity_id":"cover.eval_cr_08_west"},"expected_target_entity":"cover.eval_cr_08_west","initial_states":{"cover.eval_cr_08_east":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.eval_cr_08_east","state":"open"},"cover.eval_cr_08_north":{"attributes":{"current_position":80,"supported_features":7},"entity_id":"cover.eval_cr_08_north","state":"open"},"cover.eval_cr_08_west":{"attributes":{"current_position":40,"supported_features":7},"entity_id":"cover.eval_cr_08_west","state":"open"},"light.support_context_reference_08":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_context_reference_08","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_08_east","floor":"Ground Floor","room":"East Lounge"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_08_west","floor":"Ground Floor","room":"West Lounge"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_08_north","floor":"Ground Floor","room":"North Lounge"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_context_reference_08","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["cover.eval_cr_08_east","cover.eval_cr_08_west","cover.eval_cr_08_north"],"prior_turns":["The East, West, and North Lounge curtains all need review."]},"split":"eval","state_dependencies":["cover.eval_cr_08_east","cover.eval_cr_08_west","cover.eval_cr_08_north"],"unrelated_entity_id":"light.support_context_reference_08"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Raise the other one to 25.","base_id":"eval-context_reference-09","candidate_entity_ids":["climate.eval_cr_09_basement","climate.eval_cr_09_attic"],"category":"context_reference","clarification_answer":"The Attic AC on the Third Floor, to 25 Celsius.","clear_command":"Set the AC in the Attic on the Third Floor to 25 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"Third Floor","room":"Attic","unit":"Celsius","value":"25"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":25.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_09_attic","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_09_attic","state":"cool"},"entity_id":"climate.eval_cr_09_attic"},"expected_target_entity":"climate.eval_cr_09_attic","initial_states":{"climate.eval_cr_09_attic":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_09_attic","state":"cool"},"climate.eval_cr_09_basement":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":20.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_09_basement","state":"dry"},"light.support_context_reference_09":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_context_reference_09","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_cr_09_basement","floor":"Basement","room":"Storage Room"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_cr_09_attic","floor":"Third Floor","room":"Attic"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_context_reference_09","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["climate.eval_cr_09_basement","climate.eval_cr_09_attic"],"prior_turns":["Compare the Storage Room AC with the Attic AC."]},"split":"eval","state_dependencies":["climate.eval_cr_09_basement","climate.eval_cr_09_attic"],"unrelated_entity_id":"light.support_context_reference_09"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Make that one warmer.","base_id":"eval-context_reference-10","candidate_entity_ids":["light.eval_cr_10_reading","light.eval_cr_10_library"],"category":"context_reference","clarification_answer":"Set the Reading Room light to 3000 Kelvin.","clear_command":"Set the Light in the Reading Room on the First Floor to 3000 Kelvin.","confirmed_instruction":{"action":"set","attribute":"colorTemperature","device":"Light","floor":"First Floor","room":"Reading Room","unit":"Kelvin","value":"3000"},"expected_delta":{"after":{"attributes":{"brightness":140,"color_temp_kelvin":3000,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_10_reading","state":"on"},"before":{"attributes":{"brightness":140,"color_temp_kelvin":4200,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_10_reading","state":"on"},"entity_id":"light.eval_cr_10_reading"},"expected_target_entity":"light.eval_cr_10_reading","initial_states":{"cover.support_context_reference_10":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_context_reference_10","state":"closed"},"light.eval_cr_10_library":{"attributes":{"brightness":166,"color_temp_kelvin":3800,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_10_library","state":"on"},"light.eval_cr_10_reading":{"attributes":{"brightness":140,"color_temp_kelvin":4200,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_10_reading","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_10_reading","floor":"First Floor","room":"Reading Room"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_10_library","floor":"First Floor","room":"Library"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_context_reference_10","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.eval_cr_10_reading","light.eval_cr_10_library"],"prior_turns":["Look at the Reading Room light and the Library light."]},"split":"eval","state_dependencies":["light.eval_cr_10_reading","light.eval_cr_10_library"],"unrelated_entity_id":"cover.support_context_reference_10"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Turn it on.","base_id":"eval-context_reference-11","candidate_entity_ids":["light.eval_cr_11_sunroom","cover.eval_cr_11_sunroom"],"category":"context_reference","clarification_answer":"Open the Sunroom curtain, not the light.","clear_command":"Open the Curtain in the Sunroom on the Ground Floor.","confirmed_instruction":{"action":"turnOn","attribute":"*","device":"Curtain","floor":"Ground Floor","room":"Sunroom","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"current_position":100,"supported_features":7},"entity_id":"cover.eval_cr_11_sunroom","state":"open"},"before":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.eval_cr_11_sunroom","state":"closed"},"entity_id":"cover.eval_cr_11_sunroom"},"expected_target_entity":"cover.eval_cr_11_sunroom","initial_states":{"climate.support_context_reference_11":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_context_reference_11","state":"cool"},"cover.eval_cr_11_sunroom":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.eval_cr_11_sunroom","state":"closed"},"light.eval_cr_11_sunroom":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_11_sunroom","state":"off"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_11_sunroom","floor":"Ground Floor","room":"Sunroom"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_cr_11_sunroom","floor":"Ground Floor","room":"Sunroom"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_context_reference_11","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.eval_cr_11_sunroom","cover.eval_cr_11_sunroom"],"prior_turns":["The Sunroom light is off and the Sunroom curtain is closed."]},"split":"eval","state_dependencies":["light.eval_cr_11_sunroom","cover.eval_cr_11_sunroom"],"unrelated_entity_id":"climate.support_context_reference_11"} +{"ambiguity_dimensions":["context_entity"],"ambiguity_expected":true,"ambiguous_command":"Turn that off.","base_id":"eval-context_reference-12","candidate_entity_ids":["light.eval_cr_12_study","climate.eval_cr_12_study"],"category":"context_reference","clarification_answer":"Turn off the Study light on the Second Floor.","clear_command":"Turn off the Light in the Study on the Second Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"Second Floor","room":"Study","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_12_study","state":"off"},"before":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_12_study","state":"on"},"entity_id":"light.eval_cr_12_study"},"expected_target_entity":"light.eval_cr_12_study","initial_states":{"climate.eval_cr_12_study":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_cr_12_study","state":"cool"},"cover.support_context_reference_12":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_context_reference_12","state":"closed"},"light.eval_cr_12_study":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_cr_12_study","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_cr_12_study","floor":"Second Floor","room":"Study"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_cr_12_study","floor":"Second Floor","room":"Study"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_context_reference_12","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":["light.eval_cr_12_study","climate.eval_cr_12_study"],"prior_turns":["The Study light and Study AC on the Second Floor are both running."]},"split":"eval","state_dependencies":["light.eval_cr_12_study","climate.eval_cr_12_study"],"unrelated_entity_id":"cover.support_context_reference_12"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Turn off the bedroom light—or perhaps the living room light; I am not sure which one.","base_id":"eval-negation_correction-01","candidate_entity_ids":["light.eval_nc_01_bedroom","light.eval_nc_01_living"],"category":"negation_correction","clarification_answer":"Confirm the Bedroom light on the Ground Floor.","clear_command":"Turn off the Light in the Bedroom on the Ground Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"Ground Floor","room":"Bedroom","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_01_bedroom","state":"off"},"before":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_01_bedroom","state":"on"},"entity_id":"light.eval_nc_01_bedroom"},"expected_target_entity":"light.eval_nc_01_bedroom","initial_states":{"cover.support_negation_correction_01":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_negation_correction_01","state":"closed"},"light.eval_nc_01_bedroom":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_01_bedroom","state":"on"},"light.eval_nc_01_living":{"attributes":{"brightness":178,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_01_living","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_01_bedroom","floor":"Ground Floor","room":"Bedroom"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_01_living","floor":"Ground Floor","room":"Living Room"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_negation_correction_01","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_negation_correction_01"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the office AC to 22—no, perhaps 24; please check which value I want.","base_id":"eval-negation_correction-02","candidate_entity_ids":["climate.eval_nc_02_office"],"category":"negation_correction","clarification_answer":"Use 24 Celsius for the Office AC.","clear_command":"Set the AC in the Office on the First Floor to 24 Celsius.","confirmed_instruction":{"action":"set","attribute":"temperature","device":"AC","floor":"First Floor","room":"Office","unit":"Celsius","value":"24"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_02_office","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":26.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_02_office","state":"cool"},"entity_id":"climate.eval_nc_02_office"},"expected_target_entity":"climate.eval_nc_02_office","initial_states":{"climate.eval_nc_02_office":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":26.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_02_office","state":"cool"},"cover.support_negation_correction_02":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_negation_correction_02","state":"closed"},"light.eval_nc_02_office":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_02_office","state":"on"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_nc_02_office","floor":"First Floor","room":"Office"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_02_office","floor":"First Floor","room":"Office"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_negation_correction_02","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_negation_correction_02"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Close the patio curtain—or did I mean the balcony curtain? Do not choose for me.","base_id":"eval-negation_correction-03","candidate_entity_ids":["cover.eval_nc_03_patio","cover.eval_nc_03_balcony"],"category":"negation_correction","clarification_answer":"Close the Balcony curtain on the First Floor.","clear_command":"Close the Curtain in the Balcony on the First Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Curtain","floor":"First Floor","room":"Balcony","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.eval_nc_03_balcony","state":"closed"},"before":{"attributes":{"current_position":90,"supported_features":7},"entity_id":"cover.eval_nc_03_balcony","state":"open"},"entity_id":"cover.eval_nc_03_balcony"},"expected_target_entity":"cover.eval_nc_03_balcony","initial_states":{"cover.eval_nc_03_balcony":{"attributes":{"current_position":90,"supported_features":7},"entity_id":"cover.eval_nc_03_balcony","state":"open"},"cover.eval_nc_03_patio":{"attributes":{"current_position":80,"supported_features":7},"entity_id":"cover.eval_nc_03_patio","state":"open"},"light.support_negation_correction_03":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_negation_correction_03","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_03_patio","floor":"Ground Floor","room":"Patio"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_03_balcony","floor":"First Floor","room":"Balcony"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_negation_correction_03","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_negation_correction_03"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the kitchen light to 30 percent—no, perhaps 60 percent; I have not decided which.","base_id":"eval-negation_correction-04","candidate_entity_ids":["light.eval_nc_04_kitchen"],"category":"negation_correction","clarification_answer":"Confirm 60 percent brightness for the Kitchen light.","clear_command":"Set the Light in the Kitchen on the Ground Floor to 60 percent brightness.","confirmed_instruction":{"action":"set","attribute":"brightness","device":"Light","floor":"Ground Floor","room":"Kitchen","unit":"Percent","value":"60"},"expected_delta":{"after":{"attributes":{"brightness":153,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_04_kitchen","state":"on"},"before":{"attributes":{"brightness":204,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_04_kitchen","state":"on"},"entity_id":"light.eval_nc_04_kitchen"},"expected_target_entity":"light.eval_nc_04_kitchen","initial_states":{"climate.support_negation_correction_04":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_negation_correction_04","state":"cool"},"cover.eval_nc_04_kitchen":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.eval_nc_04_kitchen","state":"open"},"light.eval_nc_04_kitchen":{"attributes":{"brightness":204,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_04_kitchen","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_04_kitchen","floor":"Ground Floor","room":"Kitchen"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_04_kitchen","floor":"Ground Floor","room":"Kitchen"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_negation_correction_04","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"climate.support_negation_correction_04"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Turn on the first-floor bedroom AC—no, perhaps the second-floor one; ask me which.","base_id":"eval-negation_correction-05","candidate_entity_ids":["climate.eval_nc_05_bedroom_first","climate.eval_nc_05_bedroom_second"],"category":"negation_correction","clarification_answer":"Use Cool mode for the Bedroom AC on the Second Floor.","clear_command":"Set the Bedroom AC on the Second Floor to Cool mode.","confirmed_instruction":{"action":"set","attribute":"mode","device":"AC","floor":"Second Floor","room":"Bedroom","unit":"*","value":"Cool"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_05_bedroom_second","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_05_bedroom_second","state":"off"},"entity_id":"climate.eval_nc_05_bedroom_second"},"expected_target_entity":"climate.eval_nc_05_bedroom_second","initial_states":{"climate.eval_nc_05_bedroom_first":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_05_bedroom_first","state":"off"},"climate.eval_nc_05_bedroom_second":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_05_bedroom_second","state":"off"},"light.support_negation_correction_05":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_negation_correction_05","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_nc_05_bedroom_first","floor":"First Floor","room":"Bedroom"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_nc_05_bedroom_second","floor":"Second Floor","room":"Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_negation_correction_05","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_negation_correction_05"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the bedroom curtain to 20 percent—wait, perhaps 80 percent; confirm the value.","base_id":"eval-negation_correction-06","candidate_entity_ids":["cover.eval_nc_06_bedroom"],"category":"negation_correction","clarification_answer":"Confirm 80 percent for the Bedroom curtain.","clear_command":"Set the Curtain in the Bedroom on the First Floor to 80 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"First Floor","room":"Bedroom","unit":"Percent","value":"80"},"expected_delta":{"after":{"attributes":{"current_position":80,"supported_features":7},"entity_id":"cover.eval_nc_06_bedroom","state":"open"},"before":{"attributes":{"current_position":45,"supported_features":7},"entity_id":"cover.eval_nc_06_bedroom","state":"open"},"entity_id":"cover.eval_nc_06_bedroom"},"expected_target_entity":"cover.eval_nc_06_bedroom","initial_states":{"climate.support_negation_correction_06":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_negation_correction_06","state":"cool"},"cover.eval_nc_06_bedroom":{"attributes":{"current_position":45,"supported_features":7},"entity_id":"cover.eval_nc_06_bedroom","state":"open"},"light.eval_nc_06_bedroom":{"attributes":{"brightness":89,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_06_bedroom","state":"on"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_06_bedroom","floor":"First Floor","room":"Bedroom"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_06_bedroom","floor":"First Floor","room":"Bedroom"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_negation_correction_06","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"climate.support_negation_correction_06"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Turn off the hall light—or the entry light; I need to confirm which one.","base_id":"eval-negation_correction-07","candidate_entity_ids":["light.eval_nc_07_hall","light.eval_nc_07_entry"],"category":"negation_correction","clarification_answer":"Turn off the Entry light on the Ground Floor.","clear_command":"Turn off the Light in the Entry on the Ground Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"Light","floor":"Ground Floor","room":"Entry","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_07_entry","state":"off"},"before":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_07_entry","state":"on"},"entity_id":"light.eval_nc_07_entry"},"expected_target_entity":"light.eval_nc_07_entry","initial_states":{"cover.support_negation_correction_07":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_negation_correction_07","state":"closed"},"light.eval_nc_07_entry":{"attributes":{"brightness":166,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_07_entry","state":"on"},"light.eval_nc_07_hall":{"attributes":{"brightness":140,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_07_hall","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_07_hall","floor":"Ground Floor","room":"Hall"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_07_entry","floor":"Ground Floor","room":"Entry"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_negation_correction_07","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_negation_correction_07"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the studio AC to Cool—no, perhaps Dry; confirm the mode before acting.","base_id":"eval-negation_correction-08","candidate_entity_ids":["climate.eval_nc_08_studio"],"category":"negation_correction","clarification_answer":"Use Cool mode for the Studio AC.","clear_command":"Set the AC in the Studio on the Second Floor to Cool mode.","confirmed_instruction":{"action":"set","attribute":"mode","device":"AC","floor":"Second Floor","room":"Studio","unit":"*","value":"Cool"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_08_studio","state":"cool"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_08_studio","state":"heat"},"entity_id":"climate.eval_nc_08_studio"},"expected_target_entity":"climate.eval_nc_08_studio","initial_states":{"climate.eval_nc_08_studio":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_08_studio","state":"heat"},"cover.support_negation_correction_08":{"attributes":{"current_position":0,"supported_features":7},"entity_id":"cover.support_negation_correction_08","state":"closed"},"light.eval_nc_08_studio":{"attributes":{"brightness":128,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_08_studio","state":"on"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_nc_08_studio","floor":"Second Floor","room":"Studio"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_08_studio","floor":"Second Floor","room":"Studio"},{"aliases":[],"device":"Utility Curtain","domain":"cover","entity_id":"cover.support_negation_correction_08","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"cover.support_negation_correction_08"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Open the east curtain—or the west curtain—to 70 percent; ask which side.","base_id":"eval-negation_correction-09","candidate_entity_ids":["cover.eval_nc_09_east","cover.eval_nc_09_west"],"category":"negation_correction","clarification_answer":"Use the East Lounge curtain at 70 percent.","clear_command":"Set the Curtain in the East Lounge on the Ground Floor to 70 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"East Lounge","unit":"Percent","value":"70"},"expected_delta":{"after":{"attributes":{"current_position":70,"supported_features":7},"entity_id":"cover.eval_nc_09_east","state":"open"},"before":{"attributes":{"current_position":10,"supported_features":7},"entity_id":"cover.eval_nc_09_east","state":"open"},"entity_id":"cover.eval_nc_09_east"},"expected_target_entity":"cover.eval_nc_09_east","initial_states":{"cover.eval_nc_09_east":{"attributes":{"current_position":10,"supported_features":7},"entity_id":"cover.eval_nc_09_east","state":"open"},"cover.eval_nc_09_west":{"attributes":{"current_position":20,"supported_features":7},"entity_id":"cover.eval_nc_09_west","state":"open"},"light.support_negation_correction_09":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_negation_correction_09","state":"off"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_09_east","floor":"Ground Floor","room":"East Lounge"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_09_west","floor":"Ground Floor","room":"West Lounge"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_negation_correction_09","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_negation_correction_09"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Make the dining light Blue—no, perhaps Green; confirm the color.","base_id":"eval-negation_correction-10","candidate_entity_ids":["light.eval_nc_10_dining"],"category":"negation_correction","clarification_answer":"Confirm Green for the Dining Room light.","clear_command":"Set the Light in the Dining Room on the Ground Floor to Green.","confirmed_instruction":{"action":"set","attribute":"color","device":"Light","floor":"Ground Floor","room":"Dining Room","unit":"*","value":"Green"},"expected_delta":{"after":{"attributes":{"brightness":191,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[0,128,0],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_10_dining","state":"on"},"before":{"attributes":{"brightness":191,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[255,255,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_10_dining","state":"on"},"entity_id":"light.eval_nc_10_dining"},"expected_target_entity":"light.eval_nc_10_dining","initial_states":{"climate.support_negation_correction_10":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_negation_correction_10","state":"cool"},"cover.eval_nc_10_dining":{"attributes":{"current_position":65,"supported_features":7},"entity_id":"cover.eval_nc_10_dining","state":"open"},"light.eval_nc_10_dining":{"attributes":{"brightness":191,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"rgb_color":[255,255,255],"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_10_dining","state":"on"}},"inventory":[{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_10_dining","floor":"Ground Floor","room":"Dining Room"},{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_10_dining","floor":"Ground Floor","room":"Dining Room"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_negation_correction_10","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"climate.support_negation_correction_10"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Turn off the nursery AC—or the guest bedroom AC; do not guess which one.","base_id":"eval-negation_correction-11","candidate_entity_ids":["climate.eval_nc_11_nursery","climate.eval_nc_11_guest"],"category":"negation_correction","clarification_answer":"Turn off the Guest Bedroom AC and leave the Nursery AC unchanged.","clear_command":"Turn off the AC in the Guest Bedroom on the First Floor.","confirmed_instruction":{"action":"turnOff","attribute":"*","device":"AC","floor":"First Floor","room":"Guest Bedroom","unit":"*","value":"*"},"expected_delta":{"after":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_11_guest","state":"off"},"before":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_11_guest","state":"cool"},"entity_id":"climate.eval_nc_11_guest"},"expected_target_entity":"climate.eval_nc_11_guest","initial_states":{"climate.eval_nc_11_guest":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_11_guest","state":"cool"},"climate.eval_nc_11_nursery":{"attributes":{"fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":23.0,"temperature_unit":"°C"},"entity_id":"climate.eval_nc_11_nursery","state":"cool"},"light.support_negation_correction_11":{"attributes":{"brightness":0,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.support_negation_correction_11","state":"off"}},"inventory":[{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_nc_11_nursery","floor":"First Floor","room":"Nursery"},{"aliases":[],"device":"AC","domain":"climate","entity_id":"climate.eval_nc_11_guest","floor":"First Floor","room":"Guest Bedroom"},{"aliases":[],"device":"Utility Light","domain":"light","entity_id":"light.support_negation_correction_11","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"light.support_negation_correction_11"} +{"ambiguity_dimensions":["conflicting_entity_or_value"],"ambiguity_expected":true,"ambiguous_command":"Set the lounge curtain to 40 percent—no, perhaps halfway; confirm the position.","base_id":"eval-negation_correction-12","candidate_entity_ids":["cover.eval_nc_12_lounge"],"category":"negation_correction","clarification_answer":"Confirm 50 percent for the Lounge curtain.","clear_command":"Set the Curtain in the Lounge on the Ground Floor to 50 percent.","confirmed_instruction":{"action":"set","attribute":"position","device":"Curtain","floor":"Ground Floor","room":"Lounge","unit":"Percent","value":"50"},"expected_delta":{"after":{"attributes":{"current_position":50,"supported_features":7},"entity_id":"cover.eval_nc_12_lounge","state":"open"},"before":{"attributes":{"current_position":15,"supported_features":7},"entity_id":"cover.eval_nc_12_lounge","state":"open"},"entity_id":"cover.eval_nc_12_lounge"},"expected_target_entity":"cover.eval_nc_12_lounge","initial_states":{"climate.support_negation_correction_12":{"attributes":{"fan_mode":"medium","fan_modes":["low","medium","high"],"hvac_modes":["off","cool","heat","dry","fan_only","auto"],"max_temp":30.0,"min_temp":16.0,"supported_features":9,"temperature":24.0,"temperature_unit":"°C"},"entity_id":"climate.support_negation_correction_12","state":"cool"},"cover.eval_nc_12_lounge":{"attributes":{"current_position":15,"supported_features":7},"entity_id":"cover.eval_nc_12_lounge","state":"open"},"light.eval_nc_12_lounge":{"attributes":{"brightness":115,"max_color_temp_kelvin":6500,"min_color_temp_kelvin":3000,"supported_color_modes":["brightness","color_temp","rgb"]},"entity_id":"light.eval_nc_12_lounge","state":"on"}},"inventory":[{"aliases":[],"device":"Curtain","domain":"cover","entity_id":"cover.eval_nc_12_lounge","floor":"Ground Floor","room":"Lounge"},{"aliases":[],"device":"Light","domain":"light","entity_id":"light.eval_nc_12_lounge","floor":"Ground Floor","room":"Lounge"},{"aliases":[],"device":"Utility AC","domain":"climate","entity_id":"climate.support_negation_correction_12","floor":"Basement","room":"Utility Room"}],"license":"CC-BY-4.0","schema_version":1,"session_context":{"last_referenced_entities":[],"prior_turns":[]},"split":"eval","state_dependencies":[],"unrelated_entity_id":"climate.support_negation_correction_12"} diff --git a/cases/domux-clarify-commit/data/snapshot_manifest.json b/cases/domux-clarify-commit/data/snapshot_manifest.json new file mode 100644 index 0000000..aa2d311 --- /dev/null +++ b/cases/domux-clarify-commit/data/snapshot_manifest.json @@ -0,0 +1,87 @@ +{ + "schema_version": 1, + "model_id": "iFlytekOpenSource/Domux", + "revision": "6c71a32f4d624cadfd9fce9d10240d8068e53456", + "source": "huggingface_hub.HfApi.model_info(files_metadata=True)", + "files_sha256": "5a13462b24fc9b00d132c42718e037bc42fc51a3c6752041998e085579f01416", + "files": [ + { + "name": ".gitattributes", + "size_bytes": 1570, + "sha256": "34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930", + "hub_etag": "52373fe24473b1aa44333d318f578ae6bf04b49b" + }, + { + "name": "README.md", + "size_bytes": 8025, + "sha256": "6df6f95e2a87be1f0ba202d3bf5a630550876d8991d41401418f58e70c7cbd00", + "hub_etag": "d612ee3ed4918175d1ba5ab2d6a7e9e6993ccfc1" + }, + { + "name": "chat_template.jinja", + "size_bytes": 17592, + "sha256": "79ee69df203d90d4fa85d289cbd38f12aa1b09d015aac9784ddce23974f47a32", + "hub_etag": "b2caa00ebefc1d0e2846c15fca399b95420f1b1b" + }, + { + "name": "config.json", + "size_bytes": 5007, + "sha256": "55be922b1a46e06ba69b62215ae08d0967badf8f08eb27bee58ddc33f3182479", + "hub_etag": "96dae7e44d3d01c7aa039191aa4c945453979256" + }, + { + "name": "generation_config.json", + "size_bytes": 203, + "sha256": "9a91fc781041964400bf9a92a422bce6a963723d47e56065bb6a28e0b275384d", + "hub_etag": "e352f58545ce62e094ac3f1729d5c8fff8c5e7d5" + }, + { + "name": "model-00001-of-00004.safetensors", + "size_bytes": 1422130752, + "sha256": "f26e99e1fa290296828da6d00423b5557125badf7dc8d05f84dfd4f71e35b688", + "hub_etag": "f26e99e1fa290296828da6d00423b5557125badf7dc8d05f84dfd4f71e35b688" + }, + { + "name": "model-00002-of-00004.safetensors", + "size_bytes": 4697620616, + "sha256": "2a4f634f5fd5b23ab365a79cf94095f2a173f0d187bfb22817c09e0730dd77aa", + "hub_etag": "2a4f634f5fd5b23ab365a79cf94095f2a173f0d187bfb22817c09e0730dd77aa" + }, + { + "name": "model-00003-of-00004.safetensors", + "size_bytes": 2115862072, + "sha256": "bbad30c0bf6ccbea8282c5d79ebc6ea7ff126226239d34431d1f7ae0c5777306", + "hub_etag": "bbad30c0bf6ccbea8282c5d79ebc6ea7ff126226239d34431d1f7ae0c5777306" + }, + { + "name": "model-00004-of-00004.safetensors", + "size_bytes": 2011006798, + "sha256": "e9509b71370663bdf5e3b3fe6f649f9d490bc6345857aecaaf5360ac5fea074d", + "hub_etag": "e9509b71370663bdf5e3b3fe6f649f9d490bc6345857aecaaf5360ac5fea074d" + }, + { + "name": "model.safetensors.index.json", + "size_bytes": 205883, + "sha256": "7300d2c559eb88dd7f2f29da0ad661bcc173dc8446e724ed6946f0d5844093d3", + "hub_etag": "e84c00c17aba40c1866e65ce4c19a29675cd1e44" + }, + { + "name": "processor_config.json", + "size_bytes": 1689, + "sha256": "32bdf45d2ad4cc29a0822ddd157a182de76644f0419a6228d151495256e9813c", + "hub_etag": "5465974d23e1eca2c46c2809b26c997946ce0d90" + }, + { + "name": "tokenizer.json", + "size_bytes": 32169626, + "sha256": "cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f", + "hub_etag": "cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f" + }, + { + "name": "tokenizer_config.json", + "size_bytes": 2741, + "sha256": "a284d1243b62be31faa9c13e1c28cece940c4abaa7bd9ad87b94f61b40687200", + "hub_etag": "af7f25861136b515f4bf64d9d0a6cf9875a6d508" + } + ] +} diff --git a/cases/domux-clarify-commit/evaluate.py b/cases/domux-clarify-commit/evaluate.py new file mode 100644 index 0000000..84c1e42 --- /dev/null +++ b/cases/domux-clarify-commit/evaluate.py @@ -0,0 +1,1462 @@ +#!/usr/bin/env python3 +"""Replay frozen Domux outputs through the pre-registered B0/B1/B2 protocol. + +The evaluator never calls a model. It consumes the immutable JSONL emitted by +``run_model.py``, keeps all 48 evaluation base IDs in every eligible +denominator, and creates a fresh in-memory adapter and prepared action for each +mutation trial. The two output files are deterministic and intentionally omit +timestamps and local paths. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import statistics +from collections import Counter +from dataclasses import dataclass, replace +from pathlib import Path +from typing import Any, Iterable, Mapping, Sequence + +from clarify_commit import ( + ClarifyPrepareStore, + DomuxInstruction, + EntityRegistry, + EntitySpec, + GroundedRequest, + GroundingError, + InMemoryHAAdapter, + ParseError, + PreparedActionStore, + SessionContext, + altered_confirmation, + build_plan, + canonical_json, + controlled_projection, + digest_json, + ground_domux_request, + projection_matches, + resolve_unique_request, +) + + +CASE_DIR = Path(__file__).resolve().parent +DEFAULT_DATASET = CASE_DIR / "data" / "scenarios.jsonl" +DEFAULT_PROTOCOL = CASE_DIR / "data" / "protocol.json" +DEFAULT_FREEZE = CASE_DIR / "data" / "freeze.json" +DEFAULT_EVIDENCE = CASE_DIR / "evidence" / "domux_raw.jsonl" +DEFAULT_EVIDENCE_METADATA = CASE_DIR / "evidence" / "domux_metadata.json" + +FORMAL_BASE_COUNT = 48 +CONTEXT_BASE_COUNT = 12 +REGISTERED_SNAPSHOT_MANIFEST_SHA256 = "5a13462b24fc9b00d132c42718e037bc42fc51a3c6752041998e085579f01416" +REGISTERED_SNAPSHOT_FILES = json.loads( + (CASE_DIR / "data" / "snapshot_manifest.json").read_text(encoding="utf-8") +)["files"] +VARIANTS = ("clear", "ambiguous") +SLOTS = ("action", "device", "attribute", "value", "unit", "room", "floor") + +B0 = "B0_unique_or_abstain" +B1 = "B1_clarify_and_prepare" +B2 = "B2_clarify_and_commit" +ARMS = (B0, B1, B2) + +MUTATIONS = ( + "clean", + "replay", + "expiry", + "target_drift", + "session_swap", + "plan_swap", + "candidate_change", + "context_state_change", + "unrelated_state_change", +) +UNIVERSAL_GUARD_MUTATIONS = ( + "replay", + "expiry", + "target_drift", + "session_swap", + "plan_swap", + "candidate_change", + "unrelated_state_change", +) +B1_INTENTIONAL_GAPS = frozenset({ + "replay", + "expiry", + "target_drift", + "candidate_change", + "context_state_change", +}) +BLOCKING_MUTATIONS = frozenset({ + "expiry", + "target_drift", + "session_swap", + "plan_swap", + "candidate_change", + "context_state_change", +}) + + +class EvaluationInputError(ValueError): + """Frozen inputs do not satisfy the pre-registered evaluation contract.""" + + +class LogicalClock: + """A deterministic clock; expiry trials advance it without sleeping.""" + + def __init__(self, value: float = 1000.0): + self.value = value + + def __call__(self) -> float: + return self.value + + +@dataclass(frozen=True) +class ProbeOutcome: + base_id: str + variant: str + evidence_status: str + disposition: str + reason: str + instructions: tuple[DomuxInstruction, ...] + candidate_ids: tuple[str, ...] + grounded: GroundedRequest | None + raw_output: str + raw_output_sha256: str + latency_ms: float | None + error_type: str | None = None + + @property + def is_unique(self) -> bool: + return self.disposition == "unique" + + @property + def is_clarification(self) -> bool: + return self.disposition == "clarify" + + +def _sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def _sha256_text(value: str) -> str: + return _sha256_bytes(value.encode("utf-8")) + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _stable_float(value: float) -> float: + """Keep useful precision while making serialized statistics easy to diff.""" + + return float(f"{value:.15g}") + + +def _read_json(path: Path, label: str) -> dict[str, Any]: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise EvaluationInputError(f"cannot read valid {label} JSON") from exc + if not isinstance(value, dict): + raise EvaluationInputError(f"{label} must be a JSON object") + return value + + +def _read_jsonl(path: Path, label: str) -> list[dict[str, Any]]: + try: + lines = path.read_text(encoding="utf-8").splitlines() + except OSError as exc: + raise EvaluationInputError(f"cannot read {label} JSONL") from exc + rows: list[dict[str, Any]] = [] + for line_number, line in enumerate(lines, start=1): + if not line.strip(): + continue + try: + value = json.loads(line) + except json.JSONDecodeError as exc: + raise EvaluationInputError(f"{label} line {line_number} is invalid JSON") from exc + if not isinstance(value, dict): + raise EvaluationInputError(f"{label} line {line_number} is not an object") + rows.append(value) + return rows + + +def _canonical_lines(rows: Iterable[Mapping[str, object]]) -> bytes: + return "".join(canonical_json(row) + "\n" for row in rows).encode("utf-8") + + +def _instruction(fields: Mapping[str, object]) -> DomuxInstruction: + try: + values = [str(fields[slot]) for slot in SLOTS] + except KeyError as exc: + raise EvaluationInputError(f"confirmed instruction is missing {exc.args[0]}") from exc + return DomuxInstruction.from_fields(values) + + +def _registry(row: Mapping[str, object]) -> EntityRegistry: + inventory = row.get("inventory") + if not isinstance(inventory, list): + raise EvaluationInputError("scenario inventory must be a list") + entities: list[EntitySpec] = [] + for item in inventory: + if not isinstance(item, dict): + raise EvaluationInputError("inventory entries must be objects") + try: + entities.append(EntitySpec( + entity_id=str(item["entity_id"]), + domain=str(item["domain"]), + device=str(item["device"]), + room=str(item["room"]), + floor=str(item["floor"]), + aliases=tuple(str(alias) for alias in item.get("aliases", ())), + )) + except (KeyError, TypeError, ValueError) as exc: + raise EvaluationInputError("invalid inventory entry") from exc + return EntityRegistry(entities) + + +def _context(row: Mapping[str, object]) -> SessionContext: + value = row.get("session_context") + if not isinstance(value, dict): + raise EvaluationInputError("session_context must be an object") + recent = value.get("last_referenced_entities", ()) + if not isinstance(recent, list): + raise EvaluationInputError("last_referenced_entities must be a list") + return SessionContext(tuple(str(entity_id) for entity_id in recent)) + + +def _projection_snapshot( + adapter: InMemoryHAAdapter, + registry: EntityRegistry, +) -> dict[str, dict[str, object]]: + return { + entity.entity_id: controlled_projection(adapter.get_state(entity.entity_id), entity.domain) + for entity in registry.entities + } + + +def _fixture_projection(row: Mapping[str, object], registry: EntityRegistry) -> dict[str, dict[str, object]]: + states = row.get("initial_states") + if not isinstance(states, dict): + raise EvaluationInputError("initial_states must be an object") + return { + entity.entity_id: controlled_projection(states[entity.entity_id], entity.domain) + for entity in registry.entities + } + + +def _validate_protocol(protocol: Mapping[str, object]) -> None: + try: + population = protocol["analysis_population"] + mutations = protocol["mutations"] + arms = protocol["arms"] + model = protocol["model"] + except KeyError as exc: + raise EvaluationInputError(f"protocol is missing {exc.args[0]}") from exc + if not isinstance(population, dict) or population.get("evaluation_bases") != FORMAL_BASE_COUNT: + raise EvaluationInputError("formal evaluation denominator must be exactly 48 base IDs") + if population.get("unit") != "base_id" or population.get("paired_probes_per_base") != list(VARIANTS): + raise EvaluationInputError("protocol must retain the paired base_id analysis unit") + if not isinstance(arms, dict) or tuple(arms) != ARMS: + raise EvaluationInputError("protocol B0/B1/B2 arm definitions changed") + if not isinstance(mutations, dict) or tuple(mutations) != MUTATIONS: + raise EvaluationInputError("protocol mutation declarations changed") + if not isinstance(model, dict) or model.get("snapshot_files_sha256") != REGISTERED_SNAPSHOT_MANIFEST_SHA256: + raise EvaluationInputError("protocol model snapshot manifest changed") + if model.get("snapshot_manifest_file") != "snapshot_manifest.json": + raise EvaluationInputError("protocol snapshot manifest path changed") + for mutation in MUTATIONS: + expected = CONTEXT_BASE_COUNT if mutation == "context_state_change" else FORMAL_BASE_COUNT + value = mutations[mutation] + if not isinstance(value, dict) or value.get("eligible_bases") != expected: + raise EvaluationInputError(f"protocol denominator changed for {mutation}") + b1 = arms[B1] + b2 = arms[B2] + if not isinstance(b1, dict) or not isinstance(b2, dict): + raise EvaluationInputError("protocol arm definitions must be objects") + if any(b1.get(key) is not False for key in ("candidate_revalidation", "state_revalidation", "ttl", "one_time_nonce")): + raise EvaluationInputError("B1 must omit only the pre-registered temporal guards") + if any(b2.get(key) is not True for key in ("candidate_revalidation", "state_revalidation", "one_time_nonce")): + raise EvaluationInputError("B2 guard declarations changed") + if b2.get("ttl_seconds") != 30: + raise EvaluationInputError("B2 TTL must remain 30 seconds") + + +def _validate_rows(rows: Sequence[dict[str, Any]]) -> list[dict[str, Any]]: + # Preserve file order here because the freeze manifest hashes the exact + # canonical evaluation sequence. Trial output is sorted separately. + eval_rows = [row for row in rows if row.get("split") == "eval"] + if len(eval_rows) != FORMAL_BASE_COUNT: + raise EvaluationInputError("dataset must contain exactly 48 evaluation bases") + base_ids = [str(row.get("base_id", "")) for row in eval_rows] + if any(not base_id for base_id in base_ids) or len(set(base_ids)) != FORMAL_BASE_COUNT: + raise EvaluationInputError("evaluation base IDs must be non-empty and unique") + category_counts = Counter(str(row.get("category")) for row in eval_rows) + if category_counts != Counter({ + "duplicate_entity": 12, + "missing_slot": 12, + "context_reference": 12, + "negation_correction": 12, + }): + raise EvaluationInputError("evaluation strata must remain balanced at 12 bases each") + for row in eval_rows: + base_id = str(row["base_id"]) + for key in ("clear_command", "ambiguous_command", "confirmed_instruction", "expected_target_entity", "expected_delta"): + if key not in row: + raise EvaluationInputError(f"scenario {base_id} is missing {key}") + registry = _registry(row) + target = str(row["expected_target_entity"]) + registry.get(target) + candidate_ids = row.get("candidate_entity_ids") + if not isinstance(candidate_ids, list) or not 1 <= len(candidate_ids) <= 3: + raise EvaluationInputError(f"scenario {base_id} candidate set is outside one to three") + if target not in candidate_ids or len(set(candidate_ids)) != len(candidate_ids): + raise EvaluationInputError(f"scenario {base_id} has an invalid target/candidate relation") + if set(candidate_ids) - {entity.entity_id for entity in registry.entities}: + raise EvaluationInputError(f"scenario {base_id} candidate is outside the registry") + if str(row.get("unrelated_entity_id")) in candidate_ids: + raise EvaluationInputError(f"scenario {base_id} unrelated entity is candidate-bound") + states = row.get("initial_states") + if not isinstance(states, dict) or set(states) != {entity.entity_id for entity in registry.entities}: + raise EvaluationInputError(f"scenario {base_id} state fixture is not closed") + expected_delta = row["expected_delta"] + if not isinstance(expected_delta, dict) or expected_delta.get("entity_id") != target: + raise EvaluationInputError(f"scenario {base_id} expected delta target changed") + if expected_delta.get("before") != states[target]: + raise EvaluationInputError(f"scenario {base_id} expected before-state changed") + return eval_rows + + +def _validate_freeze( + freeze: Mapping[str, object], + dataset_path: Path, + protocol_path: Path, + eval_rows: Sequence[Mapping[str, object]], +) -> None: + snapshot_manifest_path = protocol_path.parent / "snapshot_manifest.json" + checks = { + "full_sha256": _sha256_file(dataset_path), + "protocol_sha256": _sha256_file(protocol_path), + "evaluation_sha256": _sha256_bytes(_canonical_lines(eval_rows)), + "snapshot_manifest_sha256": _sha256_file(snapshot_manifest_path), + } + for field, observed in checks.items(): + if freeze.get(field) != observed: + raise EvaluationInputError(f"frozen input hash mismatch: {field}") + if freeze.get("evaluation_count") != FORMAL_BASE_COUNT: + raise EvaluationInputError("freeze evaluation denominator is not 48") + + +def _load_evidence( + evidence_rows: Sequence[dict[str, Any]], + eval_rows: Sequence[dict[str, Any]], +) -> dict[tuple[str, str], dict[str, Any]]: + expected: dict[tuple[str, str], str] = {} + for row in eval_rows: + for variant in VARIANTS: + expected[(str(row["base_id"]), variant)] = str(row[f"{variant}_command"]) + + indexed: dict[tuple[str, str], dict[str, Any]] = {} + for item in evidence_rows: + key = (str(item.get("base_id", "")), str(item.get("variant", ""))) + if key not in expected: + raise EvaluationInputError("frozen evidence contains an unknown base/variant") + if key in indexed: + raise EvaluationInputError("frozen evidence contains a duplicate base/variant") + command = item.get("command") + if command != expected[key]: + raise EvaluationInputError("frozen evidence command differs from the frozen dataset") + if "query_sha256" in item and item["query_sha256"] != _sha256_text(str(command)): + raise EvaluationInputError("frozen evidence query hash mismatch") + status = item.get("status") + if status not in {"ok", "error"}: + raise EvaluationInputError("frozen evidence status must be ok or error") + raw_output = item.get("raw_output", "") + if not isinstance(raw_output, str): + raise EvaluationInputError("frozen raw_output must be a string") + if status == "ok" and "raw_output" not in item: + raise EvaluationInputError("successful frozen evidence is missing raw_output") + if "raw_output_sha256" in item and item["raw_output_sha256"] != _sha256_text(raw_output): + raise EvaluationInputError("frozen evidence raw-output hash mismatch") + latency = item.get("latency_ms") + if latency is not None and (not isinstance(latency, (int, float)) or isinstance(latency, bool) or latency < 0): + raise EvaluationInputError("frozen evidence latency must be a non-negative number or null") + indexed[key] = item + missing = sorted(set(expected) - set(indexed)) + if missing: + raise EvaluationInputError( + f"frozen evidence is incomplete: {len(missing)} of 96 paired probes missing" + ) + if len(indexed) != FORMAL_BASE_COUNT * len(VARIANTS): + raise EvaluationInputError("frozen evidence must contain exactly 96 paired probes") + return indexed + + +def _validated_model_run( + metadata: Mapping[str, object], + protocol: Mapping[str, object], + dataset_sha256: str, + evidence_sha256: str, + evidence_failure_count: int = 0, +) -> dict[str, object]: + registered = protocol["model"] + result: dict[str, object] = { + "registered_configuration": { + "model_id": registered["id"], + "revision": registered["revision"], + "precision": registered["primary_precision"], + "do_sample": registered["do_sample"], + "temperature": registered["temperature"], + "max_new_tokens": registered["max_new_tokens"], + "seed": registered["seed"], + "warmup_count": registered["warmup"]["count"], + "warmup_source": registered["warmup"]["source"], + }, + "metadata_verified": True, + "observed_environment": None, + } + required_equal = { + "model_id": registered["id"], + "tested_revision": registered["revision"], + "precision": registered["primary_precision"], + "dataset_sha256": dataset_sha256, + "sample_count": FORMAL_BASE_COUNT * len(VARIANTS), + "base_count": FORMAL_BASE_COUNT, + "warmup_count": registered["warmup"]["count"], + "raw_evidence_sha256": evidence_sha256, + "snapshot_manifest_sha256": REGISTERED_SNAPSHOT_MANIFEST_SHA256, + "snapshot_revision_verified": True, + "runner_sha256": _sha256_file(CASE_DIR / "run_model.py"), + "grounding_policy_sha256": _sha256_file(CASE_DIR / "clarify_commit.py"), + "evaluator_sha256": _sha256_file(Path(__file__).resolve()), + "run_mode": "formal", + "selective_reruns": 0, + "sample_failures": evidence_failure_count, + } + for field, expected in required_equal.items(): + if metadata.get(field) != expected: + raise EvaluationInputError(f"model-run metadata mismatch: {field}") + generation = metadata.get("generation") + if not isinstance(generation, dict): + raise EvaluationInputError("model-run metadata is missing generation settings") + for field in ("do_sample", "max_new_tokens", "seed"): + if generation.get(field) != registered[field]: + raise EvaluationInputError(f"model-run generation mismatch: {field}") + # run_model.py records temperature=0 for deterministic decoding even though + # transformers ignores it when sampling is disabled. + if generation.get("temperature") != registered["temperature"]: + raise EvaluationInputError("model-run generation mismatch: temperature") + if metadata.get("offline_inference") is not True: + raise EvaluationInputError("formal evidence must be recorded in offline inference mode") + snapshot_files = metadata.get("snapshot_files") + if not isinstance(snapshot_files, list) or len(snapshot_files) != 13: + raise EvaluationInputError("model-run metadata has an invalid snapshot manifest") + if any( + not isinstance(item, dict) + or set(item) != {"name", "size_bytes", "sha256", "hub_etag"} + or not isinstance(item.get("name"), str) + or "/" in str(item.get("name")) + or "\\" in str(item.get("name")) + for item in snapshot_files + ): + raise EvaluationInputError("model-run metadata has malformed snapshot entries") + if [item["name"] for item in snapshot_files] != sorted(item["name"] for item in snapshot_files): + raise EvaluationInputError("model-run snapshot manifest is not canonically ordered") + if snapshot_files != REGISTERED_SNAPSHOT_FILES: + raise EvaluationInputError("model-run snapshot entries differ from the registered revision") + observed_manifest_sha256 = _sha256_text(canonical_json(snapshot_files)) + if observed_manifest_sha256 != REGISTERED_SNAPSHOT_MANIFEST_SHA256: + raise EvaluationInputError("model-run snapshot files do not match the registered manifest") + + safe_fields = ( + "artifact_type", + "runtime", + "python_version", + "platform", + "torch_version", + "transformers_version", + "huggingface_hub_version", + "accelerate_version", + "bitsandbytes_version", + "precision", + "compute_dtype", + "gpu", + "gpu_total_memory_bytes", + "visible_gpu_count", + "sample_count", + "base_count", + "warmup_count", + "warmup_source", + "generation", + "offline_inference", + "run_mode", + "selective_reruns", + "sample_failures", + ) + result["observed_environment"] = { + field: metadata.get(field) for field in safe_fields + } + result["evidence_binding"] = { + "metadata_raw_evidence_sha256": metadata["raw_evidence_sha256"], + "observed_raw_evidence_sha256": evidence_sha256, + "match": True, + } + result["code_binding"] = { + "runner_sha256": metadata["runner_sha256"], + "grounding_policy_sha256": metadata["grounding_policy_sha256"], + "evaluator_sha256": metadata["evaluator_sha256"], + "match": True, + } + return result + + +def analyze_probe(row: Mapping[str, object], variant: str, evidence: Mapping[str, object]) -> ProbeOutcome: + """Classify a frozen model output without consulting the gold ambiguity label.""" + + base_id = str(row["base_id"]) + raw_output = str(evidence.get("raw_output", "")) + raw_hash = _sha256_text(raw_output) + latency_value = evidence.get("latency_ms") + latency = float(latency_value) if isinstance(latency_value, (int, float)) and not isinstance(latency_value, bool) else None + if evidence.get("status") != "ok": + return ProbeOutcome( + base_id=base_id, + variant=variant, + evidence_status="error", + disposition="unavailable", + reason="inference_error", + instructions=(), + candidate_ids=(), + grounded=None, + raw_output=raw_output, + raw_output_sha256=raw_hash, + latency_ms=latency, + error_type=str(evidence.get("error_type") or "InferenceError"), + ) + + try: + registry = _registry(row) + context = _context(row) + grounded = ground_domux_request( + str(row[f"{variant}_command"]), + raw_output, + registry, + context, + ) + except ParseError: + return ProbeOutcome( + base_id=base_id, + variant=variant, + evidence_status="ok", + disposition="safe_abstain", + reason="malformed_model_output", + instructions=(), + candidate_ids=(), + grounded=None, + raw_output=raw_output, + raw_output_sha256=raw_hash, + latency_ms=latency, + error_type="ParseError", + ) + candidate_ids = tuple(entity.entity_id for entity in grounded.candidates) + disposition = "clarify" if grounded.clarification.required else "unique" + reason = grounded.clarification.reason + if disposition == "unique": + try: + build_plan( + grounded.source_instructions[0], + grounded.candidates[0], + row["initial_states"][grounded.candidates[0].entity_id], + ) + except (GroundingError, KeyError, TypeError, ValueError): + disposition = "safe_abstain" + reason = "non_executable_instruction" + return ProbeOutcome( + base_id=base_id, + variant=variant, + evidence_status="ok", + disposition=disposition, + reason=reason, + instructions=grounded.source_instructions, + candidate_ids=candidate_ids, + grounded=grounded, + raw_output=raw_output, + raw_output_sha256=raw_hash, + latency_ms=latency, + ) + + +def _candidate_coverage(row: Mapping[str, object], outcome: ProbeOutcome) -> bool: + return ( + str(row["expected_target_entity"]) in outcome.candidate_ids + and 1 <= len(outcome.candidate_ids) <= 3 + ) + + +def _expected_projection(row: Mapping[str, object], registry: EntityRegistry, which: str) -> dict[str, object]: + target = str(row["expected_target_entity"]) + entity = registry.get(target) + return controlled_projection(row["expected_delta"][which], entity.domain) + + +def _wrong_target_transition( + target: str, + before: Mapping[str, Mapping[str, object]], + after: Mapping[str, Mapping[str, object]], + calls: Sequence[Mapping[str, object]], +) -> bool: + if any(str(call.get("data", {}).get("entity_id")) != target for call in calls): + return True + return any(before[entity_id] != after[entity_id] for entity_id in before if entity_id != target) + + +def _exact_delta( + row: Mapping[str, object], + registry: EntityRegistry, + before: Mapping[str, Mapping[str, object]], + after: Mapping[str, Mapping[str, object]], + calls: Sequence[Mapping[str, object]], +) -> bool: + target = str(row["expected_target_entity"]) + return ( + before[target] == _expected_projection(row, registry, "before") + and after[target] == _expected_projection(row, registry, "after") + and len(calls) == 1 + and str(calls[0].get("data", {}).get("entity_id")) == target + and not _wrong_target_transition(target, before, after, calls) + ) + + +def _base_trial(row: Mapping[str, object], arm: str, mutation: str, outcome: ProbeOutcome) -> dict[str, object]: + return { + "schema_version": 1, + "record_type": "execution_trial", + "base_id": str(row["base_id"]), + "category": str(row["category"]), + "arm": arm, + "mutation": mutation, + "evidence_status": outcome.evidence_status, + "input": str(row["ambiguous_command"]), + "raw_output": outcome.raw_output, + "raw_output_sha256": outcome.raw_output_sha256, + "language_disposition": outcome.disposition, + "reset_verified": False, + "preconfirm_sut_calls": 0, + "setup_calls": 0, + "sut_calls": 0, + "accepted": None, + "dispatched": None, + "commit_reason": None, + "exact_delta_success": False, + "wrong_target_transition": False, + "oracle_pass": False, + "setup_failure": None, + "interpretation": "unclassified", + } + + +def _trial_interpretation(trial: Mapping[str, object]) -> str: + if trial.get("oracle_pass") is True: + return "pass" + arm = trial["arm"] + mutation = trial["mutation"] + if ( + arm == B1 + and mutation in B1_INTENTIONAL_GAPS + and trial.get("setup_failure") is None + and trial.get("accepted") is True + and trial.get("dispatched") is True + ): + return "expected_b1_guard_limitation" + if arm == B2: + return "unexpected_b2_suite_failure" + return "unexpected_baseline_failure" + + +def _prepare_trial( + row: Mapping[str, object], + outcome: ProbeOutcome, + arm: str, + mutation: str, + ttl_seconds: float, +) -> tuple[ + InMemoryHAAdapter, + EntityRegistry, + PreparedActionStore, + Any, + LogicalClock, + dict[str, dict[str, object]], +]: + if outcome.evidence_status != "ok": + raise GroundingError("inference_error") + if not outcome.is_clarification: + raise GroundingError("clarification_not_triggered") + if not _candidate_coverage(row, outcome): + raise GroundingError("candidate_coverage_failure") + if outcome.grounded is None: + raise GroundingError("grounded_request_missing") + + registry = _registry(row) + adapter = InMemoryHAAdapter(row["initial_states"]) + expected_initial = _fixture_projection(row, registry) + observed_initial = _projection_snapshot(adapter, registry) + if observed_initial != expected_initial: + raise GroundingError("reset_projection_mismatch") + + instruction = _instruction(row["confirmed_instruction"]) + target = str(row["expected_target_entity"]) + answer = row.get("clarification_answer") + if not isinstance(answer, str) or not answer.strip(): + raise GroundingError("clarification_answer_missing") + + clock = LogicalClock() + nonce = f"{arm}:{row['base_id']}:{mutation}" + store_type = ClarifyPrepareStore if arm == B1 else PreparedActionStore + store = store_type( + ttl_seconds=ttl_seconds, + clock=clock, + nonce_factory=lambda: nonce, + ) + state_dependencies = row.get("state_dependencies", ()) + if not isinstance(state_dependencies, list): + raise GroundingError("invalid_state_dependencies") + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=outcome.grounded, + registry=registry, + adapter=adapter, + clarification_answer=answer, + confirmed_instruction=instruction, + state_dependencies=tuple(str(entity_id) for entity_id in state_dependencies), + ) + snapshot = store.snapshot(action.nonce) + plan = snapshot["plan"] + if ( + action.entity_id != target + or plan["entity_id"] != target + or not projection_matches( + _expected_projection(row, registry, "after"), + plan["expected_projection"], + ) + ): + raise GroundingError("confirmed_plan_delta_mismatch") + if adapter.sut_calls: + raise GroundingError("preconfirm_dispatch") + return adapter, registry, store, action, clock, observed_initial + + +def _run_prepared_trial( + row: Mapping[str, object], + outcome: ProbeOutcome, + arm: str, + mutation: str, + ttl_seconds: float, +) -> dict[str, object]: + trial = _base_trial(row, arm, mutation, outcome) + try: + adapter, registry, store, action, clock, initial = _prepare_trial( + row, outcome, arm, mutation, ttl_seconds, + ) + except (EvaluationInputError, GroundingError, KeyError, TypeError, ValueError) as exc: + reason = str(exc) + allowed = { + "inference_error", + "clarification_not_triggered", + "candidate_coverage_failure", + "grounded_request_missing", + "reset_projection_mismatch", + "clarification_answer_missing", + "confirmed_plan_delta_mismatch", + "preconfirm_dispatch", + "invalid_state_dependencies", + } + trial["setup_failure"] = ( + reason + if reason in allowed or isinstance(exc, (EvaluationInputError, GroundingError)) + else type(exc).__name__ + ) + trial["interpretation"] = _trial_interpretation(trial) + return trial + + trial["reset_verified"] = True + trial["initial_projection_sha256"] = digest_json(initial) + trial["request_digest"] = action.request_digest + trial["clarification_digest"] = action.clarification_digest + target = str(row["expected_target_entity"]) + active_registry = registry + confirmation = action.confirmation() + + if mutation == "expiry": + clock.value = action.expires_at + 1.0 + elif mutation == "target_drift": + adapter.mutate_state_for_setup(target) + elif mutation == "session_swap": + confirmation = altered_confirmation(confirmation, session_id="session-b") + elif mutation == "plan_swap": + different_digest = digest_json({"base_id": row["base_id"], "mutation": mutation}) + if different_digest == confirmation.plan_digest: + raise AssertionError("deterministic plan-swap digest collision") + confirmation = altered_confirmation(confirmation, plan_digest=different_digest) + elif mutation == "candidate_change": + candidate_ids = store.snapshot(action.nonce)["candidate_ids"] + entity = registry.get(candidate_ids[0]) + active_registry = registry.with_replacement(replace( + entity, + aliases=tuple(entity.aliases) + ("evaluation-metadata-change",), + )) + elif mutation == "context_state_change": + dependencies = tuple(str(entity_id) for entity_id in row.get("state_dependencies", ())) + changed = next((entity_id for entity_id in dependencies if entity_id != target), None) + if changed is None: + trial["setup_failure"] = "missing_non_target_context_dependency" + trial["setup_calls"] = len(adapter.setup_calls) + trial["interpretation"] = _trial_interpretation(trial) + return trial + adapter.mutate_state_for_setup(changed) + elif mutation == "unrelated_state_change": + adapter.mutate_state_for_setup(str(row["unrelated_entity_id"])) + elif mutation not in {"clean", "replay"}: + raise AssertionError(f"unhandled mutation: {mutation}") + + trial["preconfirm_sut_calls"] = len(adapter.sut_calls) + before = _projection_snapshot(adapter, active_registry) + + if mutation == "replay": + first = store.commit(confirmation, registry=active_registry, adapter=adapter) + after_first = _projection_snapshot(adapter, active_registry) + first_calls = len(adapter.sut_calls) + first_exact = ( + first.accepted + and first.dispatched + and first.reason == "committed" + and _exact_delta(row, active_registry, before, after_first, adapter.sut_calls) + ) + result = store.commit(confirmation, registry=active_registry, adapter=adapter) + after = _projection_snapshot(adapter, active_registry) + additional_calls = len(adapter.sut_calls) - first_calls + oracle_pass = bool( + first_exact + and not result.accepted + and not result.dispatched + and result.reason == "replayed_nonce" + and additional_calls == 0 + and after == after_first + ) + trial["first_commit_exact_delta"] = bool(first_exact) + trial["replay_additional_sut_calls"] = additional_calls + trial["exact_delta_success"] = bool(first_exact) + else: + result = store.commit(confirmation, registry=active_registry, adapter=adapter) + after = _projection_snapshot(adapter, active_registry) + if mutation in BLOCKING_MUTATIONS: + expected_reason = { + "expiry": "expired", + "target_drift": "state_changed", + "session_swap": "session_mismatch", + "plan_swap": "plan_mismatch", + "candidate_change": "candidate_set_changed", + "context_state_change": "state_changed", + }[mutation] + oracle_pass = bool( + not result.accepted + and not result.dispatched + and result.reason == expected_reason + and len(adapter.sut_calls) == 0 + and after == before + ) + else: + exact = _exact_delta(row, active_registry, before, after, adapter.sut_calls) + oracle_pass = bool( + result.accepted + and result.dispatched + and result.reason == "committed" + and exact + ) + trial["exact_delta_success"] = exact + + trial.update({ + "setup_calls": len(adapter.setup_calls), + "sut_calls": len(adapter.sut_calls), + "accepted": result.accepted, + "dispatched": result.dispatched, + "commit_reason": result.reason, + "wrong_target_transition": _wrong_target_transition( + target, before, after, adapter.sut_calls, + ), + "oracle_pass": oracle_pass, + }) + trial["interpretation"] = _trial_interpretation(trial) + return trial + + +def _run_b0_trial(row: Mapping[str, object], outcome: ProbeOutcome) -> dict[str, object]: + """Run the strong no-clarification baseline: execute only a unique plan.""" + + trial = _base_trial(row, B0, "clean", outcome) + registry = _registry(row) + adapter = InMemoryHAAdapter(row["initial_states"]) + before = _projection_snapshot(adapter, registry) + trial["reset_verified"] = before == _fixture_projection(row, registry) + trial["initial_projection_sha256"] = digest_json(before) + if not outcome.is_unique: + valid_abstention = outcome.evidence_status == "ok" + trial["safe_abstention"] = valid_abstention + trial["oracle_pass"] = valid_abstention + trial["setup_failure"] = "inference_error" if outcome.evidence_status != "ok" else None + trial["interpretation"] = ( + "expected_b0_safe_abstention" if outcome.evidence_status == "ok" + else "unexpected_baseline_failure" + ) + return trial + + try: + if outcome.grounded is None: + raise GroundingError("grounded request missing") + resolved = resolve_unique_request(outcome.grounded, registry) + entity = resolved.chosen + plan = build_plan( + resolved.confirmed_instruction, + entity, + adapter.get_state(entity.entity_id), + ) + receipt = adapter.call_service(plan.domain, plan.service, plan.service_data) + postcondition = projection_matches( + controlled_projection(receipt.after, plan.domain), + plan.expected_projection, + ) + except (GroundingError, KeyError, TypeError, ValueError): + trial["setup_failure"] = "direct_execution_failure" + trial["interpretation"] = "unexpected_baseline_failure" + return trial + + after = _projection_snapshot(adapter, registry) + exact = _exact_delta(row, registry, before, after, adapter.sut_calls) + target = str(row["expected_target_entity"]) + trial.update({ + "safe_abstention": False, + "sut_calls": len(adapter.sut_calls), + "accepted": True, + "dispatched": True, + "commit_reason": "direct_unique_dispatch" if postcondition else "postcondition_mismatch", + "exact_delta_success": exact and postcondition, + "wrong_target_transition": _wrong_target_transition(target, before, after, adapter.sut_calls), + "oracle_pass": False, + "interpretation": "b0_unclarified_dispatch_on_gold_ambiguous_probe", + }) + return trial + + +def wilson_interval(successes: int, denominator: int, z: float = 1.959963984540054) -> dict[str, float]: + if denominator <= 0 or not 0 <= successes <= denominator: + raise ValueError("Wilson inputs require 0 <= successes <= a positive denominator") + proportion = successes / denominator + z2 = z * z + scale = 1.0 + z2 / denominator + center = (proportion + z2 / (2.0 * denominator)) / scale + half = z / scale * math.sqrt( + proportion * (1.0 - proportion) / denominator + z2 / (4.0 * denominator * denominator) + ) + lower = 0.0 if successes == 0 else max(0.0, center - half) + upper = 1.0 if successes == denominator else min(1.0, center + half) + return { + "lower": _stable_float(lower), + "upper": _stable_float(upper), + } + + +def binary_metric(values: Sequence[bool]) -> dict[str, object]: + successes = sum(bool(value) for value in values) + denominator = len(values) + if denominator == 0: + raise ValueError("binary metric cannot have an empty denominator") + return { + "successes": successes, + "denominator": denominator, + "rate": _stable_float(successes / denominator), + "wilson_95": wilson_interval(successes, denominator), + } + + +def exact_mcnemar( + arm_a: str, + arm_b: str, + outcomes_a: Sequence[bool], + outcomes_b: Sequence[bool], +) -> dict[str, object]: + if len(outcomes_a) != len(outcomes_b) or not outcomes_a: + raise ValueError("McNemar inputs must be non-empty paired vectors") + both = sum(a and b for a, b in zip(outcomes_a, outcomes_b)) + a_only = sum(a and not b for a, b in zip(outcomes_a, outcomes_b)) + b_only = sum(not a and b for a, b in zip(outcomes_a, outcomes_b)) + neither = len(outcomes_a) - both - a_only - b_only + discordant = a_only + b_only + if discordant == 0: + p_value = 1.0 + else: + tail = sum(math.comb(discordant, index) for index in range(min(a_only, b_only) + 1)) + p_value = min(1.0, 2.0 * tail / (2 ** discordant)) + return { + "arm_a": arm_a, + "arm_b": arm_b, + "paired_bases": len(outcomes_a), + "both_success": both, + "a_only_success": a_only, + "b_only_success": b_only, + "neither_success": neither, + "discordant_pairs": discordant, + "risk_difference_b_minus_a": _stable_float((b_only - a_only) / len(outcomes_a)), + "exact_two_sided_p": _stable_float(p_value), + } + + +def holm_adjust(comparisons: Sequence[dict[str, object]], alpha: float = 0.05) -> list[dict[str, object]]: + if not comparisons: + return [] + ordered = sorted( + enumerate(comparisons), + key=lambda item: (float(item[1]["exact_two_sided_p"]), str(item[1].get("comparison_id", ""))), + ) + adjusted: list[float] = [1.0] * len(comparisons) + running = 0.0 + total = len(comparisons) + for rank, (original_index, comparison) in enumerate(ordered): + candidate = min(1.0, (total - rank) * float(comparison["exact_two_sided_p"])) + running = max(running, candidate) + adjusted[original_index] = running + result: list[dict[str, object]] = [] + for index, comparison in enumerate(comparisons): + updated = dict(comparison) + updated["holm_adjusted_p"] = _stable_float(adjusted[index]) + updated["reject_at_0_05"] = adjusted[index] <= alpha + result.append(updated) + return result + + +def _latency_summary( + eval_rows: Sequence[Mapping[str, object]], + outcomes: Mapping[tuple[str, str], ProbeOutcome], +) -> dict[str, object]: + paired: list[float] = [] + missing: list[str] = [] + for row in eval_rows: + base_id = str(row["base_id"]) + values = [outcomes[(base_id, variant)].latency_ms for variant in VARIANTS] + if any(value is None for value in values): + missing.append(base_id) + else: + paired.append(float(statistics.median(value for value in values if value is not None))) + ordered = sorted(paired) + p95 = ordered[math.ceil(0.95 * len(ordered)) - 1] if ordered else None + return { + "unit": "within-base median of clear and ambiguous latency", + "formal_base_denominator": FORMAL_BASE_COUNT, + "complete_pairs": len(paired), + "missing_pairs": len(missing), + "median_ms": None if not paired else _stable_float(float(statistics.median(paired))), + "p95_ms_nearest_rank": None if p95 is None else _stable_float(p95), + } + + +def _language_trial( + row: Mapping[str, object], + outcome: ProbeOutcome, + evidence: Mapping[str, object], +) -> dict[str, object]: + gold = "unique" if outcome.variant == "clear" else "clarify" + value: dict[str, object] = { + "schema_version": 1, + "record_type": "language_probe", + "base_id": outcome.base_id, + "category": str(row["category"]), + "variant": outcome.variant, + "gold_disposition": gold, + "observed_disposition": outcome.disposition, + "reason": outcome.reason, + "correct": outcome.disposition == gold, + "evidence_status": outcome.evidence_status, + "input": str(evidence["command"]), + "raw_output": str(evidence.get("raw_output", "")), + "raw_output_sha256": outcome.raw_output_sha256, + "instruction_count": len(outcome.instructions), + "candidate_ids": list(outcome.candidate_ids), + "candidate_count": len(outcome.candidate_ids), + "error_type": outcome.error_type, + "latency_ms": outcome.latency_ms, + } + if outcome.variant == "ambiguous": + value["candidate_coverage"] = _candidate_coverage(row, outcome) + value["gold_candidate_set_match"] = set(outcome.candidate_ids) == set(row["candidate_entity_ids"]) + return value + + +def run_evaluation( + eval_rows: Sequence[dict[str, Any]], + protocol: Mapping[str, object], + evidence: Mapping[tuple[str, str], Mapping[str, object]], + *, + integrity: Mapping[str, object] | None = None, + model_run: Mapping[str, object] | None = None, +) -> tuple[list[dict[str, object]], dict[str, object]]: + """Evaluate already validated inputs and return deterministic trials/report.""" + + ttl_seconds = float(protocol["arms"][B2]["ttl_seconds"]) + outcomes: dict[tuple[str, str], ProbeOutcome] = {} + language_trials: list[dict[str, object]] = [] + for row in eval_rows: + base_id = str(row["base_id"]) + for variant in VARIANTS: + outcome = analyze_probe(row, variant, evidence[(base_id, variant)]) + outcomes[(base_id, variant)] = outcome + language_trials.append(_language_trial( + row, + outcome, + evidence[(base_id, variant)], + )) + + execution_trials: list[dict[str, object]] = [] + for row in eval_rows: + ambiguous = outcomes[(str(row["base_id"]), "ambiguous")] + execution_trials.append(_run_b0_trial(row, ambiguous)) + for arm in (B1, B2): + for mutation in MUTATIONS: + if mutation == "context_state_change" and row["category"] != "context_reference": + continue + execution_trials.append(_run_prepared_trial( + row, ambiguous, arm, mutation, ttl_seconds, + )) + + by_trial = { + (str(trial["base_id"]), str(trial["arm"]), str(trial["mutation"])): trial + for trial in execution_trials + } + base_ids = [str(row["base_id"]) for row in eval_rows] + + sensitivity = [outcomes[(base_id, "ambiguous")].is_clarification for base_id in base_ids] + specificity = [outcomes[(base_id, "clear")].is_unique for base_id in base_ids] + false_clarification = [outcomes[(base_id, "clear")].is_clarification for base_id in base_ids] + discrimination = [ambiguous and clear for ambiguous, clear in zip(sensitivity, specificity)] + coverage = [ + _candidate_coverage(row, outcomes[(str(row["base_id"]), "ambiguous")]) + for row in eval_rows + ] + completion = [ + bool(by_trial[(base_id, B2, "clean")]["exact_delta_success"]) + for base_id in base_ids + ] + + language_metrics = { + "sensitivity": binary_metric(sensitivity), + "specificity": binary_metric(specificity), + "false_clarification_rate": binary_metric(false_clarification), + "paired_discrimination": binary_metric(discrimination), + "candidate_coverage": binary_metric(coverage), + "clarification_completion": binary_metric(completion), + "inference_error_rate": binary_metric([ + any(outcomes[(base_id, variant)].evidence_status != "ok" for variant in VARIANTS) + for base_id in base_ids + ]), + "latency": _latency_summary(eval_rows, outcomes), + "independent_unit": "48 paired base IDs; the 96 probes are not independent samples", + "production_ppv_warning": "The deliberately balanced 1:1 probes do not estimate production PPV.", + } + + execution_metrics: dict[str, object] = {} + universal_vectors: dict[str, list[bool]] = {} + for arm in ARMS: + clean = [by_trial[(base_id, arm, "clean")] for base_id in base_ids] + arm_metrics: dict[str, object] = { + "ambiguous_clean_exact_delta_success": binary_metric([ + bool(trial["exact_delta_success"]) for trial in clean + ]), + "dispatch_coverage": binary_metric([ + trial["sut_calls"] == 1 and trial["dispatched"] is True for trial in clean + ]), + "wrong_target_transition_rate": binary_metric([ + bool(trial["wrong_target_transition"]) for trial in clean + ]), + "zero_preconfirm_calls": binary_metric([ + trial["preconfirm_sut_calls"] == 0 for trial in clean + ]), + } + if arm == B0: + arm_metrics["safe_abstention_rate"] = binary_metric([ + bool(trial.get("safe_abstention")) for trial in clean + ]) + arm_metrics["universal_guard_rate"] = { + "status": "not_applicable", + "reason": "B0 creates no prepared authorization to mutate", + } + arm_metrics["context_guard_rate"] = { + "status": "not_applicable", + "reason": "B0 creates no prepared authorization to mutate", + } + else: + per_mutation: dict[str, object] = {} + for mutation in MUTATIONS: + eligible_ids = [ + base_id for base_id in base_ids + if mutation != "context_state_change" + or by_trial[(base_id, arm, "clean")]["category"] == "context_reference" + ] + mutation_trials = [by_trial[(base_id, arm, mutation)] for base_id in eligible_ids] + per_mutation[mutation] = binary_metric([ + bool(trial["oracle_pass"]) for trial in mutation_trials + ]) + universal = [ + all(bool(by_trial[(base_id, arm, mutation)]["oracle_pass"]) + for mutation in UNIVERSAL_GUARD_MUTATIONS) + for base_id in base_ids + ] + universal_vectors[arm] = universal + context_ids = [ + str(row["base_id"]) for row in eval_rows if row["category"] == "context_reference" + ] + arm_metrics["mutation_oracle_rates"] = per_mutation + arm_metrics["universal_guard_rate"] = binary_metric(universal) + arm_metrics["context_guard_rate"] = binary_metric([ + bool(by_trial[(base_id, arm, "context_state_change")]["oracle_pass"]) + for base_id in context_ids + ]) + arm_metrics["mutation_denominator_note"] = ( + "Each mutation is a paired per-base diagnostic, not an independent pooled sample." + ) + execution_metrics[arm] = arm_metrics + + b0_clean = [bool(by_trial[(base_id, B0, "clean")]["exact_delta_success"]) for base_id in base_ids] + b1_clean = [bool(by_trial[(base_id, B1, "clean")]["exact_delta_success"]) for base_id in base_ids] + comparisons = [ + { + "comparison_id": "B1_vs_B0_ambiguous_clean_exact_delta", + **exact_mcnemar(B0, B1, b0_clean, b1_clean), + }, + { + "comparison_id": "B2_vs_B1_universal_guard", + **exact_mcnemar(B1, B2, universal_vectors[B1], universal_vectors[B2]), + }, + ] + comparisons = holm_adjust(comparisons) + + b1_expected: dict[str, object] = {} + for mutation in sorted(B1_INTENTIONAL_GAPS): + eligible = [ + trial for trial in execution_trials + if trial["arm"] == B1 and trial["mutation"] == mutation + ] + b1_expected[mutation] = { + "observed_expected_limitations": sum( + trial["interpretation"] == "expected_b1_guard_limitation" for trial in eligible + ), + "eligible_bases": len(eligible), + } + b2_failures: dict[str, int] = {} + for mutation in MUTATIONS: + count = sum( + trial["arm"] == B2 + and trial["mutation"] == mutation + and not bool(trial["oracle_pass"]) + for trial in execution_trials + ) + b2_failures[mutation] = count + + claimed_b1 = ("clean", "session_swap", "plan_swap", "unrelated_state_change") + unexpected_b1 = { + mutation: sum( + trial["arm"] == B1 + and trial["mutation"] == mutation + and not bool(trial["oracle_pass"]) + for trial in execution_trials + ) + for mutation in claimed_b1 + } + b2_suite_pass = not any(b2_failures.values()) + interpretation = { + "expected_baseline_outcomes": { + "B0_safe_abstentions": sum( + trial["arm"] == B0 and trial.get("safe_abstention") is True + for trial in execution_trials + ), + "B1_deliberately_missing_guard_failures": b1_expected, + "note": "Expected B0 abstention and B1 omitted-guard failures are comparison results, not B2 suite failures.", + }, + "unexpected_failures": { + "B1_claimed_behavior": unexpected_b1, + "B2_full_suite": b2_failures, + }, + "b2_suite_pass": b2_suite_pass, + } + + trials = sorted( + [*language_trials, *execution_trials], + key=lambda trial: ( + 0 if trial["record_type"] == "language_probe" else 1, + str(trial["base_id"]), + VARIANTS.index(str(trial["variant"])) if trial["record_type"] == "language_probe" else ARMS.index(str(trial["arm"])), + -1 if trial["record_type"] == "language_probe" else MUTATIONS.index(str(trial["mutation"])), + ), + ) + report: dict[str, object] = { + "schema_version": 1, + "protocol_id": protocol["protocol_id"], + "status": "complete", + "population": { + "independent_unit": "base_id", + "evaluation_bases": FORMAL_BASE_COUNT, + "paired_probes": FORMAL_BASE_COUNT * len(VARIANTS), + "context_guard_bases": CONTEXT_BASE_COUNT, + "selective_reruns": 0, + "infrastructure_errors_retained": True, + }, + "input_integrity": dict(integrity or {}), + "model_run": dict(model_run or { + "registered_configuration": {}, + "metadata_verified": False, + "observed_environment": None, + }), + "methods": { + "binary_intervals": protocol["statistics"]["binary_intervals"], + "paired_test": protocol["statistics"]["paired_test"], + "multiplicity": protocol["statistics"]["multiplicity"], + "latency_unit": protocol["statistics"]["latency_unit"], + "trial_reset": list(protocol["trial_reset"]), + "pseudo_replication_guards": list( + protocol["statistics"]["pseudo_replication_guards"] + ), + }, + "metrics": { + "language": language_metrics, + "execution": execution_metrics, + }, + "primary_inference": { + "method": "two-sided exact McNemar on paired base outcomes", + "multiplicity": "Holm correction across two pre-registered comparisons", + "comparisons": comparisons, + }, + "interpretation": interpretation, + "quality_gate": { + "result": "pass" if b2_suite_pass else "fail", + "criterion": "Every eligible B2 clean/guard trial passes its pre-registered oracle.", + }, + "trial_counts": { + "language_probe_records": len(language_trials), + "execution_trial_records": len(execution_trials), + "total_records": len(trials), + }, + "determinism": { + "timestamps_in_outputs": False, + "absolute_paths_in_outputs": False, + "fresh_reset_and_prepare_per_mutation": True, + }, + } + return trials, report + + +def evaluate_to_directory( + *, + dataset_path: Path, + protocol_path: Path, + freeze_path: Path, + evidence_path: Path, + output_dir: Path, + metadata_path: Path | None = None, +) -> dict[str, object]: + dataset_rows = _read_jsonl(dataset_path, "dataset") + protocol = _read_json(protocol_path, "protocol") + freeze = _read_json(freeze_path, "freeze manifest") + _validate_protocol(protocol) + eval_rows = _validate_rows(dataset_rows) + _validate_freeze(freeze, dataset_path, protocol_path, eval_rows) + evidence_rows = _read_jsonl(evidence_path, "frozen evidence") + evidence = _load_evidence(evidence_rows, eval_rows) + if metadata_path is None: + raise EvaluationInputError("formal evaluation requires model-run metadata") + metadata = _read_json(metadata_path, "model-run metadata") + evidence_sha256 = _sha256_file(evidence_path) + model_run = _validated_model_run( + metadata, + protocol, + _sha256_file(dataset_path), + evidence_sha256, + sum(item.get("status") != "ok" for item in evidence_rows), + ) + integrity = { + "dataset_sha256": _sha256_file(dataset_path), + "evaluation_sha256": _sha256_bytes(_canonical_lines(eval_rows)), + "protocol_sha256": _sha256_file(protocol_path), + "evidence_sha256": evidence_sha256, + "freeze_verified": True, + "evidence_pairs_verified": FORMAL_BASE_COUNT * len(VARIANTS), + "code_binding": model_run["code_binding"], + } + integrity["model_run_metadata_sha256"] = _sha256_file(metadata_path) + trials, report = run_evaluation( + eval_rows, + protocol, + evidence, + integrity=integrity, + model_run=model_run, + ) + output_dir.mkdir(parents=True, exist_ok=True) + (output_dir / "trials.jsonl").write_text( + "".join(canonical_json(trial) + "\n" for trial in trials), + encoding="utf-8", + ) + (output_dir / "report.json").write_text( + json.dumps(report, ensure_ascii=False, sort_keys=True, indent=2) + "\n", + encoding="utf-8", + ) + return report + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--from-frozen-evidence", + nargs="?", + const=DEFAULT_EVIDENCE, + type=Path, + required=True, + metavar="JSONL", + help="raw JSONL from run_model.py; omit JSONL to use evidence/domux_raw.jsonl", + ) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--dataset", type=Path, default=DEFAULT_DATASET) + parser.add_argument("--protocol", type=Path, default=DEFAULT_PROTOCOL) + parser.add_argument("--freeze", type=Path, default=DEFAULT_FREEZE) + parser.add_argument( + "--evidence-metadata", + type=Path, + default=DEFAULT_EVIDENCE_METADATA, + help="required JSON metadata emitted alongside run_model.py evidence", + ) + args = parser.parse_args(argv) + try: + report = evaluate_to_directory( + dataset_path=args.dataset, + protocol_path=args.protocol, + freeze_path=args.freeze, + evidence_path=args.from_frozen_evidence, + output_dir=args.output_dir, + metadata_path=args.evidence_metadata, + ) + except EvaluationInputError as exc: + parser.error(str(exc)) + print(canonical_json({ + "status": report["status"], + "quality_gate": report["quality_gate"]["result"], + "evaluation_bases": report["population"]["evaluation_bases"], + "trial_records": report["trial_counts"]["total_records"], + })) + return 0 if report["quality_gate"]["result"] == "pass" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/cases/domux-clarify-commit/evidence/ha_acceptance.json b/cases/domux-clarify-commit/evidence/ha_acceptance.json new file mode 100644 index 0000000..91d06e8 --- /dev/null +++ b/cases/domux-clarify-commit/evidence/ha_acceptance.json @@ -0,0 +1,650 @@ +{ + "execution_source_bindings": { + "binding_bundle_sha256": "9359a27e970603a1635d1c6307b5bbd651bda86a70ab6098a189c7bdce4d5fa1", + "digest_algorithm": "sha256", + "host_python": { + "implementation": "cpython", + "version": "3.12.12" + }, + "inputs": { + "data/scenarios.jsonl": { + "sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "size_bytes": 161098 + }, + "evidence/v1/domux_raw.jsonl": { + "sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "size_bytes": 49637 + } + }, + "module_origins_verified": true, + "path_base": "case_directory", + "pre_post_execution_match": true, + "schema_version": 1, + "sources": { + "clarify_commit.py": { + "sha256": "67e1dff8b413f7dffbd5ef1d826973153b86e35cb0da3b75aeec52d11160eaff", + "size_bytes": 226611 + }, + "ha_acceptance.py": { + "sha256": "4820b87a784e5a63de27ae536d747666b65678e22578397a8483dcd30336a75a", + "size_bytes": 88780 + } + } + }, + "home_assistant": { + "auth": { + "issue_http": 200, + "refresh_after_revoke_http": 400, + "revoke_http": 200, + "token_type": "Bearer", + "ttl_seconds": 1800 + }, + "health": { + "authenticated_api_http": 200, + "message": "API running.", + "unauthenticated_api_http": 401 + }, + "onboarding": { + "final": { + "analytics": true, + "core_config": true, + "integration": true, + "user": true + }, + "initial": { + "analytics": false, + "core_config": false, + "integration": false, + "user": false + }, + "requests": { + "analytics_http": 200, + "core_config_http": 200, + "integration_http": 200, + "users_http": 200 + } + }, + "phases": { + "service_call_accounting": { + "direct_rest_events": [ + { + "domain": "light", + "http_status": 200, + "phase": "setup", + "request_path": "/api/services/light/turn_on", + "service": "turn_on" + }, + { + "domain": "light", + "http_status": 200, + "phase": "setup", + "request_path": "/api/services/light/turn_on", + "service": "turn_on" + }, + { + "domain": "cover", + "http_status": 200, + "phase": "setup", + "request_path": "/api/services/cover/set_cover_position", + "service": "set_cover_position" + }, + { + "domain": "climate", + "http_status": 200, + "phase": "setup", + "request_path": "/api/services/climate/set_hvac_mode", + "service": "set_hvac_mode" + }, + { + "domain": "climate", + "http_status": 200, + "phase": "setup", + "request_path": "/api/services/climate/set_temperature", + "service": "set_temperature" + }, + { + "domain": "light", + "http_status": 200, + "phase": "external_fault_injection", + "request_path": "/api/services/light/turn_on", + "service": "turn_on" + } + ], + "external_fault_injection": 1, + "setup_direct_rest": 5, + "sut_dispatches": 3, + "total": 9 + }, + "setup": { + "classification": "direct_rest_state_normalization", + "dispatches": [ + { + "domain": "light", + "http": 200, + "service": "turn_on" + }, + { + "domain": "light", + "http": 200, + "service": "turn_on" + }, + { + "domain": "cover", + "http": 200, + "service": "set_cover_position" + }, + { + "domain": "climate", + "http": 200, + "service": "set_hvac_mode" + }, + { + "domain": "climate", + "http": 200, + "service": "set_temperature" + } + ], + "included_in_sut_dispatch_count": false, + "purpose": "setup_only" + }, + "sut": { + "adapter": "HomeAssistantRESTAdapter", + "case_count": 4, + "cases": [ + { + "case": "recorded_ambiguous_light_off", + "controlled_after": { + "brightness": null, + "color_temp_kelvin": null, + "entity_id": "light.ceiling_lights", + "rgb_color": null, + "state": "off" + }, + "controlled_before": { + "brightness": 178, + "color_temp_kelvin": 3000, + "entity_id": "light.ceiling_lights", + "rgb_color": [ + 255, + 177, + 110 + ], + "state": "on" + }, + "domain": "light", + "domux_evidence": { + "artifact": "evidence/v1/domux_raw.jsonl", + "artifact_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "base_id": "eval-duplicate_entity-01", + "line_number": 2, + "pair_verified": true, + "query_sha256": "f27717f08a911d7db2dcafcec7dc4fb5363b9cff40840596c17d548101b6fdcf", + "raw_output_sha256": "c5fee12f0a6f2de9ca00f1a3d64485625ddcceba18e34a799ddbfe38196dd76e", + "validation": "whole_artifact_and_per_field_sha256", + "variant": "ambiguous" + }, + "grounding": { + "candidate_ids": [ + "light.ceiling_lights", + "light.bed_light" + ], + "clarification_required": true, + "resolution": "resolve_clarification_submission", + "selected_entity_id": "light.ceiling_lights" + }, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "outcome": "COMMITTED", + "postcondition": { + "all_registered_entities_exact": true, + "matched_prepared_projection": true, + "reason": "committed", + "status": "COMMITTED" + }, + "replay": { + "accepted": false, + "dispatched": false, + "reason": "replayed_nonce", + "sut_dispatch_delta": 0 + }, + "scenario_provenance": { + "artifact": "data/scenarios.jsonl", + "artifact_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "base_id": "eval-duplicate_entity-01", + "binding_sha256": "39c46c2421f31df71a05882b7fdf43a0c79ce658250c70ee4ecb53c4fff80518", + "clarification_answer": "The Living Room light on the Ground Floor.", + "clarification_answer_sha256": "136300ffc636a6a93b5960d39f4b34d3526d46c7c5cf13e96426c24a5b045662", + "confirmed_instruction": { + "action": "turnOff", + "attribute": "*", + "device": "Light", + "floor": "Ground Floor", + "room": "Living Room", + "unit": "*", + "value": "*" + }, + "confirmed_instruction_sha256": "0d5831300f6f011e923f1ef0b69481f59157762fc0221cc3b562c1344b9af9d0", + "expected_target_entity_id": "light.eval_de_01_living", + "frozen_candidate_count": 3, + "ha_matching_candidate_count": 2, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "inventory_limitation": { + "full_scenario_inventory_reproduced": false, + "profile": "semantic_target_mapping_subset_not_full_scenario_inventory" + }, + "line_number": 17, + "post_clarification_model_call": false, + "row_sha256": "8b8f6498aef6bb4cb7fd5af3ac9adbef3e2cf0013c7a22451fc339e5f85cd871", + "scenario_target_to_ha_demo_entity": { + "ha_demo_entity_id": "light.ceiling_lights", + "scenario_target_entity_id": "light.eval_de_01_living", + "semantic_fields_match": true + }, + "source": "frozen_synthetic_scenario_gold", + "target_inventory_semantics": { + "aliases": [], + "device": "Light", + "domain": "light", + "floor": "Ground Floor", + "room": "Living Room" + }, + "used_for_resolution": true, + "variant": "ambiguous", + "variant_command_sha256": "f27717f08a911d7db2dcafcec7dc4fb5363b9cff40840596c17d548101b6fdcf" + }, + "service_shape": { + "data": { + "entity_id": "light.ceiling_lights" + }, + "domain": "light", + "service": "turn_off" + } + }, + { + "case": "recorded_unique_cover_position", + "controlled_after": { + "current_position": 20, + "entity_id": "cover.hall_window", + "state": "open" + }, + "controlled_before": { + "current_position": 80, + "entity_id": "cover.hall_window", + "state": "open" + }, + "domain": "cover", + "domux_evidence": { + "artifact": "evidence/v1/domux_raw.jsonl", + "artifact_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "base_id": "eval-duplicate_entity-02", + "line_number": 3, + "pair_verified": true, + "query_sha256": "cd4494727bc97d234d9c50f345468ab88b3571230ef610b5ef367d307f30e784", + "raw_output_sha256": "1435c4aa085aa8a5470b0cafc14629f15ea12f5ea0fdb255fe0f3fcb90a85edf", + "validation": "whole_artifact_and_per_field_sha256", + "variant": "clear" + }, + "grounding": { + "candidate_ids": [ + "cover.hall_window" + ], + "clarification_required": false, + "resolution": "resolve_unique_request", + "selected_entity_id": "cover.hall_window" + }, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "outcome": "COMMITTED", + "postcondition": { + "all_registered_entities_exact": true, + "matched_prepared_projection": true, + "reason": "committed", + "status": "COMMITTED" + }, + "replay": { + "accepted": false, + "dispatched": false, + "reason": "replayed_nonce", + "sut_dispatch_delta": 0 + }, + "scenario_provenance": { + "artifact": "data/scenarios.jsonl", + "artifact_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "base_id": "eval-duplicate_entity-02", + "binding_sha256": "98e9f02e52d1c81f6a7a18a785885ab64ad93c57a3bd50de8f4f7a68b437ba83", + "clarification_answer": "The Hall curtain on the First Floor.", + "clarification_answer_sha256": "2398268aa0cca19e8cfeb11b9c26f42dc893b16693bb6dd7bef9fbf2923ceb92", + "confirmed_instruction": { + "action": "set", + "attribute": "position", + "device": "Curtain", + "floor": "First Floor", + "room": "Hall", + "unit": "Percent", + "value": "20" + }, + "confirmed_instruction_sha256": "1435c4aa085aa8a5470b0cafc14629f15ea12f5ea0fdb255fe0f3fcb90a85edf", + "expected_target_entity_id": "cover.eval_de_02_upstairs_hall", + "frozen_candidate_count": 2, + "ha_matching_candidate_count": 1, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "inventory_limitation": { + "full_scenario_inventory_reproduced": false, + "profile": "semantic_target_mapping_subset_not_full_scenario_inventory" + }, + "line_number": 18, + "post_clarification_model_call": false, + "row_sha256": "24f7b968a30cba095932b3f5da47ce8f8fc65d521546a830de1b6ea149db7fb4", + "scenario_target_to_ha_demo_entity": { + "ha_demo_entity_id": "cover.hall_window", + "scenario_target_entity_id": "cover.eval_de_02_upstairs_hall", + "semantic_fields_match": true + }, + "source": "frozen_synthetic_scenario_gold", + "target_inventory_semantics": { + "aliases": [], + "device": "Curtain", + "domain": "cover", + "floor": "First Floor", + "room": "Hall" + }, + "used_for_resolution": false, + "variant": "clear", + "variant_command_sha256": "cd4494727bc97d234d9c50f345468ab88b3571230ef610b5ef367d307f30e784" + }, + "service_shape": { + "data": { + "entity_id": "cover.hall_window", + "position": 20 + }, + "domain": "cover", + "service": "set_cover_position" + } + }, + { + "case": "recorded_unique_climate_temperature", + "controlled_after": { + "entity_id": "climate.hvac", + "fan_mode": "on_high", + "state": "cool", + "temperature": 22.0 + }, + "controlled_before": { + "entity_id": "climate.hvac", + "fan_mode": "on_high", + "state": "cool", + "temperature": 24.0 + }, + "domain": "climate", + "domux_evidence": { + "artifact": "evidence/v1/domux_raw.jsonl", + "artifact_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "base_id": "eval-duplicate_entity-03", + "line_number": 5, + "pair_verified": true, + "query_sha256": "c892af8646f0e3d0e52226ddf5d5c0ac4fed0d977ced9592e693b2abaf88962b", + "raw_output_sha256": "37282efde74972be3bc4bdab6351683ed9be31dcefbff39e1527ce81a0da58a4", + "validation": "whole_artifact_and_per_field_sha256", + "variant": "clear" + }, + "grounding": { + "candidate_ids": [ + "climate.hvac" + ], + "clarification_required": false, + "resolution": "resolve_unique_request", + "selected_entity_id": "climate.hvac" + }, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "outcome": "COMMITTED", + "postcondition": { + "all_registered_entities_exact": true, + "matched_prepared_projection": true, + "reason": "committed", + "status": "COMMITTED" + }, + "replay": { + "accepted": false, + "dispatched": false, + "reason": "replayed_nonce", + "sut_dispatch_delta": 0 + }, + "scenario_provenance": { + "artifact": "data/scenarios.jsonl", + "artifact_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "base_id": "eval-duplicate_entity-03", + "binding_sha256": "4e149daeff09b5307563d5e96661e417f9349f9cc3129f80877ed2c7d4ac46c3", + "clarification_answer": "The Bedroom AC on the Second Floor, in Celsius.", + "clarification_answer_sha256": "32d9503216df62eda6dbbced0b276a73330754bbaa1c12c10175f079f91d6431", + "confirmed_instruction": { + "action": "set", + "attribute": "temperature", + "device": "AC", + "floor": "Second Floor", + "room": "Bedroom", + "unit": "Celsius", + "value": "22" + }, + "confirmed_instruction_sha256": "37282efde74972be3bc4bdab6351683ed9be31dcefbff39e1527ce81a0da58a4", + "expected_target_entity_id": "climate.eval_de_03_bedroom_second", + "frozen_candidate_count": 2, + "ha_matching_candidate_count": 1, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "inventory_limitation": { + "full_scenario_inventory_reproduced": false, + "profile": "semantic_target_mapping_subset_not_full_scenario_inventory" + }, + "line_number": 19, + "post_clarification_model_call": false, + "row_sha256": "ef0be54be7aaae140c846076fe9568dbea981fd6a96575b12cb6633057756bdf", + "scenario_target_to_ha_demo_entity": { + "ha_demo_entity_id": "climate.hvac", + "scenario_target_entity_id": "climate.eval_de_03_bedroom_second", + "semantic_fields_match": true + }, + "source": "frozen_synthetic_scenario_gold", + "target_inventory_semantics": { + "aliases": [], + "device": "AC", + "domain": "climate", + "floor": "Second Floor", + "room": "Bedroom" + }, + "used_for_resolution": false, + "variant": "clear", + "variant_command_sha256": "c892af8646f0e3d0e52226ddf5d5c0ac4fed0d977ced9592e693b2abaf88962b" + }, + "service_shape": { + "data": { + "entity_id": "climate.hvac", + "temperature": 22.0 + }, + "domain": "climate", + "service": "set_temperature" + } + }, + { + "binding": { + "after_external_mutation_state_digest": "80e68977cce90ffd209341d5cdd5fb029c287f2cc719802fd12abbc6ec6a7f06", + "before_external_mutation_state_digest": "dcb0369d3d033fa784ccf60d4aa5d20d00ef1ddb458817e281687679dca574c6", + "changed_after_external_mutation": true, + "matched_before_external_mutation": true, + "prepared_state_digest": "dcb0369d3d033fa784ccf60d4aa5d20d00ef1ddb458817e281687679dca574c6" + }, + "case": "recorded_study_light_state_drift_rejected", + "controlled_after_external_mutation": { + "brightness": 64, + "color_temp_kelvin": 3000, + "entity_id": "light.bed_light", + "rgb_color": [ + 255, + 177, + 110 + ], + "state": "on" + }, + "controlled_before_external_mutation": { + "brightness": 166, + "color_temp_kelvin": 3000, + "entity_id": "light.bed_light", + "rgb_color": [ + 255, + 177, + 110 + ], + "state": "on" + }, + "domain": "light", + "domux_evidence": { + "artifact": "evidence/v1/domux_raw.jsonl", + "artifact_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "base_id": "eval-duplicate_entity-04", + "line_number": 7, + "pair_verified": true, + "query_sha256": "01411704354a221b89abc4c31c1d577fa7aecce126d9fd3db966c68bcfe27973", + "raw_output_sha256": "6caea436fc1c6256aa56198c93270913e9323f6a0935267c55389eef078e57a4", + "validation": "whole_artifact_and_per_field_sha256", + "variant": "clear" + }, + "external_mutation": { + "classification": "out_of_band_fault_injection", + "data": { + "brightness_pct": 25.0, + "entity_id": "light.bed_light" + }, + "domain": "light", + "http_status": 200, + "included_in_sut_dispatch_count": false, + "observed_path": "/api/states/light.bed_light", + "request_path": "/api/services/light/turn_on", + "service": "turn_on", + "transport": "home_assistant_rest_api" + }, + "grounding": { + "candidate_ids": [ + "light.bed_light" + ], + "clarification_required": false, + "resolution": "resolve_unique_request", + "selected_entity_id": "light.bed_light" + }, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "outcome": "REJECTED_BEFORE_DISPATCH", + "rejection": { + "accepted": false, + "acknowledged": false, + "dispatched": false, + "outcome_unknown": false, + "reason": "state_changed", + "status": "INVALIDATED", + "sut_dispatch_delta": 0 + }, + "scenario_provenance": { + "artifact": "data/scenarios.jsonl", + "artifact_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "base_id": "eval-duplicate_entity-04", + "binding_sha256": "675d850e8d4ccb07709daa3d2b644a6c03f12aa5c239cf62dce62aa72e542b42", + "clarification_answer": "Use the Light in the Study on the Ground Floor.", + "clarification_answer_sha256": "57973f7724e2aeb7919bbe4ca3deae6cf3c901d4359d13239bda446d894beb8b", + "confirmed_instruction": { + "action": "set", + "attribute": "brightness", + "device": "Light", + "floor": "Ground Floor", + "room": "Study", + "unit": "Percent", + "value": "35" + }, + "confirmed_instruction_sha256": "6caea436fc1c6256aa56198c93270913e9323f6a0935267c55389eef078e57a4", + "expected_target_entity_id": "light.eval_de_04_study", + "frozen_candidate_count": 3, + "ha_matching_candidate_count": 1, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory", + "inventory_limitation": { + "full_scenario_inventory_reproduced": false, + "profile": "semantic_target_mapping_subset_not_full_scenario_inventory" + }, + "line_number": 20, + "post_clarification_model_call": false, + "row_sha256": "d9b32826a3e0229d5943e69a8ba3192635194f5ba87120945a6a92274e24c78e", + "scenario_target_to_ha_demo_entity": { + "ha_demo_entity_id": "light.bed_light", + "scenario_target_entity_id": "light.eval_de_04_study", + "semantic_fields_match": true + }, + "source": "frozen_synthetic_scenario_gold", + "target_inventory_semantics": { + "aliases": [], + "device": "Light", + "domain": "light", + "floor": "Ground Floor", + "room": "Study" + }, + "used_for_resolution": false, + "variant": "clear", + "variant_command_sha256": "01411704354a221b89abc4c31c1d577fa7aecce126d9fd3db966c68bcfe27973" + }, + "service_shape": { + "data": { + "brightness_pct": 35.0, + "entity_id": "light.bed_light" + }, + "domain": "light", + "service": "turn_on" + } + } + ], + "classification": "clarify_commit_sut", + "domux_evidence": { + "artifact": "evidence/v1/domux_raw.jsonl", + "artifact_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "pair_count": 4, + "validation": "whole_artifact_and_per_field_sha256" + }, + "external_fault_injection_count": 1, + "pipeline": [ + "ground_domux_request", + "resolve_clarification_submission_or_unique", + "PreparedActionStore.prepare", + "PreparedActionStore.commit", + "HomeAssistantRESTAdapter.call_service" + ], + "rejected_before_dispatch_count": 1, + "scenario_evidence": { + "artifact": "data/scenarios.jsonl", + "artifact_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "case_count": 4, + "ha_registry_profile": "semantic_target_mapping_subset_not_full_scenario_inventory" + }, + "successful_transition_count": 3, + "sut_dispatch_total": 3 + }, + "teardown": { + "classification": "credential_cleanup", + "refresh_after_revoke_http": 400, + "refresh_revoke_http": 200 + } + }, + "readiness": { + "endpoint": "/api/onboarding", + "http": 200 + } + }, + "image": { + "architecture": "amd64", + "docker_healthcheck": false, + "manifest_digest": "sha256:8e9751cb66d3ba6624f5360a7d31b0c6821f7f5b3fb8ba0d10d58f0f481c540c", + "operating_system": "linux", + "repository": "ghcr.io/home-assistant/home-assistant", + "version": "2026.8.3" + }, + "isolation": { + "container_count": 1, + "cpu_limit": 1.5, + "memory_limit_bytes": 2147483648, + "named_volume_count": 1, + "pids_limit": 512, + "random_loopback_binding": true, + "restart_policy": "no" + }, + "schema_version": 4, + "status": "passed" +} diff --git a/cases/domux-clarify-commit/evidence/v1/domux_raw.jsonl b/cases/domux-clarify-commit/evidence/v1/domux_raw.jsonl new file mode 100644 index 0000000..dd53588 --- /dev/null +++ b/cases/domux-clarify-commit/evidence/v1/domux_raw.jsonl @@ -0,0 +1,96 @@ +{"base_id":"eval-duplicate_entity-01","command":"Turn off the Light in the Living Room on the Ground Floor.","generated_token_ids_sha256":"d00b60c15cd4f86bd8b64098056255cc5441676a3fdfa96a4b6410c05d833e30","input_tokens":63,"latency_ms":685.781,"output_tokens":17,"query_sha256":"e76104feb1401fb711296d499558fe1e3401b21ff4b5576ca558f27510f9a699","raw_output":"turnOff|Light|*|*|*|Living Room|Ground Floor","raw_output_sha256":"0d5831300f6f011e923f1ef0b69481f59157762fc0221cc3b562c1344b9af9d0","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-01","command":"Turn off the light.","generated_token_ids_sha256":"7c2127ce4aef5440a46c2dbd5bc0ff9cabf4ce4ef5848b45eb45abafbffd4e8f","input_tokens":55,"latency_ms":602.55,"output_tokens":15,"query_sha256":"f27717f08a911d7db2dcafcec7dc4fb5363b9cff40840596c17d548101b6fdcf","raw_output":"turnOff|Light|*|*|*|*|*","raw_output_sha256":"c5fee12f0a6f2de9ca00f1a3d64485625ddcceba18e34a799ddbfe38196dd76e","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-02","command":"Set the Curtain in the Hall on the First Floor to 20 percent.","generated_token_ids_sha256":"e6e43a902c44bf902d0561d244fba8304d43acfffadf71162caf0b38a6574554","input_tokens":66,"latency_ms":682.779,"output_tokens":17,"query_sha256":"cd4494727bc97d234d9c50f345468ab88b3571230ef610b5ef367d307f30e784","raw_output":"set|Curtain|position|20|Percent|Hall|First Floor","raw_output_sha256":"1435c4aa085aa8a5470b0cafc14629f15ea12f5ea0fdb255fe0f3fcb90a85edf","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-02","command":"Set the hall curtain to 20 percent.","generated_token_ids_sha256":"230b73d4ee58fe4a08e8d44d74511f7b17a81aa024ce1e002da73a2fc8455f19","input_tokens":60,"latency_ms":638.952,"output_tokens":16,"query_sha256":"72305d429984ae50b03802d95601875d66180d605ec3f4566b4333648576d9f1","raw_output":"set|Curtain|position|20|Percent|Hall|*","raw_output_sha256":"49c6a5712612bbe64e30d565c48cf414e23b2e583a552329522ed495a7ae5802","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-03","command":"Set the AC in the Bedroom on the Second Floor to 22 Celsius.","generated_token_ids_sha256":"9bae80fe9d08c7146f03624cd744a6d4ae80af1844711299364a343bf80a2735","input_tokens":66,"latency_ms":641.124,"output_tokens":16,"query_sha256":"c892af8646f0e3d0e52226ddf5d5c0ac4fed0d977ced9592e693b2abaf88962b","raw_output":"set|AC|temperature|22|Celsius|Bedroom|Second Floor","raw_output_sha256":"37282efde74972be3bc4bdab6351683ed9be31dcefbff39e1527ce81a0da58a4","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-03","command":"Set the bedroom AC to 22 degrees.","generated_token_ids_sha256":"43e70517710b99fc47468e70f5a8fb4fd6048c536c4a23696100c269dac48747","input_tokens":60,"latency_ms":600.227,"output_tokens":15,"query_sha256":"0236e69805263c4629cdf579541ea6f799d73d72a01b87903ae0aff5975f98b0","raw_output":"set|AC|temperature|22|Celsius|Bedroom|*","raw_output_sha256":"b02329a6f2a8b93cef8f4949870cafbbc4388304b997bc92762767402ad81c87","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-04","command":"Set the Light in the Study on the Ground Floor to 35 percent brightness.","generated_token_ids_sha256":"db74e1edaf9e7daec5e25ef6f89e4bff788c2110b66d0a88712f9f5e34ee47c0","input_tokens":67,"latency_ms":644.55,"output_tokens":16,"query_sha256":"01411704354a221b89abc4c31c1d577fa7aecce126d9fd3db966c68bcfe27973","raw_output":"set|Light|brightness|35|Percent|Study|Ground Floor","raw_output_sha256":"6caea436fc1c6256aa56198c93270913e9323f6a0935267c55389eef078e57a4","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-04","command":"Set the reading light to 35 percent brightness.","generated_token_ids_sha256":"692eba260d8300c1f08b937831171ad3e1fb195bc03e6757d7a79e3807ee2586","input_tokens":61,"latency_ms":643.825,"output_tokens":16,"query_sha256":"d5eb6bb8bbb66a68cf33f96d753ab17d2c97f4be5b5120f7b1d9e5f9ff20e4f6","raw_output":"set|Reading Light|brightness|35|Percent|*|*","raw_output_sha256":"3f0e0454d295edc5611e2555128ee61365956c5f1a2a16ed0817318049da8941","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-05","command":"Set the Curtain in the North Balcony on the First Floor to 70 percent.","generated_token_ids_sha256":"008b4cd05c0a3bf6e78f0e60bfaf9320a6c5fb19cccfb89a82450c2857c763a0","input_tokens":68,"latency_ms":759.512,"output_tokens":19,"query_sha256":"6d0e6ad76eb1f3fdf06179c7ab49552bacd8220ca94c370186017b8c9b02fac6","raw_output":"set|Curtain|position|70|Percent|North Balcony|First Floor","raw_output_sha256":"c0a7fd2bfe4d730a8b50a3dfea92016f3776249535656c331fa5b5b7496beaf5","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-05","command":"Open the balcony curtain to 70 percent.","generated_token_ids_sha256":"cde613a31f31629d9467e95380363e1a2e0e49b1d79abc0d40eeb0c74a126f03","input_tokens":60,"latency_ms":677.206,"output_tokens":17,"query_sha256":"5a77e78a196c8b85a8e8031d900c5e74888447c88464cf02850682fe220a8268","raw_output":"set|Curtain|position|70|Percent|Balcony|*","raw_output_sha256":"7df4bc75c527d92f81b88e41fd4784d9b9b80b94e985d188cf6662dd307f2977","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-06","command":"Set the AC in the Guest Bedroom on the Ground Floor to Dry mode.","generated_token_ids_sha256":"c885c73bc55ea9eb1728071f51a893e493e4bd4d4354f0c68492cab03324685a","input_tokens":65,"latency_ms":637.239,"output_tokens":16,"query_sha256":"c0fc2678d871d16b919b33f748d468b4749bfc4381949f35e558d87fee700243","raw_output":"set|AC|mode|Dry|*|Guest Bedroom|Ground Floor","raw_output_sha256":"6aafc2d7ac38dc405318129698da6a10b496ffaa5032b6569f34b94e8fd521b0","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-06","command":"Set the guest bedroom AC to Dry mode.","generated_token_ids_sha256":"2df50df12bf7ffa254467f6450123445436f9ab1933e38bfc814f06f029a38fe","input_tokens":59,"latency_ms":598.508,"output_tokens":15,"query_sha256":"fe12d9fb43f9dd424d3ae91ca3e00185090c1cfcfb46ab67bbab6e841b2a60bb","raw_output":"set|AC|mode|Dry|*|Guest Bedroom|*","raw_output_sha256":"44d12f7fe2629277de05c3908fb43b5fc3564349e5bd207d3d0ae2d370a2259c","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-07","command":"Turn on the Light in the Staircase on the First Floor.","generated_token_ids_sha256":"38a57e384cec3c607a250f0619a5ec6fd70c06380bc22c2c7e56f2bfa196e7ac","input_tokens":63,"latency_ms":718.323,"output_tokens":18,"query_sha256":"cfae18ab050d6d6c26200f630a8c1c1acd4e20f167bcc40288a24a6a12cc2311","raw_output":"turnOn|Light|*|*|*|Staircase|First Floor","raw_output_sha256":"7c5e5fb21934bbb587556142ba0babbe82285b4f88781bce541b00be1332b28e","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-07","command":"Turn on the staircase light.","generated_token_ids_sha256":"761e4491f8da0cf9b7c7e7751fba93fa1141969ab4848c66ccdad29a9e0276e0","input_tokens":56,"latency_ms":674.959,"output_tokens":17,"query_sha256":"dfa3838f58e83b4ffab10439f07082aaf9614fad64eb6759ec47162d82939d06","raw_output":"turnOn|Light|*|*|*|Staircase|*","raw_output_sha256":"c42f2203000bc22fc9c67fbae57f39e3d3adb7895b16397421ab9e892c4b7d3d","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-08","command":"Set the Curtain in the Media Room on the Ground Floor to 75 percent.","generated_token_ids_sha256":"ad5a12c4bf2aa43373d6522eafedca0757e78d5976ed917e6ed0e27aa574b0bd","input_tokens":67,"latency_ms":714.694,"output_tokens":18,"query_sha256":"64c65e6e424942f72282cf720dd89c8350f83fc356e3d2a39c5dc2ba26255c3b","raw_output":"set|Curtain|position|75|Percent|Media Room|Ground Floor","raw_output_sha256":"6639aa302ad9483640ea6866efac861db772e4583a32233ff80950e1b09ece3b","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-08","command":"Set the main curtain to 75 percent.","generated_token_ids_sha256":"4effc81ab00954d35322ed55a36643c7d361425b168afe3f20e4a162cca7bfd0","input_tokens":60,"latency_ms":639.597,"output_tokens":16,"query_sha256":"2808a827a4d3e0080603405a61e6ff279c5bd3ac3f94bf175e87a0567885aeb6","raw_output":"set|Main Curtain|position|75|Percent|*|*","raw_output_sha256":"789831186b66553c11c15241025fef603afed553ec9addf35f962d1e9862d7d5","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-09","command":"Turn off the AC in Studio B on the Second Floor.","generated_token_ids_sha256":"f6ae5cf6a663030eff58a8dbb4045ecb9a9362437e61a15b3842dffafb864fa6","input_tokens":62,"latency_ms":680.067,"output_tokens":17,"query_sha256":"ca9969927f7eecd2622814354dae8bcd0d39396109167b4ad200dac8c7194613","raw_output":"turnOff|AC|*|*|*|Studio B|Second Floor","raw_output_sha256":"b46fb1db55332b290d780612302631c07668c3a58152ec3139886ee06a8a1888","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-09","command":"Turn off the studio AC.","generated_token_ids_sha256":"650227ac8cdde28d011029b75a03a7a3c431c1e70981ff74f4e5f6ffae8e214e","input_tokens":56,"latency_ms":608.824,"output_tokens":15,"query_sha256":"c2a882a1f0a190faee0e5ee4b28174a8121122dc6397dd151c0f09cb10174f37","raw_output":"turnOff|AC|*|*|*|Studio|*","raw_output_sha256":"30173d85fe47d9f7e46060628db1873df7d4e2ab528a8163e1ac3f047de7907f","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-10","command":"Set the Light in the Dining Room on the Ground Floor to Warm White.","generated_token_ids_sha256":"6e9a15b3d0d453a70e554a6cc9bed12c076e8be8de72be17c997331c773556af","input_tokens":65,"latency_ms":675.876,"output_tokens":17,"query_sha256":"47585e327502dd81b6d0b6f6bcf4ea1fe3fa0ac17d425f9b37e27553cb197f80","raw_output":"set|Light|color|Warm White|*|Dining Room|Ground Floor","raw_output_sha256":"2ea998b0be86371c9d190f46c9b76edb71d74d01d2fb5a8ddc19838806860245","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-10","command":"Make the light Warm White.","generated_token_ids_sha256":"c44403297857c6713d313e6c728abb869ac315b615961d0e738e7944468c9011","input_tokens":56,"latency_ms":591.021,"output_tokens":15,"query_sha256":"f6144193b11ae80424796f77f20b7eeadb023bb100e7fd8dca32d5d32db3bcf2","raw_output":"set|Light|color|Warm White|*|*|*","raw_output_sha256":"c66d20c36cb83536b10023dedb1348cc618cd5a7aa1a623f05b633813ccc4986","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-11","command":"Close the Curtain in Bedroom 2 on the First Floor.","generated_token_ids_sha256":"f61167ecf59d0c2388a70ea9193710a056a0c8e3fb472c820fd0ae8d738ee01d","input_tokens":62,"latency_ms":748.076,"output_tokens":19,"query_sha256":"4a57324cf0e569b6d935cdf1ae3720aca754cfae17ffa9b817f14908df3da696","raw_output":"turnOff|Curtain|*|*|*|Bedroom 2|First Floor","raw_output_sha256":"bd81cde577be6944a62e4f2e3267469ce4f7df63dfab60473505fcf17808a433","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-11","command":"Close the bedroom curtain.","generated_token_ids_sha256":"085167737b1b578c3cc1de61c44573455e9776ea87529bd3a3351ccfa753b2d1","input_tokens":55,"latency_ms":631.342,"output_tokens":16,"query_sha256":"2d44a707414ca0a85caf32edf3fed546f1ebbb91a98b71710591d9058aa030e8","raw_output":"turnOff|Curtain|*|*|*|Bedroom|*","raw_output_sha256":"53b4482c1db140656195473a39e2c3e988dde73ba81f2330d9941553d4a831cc","status":"ok","variant":"ambiguous"} +{"base_id":"eval-duplicate_entity-12","command":"Set the AC in the West Office on the First Floor to 25 Celsius.","generated_token_ids_sha256":"c8f64f6ef62913d49be230547dee7c697cc9428f318783f1aee2ca90e80fe161","input_tokens":67,"latency_ms":671.453,"output_tokens":17,"query_sha256":"925ffa49bb020bb8e87475af2a43bc90ae3dc10b9a7ed18a7135757fe5ca9735","raw_output":"set|AC|temperature|25|Celsius|West Office|First Floor","raw_output_sha256":"8a2538265852cf6af81ba4960b8c1a15ef0c4d9f8829eeafb49fcbc6572c56b3","status":"ok","variant":"clear"} +{"base_id":"eval-duplicate_entity-12","command":"Set the office AC to 25 degrees.","generated_token_ids_sha256":"7bcf6b0cea77a63d5ebace38f88b76c0265a284f4f44e52afb42c52a324aee2c","input_tokens":60,"latency_ms":594.597,"output_tokens":15,"query_sha256":"70c2ddd41c68e2853f848542e494041d6be4899592097e66d2b08414eb4ca319","raw_output":"set|AC|temperature|25|Celsius|Office|*","raw_output_sha256":"c753615867b6aa82860fa57903c2fe0051ac454703f4f683110b79e2393de7c9","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-01","command":"Set the Light in the Kitchen on the Ground Floor to 40 percent brightness.","generated_token_ids_sha256":"2cb7cfa4844b14614f34cdc8d048b7e91d9ad9817f5e3482263d3f2b1d0f2361","input_tokens":67,"latency_ms":635.394,"output_tokens":16,"query_sha256":"aaf03b6752bed7bef89ca0602ec9276041bdb11964d0bd28923ca6ac3c2a59e6","raw_output":"set|Light|brightness|40|Percent|Kitchen|Ground Floor","raw_output_sha256":"8c2bbeddef57eea339245e602555421c56313c847e18227877b5b347996878dd","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-01","command":"Set the light to 40 percent.","generated_token_ids_sha256":"265f3d81f59b5bfccb1ef1630a40bf272210054956c40807c010995db2cd4174","input_tokens":59,"latency_ms":591.985,"output_tokens":15,"query_sha256":"48b7a8ff77c4beac154b0fb45ede02e9217f5e4e116df0b4c81e4a7f18d409c9","raw_output":"set|Light|brightness|40|Percent|*|*","raw_output_sha256":"ca97f9ddd1f84cb7dd78af1002f9e702325d18c57fe9a5ac37fae1403eac4657","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-02","command":"Set the AC in the Office on the First Floor to 23 Celsius.","generated_token_ids_sha256":"4543ca4e66e15e11397d7d8fa0fcfae3ed1de3f77237d5a104731d9ade37d39d","input_tokens":66,"latency_ms":631.55,"output_tokens":16,"query_sha256":"4e9bf4cecc6eb885a2ba3c367440dffb0595472534f87104d42581d23134814f","raw_output":"set|AC|temperature|23|Celsius|Office|First Floor","raw_output_sha256":"e1df732c23315a5521a737dc8f685bf6f896b8c020e912debe8c2852bb99a838","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-02","command":"Set the office AC.","generated_token_ids_sha256":"5306b293077b20577aa046e0e6636cc2f73aed1fd2f21afd3cee404f8eea4838","input_tokens":55,"latency_ms":554.805,"output_tokens":14,"query_sha256":"916f13394140bf89c88c21abaed2a0707657e0eb71b8bb1a5ad9412255fb9ccc","raw_output":"set|AC|mode|Cool|*|Office|*","raw_output_sha256":"b31eff0bc3004077a79922c82b0e9db98424ce75d7fdea45325bcfa711d2e130","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-03","command":"Set the Curtain in the Bedroom on the Second Floor to 65 percent.","generated_token_ids_sha256":"626be39b29978a1d0db96e451c3c1d5a9d81ac676b1f6180d1be73f68a044dbe","input_tokens":66,"latency_ms":674.476,"output_tokens":17,"query_sha256":"5a0b8e3cac9b67b1e4bcf40f4c3e4fbed054e7c17f30f50a99f90a66f4ca926e","raw_output":"set|Curtain|position|65|Percent|Bedroom|Second Floor","raw_output_sha256":"ccebbf6112f51c9723fb64b2c9b7304fb53b95fd798d63e8bf678255e01a212d","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-03","command":"Adjust the bedroom curtain.","generated_token_ids_sha256":"4ee3c2f105566a37ab371dfdd5b51fcd883b8cc93ba4ca9515c9d368527df4cd","input_tokens":55,"latency_ms":630.258,"output_tokens":16,"query_sha256":"ba50b847d861c459b13517a6b5c1a5f404352b8d0d6cfaf025e2bb1c806df6d8","raw_output":"adjustUp|Curtain|position|*|*|Bedroom|*","raw_output_sha256":"f7578a93a34b770c2fee21c46e1606100d645d359b8245f5857c095186584c5d","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-04","command":"Set the Light in the Living Room on the Ground Floor to Blue.","generated_token_ids_sha256":"ac2c34ff15b8f42453d426ea1eba9f83376a01186514b3d7bbeefe1b731edb00","input_tokens":64,"latency_ms":629.788,"output_tokens":16,"query_sha256":"81933eb8cd9dc3d3a8e739e96d9242828eecd0bd8c2fe4e9bbc6f707106addb9","raw_output":"set|Light|color|Blue|*|Living Room|Ground Floor","raw_output_sha256":"b815b66bc2424600874eac6f57b9c3f4222676abaec555030fbfe06883eb550c","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-04","command":"Change the living room light color.","generated_token_ids_sha256":"346a0d232df77458b9c853138d71cb54ff69b065ff01d77034a11c17a67114fc","input_tokens":57,"latency_ms":586.897,"output_tokens":15,"query_sha256":"e31a70e30e96d3de8c78944a17170f04fa56d00a6dcd5d2b03c02cdf8ca7cc05","raw_output":"set|Light|color|*|*|Living Room|*","raw_output_sha256":"6707054aac377fcd78963243453af82b6b7981f8d803709b037c97bb547816f6","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-05","command":"Set the AC in the Landing on the First Floor to 21 Celsius.","generated_token_ids_sha256":"ecdb7e7b3bfecc462034a9ac70b6313f84d368490e77f89358ae59fb31de5281","input_tokens":66,"latency_ms":634.845,"output_tokens":16,"query_sha256":"ed7be62ab20f30eaf83050f622c9ce40fa88b09ebbd06bb46309313ae941e8e1","raw_output":"set|AC|temperature|21|Celsius|Landing|First Floor","raw_output_sha256":"03e8c95e3aa997cff815bb20ce3f15c3d3f4c05e46db5c40b0168a11d54a9fc8","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-05","command":"Set the upstairs AC to 21 degrees.","generated_token_ids_sha256":"4ccb630859c8bd6b6504854c09e22aafd078aa8084f5df93c585e5913b791c42","input_tokens":60,"latency_ms":629.467,"output_tokens":16,"query_sha256":"b843a955cd2ca8cff4276a5be4e3be6758bc77b83fa8e9df89dd47ea8bf8edee","raw_output":"set|AC|temperature|21|Celsius|*|Upstairs","raw_output_sha256":"46527a4f7923d40d9c85a6b3d18b89ea61c35ba56100a1dceb17d39d7b7e2ffb","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-06","command":"Turn off the AC in the Study on the Ground Floor.","generated_token_ids_sha256":"6ada5e1156998d56b3cb75abc6e446c324fe2cd826db59b398e5b82bf1ebd77e","input_tokens":62,"latency_ms":633.12,"output_tokens":16,"query_sha256":"e678b6bad5dfe20bc829f836be00825d5721c56902a5673b8b8f6f9a3caf5a8b","raw_output":"turnOff|AC|*|*|*|Study|Ground Floor","raw_output_sha256":"3bc1bd5fcd9fb6cc61d1f03f8d10bcaed4d42c4f344fb51a1d2eabddc13f2299","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-06","command":"Turn off the device in the study.","generated_token_ids_sha256":"505ac663d2dae57704eae95031fb8f3b65a97d4bf9e597ea0b1f40aac35f1c35","input_tokens":58,"latency_ms":594.626,"output_tokens":15,"query_sha256":"846b5c9a4fbfa5dfa0fe4ccf5195c57e3017d97f92ba2be7de72264347512b96","raw_output":"turnOff|Light|*|*|*|Study|*","raw_output_sha256":"0a6418f3047957be1fc0f03d1de4975187e64876b900b85913b6e3d283f11473","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-07","command":"Set the Curtain in the Patio on the Ground Floor to 30 percent.","generated_token_ids_sha256":"af84d7fb8f164a6c9dfdef13f1387ad9afc808f20ddecf7114c9bf7a68fbc761","input_tokens":66,"latency_ms":711.872,"output_tokens":18,"query_sha256":"3c6e5fc8a9852e5e806e4dfb1f468abd30580d3d07194497ba7585938d74830b","raw_output":"set|Curtain|position|30|Percent|Patio|Ground Floor","raw_output_sha256":"0a5c74b302799e40ba732af5bf1fde34fea690c1be9d01d1b73d02fe92534dc0","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-07","command":"Set the curtain to 30 percent.","generated_token_ids_sha256":"e886b7aa02ce426f818bbded6b84a0ed07bd73483c555acb0652ecc3ddca60ef","input_tokens":59,"latency_ms":631.205,"output_tokens":16,"query_sha256":"294990f1c29dca693d82a1b8d8cc188ad7df4b65c2b7868632fbda1715ac37af","raw_output":"set|Curtain|position|30|Percent|*|*","raw_output_sha256":"6770fc640cb2fd348f0c27445c9d2ae0df6d156cfd638ff08a764ee86663b786","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-08","command":"Set the Light in the Hall on the Ground Floor to 55 percent brightness.","generated_token_ids_sha256":"a8a385e8187dfb35638bbdf8cb0ed08b294768bdcbc11417597eeca3f4fea8a7","input_tokens":67,"latency_ms":630.769,"output_tokens":16,"query_sha256":"8ad94f073c991e4f2c2350afbdc108a0ad9fc6df689aee0f20f4300fce314926","raw_output":"set|Light|brightness|55|Percent|Hall|Ground Floor","raw_output_sha256":"395520e32859717c36fe603e0fbe1e28e70c0c0e7cdd8649c141ebce40e306f5","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-08","command":"Set brightness to 55 percent.","generated_token_ids_sha256":"e31f50c9d08320ce4813d89394fc19c32470c76f1dec227c79bb1746725b4955","input_tokens":58,"latency_ms":592.91,"output_tokens":15,"query_sha256":"e622117ec8fb71de814419ab2e64ac019cd8e88dbff9860a1761de1d247137b9","raw_output":"set|Light|brightness|55|Percent|*|*","raw_output_sha256":"e79c150801f85b0e3e5951cc02cbbbdb4d28cc4f451ee64a832b30278b900c14","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-09","command":"Set the AC in the Bedroom on the First Floor to Heat mode.","generated_token_ids_sha256":"09b4ed55d99fdd231dc920f4db1c2fc30d825637f1cc88ceae819238f018111b","input_tokens":64,"latency_ms":597.403,"output_tokens":15,"query_sha256":"0c327d35522c12898f5558639839a89536d480a9fce67aaff10c111afa1ce3c4","raw_output":"set|AC|mode|Heat|*|Bedroom|First Floor","raw_output_sha256":"46e3e9ddddb7775dd62ad45dbf650b2c8c31f9ce7f127a285906393370962227","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-09","command":"Change the bedroom AC mode.","generated_token_ids_sha256":"d8d4d8606bfd52456272bc95946a7fc8facf564b866a065582c99206d3f3eb8a","input_tokens":56,"latency_ms":553.08,"output_tokens":14,"query_sha256":"1e3d91e4fa358516ac06bafd1409e8522f7aa84e8c3314b1a4cb8570b509b3b4","raw_output":"set|AC|mode|*|*|Bedroom|*","raw_output_sha256":"aec437e6f192b1abea7e26b758e0861c31e987f8e1efbcbcae6a5342c579f462","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-10","command":"Set the Curtain in the Nursery on the First Floor to 80 percent.","generated_token_ids_sha256":"c1b09c3b9963b6662248368c12e548a69a96a1e59ae5f6f68851092aeeee48a7","input_tokens":66,"latency_ms":709.301,"output_tokens":18,"query_sha256":"8f557f63113aac4d12669317dc1e72dce907d4ed578f766034042ec7f652263c","raw_output":"set|Curtain|position|80|Percent|Nursery|First Floor","raw_output_sha256":"fce6f38ba43e6c66943ceec1f5f93d8f4ee774115b7f7763904b60a8b17177c6","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-10","command":"Move the nursery curtain.","generated_token_ids_sha256":"2a59eba0359da643e92e4dc17405b1aee902447a237edd8e933e6ac85995840f","input_tokens":55,"latency_ms":669.395,"output_tokens":17,"query_sha256":"bb104f261688e593abc56d9cd46341fc7e4f483c62d204fbb3dcf1b6cd2da318","raw_output":"adjustUp|Curtain|position|*|*|Nursery|*","raw_output_sha256":"ef8e8570c8d6f3e7efd28c80d29e403898cff2491ef295d486ee6dfd3e9fa21a","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-11","command":"Set the Light in the Entry on the Ground Floor to 3500 Kelvin.","generated_token_ids_sha256":"da73cb9630b1c62ce14ead2a934ac7a55007a97fc5b274972bd5fb8bd65d48bb","input_tokens":68,"latency_ms":788.632,"output_tokens":20,"query_sha256":"57a1ccf25ff139ed6e84f7bf291650b01054a70fe9b60caab874e832a1514250","raw_output":"set|Light|colorTemperature|3500|Kelvin|Entry|Ground Floor","raw_output_sha256":"06db8a31f285d547fcb5fb0714fcf5d2604468c5296b08a3e04fc63cf47ee6f2","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-11","command":"Set the light color temperature to 3500K.","generated_token_ids_sha256":"73a979c053443b0a7d7d3cb0c6503c3604719732c3bcdffd7dfc858aeeb70522","input_tokens":63,"latency_ms":747.728,"output_tokens":19,"query_sha256":"27bd1bdab211db794ff63bbb7c879b60e6c1f9812081816109fc4064874eb56c","raw_output":"set|Light|colorTemperature|3500|Kelvin|*|*","raw_output_sha256":"81c412f7603670921d94e0ca248b1c43b7e09517428fe27aee28f2b298839a49","status":"ok","variant":"ambiguous"} +{"base_id":"eval-missing_slot-12","command":"Set the AC in the Dining Room on the Ground Floor to 24 Celsius.","generated_token_ids_sha256":"0f578c493ef77921cb47692899428c66ab49dc7c913a78c274655caabce3ce25","input_tokens":67,"latency_ms":674.994,"output_tokens":17,"query_sha256":"9c858de21667d25dcd040d51d1c842222f14d8644f682e4ff8582012ca5a10b2","raw_output":"set|AC|temperature|24|Celsius|Dining Room|Ground Floor","raw_output_sha256":"791484c40e04ba5b981a21ba281d24f4f8544ff40a7edcf9c5d3f189e5b54002","status":"ok","variant":"clear"} +{"base_id":"eval-missing_slot-12","command":"Set the downstairs temperature to 24.","generated_token_ids_sha256":"430839fb70499c2fe9f5f0dcc7de3096d7cb4ef28ccd05836d34b49475c0adc3","input_tokens":59,"latency_ms":632.011,"output_tokens":16,"query_sha256":"55bda68db2041d5166bebc20d6af2795c0549453ab21f0de86d083b4e705f18a","raw_output":"set|AC|temperature|24|Celsius|*|Downstairs","raw_output_sha256":"412d686003ec4155f1aa4e09084d717b233eaa00f9c50208bf0b0ee0bb3ea7b2","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-01","command":"Switch off the Ground Floor Light in the Living Room.","generated_token_ids_sha256":"d00b60c15cd4f86bd8b64098056255cc5441676a3fdfa96a4b6410c05d833e30","input_tokens":61,"latency_ms":667.433,"output_tokens":17,"query_sha256":"da62d3925ec6e46dfbb7599d535b16e63b396919d2f91c11c9fedbc2b036993d","raw_output":"turnOff|Light|*|*|*|Living Room|Ground Floor","raw_output_sha256":"0d5831300f6f011e923f1ef0b69481f59157762fc0221cc3b562c1344b9af9d0","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-01","command":"Turn it off.","generated_token_ids_sha256":"7c2127ce4aef5440a46c2dbd5bc0ff9cabf4ce4ef5848b45eb45abafbffd4e8f","input_tokens":54,"latency_ms":587.228,"output_tokens":15,"query_sha256":"9122d2f528d1757cf7b307cfb9ca804d711f0d45babd87911705cb7581dcfe4c","raw_output":"turnOff|Light|*|*|*|*|*","raw_output_sha256":"c5fee12f0a6f2de9ca00f1a3d64485625ddcceba18e34a799ddbfe38196dd76e","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-02","command":"Set the AC in the Office on the First Floor to 22 Celsius.","generated_token_ids_sha256":"c6f403380a4712fff1ff5b639ea07fb3ecd4535a8d0864c8010580ed4a18e3ea","input_tokens":66,"latency_ms":634.845,"output_tokens":16,"query_sha256":"7ce830dd79ecc3bbbdf0e249489713d198c253a38d9447d00272440f8e5ec801","raw_output":"set|AC|temperature|22|Celsius|Office|First Floor","raw_output_sha256":"86743eaecaf8100c3a435a05b0255fb3650b651c89d842d45a18e9f89a63f2a2","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-02","command":"Set that one to 22.","generated_token_ids_sha256":"a3c50119add5d61fa22fddfba8d2e898673b9c49871c88667f2c625a3be83fc1","input_tokens":58,"latency_ms":594.561,"output_tokens":15,"query_sha256":"13dbb5f5b153df99eb0827c058bd511ff03a0791bdb09327324e75c96e904ffc","raw_output":"set|AC|temperature|22|Celsius|*|*","raw_output_sha256":"3e126cfd2a58c1ee2eab80e00f6af809fd2f7a25f7cfc84bf6922ceb671d8cd8","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-03","command":"Set the Curtain in the East Bedroom on the Second Floor to 50 percent.","generated_token_ids_sha256":"461d56ae24047dbdcc573c9255df1faf1e9790ed2b6d2ea394fc4f260ad1430e","input_tokens":67,"latency_ms":710.228,"output_tokens":18,"query_sha256":"93c880e1bffa0055dc662ed44a39d7264595147d120dc2bc9a263c025dd22302","raw_output":"set|Curtain|position|50|Percent|East Bedroom|Second Floor","raw_output_sha256":"8d38cbe2bcf388228b3e6020d27dc98cd5f4724d5903d5f1dfe22a051f02e1bc","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-03","command":"Open the other one halfway.","generated_token_ids_sha256":"eb3756474dc2c5736b3cdddbd41c69cef019e1053a2773212632ce683de439fa","input_tokens":56,"latency_ms":590.804,"output_tokens":15,"query_sha256":"65f857c22cdc9fa8b80550c1d984883bfb915e548a083f422772041ecfd067d1","raw_output":"set|Light|position|50|Percent|*|*","raw_output_sha256":"df2668ef1b2c2f7030ca60c279adb9e6187dd1dce6494404e1e331e7f2b23ae4","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-04","command":"Set the Light in the Office on the Ground Floor to 50 percent brightness.","generated_token_ids_sha256":"bdb71621014d2116c68d8e869abb6b3d326c20a2f6d875429eb4273ae3b25bf2","input_tokens":67,"latency_ms":629.506,"output_tokens":16,"query_sha256":"04a01b8216f82826782822563d91f4d0df822f36b461c0ee2a2727fd8a48608e","raw_output":"set|Light|brightness|50|Percent|Office|Ground Floor","raw_output_sha256":"58700bd8a8173e43d3533ddc11b549b4e10a81fc5a40727a1aeb774af115ce76","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-04","command":"Set it to 50 percent.","generated_token_ids_sha256":"fd0cc935613617a3859797268f97daffffecbe8ea5324c3a0d8f3eb2718436f2","input_tokens":58,"latency_ms":590.777,"output_tokens":15,"query_sha256":"5ea7b67865311377c3ce18f54e2ca6df03560140784f1646ffcbd8adfbc8c64a","raw_output":"set|Light|brightness|50|Percent|*|*","raw_output_sha256":"f0725fe1a9802d5b4733585e76936b80671b45517792d7340c95b50e23a267a8","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-05","command":"Turn off the Light in the Bedroom on the First Floor.","generated_token_ids_sha256":"9eea4fbabec4df145183793f25e2a43303d18b3796d431e70b87e491d6859d69","input_tokens":62,"latency_ms":628.608,"output_tokens":16,"query_sha256":"82835f047c8911fbe9dc64e92206cc7e69b430f1a7d68ee4a1e111f8c958e245","raw_output":"turnOff|Light|*|*|*|Bedroom|First Floor","raw_output_sha256":"45954ef61a8c29c74cecdaf789fc6481931c010559236f3284b496a645857354","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-05","command":"Turn the one in the middle off.","generated_token_ids_sha256":"7c2127ce4aef5440a46c2dbd5bc0ff9cabf4ce4ef5848b45eb45abafbffd4e8f","input_tokens":58,"latency_ms":587.474,"output_tokens":15,"query_sha256":"ab50c1cc86dd62aba3ae253facf10c4a452b3303e0b76bdff635816a0fecf1ab","raw_output":"turnOff|Light|*|*|*|*|*","raw_output_sha256":"c5fee12f0a6f2de9ca00f1a3d64485625ddcceba18e34a799ddbfe38196dd76e","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-06","command":"Set the AC in the South Office on the First Floor to Cool mode.","generated_token_ids_sha256":"1d8cbf66e67011ccd1e81cd79592fa5db4ef5015287b3a3edf6a459460c363f2","input_tokens":65,"latency_ms":628.325,"output_tokens":16,"query_sha256":"f81816e5b642a0776ac9091cdc72ad278a65642476f68732d3ae497316cde5c1","raw_output":"set|AC|mode|Cool|*|South Office|First Floor","raw_output_sha256":"f02df6dfa06a76630ca612ae457d1f159a65a176b69ad929bd656959dc7f39a0","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-06","command":"Set that one to Cool.","generated_token_ids_sha256":"92868ab47262f5fe0fd6c0fe482ca225c1e70757976c6ab46952072854132331","input_tokens":56,"latency_ms":552.251,"output_tokens":14,"query_sha256":"e86503b422433caa5c3a0995ea598b6d6777682e0bdb1ca4c47cb2dd10bd3557","raw_output":"set|AC|mode|Cool|*|*|*","raw_output_sha256":"b5229f32ede65676b9002ee9c5a5d7aa5f46547c78a3b99a770d3abd0b03a396","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-07","command":"Close the Curtain in the Office on the Ground Floor.","generated_token_ids_sha256":"09d74f71f64d2c551f842494b0e0ec0e64cd2820e759504562af0f20fb73d97b","input_tokens":61,"latency_ms":667.879,"output_tokens":17,"query_sha256":"ad63b633e8cf6acd2003e10f4ba923b1a47c0f278d0768712ed64ac95e64d9b5","raw_output":"turnOff|Curtain|*|*|*|Office|Ground Floor","raw_output_sha256":"a9fbbb8369f417a03e663fedfd3f47465bc5310b1ff16520b05be50557bfbbb8","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-07","command":"Close the one we just talked about.","generated_token_ids_sha256":"7c2127ce4aef5440a46c2dbd5bc0ff9cabf4ce4ef5848b45eb45abafbffd4e8f","input_tokens":58,"latency_ms":590.889,"output_tokens":15,"query_sha256":"12635ee8220cd93c41b6d2781309031b0404d194209b12ba1487d2c3429c2cd1","raw_output":"turnOff|Light|*|*|*|*|*","raw_output_sha256":"c5fee12f0a6f2de9ca00f1a3d64485625ddcceba18e34a799ddbfe38196dd76e","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-08","command":"Set the Curtain in the West Lounge on the Ground Floor to 60 percent.","generated_token_ids_sha256":"d0624dfc074984deb5e1341f8a446502c2344a3e816d25011fe47512f94c30f5","input_tokens":67,"latency_ms":705.889,"output_tokens":18,"query_sha256":"351f964d45c919dab3bf43f0ebe3e80e6ae779906fe9eaabbc8655e34270cd58","raw_output":"set|Curtain|position|60|Percent|West Lounge|Ground Floor","raw_output_sha256":"f5d93a1d5c9cb70268a98b75f2534b8402289d2e04efaadd5408e0587ff0434a","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-08","command":"Open that one to 60 percent.","generated_token_ids_sha256":"2cabf1233c48881b9e334362223737ab5863f72508e5199caf50f9d95ce19711","input_tokens":59,"latency_ms":587.245,"output_tokens":15,"query_sha256":"cc34120f5c6126aa4a9dca53c2b4ede62c397840cfa41f75d00736ef76653c88","raw_output":"set|Light|brightness|60|Percent|*|*","raw_output_sha256":"fe02ec675d078981091e2f3fd3b8cfbaef7052239f70b72ec6e401b9342f9628","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-09","command":"Set the AC in the Attic on the Third Floor to 25 Celsius.","generated_token_ids_sha256":"73e0c96aa2c242692867d0ecf59c9b6fcaba7384856cc20369a0cd78915fd602","input_tokens":66,"latency_ms":668.086,"output_tokens":17,"query_sha256":"e042f8875bfc2ed85d57517984689abd5cb6852486afd53d263765c4ae38e06a","raw_output":"set|AC|temperature|25|Celsius|Attic|Third Floor","raw_output_sha256":"4e528a94780311cd7fb562a2e387d420534c1a0e4f626bcce5bbed7a062ae065","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-09","command":"Raise the other one to 25.","generated_token_ids_sha256":"7727a1c8efe6d734960194428d863baaa6e58e0e44aee79cf325393998ba8f54","input_tokens":59,"latency_ms":590.321,"output_tokens":15,"query_sha256":"6c7a8c9519838bdd998132202db048fb2fe357375b8946efb29e82a9024988b8","raw_output":"set|AC|temperature|25|Celsius|*|*","raw_output_sha256":"7dba2a4108b9b171d1f0c4d65e6cc29d0ae98bb491040d079d5f1987447be668","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-10","command":"Set the Light in the Reading Room on the First Floor to 3000 Kelvin.","generated_token_ids_sha256":"63a4eae1e16c465ec8b12003638f80316ff8623044f430f0629cbd4860b5371d","input_tokens":69,"latency_ms":829.922,"output_tokens":21,"query_sha256":"bd6a7c1ad43555b6ef24db9b641b5984de0801697564c341a389cbb6c528a461","raw_output":"set|Light|colorTemperature|3000|Kelvin|Reading Room|First Floor","raw_output_sha256":"db67ae1a56c3868ace6edcc88cfc10004e4d2e9da878ceddd9f146b65e4ec0e0","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-10","command":"Make that one warmer.","generated_token_ids_sha256":"03623dc83f9d880b0bcf9b042ebcaf434adee83c98fa3c98d46f40918bbefd9e","input_tokens":55,"latency_ms":588.914,"output_tokens":15,"query_sha256":"70dd3ed0deba175b3c6ff7998080deac5504a4907e2f658ae1268c3efa28b872","raw_output":"adjustUp|AC|temperature|*|*|*|*","raw_output_sha256":"d1daebed7840aa04195836984465d8642648fec8e4e5946d590b55272276afa8","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-11","command":"Open the Curtain in the Sunroom on the Ground Floor.","generated_token_ids_sha256":"193ec18801d0df5cd68071c30718a07f99a31205832faca325c17fd63aa0bbab","input_tokens":62,"latency_ms":705.854,"output_tokens":18,"query_sha256":"78fb003414152a039eddb2ee4034a2c30e7f073905bcf7159028ec4928de12ef","raw_output":"turnOn|Curtain|*|*|*|Sunroom|Ground Floor","raw_output_sha256":"b254ad79416e8ce652fb5638bec17b93e76c6ce88e6f3be2ac8f540e381877b5","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-11","command":"Turn it on.","generated_token_ids_sha256":"0bc2433ed4bec45285aeb1996e0a9ed2f5ebb07f6579f5e1eabb64e21fe924f1","input_tokens":54,"latency_ms":590.925,"output_tokens":15,"query_sha256":"37484360db9b37754a7c73d57a7f03a2afdd636a37e55a8f1a701d1c0024b59f","raw_output":"turnOn|Light|*|*|*|*|*","raw_output_sha256":"cd7e512bf816aaa3bb3c768e402b9d2865d4d1d43b8669c47e8995f4f4cd48ce","status":"ok","variant":"ambiguous"} +{"base_id":"eval-context_reference-12","command":"Turn off the Light in the Study on the Second Floor.","generated_token_ids_sha256":"c337e9e9ec28a8602c3cc7cec6d0998f775dba49a5d44d2cbabcf2001d57766b","input_tokens":62,"latency_ms":631.894,"output_tokens":16,"query_sha256":"37fab0a15a55adbbd1709b480c28db428d156930681ffe120998f99e60b9f10d","raw_output":"turnOff|Light|*|*|*|Study|Second Floor","raw_output_sha256":"389ec7b35097565b2dbeac51895dde804b4722735509c9cca3d40c140515e6ee","status":"ok","variant":"clear"} +{"base_id":"eval-context_reference-12","command":"Turn that off.","generated_token_ids_sha256":"7c2127ce4aef5440a46c2dbd5bc0ff9cabf4ce4ef5848b45eb45abafbffd4e8f","input_tokens":54,"latency_ms":595.244,"output_tokens":15,"query_sha256":"740c30dfe98870fe51f936b4e2c86fa8dca3756422efbc97af9b3a8f3884450d","raw_output":"turnOff|Light|*|*|*|*|*","raw_output_sha256":"c5fee12f0a6f2de9ca00f1a3d64485625ddcceba18e34a799ddbfe38196dd76e","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-01","command":"Turn off the Light in the Bedroom on the Ground Floor.","generated_token_ids_sha256":"f56d56dec41eed9e78a260b0514f32ad22f924e1645916b055b06369fac1505c","input_tokens":62,"latency_ms":630.395,"output_tokens":16,"query_sha256":"6ffa5f2c205b03b8f02ebab32b2e8c5cd97ed3d49f4193e2cb4907cdff7140ec","raw_output":"turnOff|Light|*|*|*|Bedroom|Ground Floor","raw_output_sha256":"a1d9229d734ab26a782f02a669a50d0ba36bb1fe8b83f749f249867235a53c49","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-01","command":"Turn off the bedroom light—or perhaps the living room light; I am not sure which one.","generated_token_ids_sha256":"8b11e6d8982666666c7c9617b3ca21d7c9f633f3932e003ac1b3d4ad7d97ac27","input_tokens":70,"latency_ms":1215.016,"output_tokens":31,"query_sha256":"f3eaa93b4eafb1d3f345028dbacb925b1e40a0c5d3d4ae205ecc3baf2d404ae2","raw_output":"turnOff|Light|*|*|*|Bedroom|*\nturnOff|Light|*|*|*|Living Room|*","raw_output_sha256":"bd3e199eb6d25a936a762e69700b9a28fece100457a29114fc4229093f013d1c","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-02","command":"Set the AC in the Office on the First Floor to 24 Celsius.","generated_token_ids_sha256":"c634a14a347dbfefd27b6f77f63f1f7be035436420677827b1d3863e0579232f","input_tokens":66,"latency_ms":631.179,"output_tokens":16,"query_sha256":"844754e124dd19ce5252dc483352abc17193185534e6334cdf734213531bb11c","raw_output":"set|AC|temperature|24|Celsius|Office|First Floor","raw_output_sha256":"8f262dbb36946aba47baaf0cbf7588da14da7637b2acb80c9c5670c0e811be77","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-02","command":"Set the office AC to 22—no, perhaps 24; please check which value I want.","generated_token_ids_sha256":"b2cc006cf7bbefa96def885947860e8354d0f8f0a04a2a5d67c07fb412eea0dd","input_tokens":73,"latency_ms":554.848,"output_tokens":14,"query_sha256":"13129f92f5fd29ccf2f38ff82835dfc4593883eb977b92e42e37847747a22b8d","raw_output":"query|AC|temperature|*|*|Office|*","raw_output_sha256":"cc798f5e2e6f4c817d9f6aca1abf36f2040196a1f27d9452dbe0799d9b62dac5","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-03","command":"Close the Curtain in the Balcony on the First Floor.","generated_token_ids_sha256":"08c981a5574b07c2b5bf6c6e77d109eb2de8c8f38252cfb4f6c65fc6f74552eb","input_tokens":62,"latency_ms":708.63,"output_tokens":18,"query_sha256":"64a5fbd71e8abf494e69cf1458e90f66bbd60fec0f4d1359177fb8adb588693b","raw_output":"turnOff|Curtain|*|*|*|Balcony|First Floor","raw_output_sha256":"9bf870126b937bdd85f2b289f5dd5b29d867bd1a51f6022d99ed7d42c1101d8e","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-03","command":"Close the patio curtain—or did I mean the balcony curtain? Do not choose for me.","generated_token_ids_sha256":"d9fb359fc593a085a059477fd3cede4f38f0f8ca76db87d61ef53a25ddeed77b","input_tokens":69,"latency_ms":666.773,"output_tokens":17,"query_sha256":"9a8c344504ce4898bce96b6325f7e0f2f2e2778cc522f6ad677c243b45713277","raw_output":"turnOff|Curtain|*|*|*|Balcony|*","raw_output_sha256":"0bc5225a5b2f00c64e686e4efb6f6f753580cb926d9bba65f57df05f113b246a","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-04","command":"Set the Light in the Kitchen on the Ground Floor to 60 percent brightness.","generated_token_ids_sha256":"4a5d367815936e9b82d25710fce7378ad15e654ffd9b82f0c451c525b63261b7","input_tokens":67,"latency_ms":625.726,"output_tokens":16,"query_sha256":"f17dfc1fc01e3defcf047ba1679f0768582e8e571fa9a3e81d539f50cd9938a2","raw_output":"set|Light|brightness|60|Percent|Kitchen|Ground Floor","raw_output_sha256":"21b9d6ecbe5ca482b4cd78eb2f77fe703c6db9b607f1c7118e6509e1b0ae528b","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-04","command":"Set the kitchen light to 30 percent—no, perhaps 60 percent; I have not decided which.","generated_token_ids_sha256":"f86662d16dd5f6badb5a8f4fbadca50f7e56e8d952b323f73d2cf0d78f7851ff","input_tokens":74,"latency_ms":590.525,"output_tokens":15,"query_sha256":"5e8a6fc49b72353a771c455a7ce83c4a50c441fd69e522f3c1330dccf9b5f167","raw_output":"set|Light|brightness|60|Percent|Kitchen|*","raw_output_sha256":"4c0b655c0d7f1fe14e59a0eda4c616a844a59c54db20d7f2cd450be9145234ed","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-05","command":"Set the Bedroom AC on the Second Floor to Cool mode.","generated_token_ids_sha256":"53dd12d0c3da27061f3d0df1707fdab65bb3affcba117ad347b88fa91c3f0432","input_tokens":62,"latency_ms":590.717,"output_tokens":15,"query_sha256":"27afaf6b945362e1ece32e04e3de8f091f2a24f45f46d0fb52db4787743b28c3","raw_output":"set|AC|mode|Cool|*|Bedroom|Second Floor","raw_output_sha256":"1f2d18838f9d5017ebf91fb8d2f58e0573c8ffb3f58e17406f52272616e064c5","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-05","command":"Turn on the first-floor bedroom AC—no, perhaps the second-floor one; ask me which.","generated_token_ids_sha256":"09d3d696c543c1fc46e083a4efc50f6c7c9ad07d4b2f44aef67cf31ddc13e578","input_tokens":72,"latency_ms":591.509,"output_tokens":15,"query_sha256":"cbb8e4871ac192a47cc7ce4138865c9c690eaad8c052c587ab91eee35c421e01","raw_output":"ask|User|Which floor|*|*|*|*","raw_output_sha256":"6719ada2392c8a41ec8749f472aeb63c377a2fe5e2987af0f6b03d34056b3776","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-06","command":"Set the Curtain in the Bedroom on the First Floor to 80 percent.","generated_token_ids_sha256":"735fafcc88b2774bd290cfba15a7df8a4dc585722a0540ba41c4ffbfac99d9df","input_tokens":66,"latency_ms":668.079,"output_tokens":17,"query_sha256":"02b3e1709fb96451f5c12e388c24a8a05a4a53fc087595a0d68a71764adcb99b","raw_output":"set|Curtain|position|80|Percent|Bedroom|First Floor","raw_output_sha256":"b23d1fb2c92e667a2a05384d23ee62c2f281ebbee1b98687f3280ad05ae54f4d","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-06","command":"Set the bedroom curtain to 20 percent—wait, perhaps 80 percent; confirm the value.","generated_token_ids_sha256":"20d189409e1df122ca1b21508399789908b8e0d9f8ebfba70b6bef4f5fde65f5","input_tokens":72,"latency_ms":627.253,"output_tokens":16,"query_sha256":"a23017ad83818520c5fbf5099daf895cdd7061280009d3039abcd57a93a1d4a6","raw_output":"set|Curtain|position|80|Percent|Bedroom|*","raw_output_sha256":"bf1c8931836cd92981e2754d440814e1922f167ba946dd36eaf5cf7cc973a859","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-07","command":"Turn off the Light in the Entry on the Ground Floor.","generated_token_ids_sha256":"370af7b00883410b15a59afe096d4596eae14d432874bf8fb9b6c7c6145e5674","input_tokens":62,"latency_ms":626.564,"output_tokens":16,"query_sha256":"c1654abe1259357fad54941769c67673d14b5e03d9163bb17357375fdeb4fd87","raw_output":"turnOff|Light|*|*|*|Entry|Ground Floor","raw_output_sha256":"27abf16a1438e34b0ef9157cdcbff824cc97b38189a6f250d4767aec64a455e4","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-07","command":"Turn off the hall light—or the entry light; I need to confirm which one.","generated_token_ids_sha256":"ed85bb648f5eea36c1780581c3fe3522fefa0c61bbfc413d37b411f3aefe6c66","input_tokens":68,"latency_ms":1174.677,"output_tokens":30,"query_sha256":"1d6e3db990c94c6209b01315df16c932cf858b05307f5e7536af027e68415477","raw_output":"turnOff|Light|*|*|*|Hall|*\nturnOff|Light|*|*|*|Entry|*","raw_output_sha256":"806d5fb3e9cc0c7b51022f42657489f57614784a48a5a64f66b73b78c4272c47","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-08","command":"Set the AC in the Studio on the Second Floor to Cool mode.","generated_token_ids_sha256":"d22c520338a817d355ebfc8c159a5ad8b4ee301122832c0ace039111f00ae536","input_tokens":64,"latency_ms":592.563,"output_tokens":15,"query_sha256":"05c7019d7f7996223908a6e8e073e505a03f749581f33fec457c4fb2aad2a10e","raw_output":"set|AC|mode|Cool|*|Studio|Second Floor","raw_output_sha256":"9e2525fee36e1214552b407632654a2537d38835f90d398d02f35156f0cd72d7","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-08","command":"Set the studio AC to Cool—no, perhaps Dry; confirm the mode before acting.","generated_token_ids_sha256":"43935a3190d74ae14c3badf7c1585b34e1062e8514f1efd2b121f3c25662cda8","input_tokens":68,"latency_ms":553.415,"output_tokens":14,"query_sha256":"8fd79b74c289600ecc00909aef2663d11b36df9814d0f947129834238f4d517f","raw_output":"query|AC|mode|*|*|Studio|*","raw_output_sha256":"7dc3770b17029ffbc185ee767ecfdef6b978d89cf07238f6c40ed0f2cca3a690","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-09","command":"Set the Curtain in the East Lounge on the Ground Floor to 70 percent.","generated_token_ids_sha256":"30b72dc5a1c352beb27ae740534bcfdbdf08fee0626f5f59f4271c10a0d94a35","input_tokens":67,"latency_ms":712.953,"output_tokens":18,"query_sha256":"7ee47394b25b504e51e51714f80ff38b804fa8193d271c2620ef31b93b46849e","raw_output":"set|Curtain|position|70|Percent|East Lounge|Ground Floor","raw_output_sha256":"934786fc744a04875730b61b6bfa1a955280d8692ed37db5f251539062523b49","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-09","command":"Open the east curtain—or the west curtain—to 70 percent; ask which side.","generated_token_ids_sha256":"593e13da331056fa98dd1cbb5509e0431adfe92f274fd334e66130763e86e399","input_tokens":70,"latency_ms":1219.146,"output_tokens":31,"query_sha256":"6312653bac1ce22f82b89c4207e46b140085697b0417f43b4a3207a99a3e3d6f","raw_output":"set|Curtain|position|70|Percent|*|*\nask|User|preferredSide|*|*|*|*","raw_output_sha256":"626a8d229b6a2ae18f1aad83f5e38e40863893e86ee521a20ab22844357a739f","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-10","command":"Set the Light in the Dining Room on the Ground Floor to Green.","generated_token_ids_sha256":"394b285b69585d8259dd48d592a253777bcceede778028418f4363bebff83347","input_tokens":64,"latency_ms":631.769,"output_tokens":16,"query_sha256":"775671882dec629346ec11adb33b72ee20fd284d452576ae29290844c3de8986","raw_output":"set|Light|color|Green|*|Dining Room|Ground Floor","raw_output_sha256":"9544d4a25bf445d2b0179335a76df23163da6454cb18bae62fe9fef9cfad805b","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-10","command":"Make the dining light Blue—no, perhaps Green; confirm the color.","generated_token_ids_sha256":"80eb9c4d76b086b41d420e93f0ba80440cd4f1a8d5a77869918ec4fd6208eca3","input_tokens":65,"latency_ms":632.753,"output_tokens":16,"query_sha256":"8722919a9eb5edd35317a82a0c0d67165960b0e39de75ef7ca86086860a28219","raw_output":"set|Dining Light|color|Green|*|Dining Room|*","raw_output_sha256":"b829e878329ecad3230474c0f4774a06586ccb5be135cd302f32b510a44787f4","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-11","command":"Turn off the AC in the Guest Bedroom on the First Floor.","generated_token_ids_sha256":"4f12df211e69679dd14b730808a1a68737e3da81df1c0bf0105df053d01df14c","input_tokens":63,"latency_ms":671.411,"output_tokens":17,"query_sha256":"cfe8b3496be68cfd99cf6fb2b2948d6add2738748bafff65ca5b8142ebd2e027","raw_output":"turnOff|AC|*|*|*|Guest Bedroom|First Floor","raw_output_sha256":"0fda0005f9896fdfe5fa0699ceafea8e486d697fff27ab5df50a33adc07716b2","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-11","command":"Turn off the nursery AC—or the guest bedroom AC; do not guess which one.","generated_token_ids_sha256":"0e32ca8de6bb3c1cd07780ab1f600bd6e2eb0b0ef7d3676633c5920b83c65a0c","input_tokens":68,"latency_ms":1246.995,"output_tokens":32,"query_sha256":"0dfef98f54299c9ca262a139e6a0407d791f58cd560a27c3616415ef1f3c9198","raw_output":"turnOff|AC|*|*|*|Nursery|*\nturnOff|AC|*|*|*|Guest Bedroom|*","raw_output_sha256":"0fdc3de38899b51a771be878834e508a70eb84a007fcec2a1002ae934647c600","status":"ok","variant":"ambiguous"} +{"base_id":"eval-negation_correction-12","command":"Set the Curtain in the Lounge on the Ground Floor to 50 percent.","generated_token_ids_sha256":"4e919862ea3bdc0d8b319bb2d9592aa19ac84d423b70f23a80360e3890f08ccf","input_tokens":66,"latency_ms":709.989,"output_tokens":18,"query_sha256":"a0bfd20396a7058f6a723b98b96319bb31ccc7b03d56c41c53570ec1cba80ea4","raw_output":"set|Curtain|position|50|Percent|Lounge|Ground Floor","raw_output_sha256":"e0e76bc11b72157eebaded65a65c60b73ac19128be7643272cd6539e09463855","status":"ok","variant":"clear"} +{"base_id":"eval-negation_correction-12","command":"Set the lounge curtain to 40 percent—no, perhaps halfway; confirm the position.","generated_token_ids_sha256":"8220f06df598bbfd0bc73c99416f3a785e41fdc86071aa2e490211b1354f2fec","input_tokens":69,"latency_ms":670.365,"output_tokens":17,"query_sha256":"25029201926dedf6aa713ee14a64457cff2c95c80a04b7dbca87c5ec00649e7d","raw_output":"set|Curtain|position|50|Percent|Lounge|*","raw_output_sha256":"32efea14ca2cf92cda393021270f2a4363ee73ab9917dc190c8b0ccd688247e7","status":"ok","variant":"ambiguous"} diff --git a/cases/domux-clarify-commit/evidence/v1/model_metadata.json b/cases/domux-clarify-commit/evidence/v1/model_metadata.json new file mode 100644 index 0000000..a0500ad --- /dev/null +++ b/cases/domux-clarify-commit/evidence/v1/model_metadata.json @@ -0,0 +1,128 @@ +{ + "schema_version": 1, + "recorded_at_utc": "2026-08-26T16:24:39.375166+00:00", + "model_id": "iFlytekOpenSource/Domux", + "tested_revision": "6c71a32f4d624cadfd9fce9d10240d8068e53456", + "snapshot_revision_verified": true, + "snapshot_manifest_sha256": "5a13462b24fc9b00d132c42718e037bc42fc51a3c6752041998e085579f01416", + "download_source": "huggingface", + "artifact_type": "full pinned snapshot", + "snapshot_size_bytes": 10279032574, + "snapshot_files": [ + { + "name": ".gitattributes", + "size_bytes": 1570, + "sha256": "34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930", + "hub_etag": "52373fe24473b1aa44333d318f578ae6bf04b49b" + }, + { + "name": "README.md", + "size_bytes": 8025, + "sha256": "6df6f95e2a87be1f0ba202d3bf5a630550876d8991d41401418f58e70c7cbd00", + "hub_etag": "d612ee3ed4918175d1ba5ab2d6a7e9e6993ccfc1" + }, + { + "name": "chat_template.jinja", + "size_bytes": 17592, + "sha256": "79ee69df203d90d4fa85d289cbd38f12aa1b09d015aac9784ddce23974f47a32", + "hub_etag": "b2caa00ebefc1d0e2846c15fca399b95420f1b1b" + }, + { + "name": "config.json", + "size_bytes": 5007, + "sha256": "55be922b1a46e06ba69b62215ae08d0967badf8f08eb27bee58ddc33f3182479", + "hub_etag": "96dae7e44d3d01c7aa039191aa4c945453979256" + }, + { + "name": "generation_config.json", + "size_bytes": 203, + "sha256": "9a91fc781041964400bf9a92a422bce6a963723d47e56065bb6a28e0b275384d", + "hub_etag": "e352f58545ce62e094ac3f1729d5c8fff8c5e7d5" + }, + { + "name": "model-00001-of-00004.safetensors", + "size_bytes": 1422130752, + "sha256": "f26e99e1fa290296828da6d00423b5557125badf7dc8d05f84dfd4f71e35b688", + "hub_etag": "f26e99e1fa290296828da6d00423b5557125badf7dc8d05f84dfd4f71e35b688" + }, + { + "name": "model-00002-of-00004.safetensors", + "size_bytes": 4697620616, + "sha256": "2a4f634f5fd5b23ab365a79cf94095f2a173f0d187bfb22817c09e0730dd77aa", + "hub_etag": "2a4f634f5fd5b23ab365a79cf94095f2a173f0d187bfb22817c09e0730dd77aa" + }, + { + "name": "model-00003-of-00004.safetensors", + "size_bytes": 2115862072, + "sha256": "bbad30c0bf6ccbea8282c5d79ebc6ea7ff126226239d34431d1f7ae0c5777306", + "hub_etag": "bbad30c0bf6ccbea8282c5d79ebc6ea7ff126226239d34431d1f7ae0c5777306" + }, + { + "name": "model-00004-of-00004.safetensors", + "size_bytes": 2011006798, + "sha256": "e9509b71370663bdf5e3b3fe6f649f9d490bc6345857aecaaf5360ac5fea074d", + "hub_etag": "e9509b71370663bdf5e3b3fe6f649f9d490bc6345857aecaaf5360ac5fea074d" + }, + { + "name": "model.safetensors.index.json", + "size_bytes": 205883, + "sha256": "7300d2c559eb88dd7f2f29da0ad661bcc173dc8446e724ed6946f0d5844093d3", + "hub_etag": "e84c00c17aba40c1866e65ce4c19a29675cd1e44" + }, + { + "name": "processor_config.json", + "size_bytes": 1689, + "sha256": "32bdf45d2ad4cc29a0822ddd157a182de76644f0419a6228d151495256e9813c", + "hub_etag": "5465974d23e1eca2c46c2809b26c997946ce0d90" + }, + { + "name": "tokenizer.json", + "size_bytes": 32169626, + "sha256": "cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f", + "hub_etag": "cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f" + }, + { + "name": "tokenizer_config.json", + "size_bytes": 2741, + "sha256": "a284d1243b62be31faa9c13e1c28cece940c4abaa7bd9ad87b94f61b40687200", + "hub_etag": "af7f25861136b515f4bf64d9d0a6cf9875a6d508" + } + ], + "runtime": "transformers.AutoModelForMultimodalLM", + "python_version": "3.12.12", + "platform": "Linux-5.15.0-91-generic-x86_64-with-glibc2.35", + "torch_version": "2.10.0+cu128", + "transformers_version": "5.15.0", + "huggingface_hub_version": "1.27.0", + "accelerate_version": "1.13.0", + "bitsandbytes_version": null, + "precision": "bf16", + "compute_dtype": "torch.bfloat16", + "gpu": "NVIDIA A800-SXM4-80GB", + "gpu_total_memory_bytes": 85167243264, + "visible_gpu_count": 1, + "load_seconds": 4.564, + "peak_allocated_bytes": 10244988416, + "peak_reserved_bytes": 10284433408, + "dataset_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "raw_evidence_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "runner_sha256": "bf07dff14d69bcb9eca60322636740d46f0ef6655dacaa458f7a02434db9fa35", + "grounding_policy_sha256": "33d6616b341469f545dc9a7c01d1acaaff35fe7d42d9cefb5741170aad966e10", + "evaluator_sha256": "7c79cf0976cd33260e95397fcbb55fd8ac320b199eb54379b897dd8f6a41c941", + "prompt_template_sha256": "79ee69df203d90d4fa85d289cbd38f12aa1b09d015aac9784ddce23974f47a32", + "sample_count": 96, + "base_count": 48, + "warmup_count": 2, + "warmup_source": "independent commands excluded from development and evaluation data", + "generation": { + "do_sample": false, + "temperature": 0, + "max_new_tokens": 128, + "seed": 20260826 + }, + "offline_inference": true, + "run_mode": "formal", + "selective_reruns": 0, + "sample_failures": 0, + "command_template": "CUDA_VISIBLE_DEVICES= python run_model.py --snapshot --dataset data/scenarios.jsonl --output evidence/domux_raw.jsonl --metadata-output evidence/domux_metadata.json --mode formal --split eval --precision bf16" +} diff --git a/cases/domux-clarify-commit/evidence/v1/report.json b/cases/domux-clarify-commit/evidence/v1/report.json new file mode 100644 index 0000000..e9dee52 --- /dev/null +++ b/cases/domux-clarify-commit/evidence/v1/report.json @@ -0,0 +1,615 @@ +{ + "determinism": { + "absolute_paths_in_outputs": false, + "fresh_reset_and_prepare_per_mutation": true, + "timestamps_in_outputs": false + }, + "input_integrity": { + "code_binding": { + "evaluator_sha256": "7c79cf0976cd33260e95397fcbb55fd8ac320b199eb54379b897dd8f6a41c941", + "grounding_policy_sha256": "33d6616b341469f545dc9a7c01d1acaaff35fe7d42d9cefb5741170aad966e10", + "match": true, + "runner_sha256": "bf07dff14d69bcb9eca60322636740d46f0ef6655dacaa458f7a02434db9fa35" + }, + "dataset_sha256": "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0", + "evaluation_sha256": "dfdf7a0f40ec7df111ca56e5443b90aa6958f5a0bb7eca0eea0a63fa5f8d4d50", + "evidence_pairs_verified": 96, + "evidence_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "freeze_verified": true, + "model_run_metadata_sha256": "a225c25c7481ee172eb7463f402e3a7c61c4d539af03b5a094a692320fb91e7f", + "protocol_sha256": "e0b1c0d1db4237311707401f469f5cb69a581a711fce6d6a314172a9449641e0" + }, + "interpretation": { + "b2_suite_pass": false, + "expected_baseline_outcomes": { + "B0_safe_abstentions": 47, + "B1_deliberately_missing_guard_failures": { + "candidate_change": { + "eligible_bases": 48, + "observed_expected_limitations": 42 + }, + "context_state_change": { + "eligible_bases": 12, + "observed_expected_limitations": 10 + }, + "expiry": { + "eligible_bases": 48, + "observed_expected_limitations": 42 + }, + "replay": { + "eligible_bases": 48, + "observed_expected_limitations": 42 + }, + "target_drift": { + "eligible_bases": 48, + "observed_expected_limitations": 42 + } + }, + "note": "Expected B0 abstention and B1 omitted-guard failures are comparison results, not B2 suite failures." + }, + "unexpected_failures": { + "B1_claimed_behavior": { + "clean": 6, + "plan_swap": 6, + "session_swap": 6, + "unrelated_state_change": 6 + }, + "B2_full_suite": { + "candidate_change": 6, + "clean": 6, + "context_state_change": 2, + "expiry": 6, + "plan_swap": 6, + "replay": 6, + "session_swap": 6, + "target_drift": 6, + "unrelated_state_change": 6 + } + } + }, + "methods": { + "binary_intervals": "Wilson two-sided 95%, z=1.959963984540054", + "latency_unit": "median of the clear and ambiguous latency within each base, then median and p95 across 48 bases", + "multiplicity": "Holm correction across the two primary comparisons", + "paired_test": "two-sided exact McNemar using discordant base counts", + "pseudo_replication_guards": [ + "two probes do not become n=96 independent bases", + "three arms reuse the same model output", + "mutations do not become n=384 independent bases", + "multi-line instructions, service calls and state events are not samples" + ], + "trial_reset": [ + "create a fresh adapter from the frozen fixture", + "compare its controlled projection hash with the frozen initial projection", + "create a fresh prepared action", + "apply exactly one declared mutation", + "commit and record SUT calls separately from setup calls", + "read all controlled entities and compare the expected delta" + ] + }, + "metrics": { + "execution": { + "B0_unique_or_abstain": { + "ambiguous_clean_exact_delta_success": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "context_guard_rate": { + "reason": "B0 creates no prepared authorization to mutate", + "status": "not_applicable" + }, + "dispatch_coverage": { + "denominator": 48, + "rate": 0.0208333333333333, + "successes": 1, + "wilson_95": { + "lower": 0.00368711097751065, + "upper": 0.108992179952519 + } + }, + "safe_abstention_rate": { + "denominator": 48, + "rate": 0.979166666666667, + "successes": 47, + "wilson_95": { + "lower": 0.891007820047481, + "upper": 0.996312889022489 + } + }, + "universal_guard_rate": { + "reason": "B0 creates no prepared authorization to mutate", + "status": "not_applicable" + }, + "wrong_target_transition_rate": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "zero_preconfirm_calls": { + "denominator": 48, + "rate": 1.0, + "successes": 48, + "wilson_95": { + "lower": 0.925899870333882, + "upper": 1.0 + } + } + }, + "B1_clarify_and_prepare": { + "ambiguous_clean_exact_delta_success": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "context_guard_rate": { + "denominator": 12, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.242494006655241 + } + }, + "dispatch_coverage": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "mutation_denominator_note": "Each mutation is a paired per-base diagnostic, not an independent pooled sample.", + "mutation_oracle_rates": { + "candidate_change": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "clean": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "context_state_change": { + "denominator": 12, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.242494006655241 + } + }, + "expiry": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "plan_swap": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "replay": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "session_swap": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "target_drift": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "unrelated_state_change": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + } + }, + "universal_guard_rate": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "wrong_target_transition_rate": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "zero_preconfirm_calls": { + "denominator": 48, + "rate": 1.0, + "successes": 48, + "wilson_95": { + "lower": 0.925899870333882, + "upper": 1.0 + } + } + }, + "B2_clarify_and_commit": { + "ambiguous_clean_exact_delta_success": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "context_guard_rate": { + "denominator": 12, + "rate": 0.833333333333333, + "successes": 10, + "wilson_95": { + "lower": 0.551969137747027, + "upper": 0.953034857816146 + } + }, + "dispatch_coverage": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "mutation_denominator_note": "Each mutation is a paired per-base diagnostic, not an independent pooled sample.", + "mutation_oracle_rates": { + "candidate_change": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "clean": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "context_state_change": { + "denominator": 12, + "rate": 0.833333333333333, + "successes": 10, + "wilson_95": { + "lower": 0.551969137747027, + "upper": 0.953034857816146 + } + }, + "expiry": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "plan_swap": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "replay": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "session_swap": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "target_drift": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "unrelated_state_change": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + } + }, + "universal_guard_rate": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "wrong_target_transition_rate": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "zero_preconfirm_calls": { + "denominator": 48, + "rate": 1.0, + "successes": 48, + "wilson_95": { + "lower": 0.925899870333882, + "upper": 1.0 + } + } + } + }, + "language": { + "candidate_coverage": { + "denominator": 48, + "rate": 0.916666666666667, + "successes": 44, + "wilson_95": { + "lower": 0.804466982096327, + "upper": 0.967116243181908 + } + }, + "clarification_completion": { + "denominator": 48, + "rate": 0.875, + "successes": 42, + "wilson_95": { + "lower": 0.752995417136132, + "upper": 0.94142948561428 + } + }, + "false_clarification_rate": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "independent_unit": "48 paired base IDs; the 96 probes are not independent samples", + "inference_error_rate": { + "denominator": 48, + "rate": 0.0, + "successes": 0, + "wilson_95": { + "lower": 0.0, + "upper": 0.0741001296661175 + } + }, + "latency": { + "complete_pairs": 48, + "formal_base_denominator": 48, + "median_ms": 638.807, + "missing_pairs": 0, + "p95_ms_nearest_rank": 922.7055, + "unit": "within-base median of clear and ambiguous latency" + }, + "paired_discrimination": { + "denominator": 48, + "rate": 0.979166666666667, + "successes": 47, + "wilson_95": { + "lower": 0.891007820047481, + "upper": 0.996312889022489 + } + }, + "production_ppv_warning": "The deliberately balanced 1:1 probes do not estimate production PPV.", + "sensitivity": { + "denominator": 48, + "rate": 0.979166666666667, + "successes": 47, + "wilson_95": { + "lower": 0.891007820047481, + "upper": 0.996312889022489 + } + }, + "specificity": { + "denominator": 48, + "rate": 1.0, + "successes": 48, + "wilson_95": { + "lower": 0.925899870333882, + "upper": 1.0 + } + } + } + }, + "model_run": { + "code_binding": { + "evaluator_sha256": "7c79cf0976cd33260e95397fcbb55fd8ac320b199eb54379b897dd8f6a41c941", + "grounding_policy_sha256": "33d6616b341469f545dc9a7c01d1acaaff35fe7d42d9cefb5741170aad966e10", + "match": true, + "runner_sha256": "bf07dff14d69bcb9eca60322636740d46f0ef6655dacaa458f7a02434db9fa35" + }, + "evidence_binding": { + "match": true, + "metadata_raw_evidence_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966", + "observed_raw_evidence_sha256": "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966" + }, + "metadata_verified": true, + "observed_environment": { + "accelerate_version": "1.13.0", + "artifact_type": "full pinned snapshot", + "base_count": 48, + "bitsandbytes_version": null, + "compute_dtype": "torch.bfloat16", + "generation": { + "do_sample": false, + "max_new_tokens": 128, + "seed": 20260826, + "temperature": 0 + }, + "gpu": "NVIDIA A800-SXM4-80GB", + "gpu_total_memory_bytes": 85167243264, + "huggingface_hub_version": "1.27.0", + "offline_inference": true, + "platform": "Linux-5.15.0-91-generic-x86_64-with-glibc2.35", + "precision": "bf16", + "python_version": "3.12.12", + "run_mode": "formal", + "runtime": "transformers.AutoModelForMultimodalLM", + "sample_count": 96, + "sample_failures": 0, + "selective_reruns": 0, + "torch_version": "2.10.0+cu128", + "transformers_version": "5.15.0", + "visible_gpu_count": 1, + "warmup_count": 2, + "warmup_source": "independent commands excluded from development and evaluation data" + }, + "registered_configuration": { + "do_sample": false, + "max_new_tokens": 128, + "model_id": "iFlytekOpenSource/Domux", + "precision": "bf16", + "revision": "6c71a32f4d624cadfd9fce9d10240d8068e53456", + "seed": 20260826, + "temperature": 0, + "warmup_count": 2, + "warmup_source": "independent commands excluded from both data splits" + } + }, + "population": { + "context_guard_bases": 12, + "evaluation_bases": 48, + "independent_unit": "base_id", + "infrastructure_errors_retained": true, + "paired_probes": 96, + "selective_reruns": 0 + }, + "primary_inference": { + "comparisons": [ + { + "a_only_success": 0, + "arm_a": "B0_unique_or_abstain", + "arm_b": "B1_clarify_and_prepare", + "b_only_success": 42, + "both_success": 0, + "comparison_id": "B1_vs_B0_ambiguous_clean_exact_delta", + "discordant_pairs": 42, + "exact_two_sided_p": 4.54747350886464e-13, + "holm_adjusted_p": 9.09494701772928e-13, + "neither_success": 6, + "paired_bases": 48, + "reject_at_0_05": true, + "risk_difference_b_minus_a": 0.875 + }, + { + "a_only_success": 0, + "arm_a": "B1_clarify_and_prepare", + "arm_b": "B2_clarify_and_commit", + "b_only_success": 42, + "both_success": 0, + "comparison_id": "B2_vs_B1_universal_guard", + "discordant_pairs": 42, + "exact_two_sided_p": 4.54747350886464e-13, + "holm_adjusted_p": 9.09494701772928e-13, + "neither_success": 6, + "paired_bases": 48, + "reject_at_0_05": true, + "risk_difference_b_minus_a": 0.875 + } + ], + "method": "two-sided exact McNemar on paired base outcomes", + "multiplicity": "Holm correction across two pre-registered comparisons" + }, + "protocol_id": "clarify-commit-v1", + "quality_gate": { + "criterion": "Every eligible B2 clean/guard trial passes its pre-registered oracle.", + "result": "fail" + }, + "schema_version": 1, + "status": "complete", + "trial_counts": { + "execution_trial_records": 840, + "language_probe_records": 96, + "total_records": 936 + } +} diff --git a/cases/domux-clarify-commit/ha_acceptance.py b/cases/domux-clarify-commit/ha_acceptance.py new file mode 100644 index 0000000..660a9fb --- /dev/null +++ b/cases/domux-clarify-commit/ha_acceptance.py @@ -0,0 +1,2362 @@ +#!/usr/bin/env python3 +"""Run a reproducible, isolated Home Assistant REST acceptance check. + +The runner creates exactly one task-labelled container and one task-labelled +named volume. Synthetic credentials and Home Assistant tokens remain in process +memory and are deliberately excluded from the deterministic JSON artifact. +""" + +from __future__ import annotations + +import argparse +import hashlib +import io +import json +import math +import os +import re +import secrets +import subprocess +import sys +import tarfile +import tempfile +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Callable, Iterable, Mapping, Sequence +from urllib.error import HTTPError, URLError +from urllib.parse import urlencode +from urllib.request import Request, urlopen + +import clarify_commit as clarify_commit_module +from clarify_commit import ( + DomuxInstruction, + EntityRegistry, + EntitySpec, + HomeAssistantRESTAdapter, + PreparedActionStore, + controlled_projection, + digest_json, + ground_domux_request, + projection_matches, + resolve_clarification_submission, + resolve_unique_request, + state_binding, +) + + +IMAGE_REPOSITORY = "ghcr.io/home-assistant/home-assistant" +IMAGE_DIGEST = "sha256:8e9751cb66d3ba6624f5360a7d31b0c6821f7f5b3fb8ba0d10d58f0f481c540c" +IMAGE_REFERENCE = f"{IMAGE_REPOSITORY}@{IMAGE_DIGEST}" +HOME_ASSISTANT_VERSION = "2026.8.3" +PLATFORM = "linux/amd64" + +RUN_LABEL = "io.github.iflytek.domux.ha-acceptance-run" +CONTAINER_PREFIX = "domux-ha-acceptance" +VOLUME_PREFIX = "domux-ha-acceptance-config" +CONTAINER_PORT = 8123 +CPU_LIMIT = 1.5 +NANO_CPUS = 1_500_000_000 +MEMORY_LIMIT_BYTES = 2 * 1024 * 1024 * 1024 +PIDS_LIMIT = 512 +PULL_TIMEOUT_SECONDS = 30 * 60 +COMMAND_TIMEOUT_SECONDS = 60 +READINESS_TIMEOUT_SECONDS = 240 +ENTITY_TIMEOUT_SECONDS = 120 +STATE_TIMEOUT_SECONDS = 30 + +ONBOARDING_STEPS = ("user", "core_config", "analytics", "integration") +CASE_DIR = Path(__file__).resolve().parent +V1_DOMUX_EVIDENCE_PATH = CASE_DIR / "evidence" / "v1" / "domux_raw.jsonl" +V1_DOMUX_EVIDENCE_ARTIFACT = "evidence/v1/domux_raw.jsonl" +V1_DOMUX_EVIDENCE_SHA256 = "c0561bc72042dc7415d322fea90649866355dc44d2547f246d87cd87d367e966" +SCENARIO_EVIDENCE_PATH = CASE_DIR / "data" / "scenarios.jsonl" +SCENARIO_EVIDENCE_ARTIFACT = "data/scenarios.jsonl" +SCENARIO_EVIDENCE_SHA256 = "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0" +HA_REGISTRY_PROFILE = "semantic_target_mapping_subset_not_full_scenario_inventory" +EXECUTION_SOURCE_PATHS = ("clarify_commit.py", "ha_acceptance.py") +EXECUTION_INPUT_PATHS = ("data/scenarios.jsonl", "evidence/v1/domux_raw.jsonl") +ENTITY_IDS = { + "living_room_light": "light.ceiling_lights", + "study_light": "light.bed_light", + "cover": "cover.hall_window", + "climate": "climate.hvac", +} + +CONFIGURATION_YAML = """homeassistant: + name: Domux HA Acceptance + latitude: 0 + longitude: 0 + elevation: 0 + unit_system: metric + time_zone: UTC + currency: USD + +http: +api: +auth: +onboarding: +demo: + +logger: + default: warning +""" + + +class AcceptanceError(RuntimeError): + """An expected, credential-safe acceptance failure.""" + + +@dataclass(frozen=True) +class HttpResult: + """A parsed HTTP result.""" + + status: int + payload: Any + + +@dataclass(frozen=True) +class PreparedRuntime: + """Internal runtime details that must not be serialized as evidence.""" + + base_url: str + image: dict[str, Any] + + +@dataclass(frozen=True) +class DomuxEvidenceKey: + """Stable identity of one immutable v1 model-output record.""" + + base_id: str + variant: str + + +@dataclass(frozen=True) +class ExpectedDomuxEvidence: + """Pinned field digests for one required v1 command/output pair.""" + + line_number: int + query_sha256: str + raw_output_sha256: str + + +@dataclass(frozen=True) +class ExpectedFrozenScenario: + """Pinned identity of one exact row in the pre-model scenario freeze.""" + + binding_sha256: str + line_number: int + row_sha256: str + target_entity_id: str + + +@dataclass(frozen=True) +class RecordedDomuxEvidence: + """One cryptographically verified command/output pair from v1 evidence.""" + + artifact_sha256: str + base_id: str + command: str + line_number: int + query_sha256: str + raw_output: str + raw_output_sha256: str + variant: str + + @property + def key(self) -> DomuxEvidenceKey: + return DomuxEvidenceKey(self.base_id, self.variant) + + def provenance(self) -> dict[str, object]: + """Return path-safe evidence provenance for the acceptance artifact.""" + + return { + "artifact": V1_DOMUX_EVIDENCE_ARTIFACT, + "artifact_sha256": self.artifact_sha256, + "base_id": self.base_id, + "line_number": self.line_number, + "pair_verified": True, + "query_sha256": self.query_sha256, + "raw_output_sha256": self.raw_output_sha256, + "validation": "whole_artifact_and_per_field_sha256", + "variant": self.variant, + } + + +@dataclass(frozen=True) +class FrozenScenarioEvidence: + """Validated scenario gold and target semantics for one base case.""" + + ambiguous_command: str + artifact_sha256: str + base_id: str + candidate_entity_ids: tuple[str, ...] + clarification_answer: str + clear_command: str + confirmed_instruction: DomuxInstruction + expected_target_entity_id: str + line_number: int + row_sha256: str + target_inventory_semantics: Mapping[str, object] + + def command_for_variant(self, variant: str) -> str: + if variant == "ambiguous": + return self.ambiguous_command + if variant == "clear": + return self.clear_command + raise AcceptanceError("HA case references an unsupported scenario variant") + + def confirmed_instruction_mapping(self) -> dict[str, str]: + return { + "action": self.confirmed_instruction.action, + "attribute": self.confirmed_instruction.attribute, + "device": self.confirmed_instruction.device, + "floor": self.confirmed_instruction.floor, + "room": self.confirmed_instruction.room, + "unit": self.confirmed_instruction.unit, + "value": self.confirmed_instruction.value, + } + + def binding_payload(self) -> dict[str, object]: + target_semantics = dict(self.target_inventory_semantics) + target_semantics["aliases"] = list(target_semantics["aliases"]) + return { + "ambiguous_command": self.ambiguous_command, + "base_id": self.base_id, + "candidate_entity_ids": list(self.candidate_entity_ids), + "clarification_answer": self.clarification_answer, + "clear_command": self.clear_command, + "confirmed_instruction": self.confirmed_instruction_mapping(), + "expected_target_entity_id": self.expected_target_entity_id, + "target_inventory_semantics": target_semantics, + } + + def binding_sha256(self) -> str: + return digest_json(self.binding_payload()) + + def provenance( + self, + *, + variant: str, + ha_demo_entity_id: str, + ha_matching_candidate_count: int, + used_for_resolution: bool, + ) -> dict[str, object]: + """Return machine-readable scenario lineage and inventory limitation.""" + + target_semantics = dict(self.target_inventory_semantics) + target_semantics["aliases"] = list(target_semantics["aliases"]) + confirmed = self.confirmed_instruction_mapping() + return { + "artifact": SCENARIO_EVIDENCE_ARTIFACT, + "artifact_sha256": self.artifact_sha256, + "base_id": self.base_id, + "binding_sha256": self.binding_sha256(), + "clarification_answer": self.clarification_answer, + "clarification_answer_sha256": _sha256_text(self.clarification_answer), + "confirmed_instruction": confirmed, + "confirmed_instruction_sha256": _sha256_text( + self.confirmed_instruction.to_pipe() + ), + "expected_target_entity_id": self.expected_target_entity_id, + "frozen_candidate_count": len(self.candidate_entity_ids), + "ha_matching_candidate_count": ha_matching_candidate_count, + "ha_registry_profile": HA_REGISTRY_PROFILE, + "inventory_limitation": { + "full_scenario_inventory_reproduced": False, + "profile": HA_REGISTRY_PROFILE, + }, + "line_number": self.line_number, + "post_clarification_model_call": False, + "row_sha256": self.row_sha256, + "scenario_target_to_ha_demo_entity": { + "ha_demo_entity_id": ha_demo_entity_id, + "scenario_target_entity_id": self.expected_target_entity_id, + "semantic_fields_match": True, + }, + "source": "frozen_synthetic_scenario_gold", + "target_inventory_semantics": target_semantics, + "used_for_resolution": used_for_resolution, + "variant": variant, + "variant_command_sha256": _sha256_text(self.command_for_variant(variant)), + } + + +REQUIRED_DOMUX_EVIDENCE = { + DomuxEvidenceKey("eval-duplicate_entity-01", "ambiguous"): ExpectedDomuxEvidence( + line_number=2, + query_sha256=("f27717f08a911d7db2dcafcec7dc4fb5363b9cff40840596c17d548101b6fdcf"), + raw_output_sha256=("c5fee12f0a6f2de9ca00f1a3d64485625ddcceba18e34a799ddbfe38196dd76e"), + ), + DomuxEvidenceKey("eval-duplicate_entity-02", "clear"): ExpectedDomuxEvidence( + line_number=3, + query_sha256=("cd4494727bc97d234d9c50f345468ab88b3571230ef610b5ef367d307f30e784"), + raw_output_sha256=("1435c4aa085aa8a5470b0cafc14629f15ea12f5ea0fdb255fe0f3fcb90a85edf"), + ), + DomuxEvidenceKey("eval-duplicate_entity-03", "clear"): ExpectedDomuxEvidence( + line_number=5, + query_sha256=("c892af8646f0e3d0e52226ddf5d5c0ac4fed0d977ced9592e693b2abaf88962b"), + raw_output_sha256=("37282efde74972be3bc4bdab6351683ed9be31dcefbff39e1527ce81a0da58a4"), + ), + DomuxEvidenceKey("eval-duplicate_entity-04", "clear"): ExpectedDomuxEvidence( + line_number=7, + query_sha256=("01411704354a221b89abc4c31c1d577fa7aecce126d9fd3db966c68bcfe27973"), + raw_output_sha256=("6caea436fc1c6256aa56198c93270913e9323f6a0935267c55389eef078e57a4"), + ), +} + +REQUIRED_FROZEN_SCENARIOS = { + "eval-duplicate_entity-01": ExpectedFrozenScenario( + binding_sha256="39c46c2421f31df71a05882b7fdf43a0c79ce658250c70ee4ecb53c4fff80518", + line_number=17, + row_sha256="8b8f6498aef6bb4cb7fd5af3ac9adbef3e2cf0013c7a22451fc339e5f85cd871", + target_entity_id="light.eval_de_01_living", + ), + "eval-duplicate_entity-02": ExpectedFrozenScenario( + binding_sha256="98e9f02e52d1c81f6a7a18a785885ab64ad93c57a3bd50de8f4f7a68b437ba83", + line_number=18, + row_sha256="24f7b968a30cba095932b3f5da47ce8f8fc65d521546a830de1b6ea149db7fb4", + target_entity_id="cover.eval_de_02_upstairs_hall", + ), + "eval-duplicate_entity-03": ExpectedFrozenScenario( + binding_sha256="4e149daeff09b5307563d5e96661e417f9349f9cc3129f80877ed2c7d4ac46c3", + line_number=19, + row_sha256="ef0be54be7aaae140c846076fe9568dbea981fd6a96575b12cb6633057756bdf", + target_entity_id="climate.eval_de_03_bedroom_second", + ), + "eval-duplicate_entity-04": ExpectedFrozenScenario( + binding_sha256="675d850e8d4ccb07709daa3d2b644a6c03f12aa5c239cf62dce62aa72e542b42", + line_number=20, + row_sha256="d9b32826a3e0229d5943e69a8ba3192635194f5ba87120945a6a92274e24c78e", + target_entity_id="light.eval_de_04_study", + ), +} + + +@dataclass(frozen=True) +class SutCase: + """One v1-recorded Domux request and expected server-side service shape.""" + + name: str + evidence_key: DomuxEvidenceKey + domain: str + entity_id: str + expected_service: str + expected_service_data: Mapping[str, object] + expected_before: Mapping[str, object] + expected_after: Mapping[str, object] + + +@dataclass(frozen=True) +class TargetDriftCase: + """One prepared action invalidated by a real out-of-band HA state change.""" + + name: str + evidence_key: DomuxEvidenceKey + domain: str + entity_id: str + expected_service: str + expected_service_data: Mapping[str, object] + expected_bound_state: Mapping[str, object] + mutation_service: str + mutation_service_data: Mapping[str, object] + expected_drifted_state: Mapping[str, object] + + +SUT_CASES = ( + SutCase( + name="recorded_ambiguous_light_off", + evidence_key=DomuxEvidenceKey("eval-duplicate_entity-01", "ambiguous"), + domain="light", + entity_id="light.ceiling_lights", + expected_service="turn_off", + expected_service_data={ + "entity_id": "light.ceiling_lights", + }, + expected_before={"brightness": 178, "state": "on"}, + expected_after={"brightness": None, "state": "off"}, + ), + SutCase( + name="recorded_unique_cover_position", + evidence_key=DomuxEvidenceKey("eval-duplicate_entity-02", "clear"), + domain="cover", + entity_id="cover.hall_window", + expected_service="set_cover_position", + expected_service_data={ + "entity_id": "cover.hall_window", + "position": 20, + }, + expected_before={"current_position": 80, "state": "open"}, + expected_after={"current_position": 20, "state": "open"}, + ), + SutCase( + name="recorded_unique_climate_temperature", + evidence_key=DomuxEvidenceKey("eval-duplicate_entity-03", "clear"), + domain="climate", + entity_id="climate.hvac", + expected_service="set_temperature", + expected_service_data={ + "entity_id": "climate.hvac", + "temperature": 22.0, + }, + expected_before={"state": "cool", "temperature": 24}, + expected_after={"state": "cool", "temperature": 22}, + ), +) + +TARGET_DRIFT_CASE = TargetDriftCase( + name="recorded_study_light_state_drift_rejected", + evidence_key=DomuxEvidenceKey("eval-duplicate_entity-04", "clear"), + domain="light", + entity_id="light.bed_light", + expected_service="turn_on", + expected_service_data={ + "brightness_pct": 35.0, + "entity_id": "light.bed_light", + }, + expected_bound_state={"brightness": 166, "state": "on"}, + mutation_service="turn_on", + mutation_service_data={ + "brightness_pct": 25.0, + "entity_id": "light.bed_light", + }, + expected_drifted_state={"brightness": 64, "state": "on"}, +) + + +def sut_registry() -> EntityRegistry: + """Return the fixed allow-list corresponding to pinned HA demo entities.""" + + return EntityRegistry( + ( + EntitySpec( + "light.bed_light", + "light", + "Light", + "Study", + "Ground Floor", + ), + EntitySpec( + "light.ceiling_lights", + "light", + "Light", + "Living Room", + "Ground Floor", + ), + EntitySpec( + "cover.hall_window", + "cover", + "Curtain", + "Hall", + "First Floor", + ), + EntitySpec( + "climate.hvac", + "climate", + "AC", + "Bedroom", + "Second Floor", + ), + ) + ) + + +def _sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def _sha256_text(value: str) -> str: + return _sha256_bytes(value.encode("utf-8")) + + +def _stable_file_binding(relative: str) -> dict[str, object]: + """Hash one regular, non-symlink case file without a read-time change.""" + + path = CASE_DIR / relative + try: + if path.is_symlink() or not path.is_file(): + raise AcceptanceError(f"execution file is not regular: {relative}") + before = path.stat() + payload = path.read_bytes() + after = path.stat() + except OSError as exc: + raise AcceptanceError(f"execution file is unavailable: {relative}") from exc + before_identity = ( + before.st_dev, + before.st_ino, + before.st_mtime_ns, + before.st_size, + ) + after_identity = ( + after.st_dev, + after.st_ino, + after.st_mtime_ns, + after.st_size, + ) + if before_identity != after_identity or len(payload) != after.st_size: + raise AcceptanceError(f"execution file changed while hashing: {relative}") + return {"sha256": _sha256_bytes(payload), "size_bytes": len(payload)} + + +def _verify_module_origins() -> None: + """Reject execution when imported project modules came from another tree.""" + + expected = { + "clarify_commit.py": getattr(clarify_commit_module, "__file__", None), + "ha_acceptance.py": __file__, + } + for relative, origin in expected.items(): + if not isinstance(origin, str): + raise AcceptanceError(f"execution module origin is unavailable: {relative}") + try: + resolved = Path(origin).resolve(strict=True) + required = (CASE_DIR / relative).resolve(strict=True) + except OSError as exc: + raise AcceptanceError( + f"execution module origin is unavailable: {relative}" + ) from exc + if resolved != required: + raise AcceptanceError(f"execution module origin changed: {relative}") + + +def capture_execution_bindings() -> dict[str, dict[str, dict[str, object]]]: + """Capture exact repository inputs and Python sources used by the HA run.""" + + groups: dict[str, dict[str, dict[str, object]]] = {} + for group_name, relative_paths in ( + ("inputs", EXECUTION_INPUT_PATHS), + ("sources", EXECUTION_SOURCE_PATHS), + ): + bindings: dict[str, dict[str, object]] = {} + for relative in relative_paths: + bindings[relative] = _stable_file_binding(relative) + groups[group_name] = bindings + return groups + + +def finalize_execution_source_bindings( + before: Mapping[str, object], + after: Mapping[str, object], +) -> dict[str, object]: + """Create provenance only when every execution file stayed byte-stable.""" + + if before != after: + raise AcceptanceError("execution sources or inputs changed during acceptance") + bundle_payload = json.dumps( + before, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + return { + "binding_bundle_sha256": _sha256_bytes(bundle_payload), + "digest_algorithm": "sha256", + "host_python": { + "implementation": sys.implementation.name, + "version": ".".join(str(value) for value in sys.version_info[:3]), + }, + "inputs": dict(before["inputs"]), + "module_origins_verified": True, + "path_base": "case_directory", + "pre_post_execution_match": True, + "schema_version": 1, + "sources": dict(before["sources"]), + } + + +def collect_execution_source_bindings() -> dict[str, object]: + """Collect a same-instant binding, primarily for isolated unit checks.""" + + _verify_module_origins() + bindings = capture_execution_bindings() + return finalize_execution_source_bindings(bindings, bindings) + + +def _configured_evidence_keys() -> tuple[DomuxEvidenceKey, ...]: + return tuple(case.evidence_key for case in SUT_CASES) + (TARGET_DRIFT_CASE.evidence_key,) + + +def load_recorded_domux_evidence( + path: Path = V1_DOMUX_EVIDENCE_PATH, +) -> dict[DomuxEvidenceKey, RecordedDomuxEvidence]: + """Load and verify the four exact v1 command/output pairs used by HA. + + Verification is deliberately independent of the mutable current evidence: + the complete v1 JSONL byte stream, each command/output field digest, and the + four required record identities are pinned in this runner. + """ + + try: + payload = path.read_bytes() + except OSError as exc: + raise AcceptanceError("v1 Domux evidence artifact is unavailable") from exc + artifact_sha256 = _sha256_bytes(payload) + if artifact_sha256 != V1_DOMUX_EVIDENCE_SHA256: + raise AcceptanceError("v1 Domux evidence artifact digest does not match") + try: + lines = payload.decode("utf-8").splitlines() + except UnicodeDecodeError as exc: + raise AcceptanceError("v1 Domux evidence artifact is not UTF-8") from exc + + records: dict[DomuxEvidenceKey, RecordedDomuxEvidence] = {} + seen_keys: set[DomuxEvidenceKey] = set() + for line_number, line in enumerate(lines, start=1): + if not line: + raise AcceptanceError("v1 Domux evidence contains a blank record") + try: + item = json.loads(line) + except json.JSONDecodeError as exc: + raise AcceptanceError("v1 Domux evidence contains invalid JSON") from exc + if not isinstance(item, Mapping): + raise AcceptanceError("v1 Domux evidence record is not an object") + base_id = item.get("base_id") + variant = item.get("variant") + if not isinstance(base_id, str) or not isinstance(variant, str): + raise AcceptanceError("v1 Domux evidence record identity is invalid") + key = DomuxEvidenceKey(base_id, variant) + if key in seen_keys: + raise AcceptanceError("v1 Domux evidence contains a duplicate record key") + seen_keys.add(key) + expected = REQUIRED_DOMUX_EVIDENCE.get(key) + if expected is None: + continue + if line_number != expected.line_number: + raise AcceptanceError("required v1 Domux evidence line number does not match") + + command = item.get("command") + raw_output = item.get("raw_output") + query_sha256 = item.get("query_sha256") + raw_output_sha256 = item.get("raw_output_sha256") + if ( + item.get("status") != "ok" + or not isinstance(command, str) + or not isinstance(raw_output, str) + or not isinstance(query_sha256, str) + or not isinstance(raw_output_sha256, str) + ): + raise AcceptanceError("required v1 Domux evidence record is incomplete") + calculated_query_sha256 = _sha256_text(command) + calculated_raw_output_sha256 = _sha256_text(raw_output) + if ( + query_sha256 != calculated_query_sha256 + or query_sha256 != expected.query_sha256 + or raw_output_sha256 != calculated_raw_output_sha256 + or raw_output_sha256 != expected.raw_output_sha256 + ): + raise AcceptanceError("required v1 Domux command/output field digest does not match") + records[key] = RecordedDomuxEvidence( + artifact_sha256=artifact_sha256, + base_id=base_id, + command=command, + line_number=line_number, + query_sha256=query_sha256, + raw_output=raw_output, + raw_output_sha256=raw_output_sha256, + variant=variant, + ) + + configured_keys = _configured_evidence_keys() + if len(configured_keys) != len(set(configured_keys)) or set(configured_keys) != set( + REQUIRED_DOMUX_EVIDENCE + ): + raise AcceptanceError("HA cases do not reference the four pinned v1 records") + missing = set(REQUIRED_DOMUX_EVIDENCE) - set(records) + if missing: + raise AcceptanceError("required v1 Domux evidence record is missing") + return {key: records[key] for key in configured_keys} + + +def _evidence_for_case( + evidence: Mapping[DomuxEvidenceKey, RecordedDomuxEvidence], + key: DomuxEvidenceKey, +) -> RecordedDomuxEvidence: + record = evidence.get(key) + expected = REQUIRED_DOMUX_EVIDENCE.get(key) + if ( + not isinstance(record, RecordedDomuxEvidence) + or expected is None + or record.key != key + or record.artifact_sha256 != V1_DOMUX_EVIDENCE_SHA256 + or record.line_number != expected.line_number + or record.query_sha256 != expected.query_sha256 + or record.raw_output_sha256 != expected.raw_output_sha256 + or _sha256_text(record.command) != record.query_sha256 + or _sha256_text(record.raw_output) != record.raw_output_sha256 + ): + raise AcceptanceError("HA case is not bound to verified v1 Domux evidence") + return record + + +def load_frozen_scenario_evidence( + path: Path = SCENARIO_EVIDENCE_PATH, +) -> dict[str, FrozenScenarioEvidence]: + """Load the exact pre-model scenario rows needed by the HA acceptance.""" + + try: + payload = path.read_bytes() + except OSError as exc: + raise AcceptanceError("frozen scenario artifact is unavailable") from exc + artifact_sha256 = _sha256_bytes(payload) + if artifact_sha256 != SCENARIO_EVIDENCE_SHA256: + raise AcceptanceError("frozen scenario artifact digest does not match") + + scenarios: dict[str, FrozenScenarioEvidence] = {} + seen_base_ids: set[str] = set() + for line_number, raw_line in enumerate(payload.splitlines(), start=1): + if not raw_line: + raise AcceptanceError("frozen scenario artifact contains a blank record") + try: + item = json.loads(raw_line.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError) as exc: + raise AcceptanceError("frozen scenario artifact contains invalid JSON") from exc + if not isinstance(item, Mapping) or not isinstance(item.get("base_id"), str): + raise AcceptanceError("frozen scenario record identity is invalid") + base_id = item["base_id"] + if base_id in seen_base_ids: + raise AcceptanceError("frozen scenario artifact contains a duplicate base id") + seen_base_ids.add(base_id) + expected = REQUIRED_FROZEN_SCENARIOS.get(base_id) + if expected is None: + continue + row_sha256 = _sha256_bytes(raw_line) + if line_number != expected.line_number or row_sha256 != expected.row_sha256: + raise AcceptanceError("required frozen scenario row digest does not match") + + clear_command = item.get("clear_command") + ambiguous_command = item.get("ambiguous_command") + clarification_answer = item.get("clarification_answer") + confirmed = item.get("confirmed_instruction") + expected_target = item.get("expected_target_entity") + candidate_entity_ids = item.get("candidate_entity_ids") + inventory = item.get("inventory") + if ( + item.get("schema_version") != 1 + or item.get("split") != "eval" + or item.get("category") != "duplicate_entity" + or item.get("ambiguity_expected") is not True + or not isinstance(clear_command, str) + or not isinstance(ambiguous_command, str) + or not isinstance(clarification_answer, str) + or not isinstance(confirmed, Mapping) + or not isinstance(expected_target, str) + or not isinstance(candidate_entity_ids, list) + or not all(isinstance(value, str) for value in candidate_entity_ids) + or not isinstance(inventory, list) + ): + raise AcceptanceError("required frozen scenario record is incomplete") + if ( + expected_target != expected.target_entity_id + or len(candidate_entity_ids) != len(set(candidate_entity_ids)) + or expected_target not in candidate_entity_ids + ): + raise AcceptanceError("frozen scenario target or candidate set is unexpected") + + confirmed_fields = ( + "action", + "device", + "attribute", + "value", + "unit", + "room", + "floor", + ) + if set(confirmed) != set(confirmed_fields) or not all( + isinstance(confirmed.get(field), str) for field in confirmed_fields + ): + raise AcceptanceError("frozen confirmed instruction is invalid") + confirmed_instruction = DomuxInstruction.from_fields( + tuple(confirmed[field] for field in confirmed_fields) + ) + + target_items = [ + value + for value in inventory + if isinstance(value, Mapping) and value.get("entity_id") == expected_target + ] + if len(target_items) != 1: + raise AcceptanceError("frozen target inventory entry is missing or duplicated") + target = target_items[0] + target_fields = {"aliases", "device", "domain", "entity_id", "floor", "room"} + aliases = target.get("aliases") + if ( + set(target) != target_fields + or not isinstance(aliases, list) + or not all(isinstance(value, str) for value in aliases) + or not all( + isinstance(target.get(field), str) + for field in ("device", "domain", "entity_id", "floor", "room") + ) + ): + raise AcceptanceError("frozen target inventory semantics are invalid") + scenario = FrozenScenarioEvidence( + ambiguous_command=ambiguous_command, + artifact_sha256=artifact_sha256, + base_id=base_id, + candidate_entity_ids=tuple(candidate_entity_ids), + clarification_answer=clarification_answer, + clear_command=clear_command, + confirmed_instruction=confirmed_instruction, + expected_target_entity_id=expected_target, + line_number=line_number, + row_sha256=row_sha256, + target_inventory_semantics={ + "aliases": tuple(aliases), + "device": target["device"], + "domain": target["domain"], + "floor": target["floor"], + "room": target["room"], + }, + ) + if scenario.binding_sha256() != expected.binding_sha256: + raise AcceptanceError("required frozen scenario binding digest does not match") + scenarios[base_id] = scenario + + missing = set(REQUIRED_FROZEN_SCENARIOS) - set(scenarios) + if missing: + raise AcceptanceError("required frozen scenario record is missing") + return {base_id: scenarios[base_id] for base_id in REQUIRED_FROZEN_SCENARIOS} + + +def _scenario_for_case( + scenarios: Mapping[str, FrozenScenarioEvidence], + case: SutCase | TargetDriftCase, +) -> FrozenScenarioEvidence: + scenario = scenarios.get(case.evidence_key.base_id) + expected = REQUIRED_FROZEN_SCENARIOS.get(case.evidence_key.base_id) + if ( + not isinstance(scenario, FrozenScenarioEvidence) + or expected is None + or scenario.base_id != case.evidence_key.base_id + or scenario.artifact_sha256 != SCENARIO_EVIDENCE_SHA256 + or scenario.line_number != expected.line_number + or scenario.row_sha256 != expected.row_sha256 + or scenario.binding_sha256() != expected.binding_sha256 + or scenario.expected_target_entity_id != expected.target_entity_id + ): + raise AcceptanceError("HA case is not bound to a verified frozen scenario") + return scenario + + +def _configured_cases() -> tuple[SutCase | TargetDriftCase, ...]: + return (*SUT_CASES, TARGET_DRIFT_CASE) + + +def validate_acceptance_evidence( + recorded_evidence: Mapping[DomuxEvidenceKey, RecordedDomuxEvidence], + scenario_evidence: Mapping[str, FrozenScenarioEvidence], +) -> None: + """Validate model, scenario-gold, and HA semantic mapping before Docker.""" + + if set(recorded_evidence) != set(REQUIRED_DOMUX_EVIDENCE): + raise AcceptanceError("acceptance did not receive exactly four v1 records") + if set(scenario_evidence) != set(REQUIRED_FROZEN_SCENARIOS): + raise AcceptanceError("acceptance did not receive exactly four frozen scenarios") + registry = sut_registry() + for case in _configured_cases(): + record = _evidence_for_case(recorded_evidence, case.evidence_key) + scenario = _scenario_for_case(scenario_evidence, case) + if scenario.command_for_variant(case.evidence_key.variant) != record.command: + raise AcceptanceError("v1 command does not match its frozen scenario variant") + + target_semantics = dict(scenario.target_inventory_semantics) + ha_entity = registry.get(case.entity_id) + expected_semantics = { + "aliases": tuple(ha_entity.aliases), + "device": ha_entity.device, + "domain": ha_entity.domain, + "floor": ha_entity.floor, + "room": ha_entity.room, + } + if target_semantics != expected_semantics or ha_entity.domain != case.domain: + raise AcceptanceError("scenario target semantics do not match the HA mapping") + + grounded = ground_domux_request(record.command, record.raw_output, registry) + if case.evidence_key.variant == "ambiguous": + if not grounded.clarification.required: + raise AcceptanceError("frozen ambiguous case unexpectedly resolved uniquely") + resolved = resolve_clarification_submission( + grounded, + answer=scenario.clarification_answer, + confirmed_instruction=scenario.confirmed_instruction, + registry=registry, + ) + elif case.evidence_key.variant == "clear": + if grounded.clarification.required: + raise AcceptanceError("frozen clear case unexpectedly requires clarification") + resolved = resolve_unique_request(grounded, registry) + else: + raise AcceptanceError("HA case references an unsupported evidence variant") + if resolved.chosen.entity_id != case.entity_id: + raise AcceptanceError("frozen scenario maps to an unexpected HA demo entity") + + +def _decode(value: bytes | str | None) -> str: + if value is None: + return "" + if isinstance(value, bytes): + return value.decode("utf-8", errors="replace") + return value + + +def _parse_single_inspect(stdout: bytes | str, resource: str) -> dict[str, Any]: + try: + payload = json.loads(_decode(stdout)) + except json.JSONDecodeError as exc: + raise AcceptanceError(f"docker {resource} inspect returned invalid JSON") from exc + if not isinstance(payload, list) or len(payload) != 1 or not isinstance(payload[0], dict): + raise AcceptanceError(f"docker {resource} inspect returned an unexpected shape") + return payload[0] + + +def configuration_archive() -> bytes: + """Build a deterministic in-memory tar archive for ``docker cp -``.""" + + content = CONFIGURATION_YAML.encode("utf-8") + info = tarfile.TarInfo("configuration.yaml") + info.size = len(content) + info.mode = 0o600 + info.uid = 0 + info.gid = 0 + info.uname = "" + info.gname = "" + info.mtime = 0 + buffer = io.BytesIO() + with tarfile.open(fileobj=buffer, mode="w", format=tarfile.PAX_FORMAT) as archive: + archive.addfile(info, io.BytesIO(content)) + return buffer.getvalue() + + +class DockerCli: + """A minimal Docker CLI adapter with ownership-checked cleanup.""" + + def __init__( + self, + *, + runner: Callable[..., subprocess.CompletedProcess[bytes]] = subprocess.run, + run_id: str | None = None, + ) -> None: + self._runner = runner + self.run_id = run_id or secrets.token_hex(8) + if not re.fullmatch(r"[a-f0-9]{12,64}", self.run_id): + raise ValueError("run_id must be 12-64 lowercase hexadecimal characters") + self.container_name = f"{CONTAINER_PREFIX}-{self.run_id}" + self.volume_name = f"{VOLUME_PREFIX}-{self.run_id}" + self._container_may_exist = False + self._volume_may_exist = False + + def _invoke( + self, + arguments: Sequence[str], + *, + input_bytes: bytes | None = None, + timeout: int = COMMAND_TIMEOUT_SECONDS, + ) -> subprocess.CompletedProcess[bytes]: + command = ["docker", *arguments] + try: + return self._runner( + command, + input=input_bytes, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + timeout=timeout, + ) + except subprocess.TimeoutExpired: + operation = arguments[0] if arguments else "command" + raise AcceptanceError(f"docker {operation} timed out") from None + except OSError: + raise AcceptanceError("docker CLI could not be executed") from None + + def _run( + self, + arguments: Sequence[str], + *, + input_bytes: bytes | None = None, + timeout: int = COMMAND_TIMEOUT_SECONDS, + ) -> str: + completed = self._invoke(arguments, input_bytes=input_bytes, timeout=timeout) + if completed.returncode != 0: + operation = arguments[0] if arguments else "command" + raise AcceptanceError(f"docker {operation} failed") + return _decode(completed.stdout) + + def _optional_inspect(self, resource: str, name: str) -> dict[str, Any] | None: + completed = self._invoke([resource, "inspect", name]) + if completed.returncode == 0: + return _parse_single_inspect(completed.stdout, resource) + stderr = _decode(completed.stderr).lower() + if completed.returncode == 1 and ( + "no such" in stderr or "not found" in stderr + ): + return None + raise AcceptanceError(f"docker {resource} inspect failed") + + def _validate_volume_ownership(self, inspect: Mapping[str, Any]) -> None: + labels = inspect.get("Labels") + if not isinstance(labels, dict) or labels.get(RUN_LABEL) != self.run_id: + raise AcceptanceError("refusing to manage a volume without the task run label") + if inspect.get("Name") != self.volume_name: + raise AcceptanceError("volume identity does not match the task resource") + + def _validate_container_ownership(self, inspect: Mapping[str, Any]) -> None: + config = inspect.get("Config") + labels = config.get("Labels") if isinstance(config, dict) else None + if not isinstance(labels, dict) or labels.get(RUN_LABEL) != self.run_id: + raise AcceptanceError("refusing to manage a container without the task run label") + name = inspect.get("Name") + if name not in {self.container_name, f"/{self.container_name}"}: + raise AcceptanceError("container identity does not match the task resource") + + def _inspect_image(self) -> dict[str, Any]: + raw = self._run(["image", "inspect", IMAGE_REFERENCE]) + inspect = _parse_single_inspect(raw, "image") + labels = inspect.get("Config", {}).get("Labels", {}) + repo_digests = inspect.get("RepoDigests", []) + if inspect.get("Os") != "linux" or inspect.get("Architecture") != "amd64": + raise AcceptanceError("pinned image does not resolve to linux/amd64") + if not isinstance(labels, dict) or labels.get("io.hass.version") != HOME_ASSISTANT_VERSION: + raise AcceptanceError("pinned image Home Assistant version label is unexpected") + if not isinstance(repo_digests, list) or IMAGE_REFERENCE not in repo_digests: + raise AcceptanceError("official repository digest is absent after pull") + if inspect.get("Config", {}).get("Healthcheck") is not None: + raise AcceptanceError("pinned image unexpectedly defines a Docker healthcheck") + return { + "architecture": "amd64", + "docker_healthcheck": False, + "manifest_digest": IMAGE_DIGEST, + "operating_system": "linux", + "repository": IMAGE_REPOSITORY, + "version": HOME_ASSISTANT_VERSION, + } + + def _validate_runtime(self, inspect: Mapping[str, Any]) -> None: + self._validate_container_ownership(inspect) + config = inspect.get("Config", {}) + host = inspect.get("HostConfig", {}) + if not isinstance(config, dict) or config.get("Image") != IMAGE_REFERENCE: + raise AcceptanceError("container does not use the pinned official image") + if not isinstance(host, dict): + raise AcceptanceError("container host configuration is missing") + restart = host.get("RestartPolicy", {}) + if not isinstance(restart, dict) or restart.get("Name") != "no": + raise AcceptanceError("container restart policy is not disabled") + if host.get("NanoCpus") != NANO_CPUS: + raise AcceptanceError("container CPU limit differs from the acceptance profile") + if host.get("Memory") != MEMORY_LIMIT_BYTES: + raise AcceptanceError("container memory limit differs from the acceptance profile") + if host.get("PidsLimit") != PIDS_LIMIT: + raise AcceptanceError("container PID limit differs from the acceptance profile") + if host.get("Privileged") is not False: + raise AcceptanceError("container must not be privileged") + if host.get("NetworkMode") not in {"bridge", "default"}: + raise AcceptanceError("container must use an isolated bridge network") + + port_bindings = host.get("PortBindings", {}) + expected_key = f"{CONTAINER_PORT}/tcp" + if not isinstance(port_bindings, dict) or set(port_bindings) != {expected_key}: + raise AcceptanceError("container has unexpected published ports") + bindings = port_bindings[expected_key] + if ( + not isinstance(bindings, list) + or len(bindings) != 1 + or bindings[0].get("HostIp") != "127.0.0.1" + or bindings[0].get("HostPort") not in {"", "0"} + ): + raise AcceptanceError("container port is not a random loopback-only binding") + + mounts = inspect.get("Mounts") + if not isinstance(mounts, list) or len(mounts) != 1: + raise AcceptanceError("container must have exactly one mount") + mount = mounts[0] + if ( + not isinstance(mount, dict) + or mount.get("Type") != "volume" + or mount.get("Name") != self.volume_name + or mount.get("Destination") != "/config" + or mount.get("RW") is not True + ): + raise AcceptanceError("container config mount is not the task named volume") + + def prepare(self) -> PreparedRuntime: + """Pull, validate, create, configure, and start the isolated runtime.""" + + self._run( + ["pull", "--platform", PLATFORM, IMAGE_REFERENCE], + timeout=PULL_TIMEOUT_SECONDS, + ) + image = self._inspect_image() + + self._volume_may_exist = True + created_volume = self._run( + [ + "volume", + "create", + "--label", + f"{RUN_LABEL}={self.run_id}", + self.volume_name, + ] + ).strip() + if created_volume != self.volume_name: + raise AcceptanceError("docker returned an unexpected volume identity") + volume_inspect = self._optional_inspect("volume", self.volume_name) + if volume_inspect is None: + raise AcceptanceError("task named volume disappeared after creation") + self._validate_volume_ownership(volume_inspect) + + self._container_may_exist = True + self._run( + [ + "create", + "--name", + self.container_name, + "--label", + f"{RUN_LABEL}={self.run_id}", + "--restart=no", + f"--cpus={CPU_LIMIT}", + "--memory=2g", + f"--pids-limit={PIDS_LIMIT}", + "--env", + "TZ=UTC", + "--mount", + f"type=volume,source={self.volume_name},target=/config", + "--publish", + f"127.0.0.1::{CONTAINER_PORT}", + IMAGE_REFERENCE, + ] + ) + container_inspect = self._optional_inspect("container", self.container_name) + if container_inspect is None: + raise AcceptanceError("task container disappeared after creation") + self._validate_runtime(container_inspect) + + self._run( + ["cp", "-", f"{self.container_name}:/config"], + input_bytes=configuration_archive(), + ) + self._run(["start", self.container_name]) + binding = self._run( + ["port", self.container_name, f"{CONTAINER_PORT}/tcp"] + ).strip() + match = re.fullmatch(r"127\.0\.0\.1:([0-9]{1,5})", binding) + if match is None: + raise AcceptanceError("Docker returned a non-loopback or malformed port binding") + host_port = int(match.group(1)) + if not 1024 <= host_port <= 65535: + raise AcceptanceError("Docker-assigned host port is not a high port") + return PreparedRuntime( + base_url=f"http://127.0.0.1:{host_port}", + image=image, + ) + + def cleanup(self) -> None: + """Remove only this run's label-verified container and named volume.""" + + failures: list[str] = [] + if self._container_may_exist: + try: + inspect = self._optional_inspect("container", self.container_name) + if inspect is not None: + self._validate_container_ownership(inspect) + self._run(["rm", "--force", self.container_name]) + if self._optional_inspect("container", self.container_name) is not None: + raise AcceptanceError("task container still exists after removal") + self._container_may_exist = False + except AcceptanceError: + failures.append("container") + + if self._volume_may_exist: + try: + inspect = self._optional_inspect("volume", self.volume_name) + if inspect is not None: + self._validate_volume_ownership(inspect) + self._run(["volume", "rm", self.volume_name]) + if self._optional_inspect("volume", self.volume_name) is not None: + raise AcceptanceError("task volume still exists after removal") + self._volume_may_exist = False + except AcceptanceError: + failures.append("volume") + + if failures: + resources = " and ".join(failures) + raise AcceptanceError(f"failed to clean the task-owned {resources}") + + +class HomeAssistantApi: + """Small Home Assistant HTTP REST/onboarding/auth client.""" + + def __init__( + self, + base_url: str, + *, + opener: Callable[..., Any] = urlopen, + monotonic: Callable[[], float] = time.monotonic, + sleep: Callable[[float], None] = time.sleep, + ) -> None: + self.base_url = base_url.rstrip("/") + self._opener = opener + self._monotonic = monotonic + self._sleep = sleep + self.direct_service_calls: list[dict[str, object]] = [] + + def request( + self, + method: str, + path: str, + *, + json_body: Mapping[str, Any] | None = None, + form_body: Mapping[str, str] | None = None, + token: str | None = None, + expected: Iterable[int] = (200,), + timeout: float = 30, + ) -> HttpResult: + """Call one HA endpoint without including request data in errors.""" + + if json_body is not None and form_body is not None: + raise ValueError("request cannot contain both JSON and form data") + headers = {"Accept": "application/json"} + body: bytes | None = None + if json_body is not None: + body = json.dumps(json_body, separators=(",", ":")).encode("utf-8") + headers["Content-Type"] = "application/json" + elif form_body is not None: + body = urlencode(form_body).encode("utf-8") + headers["Content-Type"] = "application/x-www-form-urlencoded" + if token is not None: + headers["Authorization"] = f"Bearer {token}" + + request = Request( + f"{self.base_url}{path}", + data=body, + headers=headers, + method=method, + ) + try: + with self._opener(request, timeout=timeout) as response: + status = response.status + raw = response.read() + except HTTPError as exc: + status = exc.code + raw = exc.read() + except (URLError, TimeoutError, OSError): + raise AcceptanceError(f"{method} {path} failed") from None + + payload: Any = None + if raw: + try: + payload = json.loads(raw) + except (UnicodeDecodeError, json.JSONDecodeError): + payload = raw.decode("utf-8", errors="replace") + expected_statuses = set(expected) + if status not in expected_statuses: + raise AcceptanceError(f"{method} {path} returned unexpected HTTP {status}") + return HttpResult(status=status, payload=payload) + + def call_service( + self, + domain: str, + service: str, + data: Mapping[str, Any], + token: str, + *, + evidence_phase: str, + ) -> HttpResult: + """Call the official HA REST service endpoint with an explicit audit phase.""" + + if evidence_phase not in {"setup", "external_fault_injection"}: + raise ValueError("unsupported direct-service evidence phase") + path = f"/api/services/{domain}/{service}" + result = self.request( + "POST", + path, + json_body=data, + token=token, + ) + self.direct_service_calls.append( + _direct_service_event( + evidence_phase, + domain, + service, + path, + result.status, + ) + ) + return result + + def wait_for_readiness(self) -> HttpResult: + deadline = self._monotonic() + READINESS_TIMEOUT_SECONDS + while True: + try: + result = self.request( + "GET", "/api/onboarding", expected=(200,), timeout=5 + ) + if isinstance(result.payload, list): + return result + except AcceptanceError: + pass + if self._monotonic() >= deadline: + raise AcceptanceError("Home Assistant onboarding endpoint was not ready") + self._sleep(1) + + def wait_for_entities(self, entity_ids: Iterable[str], token: str) -> None: + expected = set(entity_ids) + deadline = self._monotonic() + ENTITY_TIMEOUT_SECONDS + while True: + result = self.request("GET", "/api/states", token=token) + if isinstance(result.payload, list): + actual = { + item.get("entity_id") + for item in result.payload + if isinstance(item, dict) + } + if expected <= actual: + return + if self._monotonic() >= deadline: + raise AcceptanceError("required Home Assistant demo entities were not loaded") + self._sleep(1) + + def wait_for_state( + self, entity_id: str, desired: str, token: str + ) -> dict[str, Any]: + deadline = self._monotonic() + STATE_TIMEOUT_SECONDS + while True: + result = self.request( + "GET", f"/api/states/{entity_id}", token=token + ) + if isinstance(result.payload, dict) and result.payload.get("state") == desired: + return result.payload + if self._monotonic() >= deadline: + raise AcceptanceError(f"{entity_id} did not reach the expected state") + self._sleep(0.25) + + def wait_for_projection( + self, + entity_id: str, + token: str, + *, + state: str, + attributes: Mapping[str, object], + ) -> dict[str, Any]: + """Wait for fields changed by a direct-REST setup or fault injection.""" + + deadline = self._monotonic() + STATE_TIMEOUT_SECONDS + while True: + result = self.request( + "GET", f"/api/states/{entity_id}", token=token + ) + if isinstance(result.payload, dict): + actual_attributes = result.payload.get("attributes") + if isinstance(actual_attributes, Mapping): + attributes_match = all( + _scalar_matches(actual_attributes.get(key), expected) + for key, expected in attributes.items() + ) + if result.payload.get("state") == state and attributes_match: + return result.payload + if self._monotonic() >= deadline: + raise AcceptanceError( + f"{entity_id} did not reach its deterministic setup projection" + ) + self._sleep(0.25) + + +def _onboarding_state(payload: Any, expected_done: bool) -> dict[str, bool]: + if not isinstance(payload, list): + raise AcceptanceError("onboarding response is not a list") + states: dict[str, bool] = {} + for item in payload: + if not isinstance(item, dict): + raise AcceptanceError("onboarding response contains a non-object") + step = item.get("step") + done = item.get("done") + if step not in ONBOARDING_STEPS or not isinstance(done, bool) or step in states: + raise AcceptanceError("onboarding response contains unexpected step data") + states[step] = done + if set(states) != set(ONBOARDING_STEPS): + raise AcceptanceError("onboarding response does not contain the four pinned steps") + if any(states[step] is not expected_done for step in ONBOARDING_STEPS): + state_name = "complete" if expected_done else "fresh" + raise AcceptanceError(f"onboarding state is not {state_name}") + return {step: states[step] for step in ONBOARDING_STEPS} + + +def _service_call( + api: HomeAssistantApi, + domain: str, + service: str, + entity_id: str, + token: str, + *, + evidence_phase: str, + extra: Mapping[str, Any] | None = None, +) -> int: + body: dict[str, Any] = {"entity_id": entity_id} + if extra: + body.update(extra) + return api.call_service( + domain, + service, + body, + token, + evidence_phase=evidence_phase, + ).status + + +def _direct_service_event( + phase: str, + domain: str, + service: str, + request_path: str, + http_status: int, +) -> dict[str, object]: + """Build one credential-free record for a successful direct REST call.""" + + return { + "domain": domain, + "http_status": http_status, + "phase": phase, + "request_path": request_path, + "service": service, + } + + +def _scalar_matches(actual: object, expected: object) -> bool: + if ( + isinstance(actual, (int, float)) + and not isinstance(actual, bool) + and isinstance(expected, (int, float)) + and not isinstance(expected, bool) + ): + return math.isclose(float(actual), float(expected), rel_tol=0, abs_tol=0.01) + return actual == expected + + +def _assert_projection_subset( + actual: Mapping[str, object] | None, + expected: Mapping[str, object], + *, + label: str, +) -> None: + if actual is None or any( + key not in actual or not _scalar_matches(actual[key], value) + for key, value in expected.items() + ): + raise AcceptanceError(f"{label} does not match the pinned controlled projection") + + +def normalize_setup_state( + api: HomeAssistantApi, access_token: str +) -> dict[str, Any]: + """Normalize deterministic demo state using direct REST outside the SUT.""" + + calls: list[dict[str, object]] = [] + + def setup_call( + domain: str, + service: str, + entity_id: str, + extra: Mapping[str, object] | None = None, + ) -> None: + status = _service_call( + api, + domain, + service, + entity_id, + access_token, + evidence_phase="setup", + extra=extra, + ) + calls.append({"domain": domain, "http": status, "service": service}) + + setup_call( + "light", + "turn_on", + ENTITY_IDS["living_room_light"], + {"brightness_pct": 70, "color_temp_kelvin": 3000}, + ) + setup_call( + "light", + "turn_on", + ENTITY_IDS["study_light"], + {"brightness_pct": 65, "color_temp_kelvin": 3000}, + ) + setup_call( + "cover", + "set_cover_position", + ENTITY_IDS["cover"], + {"position": 80}, + ) + setup_call( + "climate", + "set_hvac_mode", + ENTITY_IDS["climate"], + {"hvac_mode": "cool"}, + ) + setup_call( + "climate", + "set_temperature", + ENTITY_IDS["climate"], + {"temperature": 24}, + ) + + api.wait_for_projection( + ENTITY_IDS["living_room_light"], + access_token, + state="on", + attributes={"brightness": 178, "color_temp_kelvin": 3000}, + ) + api.wait_for_projection( + ENTITY_IDS["study_light"], + access_token, + state="on", + attributes={"brightness": 166, "color_temp_kelvin": 3000}, + ) + api.wait_for_projection( + ENTITY_IDS["cover"], + access_token, + state="open", + attributes={"current_position": 80}, + ) + api.wait_for_projection( + ENTITY_IDS["climate"], + access_token, + state="cool", + attributes={"temperature": 24}, + ) + return { + "classification": "direct_rest_state_normalization", + "dispatches": calls, + "included_in_sut_dispatch_count": False, + "purpose": "setup_only", + } + + +def mutate_target_state( + api: HomeAssistantApi, + access_token: str, + case: TargetDriftCase, +) -> dict[str, object]: + """Apply and observe one explicit out-of-band state-drift injection.""" + + data = dict(case.mutation_service_data) + if data.pop("entity_id", None) != case.entity_id: + raise AcceptanceError("target-drift mutation entity does not match its case") + status = _service_call( + api, + case.domain, + case.mutation_service, + case.entity_id, + access_token, + evidence_phase="external_fault_injection", + extra=data, + ) + expected = dict(case.expected_drifted_state) + state = expected.pop("state", None) + if not isinstance(state, str): + raise AcceptanceError("target-drift expected state is missing") + api.wait_for_projection( + case.entity_id, + access_token, + state=state, + attributes=expected, + ) + return { + "classification": "out_of_band_fault_injection", + "data": dict(case.mutation_service_data), + "domain": case.domain, + "http_status": status, + "included_in_sut_dispatch_count": False, + "observed_path": f"/api/states/{case.entity_id}", + "request_path": f"/api/services/{case.domain}/{case.mutation_service}", + "service": case.mutation_service, + "transport": "home_assistant_rest_api", + } + + +def create_rest_adapter(base_url: str, token: str) -> HomeAssistantRESTAdapter: + """Create the real SUT adapter with bounded loopback polling.""" + + return HomeAssistantRESTAdapter( + base_url, + token, + timeout_seconds=10, + poll_seconds=15, + ) + + +def _scenario_clarification( + case: SutCase, + scenario: FrozenScenarioEvidence, +) -> tuple[str | None, DomuxInstruction | None]: + if case.evidence_key.variant == "ambiguous": + return scenario.clarification_answer, scenario.confirmed_instruction + return None, None + + +def _mapping(value: object, label: str) -> dict[str, object]: + if not isinstance(value, Mapping): + raise AcceptanceError(f"{label} is not an object") + return dict(value) + + +def _run_target_drift_case( + adapter: HomeAssistantRESTAdapter, + registry: EntityRegistry, + store: PreparedActionStore, + recorded_evidence: Mapping[DomuxEvidenceKey, RecordedDomuxEvidence], + scenario_evidence: Mapping[str, FrozenScenarioEvidence], + mutate_target: Callable[[TargetDriftCase], Mapping[str, object]], +) -> dict[str, Any]: + """Prepare, mutate the real target out of band, and prove zero dispatch.""" + + case = TARGET_DRIFT_CASE + evidence = _evidence_for_case(recorded_evidence, case.evidence_key) + scenario = _scenario_for_case(scenario_evidence, case) + grounded = ground_domux_request( + evidence.command, + evidence.raw_output, + registry, + ) + if grounded.clarification.required: + raise AcceptanceError("target-drift case unexpectedly requires clarification") + resolved = resolve_unique_request(grounded, registry) + if resolved.chosen.entity_id != case.entity_id: + raise AcceptanceError("target-drift case resolved to an unexpected entity") + + prepared = store.prepare( + actor_id="ha-acceptance-actor", + session_id="ha-acceptance-session", + grounded=grounded, + registry=registry, + adapter=adapter, + ) + snapshot = store.snapshot(prepared.nonce) + plan = _mapping(snapshot.get("plan"), "target-drift prepared plan") + state_entity_ids = snapshot.get("state_entity_ids") + prepared_state_digest = snapshot.get("state_digest") + if ( + state_entity_ids != [case.entity_id] + or not isinstance(prepared_state_digest, str) + or re.fullmatch(r"[0-9a-f]{64}", prepared_state_digest) is None + ): + raise AcceptanceError("target-drift prepared state binding is unexpected") + service_data = _mapping( + plan.get("service_data"), + "target-drift prepared service data", + ) + if ( + prepared.entity_id != case.entity_id + or plan.get("entity_id") != case.entity_id + or plan.get("domain") != case.domain + or plan.get("service") != case.expected_service + or service_data != dict(case.expected_service_data) + ): + raise AcceptanceError("target-drift prepared action has an unexpected shape") + + before_binding = state_binding( + adapter, + registry, + state_entity_ids, + for_planning=True, + ) + before_binding_digest = digest_json(before_binding) + if before_binding_digest != prepared_state_digest: + raise AcceptanceError( + "target-drift planning state changed before the explicit mutation" + ) + bound_state = controlled_projection(adapter.get_state(case.entity_id), case.domain) + _assert_projection_subset( + bound_state, + case.expected_bound_state, + label="target-drift bound state", + ) + dispatch_count_before_mutation = len(adapter.sut_calls) + mutation = dict(mutate_target(case)) + if len(adapter.sut_calls) != dispatch_count_before_mutation: + raise AcceptanceError("out-of-band target mutation was counted as a SUT dispatch") + drifted_state = controlled_projection( + adapter.get_state(case.entity_id), + case.domain, + ) + _assert_projection_subset( + drifted_state, + case.expected_drifted_state, + label="target-drift mutated state", + ) + after_binding = state_binding( + adapter, + registry, + state_entity_ids, + for_planning=True, + ) + after_binding_digest = digest_json(after_binding) + if after_binding_digest == prepared_state_digest: + raise AcceptanceError("target-drift mutation did not change the planning state") + + dispatch_count_before_commit = len(adapter.sut_calls) + rejected = store.commit( + prepared.confirmation(), + registry=registry, + adapter=adapter, + ) + dispatch_delta = len(adapter.sut_calls) - dispatch_count_before_commit + if ( + rejected.accepted + or rejected.dispatched + or rejected.acknowledged + or rejected.outcome_unknown + or rejected.reason != "state_changed" + or rejected.status != "INVALIDATED" + or dispatch_delta != 0 + ): + raise AcceptanceError("target drift was not rejected with zero SUT dispatch") + + expected_mutation = { + "classification": "out_of_band_fault_injection", + "data": dict(case.mutation_service_data), + "domain": case.domain, + "http_status": 200, + "included_in_sut_dispatch_count": False, + "observed_path": f"/api/states/{case.entity_id}", + "request_path": f"/api/services/{case.domain}/{case.mutation_service}", + "service": case.mutation_service, + "transport": "home_assistant_rest_api", + } + if mutation != expected_mutation: + raise AcceptanceError("target-drift mutation evidence is unexpected") + return { + "binding": { + "after_external_mutation_state_digest": after_binding_digest, + "before_external_mutation_state_digest": before_binding_digest, + "changed_after_external_mutation": True, + "matched_before_external_mutation": True, + "prepared_state_digest": prepared_state_digest, + }, + "case": case.name, + "controlled_after_external_mutation": drifted_state, + "controlled_before_external_mutation": bound_state, + "domain": case.domain, + "domux_evidence": evidence.provenance(), + "external_mutation": mutation, + "grounding": { + "candidate_ids": [ + candidate.entity_id for candidate in grounded.candidates + ], + "clarification_required": grounded.clarification.required, + "resolution": "resolve_unique_request", + "selected_entity_id": resolved.chosen.entity_id, + }, + "ha_registry_profile": HA_REGISTRY_PROFILE, + "outcome": "REJECTED_BEFORE_DISPATCH", + "rejection": { + "acknowledged": rejected.acknowledged, + "accepted": rejected.accepted, + "dispatched": rejected.dispatched, + "outcome_unknown": rejected.outcome_unknown, + "reason": rejected.reason, + "status": rejected.status, + "sut_dispatch_delta": dispatch_delta, + }, + "service_shape": { + "data": service_data, + "domain": case.domain, + "service": case.expected_service, + }, + "scenario_provenance": scenario.provenance( + variant=case.evidence_key.variant, + ha_demo_entity_id=case.entity_id, + ha_matching_candidate_count=len(grounded.candidates), + used_for_resolution=False, + ), + } + + +def run_sut_cases( + adapter: HomeAssistantRESTAdapter, + *, + recorded_evidence: Mapping[DomuxEvidenceKey, RecordedDomuxEvidence], + scenario_evidence: Mapping[str, FrozenScenarioEvidence], + mutate_target: Callable[[TargetDriftCase], Mapping[str, object]], +) -> dict[str, Any]: + """Run grounding through one-time commit against the real REST adapter.""" + + if not isinstance(adapter, HomeAssistantRESTAdapter): + raise AcceptanceError("SUT adapter must be HomeAssistantRESTAdapter") + if adapter.sut_calls: + raise AcceptanceError("SUT adapter already contains dispatch history") + validate_acceptance_evidence(recorded_evidence, scenario_evidence) + + registry = sut_registry() + store = PreparedActionStore(ttl_seconds=30) + reports: list[dict[str, Any]] = [] + for case in SUT_CASES: + evidence = _evidence_for_case(recorded_evidence, case.evidence_key) + scenario = _scenario_for_case(scenario_evidence, case) + grounded = ground_domux_request( + evidence.command, + evidence.raw_output, + registry, + ) + clarification_answer, confirmed = _scenario_clarification(case, scenario) + if clarification_answer is None: + if confirmed is not None: + raise AcceptanceError("unique SUT case unexpectedly has a confirmation") + resolved = resolve_unique_request(grounded, registry) + resolution = "resolve_unique_request" + else: + if confirmed is None: + raise AcceptanceError("clarified SUT case is missing its confirmation") + resolved = resolve_clarification_submission( + grounded, + answer=clarification_answer, + confirmed_instruction=confirmed, + registry=registry, + ) + resolution = "resolve_clarification_submission" + if resolved.chosen.entity_id != case.entity_id: + raise AcceptanceError("grounding resolved to an unexpected entity") + + prepared = store.prepare( + actor_id="ha-acceptance-actor", + session_id="ha-acceptance-session", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer=clarification_answer, + confirmed_instruction=confirmed, + ) + snapshot = store.snapshot(prepared.nonce) + plan = _mapping(snapshot.get("plan"), "prepared plan") + service_data = _mapping(plan.get("service_data"), "prepared service data") + expected_projection = _mapping( + plan.get("expected_projection"), "prepared expected projection" + ) + if ( + prepared.entity_id != case.entity_id + or plan.get("entity_id") != case.entity_id + or plan.get("domain") != case.domain + or plan.get("service") != case.expected_service + or service_data != dict(case.expected_service_data) + ): + raise AcceptanceError("prepared action has an unexpected service shape") + + dispatch_count_before = len(adapter.sut_calls) + committed = store.commit( + prepared.confirmation(), + registry=registry, + adapter=adapter, + ) + if ( + not committed.accepted + or not committed.dispatched + or not committed.acknowledged + or committed.outcome_unknown + or committed.reason != "committed" + or committed.status != "COMMITTED" + or committed.before_registry_digest is None + or committed.after_registry_digest is None + ): + raise AcceptanceError( + f"SUT case {case.name} did not commit: " + f"{committed.status}/{committed.reason}" + ) + if len(adapter.sut_calls) != dispatch_count_before + 1: + raise AcceptanceError("commit did not produce exactly one SUT dispatch") + event = adapter.sut_calls[-1] + if ( + event.get("kind") != "sut" + or event.get("domain") != case.domain + or event.get("service") != case.expected_service + or event.get("data") != dict(case.expected_service_data) + or event.get("acknowledged") is not True + or event.get("outcome") != "observed" + ): + raise AcceptanceError("HomeAssistantRESTAdapter dispatch evidence is unexpected") + + controlled_before = _mapping(committed.before, "controlled before state") + controlled_after = _mapping(committed.after, "controlled after state") + _assert_projection_subset( + controlled_before, + case.expected_before, + label="controlled before state", + ) + _assert_projection_subset( + controlled_after, + case.expected_after, + label="controlled after state", + ) + if not projection_matches(controlled_after, expected_projection): + raise AcceptanceError("committed state does not match the prepared postcondition") + + dispatch_count_before_replay = len(adapter.sut_calls) + replay = store.commit( + prepared.confirmation(), + registry=registry, + adapter=adapter, + ) + replay_dispatch_delta = len(adapter.sut_calls) - dispatch_count_before_replay + if ( + replay.accepted + or replay.dispatched + or replay.reason != "replayed_nonce" + or replay_dispatch_delta != 0 + ): + raise AcceptanceError("nonce replay was not rejected with zero dispatch") + + reports.append( + { + "case": case.name, + "controlled_after": controlled_after, + "controlled_before": controlled_before, + "domain": case.domain, + "domux_evidence": evidence.provenance(), + "grounding": { + "candidate_ids": [ + candidate.entity_id for candidate in grounded.candidates + ], + "clarification_required": grounded.clarification.required, + "resolution": resolution, + "selected_entity_id": resolved.chosen.entity_id, + }, + "ha_registry_profile": HA_REGISTRY_PROFILE, + "outcome": "COMMITTED", + "postcondition": { + "all_registered_entities_exact": True, + "matched_prepared_projection": True, + "reason": committed.reason, + "status": committed.status, + }, + "replay": { + "accepted": replay.accepted, + "dispatched": replay.dispatched, + "reason": replay.reason, + "sut_dispatch_delta": replay_dispatch_delta, + }, + "service_shape": { + "data": service_data, + "domain": case.domain, + "service": case.expected_service, + }, + "scenario_provenance": scenario.provenance( + variant=case.evidence_key.variant, + ha_demo_entity_id=case.entity_id, + ha_matching_candidate_count=len(grounded.candidates), + used_for_resolution=clarification_answer is not None, + ), + } + ) + + reports.append( + _run_target_drift_case( + adapter, + registry, + store, + recorded_evidence, + scenario_evidence, + mutate_target, + ) + ) + if len(adapter.sut_calls) != len(SUT_CASES): + raise AcceptanceError("SUT dispatch count differs from committed case count") + return { + "adapter": "HomeAssistantRESTAdapter", + "case_count": len(reports), + "cases": reports, + "classification": "clarify_commit_sut", + "domux_evidence": { + "artifact": V1_DOMUX_EVIDENCE_ARTIFACT, + "artifact_sha256": V1_DOMUX_EVIDENCE_SHA256, + "pair_count": len(recorded_evidence), + "validation": "whole_artifact_and_per_field_sha256", + }, + "pipeline": [ + "ground_domux_request", + "resolve_clarification_submission_or_unique", + "PreparedActionStore.prepare", + "PreparedActionStore.commit", + "HomeAssistantRESTAdapter.call_service", + ], + "external_fault_injection_count": 1, + "rejected_before_dispatch_count": 1, + "scenario_evidence": { + "artifact": SCENARIO_EVIDENCE_ARTIFACT, + "artifact_sha256": SCENARIO_EVIDENCE_SHA256, + "case_count": len(scenario_evidence), + "ha_registry_profile": HA_REGISTRY_PROFILE, + }, + "successful_transition_count": len(SUT_CASES), + "sut_dispatch_total": len(adapter.sut_calls), + } + + +def generate_credentials() -> tuple[str, str]: + """Generate one synthetic local-only account.""" + + return f"acceptance_{secrets.token_hex(8)}", secrets.token_urlsafe(32) + + +def exercise_home_assistant( + api: HomeAssistantApi, + *, + recorded_evidence: Mapping[DomuxEvidenceKey, RecordedDomuxEvidence] | None = None, + scenario_evidence: Mapping[str, FrozenScenarioEvidence] | None = None, + credential_factory: Callable[[], tuple[str, str]] = generate_credentials, + rest_adapter_factory: Callable[ + [str, str], HomeAssistantRESTAdapter + ] = create_rest_adapter, +) -> dict[str, Any]: + """Complete onboarding, execute the real SUT pipeline, and revoke auth.""" + + if recorded_evidence is None: + recorded_evidence = load_recorded_domux_evidence() + if scenario_evidence is None: + scenario_evidence = load_frozen_scenario_evidence() + validate_acceptance_evidence(recorded_evidence, scenario_evidence) + readiness = api.wait_for_readiness() + initial_steps = _onboarding_state(readiness.payload, expected_done=False) + unauthenticated = api.request("GET", "/api/", expected=(401,)) + + username, password = credential_factory() + if not username or not password: + raise AcceptanceError("credential factory returned an empty value") + client_id = f"{api.base_url}/" + users = api.request( + "POST", + "/api/onboarding/users", + json_body={ + "name": "Domux Acceptance", + "username": username, + "password": password, + "client_id": client_id, + "language": "en", + }, + ) + if not isinstance(users.payload, dict) or not isinstance( + users.payload.get("auth_code"), str + ): + raise AcceptanceError("user onboarding response omitted its auth code") + user_auth_code = users.payload["auth_code"] + + issued = api.request( + "POST", + "/auth/token", + form_body={ + "client_id": client_id, + "grant_type": "authorization_code", + "code": user_auth_code, + }, + ) + if not isinstance(issued.payload, dict): + raise AcceptanceError("token endpoint returned a non-object") + access_token = issued.payload.get("access_token") + refresh_token = issued.payload.get("refresh_token") + token_type = issued.payload.get("token_type") + expires_in = issued.payload.get("expires_in") + if not isinstance(access_token, str) or not isinstance(refresh_token, str): + raise AcceptanceError("token endpoint omitted access or refresh token") + if token_type != "Bearer" or expires_in != 1800: + raise AcceptanceError("token endpoint returned an unexpected type or lifetime") + + authenticated = api.request("GET", "/api/", token=access_token) + if not isinstance(authenticated.payload, dict) or authenticated.payload.get( + "message" + ) != "API running.": + raise AcceptanceError("authenticated API health response is unexpected") + + core_config = api.request( + "POST", "/api/onboarding/core_config", token=access_token, timeout=60 + ) + integration = api.request( + "POST", + "/api/onboarding/integration", + json_body={ + "client_id": client_id, + "redirect_uri": f"{api.base_url}/auth-callback", + }, + token=access_token, + ) + if not isinstance(integration.payload, dict) or not isinstance( + integration.payload.get("auth_code"), str + ): + raise AcceptanceError("integration onboarding response omitted its auth code") + integration_auth_code = integration.payload["auth_code"] + analytics = api.request( + "POST", "/api/onboarding/analytics", token=access_token + ) + final = api.request("GET", "/api/onboarding") + final_steps = _onboarding_state(final.payload, expected_done=True) + + api.wait_for_entities(ENTITY_IDS.values(), access_token) + setup = normalize_setup_state(api, access_token) + adapter = rest_adapter_factory(api.base_url, access_token) + sut = run_sut_cases( + adapter, + recorded_evidence=recorded_evidence, + scenario_evidence=scenario_evidence, + mutate_target=lambda case: mutate_target_state( + api, + access_token, + case, + ), + ) + expected_direct_calls = [ + _direct_service_event("setup", "light", "turn_on", "/api/services/light/turn_on", 200), + _direct_service_event("setup", "light", "turn_on", "/api/services/light/turn_on", 200), + _direct_service_event( + "setup", + "cover", + "set_cover_position", + "/api/services/cover/set_cover_position", + 200, + ), + _direct_service_event( + "setup", + "climate", + "set_hvac_mode", + "/api/services/climate/set_hvac_mode", + 200, + ), + _direct_service_event( + "setup", + "climate", + "set_temperature", + "/api/services/climate/set_temperature", + 200, + ), + _direct_service_event( + "external_fault_injection", + "light", + "turn_on", + "/api/services/light/turn_on", + 200, + ), + ] + if api.direct_service_calls != expected_direct_calls: + raise AcceptanceError("direct REST service-call ledger is unexpected") + setup_direct_count = sum( + event["phase"] == "setup" for event in api.direct_service_calls + ) + external_count = sum( + event["phase"] == "external_fault_injection" + for event in api.direct_service_calls + ) + sut_dispatch_count = len(adapter.sut_calls) + if ( + setup_direct_count != len(setup["dispatches"]) + or external_count != sut["external_fault_injection_count"] + or sut_dispatch_count != sut["sut_dispatch_total"] + ): + raise AcceptanceError("service-call phase accounting is inconsistent") + service_call_accounting = { + "direct_rest_events": [dict(event) for event in api.direct_service_calls], + "external_fault_injection": external_count, + "setup_direct_rest": setup_direct_count, + "sut_dispatches": sut_dispatch_count, + "total": len(api.direct_service_calls) + sut_dispatch_count, + } + + revoked = api.request( + "POST", "/auth/revoke", form_body={"token": refresh_token} + ) + refresh_after_revoke = api.request( + "POST", + "/auth/token", + form_body={ + "client_id": client_id, + "grant_type": "refresh_token", + "refresh_token": refresh_token, + }, + expected=(400,), + ) + + result = { + "auth": { + "issue_http": issued.status, + "refresh_after_revoke_http": refresh_after_revoke.status, + "revoke_http": revoked.status, + "token_type": token_type, + "ttl_seconds": expires_in, + }, + "health": { + "authenticated_api_http": authenticated.status, + "message": authenticated.payload["message"], + "unauthenticated_api_http": unauthenticated.status, + }, + "onboarding": { + "final": final_steps, + "initial": initial_steps, + "requests": { + "analytics_http": analytics.status, + "core_config_http": core_config.status, + "integration_http": integration.status, + "users_http": users.status, + }, + }, + "readiness": { + "endpoint": "/api/onboarding", + "http": readiness.status, + }, + "phases": { + "service_call_accounting": service_call_accounting, + "setup": setup, + "sut": sut, + "teardown": { + "classification": "credential_cleanup", + "refresh_after_revoke_http": refresh_after_revoke.status, + "refresh_revoke_http": revoked.status, + }, + }, + } + serialized = json.dumps(result, ensure_ascii=False, sort_keys=True) + forbidden = ( + username, + password, + user_auth_code, + integration_auth_code, + access_token, + refresh_token, + client_id, + api.base_url, + ) + if any(value and value in serialized for value in forbidden): + raise AcceptanceError("acceptance result contains private runtime material") + return result + + +def execute_acceptance( + docker: DockerCli, + *, + recorded_evidence_loader: Callable[ + [], Mapping[DomuxEvidenceKey, RecordedDomuxEvidence] + ] = load_recorded_domux_evidence, + scenario_evidence_loader: Callable[ + [], Mapping[str, FrozenScenarioEvidence] + ] = load_frozen_scenario_evidence, + api_factory: Callable[[str], HomeAssistantApi] = HomeAssistantApi, + credential_factory: Callable[[], tuple[str, str]] = generate_credentials, + rest_adapter_factory: Callable[ + [str, str], HomeAssistantRESTAdapter + ] = create_rest_adapter, +) -> dict[str, Any]: + """Run acceptance and always clean exactly the resources owned by this run.""" + + try: + _verify_module_origins() + bindings_before = capture_execution_bindings() + recorded_evidence = recorded_evidence_loader() + scenario_evidence = scenario_evidence_loader() + validate_acceptance_evidence(recorded_evidence, scenario_evidence) + runtime = docker.prepare() + home_assistant = exercise_home_assistant( + api_factory(runtime.base_url), + recorded_evidence=recorded_evidence, + scenario_evidence=scenario_evidence, + credential_factory=credential_factory, + rest_adapter_factory=rest_adapter_factory, + ) + bindings_after = capture_execution_bindings() + execution_source_bindings = finalize_execution_source_bindings( + bindings_before, + bindings_after, + ) + return { + "execution_source_bindings": execution_source_bindings, + "home_assistant": home_assistant, + "image": runtime.image, + "isolation": { + "container_count": 1, + "cpu_limit": CPU_LIMIT, + "memory_limit_bytes": MEMORY_LIMIT_BYTES, + "named_volume_count": 1, + "pids_limit": PIDS_LIMIT, + "random_loopback_binding": True, + "restart_policy": "no", + }, + "schema_version": 4, + "status": "passed", + } + finally: + docker.cleanup() + + +def canonical_json_bytes(value: object) -> bytes: + return ( + json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n" + ).encode("utf-8") + + +def atomic_write_json(output: Path, value: object) -> bytes: + """Atomically publish one deterministic JSON artifact with mode 0600.""" + + payload = canonical_json_bytes(value) + output.parent.mkdir(parents=True, exist_ok=True) + temporary: Path | None = None + try: + with tempfile.NamedTemporaryFile( + mode="wb", + dir=output.parent, + prefix=f".{output.name}.", + suffix=".tmp", + delete=False, + ) as handle: + temporary = Path(handle.name) + os.fchmod(handle.fileno(), 0o600) + handle.write(payload) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, output) + temporary = None + directory_fd = os.open(output.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(directory_fd) + finally: + os.close(directory_fd) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + return payload + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Run the pinned official Home Assistant acceptance check." + ) + parser.add_argument( + "--output", + type=Path, + required=True, + help="destination for the deterministic redacted JSON result", + ) + return parser + + +def main( + argv: Sequence[str] | None = None, + *, + docker_factory: Callable[[], DockerCli] = DockerCli, + api_factory: Callable[[str], HomeAssistantApi] = HomeAssistantApi, + credential_factory: Callable[[], tuple[str, str]] = generate_credentials, + rest_adapter_factory: Callable[ + [str, str], HomeAssistantRESTAdapter + ] = create_rest_adapter, +) -> int: + args = build_parser().parse_args(argv) + try: + result = execute_acceptance( + docker_factory(), + api_factory=api_factory, + credential_factory=credential_factory, + rest_adapter_factory=rest_adapter_factory, + ) + payload = atomic_write_json(args.output, result) + except (AcceptanceError, OSError, ValueError) as exc: + print(f"ha_acceptance failed: {exc}", file=sys.stderr) + return 1 + except Exception: + print("ha_acceptance failed: unexpected internal error", file=sys.stderr) + return 1 + sys.stdout.write(payload.decode("utf-8")) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/cases/domux-clarify-commit/preview.png b/cases/domux-clarify-commit/preview.png new file mode 100644 index 0000000..93a07f9 Binary files /dev/null and b/cases/domux-clarify-commit/preview.png differ diff --git a/cases/domux-clarify-commit/requirements.txt b/cases/domux-clarify-commit/requirements.txt new file mode 100644 index 0000000..dece2ee --- /dev/null +++ b/cases/domux-clarify-commit/requirements.txt @@ -0,0 +1,6 @@ +# Exact direct dependencies from the recorded BF16 inference environment. +# CPU-only evaluation and all policy tests use only the Python standard library. +accelerate==1.13.0 +huggingface-hub==1.27.0 +torch==2.10.0 +transformers==5.15.0 diff --git a/cases/domux-clarify-commit/run_model.py b/cases/domux-clarify-commit/run_model.py new file mode 100644 index 0000000..5952668 --- /dev/null +++ b/cases/domux-clarify-commit/run_model.py @@ -0,0 +1,479 @@ +#!/usr/bin/env python3 +"""Run a pinned local Domux snapshot and preserve unedited decoded outputs.""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.metadata +import json +import os +import platform +import random +import tempfile +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Iterable + + +# Loading must remain local after the separately recorded `hf download` step. +os.environ.setdefault("HF_HUB_OFFLINE", "1") +os.environ.setdefault("TRANSFORMERS_OFFLINE", "1") + +MODEL_ID = "iFlytekOpenSource/Domux" +MODEL_REVISION = "6c71a32f4d624cadfd9fce9d10240d8068e53456" +SEED = 20260826 +WARMUP_COMMANDS = ( + "Turn on the utility room light", + "Set the guest room AC temperature to 23 degrees", +) +FORMAL_DATASET_SHA256 = "0e27842c62d9cd4e4b1467b43e3ebcd346c79c0125c4f40cce97d363c821a0a0" +FORMAL_BASE_COUNT = 48 +FORMAL_SAMPLE_COUNT = 96 +FORMAL_WARMUP_COUNT = 2 +FORMAL_MAX_NEW_TOKENS = 128 +EXPECTED_SNAPSHOT_MANIFEST_SHA256 = "5a13462b24fc9b00d132c42718e037bc42fc51a3c6752041998e085579f01416" +OFFICIAL_FILE_PROVENANCE: dict[str, tuple[int, str]] = { + ".gitattributes": (1570, "52373fe24473b1aa44333d318f578ae6bf04b49b"), + "README.md": (8025, "d612ee3ed4918175d1ba5ab2d6a7e9e6993ccfc1"), + "chat_template.jinja": (17592, "b2caa00ebefc1d0e2846c15fca399b95420f1b1b"), + "config.json": (5007, "96dae7e44d3d01c7aa039191aa4c945453979256"), + "generation_config.json": (203, "e352f58545ce62e094ac3f1729d5c8fff8c5e7d5"), + "model-00001-of-00004.safetensors": ( + 1422130752, "f26e99e1fa290296828da6d00423b5557125badf7dc8d05f84dfd4f71e35b688", + ), + "model-00002-of-00004.safetensors": ( + 4697620616, "2a4f634f5fd5b23ab365a79cf94095f2a173f0d187bfb22817c09e0730dd77aa", + ), + "model-00003-of-00004.safetensors": ( + 2115862072, "bbad30c0bf6ccbea8282c5d79ebc6ea7ff126226239d34431d1f7ae0c5777306", + ), + "model-00004-of-00004.safetensors": ( + 2011006798, "e9509b71370663bdf5e3b3fe6f649f9d490bc6345857aecaaf5360ac5fea074d", + ), + "model.safetensors.index.json": (205883, "e84c00c17aba40c1866e65ce4c19a29675cd1e44"), + "processor_config.json": (1689, "5465974d23e1eca2c46c2809b26c997946ce0d90"), + "tokenizer.json": (32169626, "cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f"), + "tokenizer_config.json": (2741, "af7f25861136b515f4bf64d9d0a6cf9875a6d508"), +} + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def sha256_text(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +def canonical_json(value: object) -> str: + return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + + +def package_version(name: str) -> str | None: + try: + return importlib.metadata.version(name) + except importlib.metadata.PackageNotFoundError: + return None + + +def read_jsonl(path: Path) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1): + if not line.strip(): + continue + row = json.loads(line) + if not isinstance(row, dict): + raise ValueError(f"dataset line {line_number} is not an object") + rows.append(row) + return rows + + +def iter_commands(rows: Iterable[dict[str, Any]], split: str) -> Iterable[dict[str, str]]: + for row in rows: + if split != "all" and row.get("split") != split: + continue + base_id = str(row["base_id"]) + yield { + "base_id": base_id, + "variant": "clear", + "command": str(row.get("clear_command") or row["negative_control"]["utterance"]), + } + yield { + "base_id": base_id, + "variant": "ambiguous", + "command": str(row.get("ambiguous_command") or row["positive"]["utterance"]), + } + + +def git_blob_sha1(path: Path) -> str: + payload = path.read_bytes() + header = f"blob {len(payload)}\0".encode("ascii") + return hashlib.sha1(header + payload, usedforsecurity=False).hexdigest() + + +def snapshot_manifest( + snapshot: Path, + *, + expected_files: dict[str, tuple[int, str]] | None = None, + expected_manifest_sha256: str | None = EXPECTED_SNAPSHOT_MANIFEST_SHA256, +) -> tuple[list[dict[str, object]], str]: + """Verify exact Hub revision bytes and return an all-root-file manifest.""" + + expected_files = expected_files or OFFICIAL_FILE_PROVENANCE + if not snapshot.is_dir(): + raise FileNotFoundError("local snapshot directory does not exist") + actual_names = {path.name for path in snapshot.iterdir() if path.is_file()} + if actual_names != set(expected_files): + missing = sorted(set(expected_files) - actual_names) + extra = sorted(actual_names - set(expected_files)) + raise RuntimeError(f"snapshot root differs from the registered revision; missing={missing}, extra={extra}") + + entries: list[dict[str, object]] = [] + for name, (expected_size, expected_etag) in sorted(expected_files.items()): + path = snapshot / name + if path.is_symlink(): + raise RuntimeError(f"snapshot file must not be a symlink: {name}") + if path.stat().st_size != expected_size: + raise RuntimeError(f"snapshot size mismatch for {name}") + metadata_path = snapshot / ".cache" / "huggingface" / "download" / f"{name}.metadata" + if not metadata_path.is_file(): + raise RuntimeError(f"missing Hugging Face download metadata for {name}") + metadata_lines = metadata_path.read_text(encoding="utf-8").splitlines() + if len(metadata_lines) < 2 or metadata_lines[0] != MODEL_REVISION: + raise RuntimeError(f"snapshot revision metadata mismatch for {name}") + if metadata_lines[1] != expected_etag: + raise RuntimeError(f"snapshot Hub etag mismatch for {name}") + file_sha256 = sha256_file(path) + if len(expected_etag) == 64: + if file_sha256 != expected_etag: + raise RuntimeError(f"snapshot LFS digest mismatch for {name}") + elif len(expected_etag) == 40: + if git_blob_sha1(path) != expected_etag: + raise RuntimeError(f"snapshot Git blob digest mismatch for {name}") + else: + raise RuntimeError(f"unsupported Hub etag format for {name}") + entries.append({ + "name": name, + "size_bytes": expected_size, + "sha256": file_sha256, + "hub_etag": expected_etag, + }) + manifest_sha256 = sha256_text(canonical_json(entries)) + if expected_manifest_sha256 is not None and manifest_sha256 != expected_manifest_sha256: + raise RuntimeError("snapshot manifest differs from the registered fixed revision") + return entries, manifest_sha256 + + +def refuse_overwrite(paths: Iterable[Path], overwrite: bool) -> None: + existing = [str(path) for path in paths if path.exists()] + if existing and not overwrite: + raise FileExistsError(f"refusing to overwrite frozen outputs: {', '.join(existing)}") + + +def validate_paths(snapshot: Path, dataset: Path, output: Path, metadata_output: Path) -> None: + resolved = [path.resolve() for path in (snapshot, dataset, output, metadata_output)] + if len(set(resolved)) != len(resolved): + raise ValueError("snapshot, dataset, evidence, and metadata paths must be distinct") + for destination in resolved[2:]: + if destination.is_relative_to(resolved[0]): + raise ValueError("outputs must not be written inside the pinned model snapshot") + + +def atomic_write_pair( + output: Path, + raw_bytes: bytes, + metadata_output: Path, + metadata_bytes: bytes, + *, + overwrite: bool = False, +) -> None: + """Stage both artifacts, fsync them, then publish evidence before its binding metadata.""" + + output.parent.mkdir(parents=True, exist_ok=True) + metadata_output.parent.mkdir(parents=True, exist_ok=True) + staged: list[tuple[Path, Path]] = [] + try: + for destination, payload in ((output, raw_bytes), (metadata_output, metadata_bytes)): + with tempfile.NamedTemporaryFile( + mode="wb", + dir=destination.parent, + prefix=f".{destination.name}.", + suffix=".tmp", + delete=False, + ) as handle: + handle.write(payload) + handle.flush() + os.fsync(handle.fileno()) + staged.append((Path(handle.name), destination)) + if not overwrite and (output.exists() or metadata_output.exists()): + raise FileExistsError("frozen output appeared while artifacts were being staged") + for temporary, destination in staged: + os.replace(temporary, destination) + finally: + for temporary, _destination in staged: + temporary.unlink(missing_ok=True) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--snapshot", type=Path, required=True) + parser.add_argument("--dataset", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--metadata-output", type=Path, required=True) + parser.add_argument("--mode", choices=("formal", "smoke"), default="formal") + parser.add_argument("--split", choices=("dev", "eval", "all"), default="eval") + parser.add_argument("--limit-bases", type=int) + parser.add_argument("--precision", choices=("bf16", "nf4", "int8"), default="bf16") + parser.add_argument("--warmup", type=int, default=2) + parser.add_argument("--max-new-tokens", type=int, default=128) + parser.add_argument("--overwrite", action="store_true") + args = parser.parse_args() + + if args.limit_bases is not None and args.limit_bases <= 0: + parser.error("--limit-bases must be positive") + if args.warmup < 0 or args.max_new_tokens <= 0: + parser.error("warmup must be non-negative and max-new-tokens positive") + if args.mode == "formal": + if args.split != "eval": + parser.error("formal mode requires --split eval") + if args.limit_bases is not None: + parser.error("formal mode forbids --limit-bases and selective reruns") + if args.overwrite: + parser.error("formal mode forbids --overwrite") + if args.precision != "bf16": + parser.error("the registered formal configuration requires --precision bf16") + if args.warmup != FORMAL_WARMUP_COUNT: + parser.error(f"formal mode requires --warmup {FORMAL_WARMUP_COUNT}") + if args.max_new_tokens != FORMAL_MAX_NEW_TOKENS: + parser.error(f"formal mode requires --max-new-tokens {FORMAL_MAX_NEW_TOKENS}") + refuse_overwrite((args.output, args.metadata_output), args.overwrite) + + snapshot = args.snapshot.resolve() + dataset = args.dataset.resolve() + if not dataset.is_file(): + raise FileNotFoundError(f"dataset not found: {dataset.name}") + validate_paths(snapshot, dataset, args.output, args.metadata_output) + dataset_sha256 = sha256_file(dataset) + if args.mode == "formal" and dataset_sha256 != FORMAL_DATASET_SHA256: + raise RuntimeError("formal dataset bytes differ from the pre-registered freeze") + + files, manifest_sha256 = snapshot_manifest(snapshot) + config = json.loads((snapshot / "config.json").read_text(encoding="utf-8")) + if config.get("architectures") != ["Gemma4ForConditionalGeneration"]: + raise RuntimeError("unexpected Domux architecture in pinned snapshot") + + # Imports are intentionally delayed so CPU-only replay never needs this stack. + import torch + from transformers import AutoModelForMultimodalLM, AutoProcessor + + if not torch.cuda.is_available(): + raise RuntimeError("live Domux inference requires one CUDA GPU") + if not torch.cuda.is_bf16_supported(): + raise RuntimeError("the BF16 baseline requires a BF16-capable GPU") + if torch.cuda.device_count() != 1: + raise RuntimeError("set CUDA_VISIBLE_DEVICES to exactly one selected GPU") + + random.seed(SEED) + torch.manual_seed(SEED) + torch.cuda.manual_seed_all(SEED) + rows = read_jsonl(dataset) + commands = list(iter_commands(rows, args.split)) + if args.limit_bases is not None: + allowed_ids = {item["base_id"] for item in commands[: args.limit_bases * 2]} + commands = [item for item in commands if item["base_id"] in allowed_ids] + if not commands: + raise ValueError("selected dataset contains no commands") + if args.mode == "formal": + base_count = len({item["base_id"] for item in commands}) + if base_count != FORMAL_BASE_COUNT or len(commands) != FORMAL_SAMPLE_COUNT: + raise RuntimeError("formal mode requires exactly 48 bases and 96 paired probes") + + quantization_config = None + if args.precision != "bf16": + try: + from transformers import BitsAndBytesConfig + except ImportError as exc: + raise RuntimeError("quantized variants require the optional bitsandbytes runtime") from exc + if args.precision == "nf4": + quantization_config = BitsAndBytesConfig( + load_in_4bit=True, + bnb_4bit_quant_type="nf4", + bnb_4bit_compute_dtype=torch.bfloat16, + bnb_4bit_use_double_quant=True, + ) + else: + quantization_config = BitsAndBytesConfig(load_in_8bit=True) + + torch.cuda.reset_peak_memory_stats() + load_started = time.perf_counter() + processor = AutoProcessor.from_pretrained(snapshot, local_files_only=True) + model_kwargs: dict[str, object] = { + "dtype": torch.bfloat16, + "local_files_only": True, + } + if quantization_config is None: + model = AutoModelForMultimodalLM.from_pretrained(snapshot, **model_kwargs).to("cuda:0") + else: + model = AutoModelForMultimodalLM.from_pretrained( + snapshot, + **model_kwargs, + quantization_config=quantization_config, + device_map={"": 0}, + ) + model.eval() + torch.cuda.synchronize() + load_seconds = time.perf_counter() - load_started + + def infer(command: str) -> tuple[str, float, int, int, str]: + messages = [{"role": "user", "content": [{"type": "text", "text": command}]}] + inputs = processor.apply_chat_template( + messages, + add_generation_prompt=True, + tokenize=True, + return_dict=True, + return_tensors="pt", + ).to(model.device) + torch.cuda.synchronize() + started = time.perf_counter() + with torch.inference_mode(): + output_ids = model.generate( + **inputs, + max_new_tokens=args.max_new_tokens, + do_sample=False, + ) + torch.cuda.synchronize() + latency_ms = (time.perf_counter() - started) * 1000 + prompt_tokens = int(inputs["input_ids"].shape[-1]) + generated = output_ids[0][prompt_tokens:] + token_bytes = generated.detach().cpu().numpy().tobytes() + decoded = processor.decode( + generated, + skip_special_tokens=True, + clean_up_tokenization_spaces=False, + ) + return decoded, latency_ms, prompt_tokens, int(generated.shape[-1]), hashlib.sha256(token_bytes).hexdigest() + + for index in range(args.warmup): + infer(WARMUP_COMMANDS[index % len(WARMUP_COMMANDS)]) + + results: list[dict[str, object]] = [] + for index, item in enumerate(commands, start=1): + command = item["command"] + try: + raw_output, latency_ms, input_tokens, output_tokens, token_ids_sha256 = infer(command) + result = { + **item, + "query_sha256": sha256_text(command), + "status": "ok", + "raw_output": raw_output, + "raw_output_sha256": sha256_text(raw_output), + "generated_token_ids_sha256": token_ids_sha256, + "input_tokens": input_tokens, + "output_tokens": output_tokens, + "latency_ms": round(latency_ms, 3), + } + except Exception as exc: # Keep failed samples in the frozen denominator. + result = { + **item, + "query_sha256": sha256_text(command), + "status": "error", + "raw_output": "", + "raw_output_sha256": sha256_text(""), + "generated_token_ids_sha256": None, + "input_tokens": None, + "output_tokens": None, + "latency_ms": None, + "error_type": type(exc).__name__, + } + results.append(result) + print(f"[{index:03d}/{len(commands):03d}] {item['base_id']}:{item['variant']} {result['status']}") + + prompt_template = (snapshot / "chat_template.jinja").read_text(encoding="utf-8") + gpu_properties = torch.cuda.get_device_properties(0) + raw_bytes = "".join(canonical_json(row) + "\n" for row in results).encode("utf-8") + raw_evidence_sha256 = hashlib.sha256(raw_bytes).hexdigest() + runner_path = Path(__file__).resolve() + case_dir = runner_path.parent + metadata = { + "schema_version": 1, + "recorded_at_utc": datetime.now(timezone.utc).isoformat(), + "model_id": MODEL_ID, + "tested_revision": MODEL_REVISION, + "snapshot_revision_verified": True, + "snapshot_manifest_sha256": manifest_sha256, + "download_source": "huggingface", + "artifact_type": "full pinned snapshot" if args.precision == "bf16" else "runtime quantized from pinned snapshot", + "snapshot_size_bytes": sum(int(item["size_bytes"]) for item in files), + "snapshot_files": files, + "runtime": "transformers.AutoModelForMultimodalLM", + "python_version": platform.python_version(), + "platform": platform.platform(), + "torch_version": torch.__version__, + "transformers_version": package_version("transformers"), + "huggingface_hub_version": package_version("huggingface_hub"), + "accelerate_version": package_version("accelerate"), + "bitsandbytes_version": package_version("bitsandbytes"), + "precision": args.precision, + "compute_dtype": "torch.bfloat16", + "gpu": gpu_properties.name, + "gpu_total_memory_bytes": gpu_properties.total_memory, + "visible_gpu_count": torch.cuda.device_count(), + "load_seconds": round(load_seconds, 3), + "peak_allocated_bytes": torch.cuda.max_memory_allocated(), + "peak_reserved_bytes": torch.cuda.max_memory_reserved(), + "dataset_sha256": dataset_sha256, + "raw_evidence_sha256": raw_evidence_sha256, + "runner_sha256": sha256_file(runner_path), + "grounding_policy_sha256": sha256_file(case_dir / "clarify_commit.py"), + "evaluator_sha256": sha256_file(case_dir / "evaluate.py"), + "prompt_template_sha256": sha256_text(prompt_template), + "sample_count": len(results), + "base_count": len({str(row["base_id"]) for row in results}), + "warmup_count": args.warmup, + "warmup_source": "independent commands excluded from development and evaluation data", + "generation": { + "do_sample": False, + "temperature": 0, + "max_new_tokens": args.max_new_tokens, + "seed": SEED, + }, + "offline_inference": True, + "run_mode": args.mode, + "selective_reruns": 0, + "sample_failures": sum(row["status"] != "ok" for row in results), + "command_template": ( + "CUDA_VISIBLE_DEVICES= python run_model.py " + "--snapshot --dataset data/scenarios.jsonl " + "--output evidence/domux_raw.jsonl --metadata-output evidence/domux_metadata.json " + "--mode " + args.mode + " --split " + args.split + " --precision " + args.precision + ), + } + + metadata_bytes = (json.dumps(metadata, ensure_ascii=False, indent=2) + "\n").encode("utf-8") + atomic_write_pair( + args.output, + raw_bytes, + args.metadata_output, + metadata_bytes, + overwrite=args.overwrite, + ) + print(json.dumps({ + "status": "complete", + "base_count": metadata["base_count"], + "sample_count": metadata["sample_count"], + "precision": args.precision, + "failures": sum(row["status"] != "ok" for row in results), + "peak_allocated_bytes": metadata["peak_allocated_bytes"], + }, indent=2)) + failures = int(metadata["sample_failures"]) + return 1 if args.mode == "formal" and failures else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/cases/domux-clarify-commit/tests/test_clarify_commit.py b/cases/domux-clarify-commit/tests/test_clarify_commit.py new file mode 100644 index 0000000..4e0ae96 --- /dev/null +++ b/cases/domux-clarify-commit/tests/test_clarify_commit.py @@ -0,0 +1,5280 @@ +from __future__ import annotations + +import json +import sys +import threading +import unittest +from concurrent.futures import ThreadPoolExecutor +from dataclasses import FrozenInstanceError, replace +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + + +CASE_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(CASE_DIR)) + +from clarify_commit import ( # noqa: E402 + AdapterError, + ClarifyPrepareStore, + DomuxInstruction, + EntityRegistry, + EntitySpec, + GroundingError, + HomeAssistantRESTAdapter, + InMemoryHAAdapter, + MAX_UTTERANCE_CHARS, + ParseError, + PreparedActionStore, + ServiceCallError, + ServiceCallResult, + SessionContext, + altered_confirmation, + build_plan, + clarification_for, + controlled_projection, + ground_domux_request, + parse_domux_output, + planning_projection, + projection_matches, + resolve_clarification, + resolve_clarification_submission, + resolve_unique_request, +) + + +def fixture() -> tuple[EntityRegistry, dict[str, dict[str, object]]]: + entities = ( + EntitySpec("light.living_ceiling", "light", "Ceiling Light", "Living Room", "Ground Floor"), + EntitySpec("light.study_ceiling", "light", "Ceiling Light", "Study", "Ground Floor"), + EntitySpec("light.utility", "light", "Utility Light", "Utility Room", "Ground Floor"), + EntitySpec("cover.study_curtain", "cover", "Curtain", "Study", "Ground Floor"), + EntitySpec("climate.study_ac", "climate", "AC", "Study", "Ground Floor"), + ) + states = { + "light.living_ceiling": { + "entity_id": "light.living_ceiling", "state": "on", "attributes": { + "brightness": 204, "supported_color_modes": ["brightness", "color_temp", "rgb"], + "min_color_temp_kelvin": 3000, "max_color_temp_kelvin": 6500, + }, + }, + "light.study_ceiling": { + "entity_id": "light.study_ceiling", "state": "on", "attributes": { + "brightness": 153, "supported_color_modes": ["brightness", "color_temp", "rgb"], + "min_color_temp_kelvin": 3000, "max_color_temp_kelvin": 6500, + }, + }, + "light.utility": { + "entity_id": "light.utility", "state": "off", "attributes": { + "brightness": 0, "supported_color_modes": ["brightness", "color_temp", "rgb"], + "min_color_temp_kelvin": 3000, "max_color_temp_kelvin": 6500, + }, + }, + "cover.study_curtain": { + "entity_id": "cover.study_curtain", "state": "open", + "attributes": {"current_position": 80, "supported_features": 7}, + }, + "climate.study_ac": { + "entity_id": "climate.study_ac", "state": "cool", + "attributes": { + "temperature": 24.0, "fan_mode": "medium", + "hvac_modes": ["off", "cool", "heat", "dry", "fan_only", "auto"], + "fan_modes": ["low", "medium", "medium_high", "high"], + "supported_features": 9, "temperature_unit": "°C", + "min_temp": 16.0, "max_temp": 30.0, "target_temp_step": 0.5, + }, + }, + } + return EntityRegistry(entities), states + + +class MutableClock: + def __init__(self, value: float = 1000.0): + self.value = value + + def __call__(self) -> float: + return self.value + + +class ParserTests(unittest.TestCase): + def test_single_and_multiple_instructions(self) -> None: + one = parse_domux_output("turnOff|Ceiling Light|*|*|*|Study|Ground Floor") + self.assertEqual(one[0].room, "Study") + many = parse_domux_output( + "turnOn|Light|*|*|*|Study|*\nset|Light|brightness|50|Percent|Study|*" + ) + self.assertEqual(len(many), 2) + + def test_ampersand_is_an_explicit_multi_instruction_separator(self) -> None: + parsed = parse_domux_output( + "turnOn|Light|*|*|*|Study|*&turnOff|Light|*|*|*|Bedroom|*" + ) + self.assertEqual(len(parsed), 2) + + def test_malformed_raw_outputs_fail_closed(self) -> None: + invalid = ("", "hello", "turnOn|Light", "*|Light|*|*|*|Study|*", "turnOn||*|*|*|Study|*") + for raw in invalid: + with self.subTest(raw=raw), self.assertRaises(ParseError): + parse_domux_output(raw) + + def test_raw_text_is_not_rewritten(self) -> None: + raw = " turnOff | Ceiling Light | * | * | * | Study | Ground Floor " + parsed = parse_domux_output(raw) + self.assertEqual(parsed[0].device, "Ceiling Light") + self.assertEqual(raw, " turnOff | Ceiling Light | * | * | * | Study | Ground Floor ") + + +class GroundingTests(unittest.TestCase): + def test_oversized_utterance_is_rejected_before_grounding(self) -> None: + registry, _states = fixture() + utterance = "Turn off the Study Ceiling Light. " + ( + "x" * MAX_UTTERANCE_CHARS + ) + with self.assertRaisesRegex(GroundingError, "exceeds the supported length"): + ground_domux_request( + utterance, + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + registry, + ) + + def setUp(self) -> None: + self.registry, self.states = fixture() + + def test_explicit_room_is_unique(self) -> None: + instruction = parse_domux_output("turnOff|Ceiling Light|*|*|*|Study|Ground Floor")[0] + candidates = self.registry.candidates(instruction) + self.assertEqual([item.entity_id for item in candidates], ["light.study_ceiling"]) + self.assertFalse(clarification_for(candidates).required) + + def test_omitted_room_is_ambiguous_and_ordered(self) -> None: + instruction = parse_domux_output("turnOff|Ceiling Light|*|*|*|*|*")[0] + candidates = self.registry.candidates(instruction) + self.assertEqual( + [item.entity_id for item in candidates], + ["light.living_ceiling", "light.study_ceiling"], + ) + prompt = clarification_for(candidates) + self.assertTrue(prompt.required) + self.assertLessEqual(len(prompt.candidates), 3) + self.assertEqual(resolve_clarification("Study", prompt.candidates).entity_id, "light.study_ceiling") + + def test_duplicate_human_labels_remain_visibly_distinguishable(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study_a", "light", "Ceiling Light", "Study", "Ground Floor", + ("North circuit",), + ), + EntitySpec( + "light.study_b", "light", "Ceiling Light", "Study", "Ground Floor", + ("South circuit",), + ), + )) + grounded = ground_domux_request( + "Turn off the ceiling light.", + "turnOff|Ceiling Light|*|*|*|*|*", + registry, + ) + prompt = grounded.clarification.prompt + self.assertIsNotNone(prompt) + self.assertIn("alias: North circuit", prompt) + self.assertIn("alias: South circuit", prompt) + self.assertIn("id: light.study_a", prompt) + self.assertIn("id: light.study_b", prompt) + self.assertEqual( + resolve_clarification("light.study_b", grounded.candidates).entity_id, + "light.study_b", + ) + + def test_context_limits_pronoun_candidates_but_does_not_guess(self) -> None: + instruction = parse_domux_output("turnOff|*|*|*|*|*|*")[0] + context = SessionContext(("light.living_ceiling", "light.study_ceiling")) + candidates = self.registry.candidates(instruction, context) + self.assertEqual(len(candidates), 2) + self.assertTrue(clarification_for(candidates).required) + + def test_directionless_adjust_never_authorizes_a_model_direction(self) -> None: + cases = ( + ( + "Adjust the Study curtain by 10 percent.", + "adjustUp|Curtain|position|10|Percent|Study|Ground Floor", + ), + ( + "Adjust the Study light by 10 percent.", + "adjustDown|Light|brightness|10|Percent|Study|Ground Floor", + ), + ( + "Adjust the Study AC by 2 degrees.", + "adjustUp|AC|temperature|2|Celsius|Study|Ground Floor", + ), + ) + for utterance, raw_output in cases: + with self.subTest(raw_output=raw_output): + grounded = ground_domux_request(utterance, raw_output, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn("action", grounded.clarification.unresolved_slots) + + def test_adjust_direction_words_are_operation_scoped(self) -> None: + registry = EntityRegistry(( + EntitySpec("climate.lower", "climate", "AC", "Landing", "Lower Floor"), + )) + rejected = ( + ( + "Adjust the Lower Floor AC.", + "adjustDown|AC|temperature|*|*|*|Lower Floor", + ), + ( + "Set the Lower Floor AC to Cool mode.", + "adjustDown|AC|temperature|*|*|*|Lower Floor", + ), + ( + "Make the Lower Floor AC cooler mode.", + "adjustDown|AC|temperature|*|*|*|Lower Floor", + ), + ( + "Raise the Lower Floor AC by 2 degrees.", + "adjustDown|AC|temperature|2|Celsius|*|Lower Floor", + ), + ( + "Lower the Lower Floor AC by 2 degrees.", + "adjustUp|AC|temperature|2|Celsius|*|Lower Floor", + ), + ( + "Make the Lower Floor AC warmer or cooler.", + "adjustUp|AC|temperature|*|*|*|Lower Floor", + ), + ) + for utterance, raw_output in rejected: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw_output, registry) + self.assertTrue(grounded.clarification.required) + self.assertIn("action", grounded.clarification.unresolved_slots) + + accepted = ( + ( + "Make the Lower Floor AC warmer.", + "adjustUp|AC|temperature|*|*|*|Lower Floor", + ), + ( + "Make the Lower Floor AC cooler.", + "adjustDown|AC|temperature|*|*|*|Lower Floor", + ), + ) + for utterance, raw_output in accepted: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw_output, registry) + self.assertNotIn("action", grounded.clarification.unresolved_slots) + + def test_unregistered_floor_is_unresolved_not_an_ontology_guess(self) -> None: + registry = EntityRegistry(( + EntitySpec("climate.landing", "climate", "AC", "Landing", "First Floor"), + EntitySpec("climate.loft", "climate", "AC", "Loft", "Second Floor"), + )) + grounded = ground_domux_request( + "Set the upstairs AC to 21 degrees.", + "set|AC|temperature|21|Celsius|*|Upstairs", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("climate.landing", "climate.loft"), + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("floor", grounded.clarification.unresolved_slots) + resolved = resolve_clarification_submission( + grounded, + answer="Use 21 Celsius for the Landing AC on the First Floor.", + confirmed_instruction=DomuxInstruction( + "set", "AC", "temperature", "21", "Celsius", "Landing", "First Floor" + ), + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, "climate.landing") + + singleton = EntityRegistry(( + EntitySpec("climate.only", "climate", "AC", "Landing", "First Floor"), + )) + singleton_grounded = ground_domux_request( + "Set the upstairs AC to 21 degrees.", + "set|AC|temperature|21|Celsius|*|Upstairs", + singleton, + ) + self.assertEqual( + tuple(entity.entity_id for entity in singleton_grounded.candidates), + ("climate.only",), + ) + self.assertTrue(singleton_grounded.clarification.required) + self.assertIn("floor", singleton_grounded.clarification.unresolved_slots) + + omitted_by_model = ground_domux_request( + "Set the upstairs AC to 21 degrees.", + "set|AC|temperature|21|Celsius|*|*", + singleton, + ) + self.assertTrue(omitted_by_model.clarification.required) + self.assertIn("device", omitted_by_model.clarification.unresolved_slots) + with self.assertRaisesRegex(GroundingError, "explicitly identify"): + resolve_clarification_submission( + omitted_by_model, + answer="Yes.", + confirmed_instruction=DomuxInstruction( + "set", "AC", "temperature", "21", "Celsius", "Landing", "First Floor" + ), + registry=singleton, + ) + + def test_unregistered_compound_device_falls_back_only_to_clarification(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.dining", "light", "Light", "Dining Room", "Ground Floor"), + EntitySpec("cover.dining", "cover", "Curtain", "Dining Room", "Ground Floor"), + )) + grounded = ground_domux_request( + "Make the dining light Blue—no, perhaps Green; confirm the color.", + "set|Dining Light|color|Green|*|Dining Room|*", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.dining",), + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("device", grounded.clarification.unresolved_slots) + resolved = resolve_clarification_submission( + grounded, + answer="Confirm Green for the Dining Room light.", + confirmed_instruction=DomuxInstruction( + "set", "Light", "color", "Green", "*", "Dining Room", "Ground Floor" + ), + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, "light.dining") + + def test_partial_room_token_with_generic_device_never_becomes_canonical(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.dining", "light", "Light", "Dining Room", "Ground Floor"), + EntitySpec("cover.dining", "cover", "Curtain", "Dining Room", "Ground Floor"), + )) + for raw_output in ( + "set|Dining Light|color|Green|*|Dining Room|*", + "set|Light|color|Green|*|*|*", + "set|Light|color|Green|*|Dining Room|*", + ): + with self.subTest(raw_output=raw_output): + grounded = ground_domux_request( + "Set the dining light to Green.", + raw_output, + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.dining",), + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("room", grounded.clarification.unresolved_slots) + + canonical = ground_domux_request( + "Set the Dining Room light to Green.", + "set|Light|color|Green|*|Dining Room|*", + registry, + ) + self.assertFalse(canonical.clarification.required) + + registered_compound = EntityRegistry(( + EntitySpec("light.dining_named", "light", "Dining Light", "Atrium", "Ground Floor"), + )) + compound = ground_domux_request( + "Set the Dining Light in the Atrium to Green.", + "set|Dining Light|color|Green|*|Atrium|*", + registered_compound, + ) + self.assertFalse(compound.clarification.required) + + overlapping_rooms = EntityRegistry(( + EntitySpec("light.reading_room", "light", "Light", "Reading Room", "First Floor"), + EntitySpec("light.reading_nook", "light", "Light", "Reading Nook", "First Floor"), + )) + exact_overlap = ground_domux_request( + "Turn off the Reading Room light.", + "turnOff|Light|*|*|*|Reading Room|*", + overlapping_rooms, + ) + self.assertFalse(exact_overlap.clarification.required) + + def test_deictic_context_precedes_a_model_only_domain_guess(self) -> None: + context = SessionContext(("light.living_ceiling", "light.study_ceiling")) + grounded = ground_domux_request( + "Make that one warmer.", + "adjustUp|AC|temperature|*|*|*|*", + self.registry, + context, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.living_ceiling", "light.study_ceiling"), + ) + self.assertTrue(grounded.clarification.required) + resolved = resolve_clarification_submission( + grounded, + answer="Set the Living Room light to 3000 Kelvin.", + confirmed_instruction=DomuxInstruction( + "set", "Light", "colorTemperature", "3000", "Kelvin", + "Living Room", "Ground Floor", + ), + registry=self.registry, + ) + self.assertEqual(resolved.chosen.entity_id, "light.living_ceiling") + + wrong_room = ground_domux_request( + "Turn it off.", + "turnOff|AC|*|*|*|Utility Room|Ground Floor", + self.registry, + context, + ) + self.assertEqual( + tuple(entity.entity_id for entity in wrong_room.candidates), + ("light.living_ceiling", "light.study_ceiling"), + ) + self.assertNotIn("climate.study_ac", { + entity.entity_id for entity in wrong_room.candidates + }) + self.assertTrue(wrong_room.clarification.required) + + def test_deictic_context_can_be_narrowed_by_an_explicit_user_domain(self) -> None: + context = SessionContext(("light.study_ceiling", "cover.study_curtain")) + grounded = ground_domux_request( + "Turn that light off.", + "turnOff|Light|*|*|*|*|*", + self.registry, + context, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.study_ceiling",), + ) + self.assertFalse(grounded.clarification.required) + + wrong_model_domain = ground_domux_request( + "Turn that light off.", + "turnOff|AC|*|*|*|*|*", + self.registry, + context, + ) + self.assertEqual( + tuple(entity.entity_id for entity in wrong_model_domain.candidates), + ("light.study_ceiling",), + ) + self.assertTrue(wrong_model_domain.clarification.required) + + no_context_match = ground_domux_request( + "Turn that light off.", + "turnOff|Light|*|*|*|*|*", + self.registry, + SessionContext(("cover.study_curtain",)), + ) + self.assertEqual(no_context_match.candidates, ()) + self.assertTrue(no_context_match.clarification.required) + + def test_stale_deictic_context_never_falls_back_to_a_global_singleton(self) -> None: + context = SessionContext(("missing.old_entity", "light.study_ceiling")) + grounded = ground_domux_request( + "Turn it off.", + "turnOff|*|*|*|*|*|*", + self.registry, + context, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.study_ceiling",), + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("stale_context_reference", grounded.clarification.reasons) + self.assertIn("context", grounded.clarification.unresolved_slots) + + all_stale = ground_domux_request( + "Turn it off.", + "turnOff|*|*|*|*|*|*", + self.registry, + SessionContext(("missing.old_entity",)), + ) + self.assertEqual(all_stale.candidates, ()) + self.assertTrue(all_stale.clarification.required) + + def test_deictic_request_without_context_requires_an_explicit_target(self) -> None: + singleton = EntityRegistry(( + EntitySpec("light.only", "light", "Light", "Study", "Ground Floor"), + )) + unresolved = ground_domux_request( + "Turn it off.", + "turnOff|*|*|*|*|*|*", + singleton, + ) + self.assertTrue(unresolved.clarification.required) + self.assertIn("unsupported_request_grammar", unresolved.clarification.reasons) + self.assertIn("authorization", unresolved.clarification.unresolved_slots) + + explicit = ground_domux_request( + "Turn that Study light off.", + "turnOff|Light|*|*|*|Study|*", + singleton, + ) + self.assertFalse(explicit.clarification.required) + + def test_explicit_selector_zero_match_never_falls_back_to_model_or_context(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("cover.kitchen", "cover", "Curtain", "Kitchen", "Ground Floor"), + )) + for raw_output in ( + "turnOff|Light|*|*|*|*|*", + "turnOff|Light|*|*|*|Kitchen|*", + "turnOff|Light|*|*|*|Study|*", + ): + with self.subTest(raw_output=raw_output): + grounded = ground_domux_request( + "Turn off the Kitchen light.", + raw_output, + registry, + ) + self.assertEqual(grounded.candidates, ()) + self.assertTrue(grounded.clarification.required) + + contextual = ground_domux_request( + "Turn that Kitchen light off.", + "turnOff|Light|*|*|*|Kitchen|*", + registry, + SessionContext(("light.study",)), + ) + self.assertEqual(contextual.candidates, ()) + self.assertTrue(contextual.clarification.required) + + with_kitchen_light = EntityRegistry(( + *registry.entities, + EntitySpec("light.kitchen", "light", "Light", "Kitchen", "Ground Floor"), + )) + positive = ground_domux_request( + "Turn off the Kitchen light.", + "turnOff|Light|*|*|*|*|*", + with_kitchen_light, + ) + self.assertEqual( + tuple(entity.entity_id for entity in positive.candidates), + ("light.kitchen",), + ) + self.assertFalse(positive.clarification.required) + + def test_same_span_room_and_alias_interpretations_are_not_anded(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study_a", "light", "Light", "Study", "Ground Floor"), + EntitySpec( + "light.study_b", "light", "Light", "Study", "Ground Floor", ("Study",), + ), + )) + grounded = ground_domux_request( + "Turn off the Study light.", + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.study_a", "light.study_b"), + ) + self.assertTrue(grounded.clarification.required) + + def test_overlapping_room_and_specific_device_interpretations_remain_ambiguous(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("light.named", "light", "Study Light", "Atrium", "Ground Floor"), + )) + grounded = ground_domux_request( + "Turn off the Study light.", + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.named", "light.study"), + ) + self.assertTrue(grounded.clarification.required) + + def test_inventory_words_in_ordinary_prose_are_not_target_selectors(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.right", "light", "Light", "Right", "Ground Floor"), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + grounded = ground_domux_request( + "Turn the light off right now.", + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.right", "light.study"), + ) + contextual = ground_domux_request( + "Turn that light off right now.", + "turnOff|Light|*|*|*|*|*", + registry, + SessionContext(("light.right", "light.study")), + ) + self.assertEqual(len(contextual.candidates), 2) + + alias_registry = EntityRegistry(( + EntitySpec( + "light.alias", "light", "Light", "Living Room", "Ground Floor", ("Right",), + ), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + alias_collision = ground_domux_request( + "Turn the light off right now.", + "turnOff|Light|*|*|*|*|*", + alias_registry, + ) + self.assertEqual(len(alias_collision.candidates), 2) + + def test_relational_room_phrases_never_fall_back_to_a_global_singleton(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("cover.kitchen", "cover", "Curtain", "Kitchen", "Ground Floor"), + )) + for phrase in ("my room", "the right room", "any room"): + with self.subTest(phrase=phrase): + grounded = ground_domux_request( + f"Turn off the light in {phrase}.", + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("room", grounded.clarification.unresolved_slots) + with self.assertRaisesRegex(GroundingError, "repaired target"): + resolve_clarification_submission( + grounded, + answer="Yes.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ), + registry=registry, + ) + + explicit = ground_domux_request( + "Turn off the light in Study.", + "turnOff|Light|*|*|*|Study|*", + registry, + ) + self.assertFalse(explicit.clarification.required) + + def test_partial_selector_detection_is_limited_to_selector_positions(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "First Floor"), + )) + sequence_word = ground_domux_request( + "First, turn off the Study light on the First Floor.", + "turnOff|Light|*|*|*|Study|First Floor", + registry, + ) + self.assertNotIn("floor", sequence_word.clarification.unresolved_slots) + + lower_registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Lower Floor"), + )) + operation_word = ground_domux_request( + "Lower the Study light brightness.", + "adjustDown|Light|brightness|*|*|Study|*", + lower_registry, + ) + self.assertNotIn("floor", operation_word.clarification.unresolved_slots) + + def test_context_selector_is_a_true_intersection(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("light.kitchen", "light", "Light", "Kitchen", "Ground Floor"), + )) + only_study = ground_domux_request( + "Turn that Kitchen light off.", + "turnOff|Light|*|*|*|*|*", + registry, + SessionContext(("light.study",)), + ) + self.assertEqual(only_study.candidates, ()) + + both = ground_domux_request( + "Turn that Kitchen light off.", + "turnOff|Light|*|*|*|*|*", + registry, + SessionContext(("light.study", "light.kitchen")), + ) + self.assertEqual( + tuple(entity.entity_id for entity in both.candidates), + ("light.kitchen",), + ) + + stale = ground_domux_request( + "Turn that Kitchen light off.", + "turnOff|Light|*|*|*|*|*", + registry, + SessionContext(("missing.kitchen", "light.study")), + ) + self.assertEqual(stale.candidates, ()) + self.assertIn("stale_context_reference", stale.clarification.reasons) + + def test_other_reference_never_collapses_to_a_single_context_target(self) -> None: + no_context = ground_domux_request( + "Turn the other light off.", + "turnOff|Light|*|*|*|*|*", + self.registry, + ) + self.assertEqual(no_context.candidates, ()) + self.assertIn("unsupported_request_grammar", no_context.clarification.reasons) + + one_context = ground_domux_request( + "Turn the other light off.", + "turnOff|Light|*|*|*|*|*", + self.registry, + SessionContext(("light.study_ceiling",)), + ) + self.assertEqual(one_context.candidates, ()) + + heterogeneous = ground_domux_request( + "Turn the other light off.", + "turnOff|Light|*|*|*|Study|*", + self.registry, + SessionContext(("light.study_ceiling", "cover.study_curtain")), + ) + self.assertEqual(heterogeneous.candidates, ()) + + two_lights = ground_domux_request( + "Turn the other light off.", + "turnOff|Light|*|*|*|Study|*", + self.registry, + SessionContext(("light.living_ceiling", "light.study_ceiling")), + ) + self.assertEqual(len(two_lights.candidates), 2) + self.assertIn("other_reference_requires_selection", two_lights.clarification.reasons) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + two_lights, + answer="Yes.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Ceiling Light", "*", "*", "*", "Study", "Ground Floor" + ), + registry=self.registry, + ) + selected = resolve_clarification_submission( + two_lights, + answer="2", + confirmed_instruction=DomuxInstruction( + "turnOff", "Ceiling Light", "*", "*", "*", "Study", "Ground Floor" + ), + registry=self.registry, + ) + self.assertEqual(selected.chosen.entity_id, "light.study_ceiling") + + def test_multi_token_selector_spans_do_not_hide_a_second_partial_selector(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.dining", "light", "Light", "Dining Room", "Ground Floor"), + EntitySpec("cover.study", "cover", "Curtain", "Study", "Ground Floor"), + )) + contradictory = ground_domux_request( + "Set the dining light in Study to Green.", + "set|Light|color|Green|*|*|*", + registry, + ) + self.assertEqual(contradictory.candidates, ()) + self.assertIn("room", contradictory.clarification.unresolved_slots) + + exact = ground_domux_request( + "Set the Dining Room light to Green.", + "set|Light|color|Green|*|Dining Room|*", + registry, + ) + self.assertFalse(exact.clarification.required) + + named_device = EntityRegistry(( + EntitySpec("light.named", "light", "Dining Light", "Atrium", "Ground Floor"), + )) + complete_device = ground_domux_request( + "Set the Dining Light in the Atrium to Green.", + "set|Dining Light|color|Green|*|Atrium|*", + named_device, + ) + self.assertFalse(complete_device.clarification.required) + + def test_negated_relative_directions_cannot_authorize_adjustment(self) -> None: + cases = ( + ("Do not increase the Study light brightness.", "adjustUp"), + ("Don't decrease the Study light brightness.", "adjustDown"), + ("Never make the Study light brighter.", "adjustUp"), + ("No increase for the Study light brightness.", "adjustUp"), + ("Make the Study AC not warmer.", "adjustUp"), + ("No dimmer for the Study light.", "adjustDown"), + ) + for utterance, action in cases: + raw_output = ( + f"{action}|AC|temperature|*|*|Study|*" + if "AC" in utterance + else f"{action}|Light|brightness|*|*|Study|*" + ) + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw_output, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn("negative_or_cancelled_intent", grounded.clarification.reasons) + self.assertIn("action", grounded.clarification.unresolved_slots) + + for utterance, action in ( + ("Make the Study light not any brighter.", "adjustUp"), + ("Make the Study light not any dimmer.", "adjustDown"), + ("Make the Study light anything but brighter.", "adjustUp"), + ("Make the Study light other than dimmer.", "adjustDown"), + ("I don't want the Study light brighter.", "adjustUp"), + ("No need for a brighter Study light.", "adjustUp"), + ("No need for a dimmer Study light.", "adjustDown"), + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + f"{action}|Light|brightness|*|*|Study|*", + self.registry, + ) + self.assertIn("negative_or_cancelled_intent", grounded.clarification.reasons) + with self.assertRaisesRegex(GroundingError, "negated"): + resolve_clarification_submission( + grounded, + answer="Confirm the Study light on the Ground Floor.", + confirmed_instruction=DomuxInstruction( + action, "Light", "brightness", "*", "*", "Study", "Ground Floor" + ), + registry=self.registry, + ) + + def test_negative_selector_spans_never_become_positive_target_constraints(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.living", "light", "Light", "Living Room", "Ground Floor"), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + for utterance in ( + "Turn off the light, but not the light in Study.", + "Turn off the light, but not any one in Study.", + "Turn off the light, but not a light in Study.", + "Turn off the light, anything but a light in Study.", + "Turn off a light, but don't use the light in Study.", + "Turn off a light, but do not use my device in Study.", + "Turn off every light but Study.", + "Turn off every light, but Study.", + "Turn off every light (but Study).", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.living", "light.study"), + ) + self.assertIn("light.study", grounded.negated_entity_ids) + self.assertTrue(grounded.clarification.required) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer="Yes.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ), + registry=registry, + ) + + def test_punctuated_and_anaphoric_negative_selectors_fail_closed(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + for utterance in ( + 'Turn off the light, but not "Study Lamp".', + "Turn off the light, but not (Study Lamp).", + "Turn off the light, but not: Study Lamp.", + "Turn off the light, but not the light that I mean, the one in Study.", + "Turn off the light, but do not use the lamp I mean, namely Study Lamp.", + "Turn off everything but Study Lamp.", + "Turn off every light but Study Lamp.", + "Turn off every light, but Study Lamp.", + "Turn off every light (but Study Lamp).", + "Turn off all lights but Study Lamp.", + "Turn off all the lights but Study Lamp.", + "Turn off all the lights, but Study Lamp.", + "Turn off any light but Study Lamp.", + "Turn off any one light but Study Lamp.", + "Turn off any of the lights but Study Lamp.", + "Turn off either light but Study Lamp.", + "Turn off each light but Study Lamp.", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.living", "light.study"), + ) + self.assertEqual(grounded.negated_entity_ids, ("light.study",)) + self.assertTrue(grounded.clarification.required) + with self.assertRaises(GroundingError): + resolve_unique_request(grounded, registry) + + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "negative-selector-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Yes.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + for utterance in ( + "Turn off nothing but Study Lamp.", + "Turn off no light but Study Lamp.", + ): + with self.subTest(only_target=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertEqual(grounded.negated_entity_ids, ()) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.study",), + ) + + corrected = ground_domux_request( + "Turn off not Study Lamp but Living Lamp.", + "turnOff|Light|*|*|*|Living Room|Ground Floor", + registry, + ) + self.assertEqual(corrected.negated_entity_ids, ("light.study",)) + resolved = resolve_clarification_submission( + corrected, + answer="Not Study Lamp; use Living Lamp.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Living Room", "Ground Floor" + ), + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, "light.living") + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "positive-contrast-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=corrected, + registry=registry, + adapter=adapter, + clarification_answer="Not Study Lamp; use Living Lamp.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Living Room", "Ground Floor" + ), + ) + result = store.commit(action.confirmation(), registry=registry, adapter=adapter) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual(adapter.sut_calls[0]["data"]["entity_id"], "light.living") + + def test_negative_selector_scope_resets_only_at_explicit_boundaries(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + EntitySpec( + "light.utility", "light", "Light", "Utility Room", "Ground Floor", + aliases=("Utility Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed_living = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Living Room", "Ground Floor" + ) + for utterance in ( + "Do not use Study Lamp. Turn off Living Lamp.", + "Do not use Study Lamp; turn off Living Lamp.", + "Do not use Study Lamp, then turn off Living Lamp.", + "Do not use Study Lamp, turn off Living Lamp.", + "Do not use Study Lamp and then turn off Living Lamp.", + "Do not use Study Lamp: turn off Living Lamp.", + "Do not use Study Lamp—turn off Living Lamp.", + "Do not use, choose Study Lamp, then turn off Living Lamp.", + "Do not use, choose Study Lamp, turn off Living Lamp.", + "Do not use, choose Study Lamp: turn off Living Lamp.", + "Turn off not Study Lamp: use Living Lamp.", + "Turn off not Study Lamp—use Living Lamp.", + ): + with self.subTest(positive_restart=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Living Room|Ground Floor", + registry, + ) + self.assertEqual(grounded.negated_entity_ids, ("light.study",)) + self.assertNotIn( + "light.living", + grounded.negated_entity_ids, + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "explicit-boundary-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Living Lamp.", + confirmed_instruction=confirmed_living, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "light.living", + ) + + for utterance in ( + "Turn off any light but use Living Lamp.", + "Turn off any light, but use Living Lamp.", + ): + with self.subTest(direct_positive_restart=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Living Room|Ground Floor", + registry, + ) + self.assertNotIn("light.living", grounded.negated_entity_ids) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.living",), + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "direct-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Living Lamp.", + confirmed_instruction=confirmed_living, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "light.living", + ) + + excluded_pair = ground_domux_request( + "Turn off Utility Lamp, but not Study Lamp or Living Lamp.", + "turnOff|Light|*|*|*|Utility Room|Ground Floor", + registry, + ) + self.assertEqual( + excluded_pair.negated_entity_ids, + ("light.living", "light.study"), + ) + self.assertEqual( + tuple(entity.entity_id for entity in excluded_pair.candidates), + ("light.utility",), + ) + + plain_and = ground_domux_request( + "Do not use Study Lamp and turn off Living Lamp.", + "turnOff|Light|*|*|*|Living Room|Ground Floor", + registry, + ) + self.assertEqual( + plain_and.negated_entity_ids, + ("light.living", "light.study"), + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "plain-and-negation-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=plain_and, + registry=registry, + adapter=adapter, + clarification_answer="Use Living Lamp.", + confirmed_instruction=confirmed_living, + ) + self.assertEqual(adapter.sut_calls, []) + + def test_entity_id_periods_do_not_terminate_negative_scope(self) -> None: + registry = EntityRegistry(( + EntitySpec("climate.study", "climate", "AC", "Study", "Ground Floor"), + EntitySpec( + "climate.living", "climate", "AC", "Living Room", "Ground Floor" + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "cool", + "attributes": { + "temperature": 24.0, + "hvac_modes": ["off", "cool"], + "fan_modes": ["low", "medium", "high"], + "supported_features": 1, + "temperature_unit": "°C", + "min_temp": 16.0, + "max_temp": 30.0, + "target_temp_step": 0.5, + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "AC", "*", "*", "*", "Study", "Ground Floor" + ) + for utterance in ( + "Turn off the AC, but not climate.study.", + 'Turn off the AC, but not "climate.study".', + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|AC|*|*|*|Study|Ground Floor", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("climate.living", "climate.study"), + ) + self.assertIn("climate.study", grounded.negated_entity_ids) + self.assertTrue(grounded.clarification.required) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer="Use climate.study.", + confirmed_instruction=confirmed, + registry=registry, + ) + + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "entity-id-negation-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use climate.study.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + separate_sentence = ground_domux_request( + "Do not use climate.study. Turn off climate.living.", + "turnOff|AC|*|*|*|Living Room|Ground Floor", + registry, + ) + self.assertEqual(separate_sentence.negated_entity_ids, ("climate.study",)) + self.assertEqual( + tuple(entity.entity_id for entity in separate_sentence.candidates), + ("climate.living",), + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "entity-id-sentence-boundary-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=separate_sentence, + registry=registry, + adapter=adapter, + clarification_answer="Use climate.living.", + confirmed_instruction=DomuxInstruction( + "turnOff", "AC", "*", "*", "*", "Living Room", "Ground Floor" + ), + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "climate.living", + ) + + def test_coordinated_negative_actions_never_restart_authorization(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + for utterance in ( + "Turn off the light but do not use, choose, or mean Study Lamp.", + "Turn off the light but do not use or choose Study Lamp.", + "Turn off the light but do not use and choose Study Lamp.", + "Turn off the light but do not use, select, choose, or mean Study Lamp.", + "Turn off the light but do not use, choose, or mean the Study Lamp.", + "Turn off the light but do not use, choose, or mean (Study Lamp).", + 'Turn off the light but do not use, choose, or mean "Study Lamp".', + "Turn off the light but do not use, choose, or mean my Study Lamp.", + "Turn off the light but do not use choose Study Lamp.", + "Turn off the light but do not turn off, switch off, or close Study Lamp.", + "Turn off the light but do not turn off, switch off, or close the Study Lamp.", + "Turn off a light, but I don't want Study Lamp.", + "Turn off a light, but I don't need Study Lamp.", + "Turn off a light, but I don't have Study Lamp.", + "Turn off a light, but I don't want to use Study Lamp.", + "Turn off a light, but I don't need to use Study Lamp.", + "Turn off a light, but I don't want to choose Study Lamp.", + "Turn off a light, but I do not want Study Lamp.", + "Turn off a light, but I do not need to use Study Lamp.", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertIn("light.study", grounded.negated_entity_ids) + self.assertTrue(grounded.clarification.required) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer="Use Study Lamp.", + confirmed_instruction=confirmed, + registry=registry, + ) + + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "coordinated-negation-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + def test_negator_to_predicate_gaps_cannot_authorize_a_selector(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + negated = ( + "Turn off a light, but do not: use Study Lamp.", + "Turn off a light, but don't: use Study Lamp.", + "Turn off a light, but do not—use Study Lamp.", + "Turn off a light, but do not, please, use Study Lamp.", + "Turn off a light, but do not use: choose Study Lamp.", + "Turn off a light, but I don't want to use, choose Study Lamp.", + "Turn off a light, but I don't want to use, choose, or mean Study Lamp.", + "Turn off a light, but I don't need to use, choose Study Lamp.", + "Turn off a light, but I do not want to use, choose Study Lamp.", + "Turn off a light, but do not, I am sure, use Study Lamp.", + "Turn off a light, but don't, would you, use Study Lamp.", + "Turn off a light, but do not, could you, use Study Lamp.", + "Turn off a light, but do not, can you, use Study Lamp.", + "Turn off a light, but do not, for me, use Study Lamp.", + "Turn off a light, but do not, around, use Study Lamp.", + "Turn off a light, but not!!! Study Lamp.", + "Turn off a light, but not... Study Lamp.", + "Turn off a light, but do not!!! use Study Lamp.", + "Turn off a light, but don't... use Study Lamp.", + "Turn off a light, but do not?! choose Study Lamp.", + "Turn off a light, but no! Study Lamp.", + "Turn off a light, but do not! use Study Lamp.", + "Turn off a light, but don't? choose Study Lamp.", + "Turn off a light, but not! use Study Lamp.", + "Turn off a light, but do not. use Study Lamp.", + "Do not! turn off the Study light.", + "Don't! turn off the Study light.", + "Do not. turn off the Study light.", + "Do not? turn off the Study light.", + "Do not! increase the Study light brightness by 10 percent.", + "Turn off a light, no, wait, Study Lamp.", + "Turn off a light, no, wait, use Study Lamp.", + ) + for utterance in negated: + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertIn("light.study", grounded.negated_entity_ids) + self.assertTrue(grounded.clarification.required) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "negator-gap-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + for utterance in ( + "Turn off a light, wait, Study Lamp.", + "Turn off a light, wait, use Study Lamp.", + ): + with self.subTest(suspensive_wait=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn( + "negative_or_cancelled_intent", + grounded.clarification.reasons, + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + def test_generic_exclusions_cannot_be_laundered_by_clarification(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec( + "climate.living", "climate", "AC", "Living Room", "Ground Floor" + ), + )) + states = { + "light.study": { + "entity_id": "light.study", + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + }, + "climate.living": { + "entity_id": "climate.living", + "state": "cool", + "attributes": { + "temperature": 24.0, + "hvac_modes": ["off", "cool"], + "fan_modes": ["low", "medium", "high"], + "supported_features": 1, + "temperature_unit": "°C", + "min_temp": 16.0, + "max_temp": 30.0, + "target_temp_step": 0.5, + }, + }, + } + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + + def assert_blocked(utterance: str, *, domain_exclusion: bool) -> None: + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertTrue(grounded.clarification.required) + if domain_exclusion: + self.assertIn("light.study", grounded.negated_entity_ids) + else: + self.assertIn( + "unsupported_request_grammar", + grounded.clarification.reasons, + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + ttl_seconds=30, + clock=MutableClock(), + nonce_factory=lambda: "generic-exclusion-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Yes, use the Study light.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + for utterance in ( + "Turn off a device, but avoid the light.", + "Turn off a device, but use anything other than the light.", + "Turn off a device, but use anything besides the light.", + "Turn off a device, but use anything apart from the light.", + "Turn off a device instead of the light.", + "Turn off anything but the light.", + "Turn off a device without the light.", + "Turn off a device except for the light.", + "Turn off a device, but do not use any light.", + ): + with self.subTest(domain_exclusion=utterance): + assert_blocked(utterance, domain_exclusion=True) + + for utterance in ( + "Turn off a device, but avoid the device.", + "Turn off a device other than the device.", + "Turn off a device besides the device.", + "Turn off a device apart from the device.", + "Turn off a device instead of the device.", + "Turn off a device, but do not use anything.", + "Turn off a device, but do not use any device.", + "Turn off a device, but do not use a device.", + "Turn off a light. Do not use anything. Study.", + ): + with self.subTest(unresolved_generic=utterance): + assert_blocked(utterance, domain_exclusion=False) + + # Double-negation/"only" semantics are deliberately outside the + # bounded grammar. Fail closed even though the conservative parser + # records the named selector itself as negated. + double_negative = "Turn off a light, but don't use anything but Study." + grounded = ground_domux_request( + double_negative, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("light.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Yes, use the Study light.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + specific_registry = EntityRegistry(( + EntitySpec("light.right", "light", "Light", "Right", "Ground Floor"), + EntitySpec("light.left", "light", "Light", "Left", "Ground Floor"), + )) + specific_states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in specific_registry.entities + } + grounded = ground_domux_request( + "Do not use the light in Right, turn off the light in Left.", + "turnOff|Light|*|*|*|Left|Ground Floor", + specific_registry, + ) + self.assertEqual(grounded.negated_entity_ids, ("light.right",)) + adapter = InMemoryHAAdapter(specific_states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "qualified-selector-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=specific_registry, + adapter=adapter, + clarification_answer="Use the light in Left.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Left", "Ground Floor" + ), + ) + result = store.commit( + action.confirmation(), + registry=specific_registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual(adapter.sut_calls[0]["data"]["entity_id"], "light.left") + + def test_anaphoric_exclusions_bind_the_later_named_referent(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + utterances = ( + "Turn off a light, but do not use that, but by that I mean Study Lamp.", + "Turn off a light, but do not use it, but by it I mean Study Lamp.", + ( + "Turn off a light, but do not use the one that I mean, " + "but by that I mean Study Lamp." + ), + ( + "Turn off a light, but do not use the light that I mean—" + "but by that I mean Study Lamp." + ), + ( + "Turn off a light, but do not use it, " + "but by it, I mean Study Lamp." + ), + ( + "Turn off a light, but do not use that, " + "but by that, I mean Study Lamp." + ), + ( + "Turn off a light, but do not use this, " + "but by this— I mean Study Lamp." + ), + "Turn off a light, but do not use it; by it, I mean Study Lamp.", + ) + for utterance in utterances: + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertIn("light.study", grounded.negated_entity_ids) + self.assertTrue(grounded.clarification.required) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "anaphoric-exclusion-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + def test_generic_withdrawals_block_but_complete_new_commands_restart_scope(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + withdrawals = ( + "Turn off a light; I don't need any light.", + "Turn off a light; I don't want any light.", + "Turn off a light; I do not need the light.", + "Turn off a light; I do not want the light.", + "Turn off a light; no need for any light.", + "Turn off a light; I have no need for a light.", + "Turn off a light; I don't have any light.", + "Turn off a light; I would not use any light.", + "Turn off a light; I could not use any light.", + "Turn off a light; I can not use any light.", + "Turn off a light; I have no use for any light.", + "Turn off a light; I do not want to use any light.", + "Turn off a light; I don't need you to use any light.", + "Turn off a light; I do not want you to use the light.", + "Turn off a light; no need to use any light.", + "Turn off a light; no need to choose any light.", + "Turn off a light; no need to select any light.", + "Turn off a light; I need not use any light.", + "Turn off a light; we need not choose any light.", + "Turn off a light; you need not select any light.", + "Turn off a light; I want not to use any light.", + "Turn off a light; we want not to choose any light.", + "Turn off a light; I have no need to use any light.", + "Turn off a light; we have no need to choose any light.", + "Turn off a light; no use of any light.", + "Turn off a light; I have no use of any light.", + "Turn off a light; I want no use of any light.", + ) + for utterance in withdrawals: + with self.subTest(withdrawal=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertIn( + "negative_or_cancelled_intent", + grounded.clarification.reasons, + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + explanatory_colon_withdrawals = ( + "Don't do it: turn off Study Lamp.", + "Do not execute this: turn off Study Lamp.", + "Do not confirm this: turn off Study Lamp.", + "I don't want this: turn off Study Lamp.", + "I do not need it: turn off Study Lamp.", + "I have no need for this: turn off Study Lamp.", + "No need for this: turn off Study Lamp.", + "Avoid this: turn off Study Lamp.", + ) + for utterance in explanatory_colon_withdrawals: + with self.subTest(explanatory_colon_withdrawal=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertIn("light.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + target_only_restatements = ( + "Turn off a light; I don't want any light, I mean Study Lamp.", + "Turn off a light; I don't need any light, I mean Study Lamp.", + "Turn off a light; I don't have any light, I mean Study Lamp.", + "Turn off a light; I would not use any light, I mean Study Lamp.", + "Turn off a light; no need for any light, I mean Study Lamp.", + "Turn off a light; I have no use for any light, I mean Study Lamp.", + "Do not turn off Living Lamp, I mean Study Lamp.", + "I don't want Living Lamp, I mean Study Lamp.", + ) + for utterance in target_only_restatements: + with self.subTest(target_only_restart=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertIn( + "negative_or_cancelled_intent", + grounded.clarification.reasons, + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + negative_state_restatements = ( + ( + "Turn off a light; do not use or move Living Lamp, " + "I mean Study Lamp." + ), + ( + "Turn off a light; do not use and move Living Lamp, " + "I mean Study Lamp." + ), + ( + "Turn off a light; do not use Living Lamp to move it, " + "I mean Study Lamp." + ), + ) + for utterance in negative_state_restatements: + with self.subTest(negative_state_restatement=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertIn("light.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + incomplete_restarts = ( + "Turn off a light; I do not want any light; adjust Study Lamp.", + "Turn off a light; I do not want any light; move Study Lamp.", + "Turn off a light; I do not want any light; change Study Lamp.", + "Turn off a light; I do not want any light; make Study Lamp.", + "Turn off a light; I do not want any light; turn around Study Lamp.", + "Turn off a light; I do not want any light; turn to Study Lamp.", + "Turn off a light; I do not want any light; switch around Study Lamp.", + ) + for utterance in incomplete_restarts: + with self.subTest(incomplete_restart=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertIn("action", grounded.clarification.unresolved_slots) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + repaired_restart = ground_domux_request( + "Turn off a light; I do not want any light; adjust Study Lamp.", + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "repaired-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=repaired_restart, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study Lamp.", + confirmed_instruction=confirmed, + ) + result = store.commit(action.confirmation(), registry=registry, adapter=adapter) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual(adapter.sut_calls[0]["data"]["entity_id"], "light.study") + + direct_generic_exclusions = ( + "Turn off a light; I would not use any light.", + "Turn off a light; I could not use any light.", + "Turn off a light; I can not use any light.", + "Turn off a light; I do not want to use any light.", + "Turn off a light; I don't need you to use any light.", + "Turn off a light; I do not want you to use the light.", + "Turn off a light; no need to use any light.", + "Turn off a light; I need not choose any light.", + "Turn off a light; I want not to select any light.", + "Turn off a light; I have no need to choose any light.", + "Turn off a light; no use of any light.", + "Turn off a light; I want no use of any light.", + ) + for utterance in direct_generic_exclusions: + with self.subTest(direct_generic_exclusion=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual( + grounded.negated_entity_ids, + ("light.living", "light.study"), + ) + + positive_restarts = ( + "Do not turn off Living Lamp; turn off Study Lamp.", + "Do not use Living Lamp: turn off Study Lamp.", + "Don't turn off Living Lamp, then turn off Study Lamp.", + "Never turn off Living Lamp. Turn off Study Lamp.", + "Do not switch off Living Lamp—but turn off Study Lamp.", + "I don't need any light; turn off Study Lamp.", + "I would not use any light. Turn off Study Lamp.", + "I have no use for any light—turn off Study Lamp.", + "I do not want any light; can you turn off Study Lamp?", + "I do not want any light; could you please turn off Study Lamp?", + "I do not want any light; would you turn off Study Lamp?", + "I do not want any light; will you turn off Study Lamp?", + "I do not want any light; I want you to turn off Study Lamp.", + "I do not want any light; I need you to turn off Study Lamp.", + "I do not want any light; can you just turn off Study Lamp?", + "I do not want any light; could you please just turn off Study Lamp?", + "I do not want any light; I want you to just turn off Study Lamp.", + "I do not want any light; I just want you to turn off Study Lamp.", + "I need not choose any light; turn off Study Lamp.", + "No use of any light; turn off Study Lamp.", + "I don't need any light; proceed to turn off Study Lamp.", + "I don't need any light; execute turn off Study Lamp.", + "Do not proceed; turn off Study Lamp.", + "Do not execute; turn off Study Lamp.", + "Do not confirm; turn off Study Lamp.", + "Do not proceed, then turn off Study Lamp.", + "Do not execute, then turn off Study Lamp.", + "Do not confirm, then turn off Study Lamp.", + "No need to use any light; turn off Study Lamp.", + "Anything but turn off Living Lamp; turn off Study Lamp.", + "Turn off any light but use Study Lamp.", + "Turn off any light, but use Study Lamp.", + ) + for utterance in positive_restarts: + with self.subTest(positive_restart=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertNotIn( + "negative_or_cancelled_intent", + grounded.clarification.reasons, + ) + self.assertNotIn("light.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "positive-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study Lamp.", + confirmed_instruction=confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "light.study", + ) + + def test_selector_aware_positive_corrections_remain_executable(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + cases = ( + ( + "Turn off a light, do not use Living Lamp, I mean Study Lamp.", + "turnOff|Light|*|*|*|Study|Ground Floor", + "Use Study Lamp.", + ), + ( + "Turn off a light, do not use Living Lamp, just use Study Lamp.", + "turnOff|Light|*|*|*|Study|Ground Floor", + "Use Study Lamp.", + ), + ( + "Turn off a light, do not use Living Lamp, I choose Study Lamp.", + "turnOff|Light|*|*|*|Study|Ground Floor", + "Use Study Lamp.", + ), + ( + "Turn off a light, do not use Living Lamp, Study Lamp instead.", + "turnOff|Light|*|*|*|Study|Ground Floor", + "Use Study Lamp.", + ), + ( + "Turn off not Living Lamp: use Study Lamp.", + "turnOff|Light|*|*|*|Study|Ground Floor", + "Use Study Lamp.", + ), + ( + "Turn off not Living Lamp—use Study Lamp.", + "turnOff|Light|*|*|*|Study|Ground Floor", + "Use Study Lamp.", + ), + ( + "Do not use Living Lamp, increase Study Lamp brightness by 10 percent.", + "adjustUp|Light|brightness|10|Percent|Study|Ground Floor", + "Increase Study Lamp brightness by 10 percent.", + ), + ( + "Do not use Living Lamp, raise Study Lamp brightness by 10 percent.", + "adjustUp|Light|brightness|10|Percent|Study|Ground Floor", + "Raise Study Lamp brightness by 10 percent.", + ), + ( + "Do not use Living Lamp, lower Study Lamp brightness by 10 percent.", + "adjustDown|Light|brightness|10|Percent|Study|Ground Floor", + "Lower Study Lamp brightness by 10 percent.", + ), + ( + "Do not use Living Lamp, change Study Lamp brightness to 50 percent.", + "set|Light|brightness|50|Percent|Study|Ground Floor", + "Change Study Lamp brightness to 50 percent.", + ), + ) + for utterance, raw_output, clarification_answer in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw_output, registry) + self.assertEqual(grounded.negated_entity_ids, ("light.living",)) + self.assertNotIn("light.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "selector-correction-nonce", + ) + confirmed = parse_domux_output(raw_output)[0] + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer=clarification_answer, + confirmed_instruction=confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "light.study", + ) + + def test_postposed_no_corrections_withdraw_old_selector(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + EntitySpec( + "light.kitchen", "light", "Light", "Kitchen", "Ground Floor", + aliases=("Kitchen Lamp",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed_living = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Living Room", "Ground Floor" + ) + confirmed_study = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + for utterance in ( + "Turn off Study Lamp—no, Living Lamp instead.", + "Turn off Study Lamp, no, Living Lamp instead.", + "Turn off Study Lamp. No, Living Lamp instead.", + "Turn off Study Lamp—no, I mean Living Lamp.", + "Turn off Study Lamp; no, choose Living Lamp instead.", + "Turn off Study Lamp? No, use the Living Lamp instead.", + "Turn off Study Lamp - no, Living Lamp instead.", + "Turn off Study Lamp–no, Living Lamp instead.", + "Turn off Study Lamp—no: Living Lamp instead.", + "Turn off Study Lamp—no. Living Lamp instead.", + "Turn off Study Lamp—no! Living Lamp instead.", + "Turn off Study Lamp—no? Living Lamp instead.", + "Turn off Study Lamp—no; Living Lamp instead.", + "Turn off Study Lamp—no。 Living Lamp instead.", + "Turn off Study Lamp—no！ Living Lamp instead.", + "Turn off Study Lamp—no？ Living Lamp instead.", + "Turn off Study Lamp—no； Living Lamp instead.", + "Turn off Study Lamp，no, Living Lamp instead.", + "Turn off Study Lamp—no, turn off Living Lamp instead.", + "Turn off Study Lamp—no, please turn off the Living Lamp instead.", + "Turn off Study Lamp—no, switch off Living Lamp instead.", + ): + with self.subTest(positive_replacement=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Living Room|Ground Floor", + registry, + ) + self.assertEqual(grounded.negated_entity_ids, ("light.study",)) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.living",), + ) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer="Use Study Lamp.", + confirmed_instruction=confirmed_study, + registry=registry, + ) + + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "postposed-correction-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Living Lamp.", + confirmed_instruction=confirmed_living, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "light.living", + ) + + value_correction = ground_domux_request( + "Set Study Lamp to 30 percent—no, perhaps 60 percent; confirm the value.", + "set|Light|brightness|60|Percent|Study|Ground Floor", + registry, + ) + self.assertEqual(value_correction.negated_entity_ids, ()) + self.assertEqual( + tuple(entity.entity_id for entity in value_correction.candidates), + ("light.study",), + ) + negative_value_correction = ground_domux_request( + "Set Study Lamp to 30 percent—no, not 30, 60 percent instead.", + "set|Light|brightness|60|Percent|Study|Ground Floor", + registry, + ) + self.assertNotIn( + "negative_or_cancelled_intent", + negative_value_correction.clarification.reasons, + ) + self.assertNotIn("light.study", negative_value_correction.negated_entity_ids) + + chained = ground_domux_request( + "Turn off Study Lamp—no, Living Lamp, no, Kitchen Lamp instead.", + "turnOff|Light|*|*|*|Kitchen|Ground Floor", + registry, + ) + self.assertEqual( + chained.negated_entity_ids, + ("light.living", "light.study"), + ) + self.assertEqual( + tuple(entity.entity_id for entity in chained.candidates), + ("light.kitchen",), + ) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + chained, + answer="Use Living Lamp.", + confirmed_instruction=confirmed_living, + registry=registry, + ) + confirmed_kitchen = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Kitchen", "Ground Floor" + ) + chained_adapter = InMemoryHAAdapter(states) + chained_store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "chained-correction-nonce", + ) + chained_action = chained_store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=chained, + registry=registry, + adapter=chained_adapter, + clarification_answer="Use Kitchen Lamp.", + confirmed_instruction=confirmed_kitchen, + ) + chained_result = chained_store.commit( + chained_action.confirmation(), + registry=registry, + adapter=chained_adapter, + ) + self.assertEqual( + (chained_result.status, len(chained_adapter.sut_calls)), + ("COMMITTED", 1), + ) + self.assertEqual( + chained_adapter.sut_calls[0]["data"]["entity_id"], + "light.kitchen", + ) + + corrected_back = ground_domux_request( + "Turn off Study Lamp—no, Living Lamp, no, Study Lamp instead.", + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertEqual(corrected_back.negated_entity_ids, ("light.living",)) + self.assertEqual( + tuple(entity.entity_id for entity in corrected_back.candidates), + ("light.study",), + ) + corrected_back_adapter = InMemoryHAAdapter(states) + corrected_back_store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "corrected-back-nonce", + ) + corrected_back_action = corrected_back_store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=corrected_back, + registry=registry, + adapter=corrected_back_adapter, + clarification_answer="Use Study Lamp.", + confirmed_instruction=confirmed_study, + ) + corrected_back_result = corrected_back_store.commit( + corrected_back_action.confirmation(), + registry=registry, + adapter=corrected_back_adapter, + ) + self.assertEqual( + (corrected_back_result.status, len(corrected_back_adapter.sut_calls)), + ("COMMITTED", 1), + ) + self.assertEqual( + corrected_back_adapter.sut_calls[0]["data"]["entity_id"], + "light.study", + ) + + for utterance in ( + "Turn off Study Lamp, no, wait, Living Lamp.", + "Turn off Study Lamp—no, not Living Lamp instead.", + "Do not use Study Lamp—no, Living Lamp instead.", + ): + with self.subTest(non_positive_replacement=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Living Room|Ground Floor", + registry, + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "blocked-postposed-correction-nonce", + ) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Use Living Lamp.", + confirmed_instruction=confirmed_living, + ) + self.assertEqual(adapter.sut_calls, []) + + for utterance in ( + "Turn off Study Lamp—no, not Living Lamp instead.", + "Turn off Study Lamp—no, please don't use Living Lamp instead.", + "Turn off Study Lamp—no, please do not use Living Lamp instead.", + "Turn off Study Lamp—no, avoid Living Lamp instead.", + "Turn off Study Lamp—no, anything but Living Lamp instead.", + "Turn off Study Lamp—no, wait, I mean Living Lamp instead.", + ): + with self.subTest(block_candidate_expansion=utterance): + unsafe_replacement = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Kitchen|Ground Floor", + registry, + ) + self.assertIn( + "negative_or_cancelled_intent", + unsafe_replacement.clarification.reasons, + ) + expansion_adapter = InMemoryHAAdapter(states) + expansion_store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + expansion_store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=unsafe_replacement, + registry=registry, + adapter=expansion_adapter, + clarification_answer="Use Kitchen Lamp.", + confirmed_instruction=confirmed_kitchen, + ) + self.assertEqual(expansion_adapter.sut_calls, []) + + def test_generic_domain_restart_requires_positive_domain_evidence(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.study", "light", "Light", "Study", "Ground Floor", + aliases=("Study Lamp",), + ), + EntitySpec( + "light.living", "light", "Light", "Living Room", "Ground Floor", + aliases=("Living Lamp",), + ), + EntitySpec( + "cover.study", "cover", "Curtain", "Study", "Ground Floor", + aliases=("Study Curtain",), + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on" if entity.domain == "light" else "open", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + } if entity.domain == "light" else { + "current_position": 100, + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + blocked_restarts = ( + ( + "Turn off a device; I don't need any light; turn off.", + "turnOff|Light|*|*|*|*|*", + ), + ( + "Turn off a device; I don't need any light; turn off a device.", + "turnOff|Light|*|*|*|*|*", + ), + ( + "Turn off a device; I don't need any light; turn off Study.", + "turnOff|Light|*|*|*|Study|Ground Floor", + ), + *( + ( + f"Turn off a device; I don't need any light; " + f"turn off a device {preposition} light.", + "turnOff|Light|*|*|*|Study|Ground Floor", + ) + for preposition in ( + "about", "around", "at", "by", "for", "from", "in", "on", "to" + ) + ), + ) + for utterance, raw_output in blocked_restarts: + with self.subTest(blocked_restart=utterance): + grounded = ground_domux_request(utterance, raw_output, registry) + self.assertEqual( + grounded.negated_entity_ids, + ("light.living", "light.study"), + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study light on Ground Floor.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + universal_blocked_restarts = ( + "Turn off a device; I don't need any device; turn off.", + "Turn off a device; I don't want any device; turn off a device.", + "Turn off a device; do not use any device; turn off.", + "Turn off a device; I don't need anything; turn off.", + ) + for utterance in universal_blocked_restarts: + with self.subTest(universal_blocked_restart=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual( + grounded.negated_entity_ids, + ("cover.study", "light.living", "light.study"), + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study light on Ground Floor.", + confirmed_instruction=confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + for utterance in ( + "Turn off a device; I don't need any light; turn off Study Lamp.", + "Turn off a device; I don't need any light; turn off a light.", + ): + with self.subTest(domain_reauthorized=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertEqual(grounded.negated_entity_ids, ()) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "domain-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study light on Ground Floor.", + confirmed_instruction=confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "light.study", + ) + + for utterance in ( + "I don't need any device; turn off Study Lamp.", + "I don't need anything; turn off Study Lamp.", + ): + with self.subTest(universal_reauthorized=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertNotIn("light.study", grounded.negated_entity_ids) + self.assertIn("cover.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "universal-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off Study light on Ground Floor.", + confirmed_instruction=confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + + set_confirmed = DomuxInstruction( + "set", "Light", "brightness", "25", "Percent", "Study", "Ground Floor" + ) + attribute_first_restarts = ( + "I don't want any light; set the brightness of Study light to 25 percent.", + "I don't want any light; change the brightness on Study light to 25 percent.", + ( + "I don't want any light; could you set the brightness of " + "Study light to 25 percent?" + ), + "I don't want any light; set brightness to 25 percent on Study light.", + ) + for utterance in attribute_first_restarts: + with self.subTest(attribute_first_restart=utterance): + grounded = ground_domux_request( + utterance, + set_confirmed.to_pipe(), + registry, + ) + self.assertNotIn("light.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "attribute-first-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer=( + "Set Study light brightness to 25 percent on Ground Floor." + ), + confirmed_instruction=set_confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + + def test_generic_device_aliases_share_one_exclusion_and_restart_policy(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Lamp", "Study", "Ground Floor"), + EntitySpec("cover.study", "cover", "Blind", "Study", "Ground Floor"), + EntitySpec( + "climate.study", "climate", "Air Conditioner", "Study", "Ground Floor" + ), + )) + states = { + "light.study": { + "entity_id": "light.study", + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + }, + "cover.study": { + "entity_id": "cover.study", + "state": "open", + "attributes": {"current_position": 100, "supported_features": 7}, + }, + "climate.study": { + "entity_id": "climate.study", + "state": "cool", + "attributes": { + "temperature": 24.0, + "hvac_modes": ["off", "cool", "heat"], + "supported_features": 1, + "temperature_unit": "°C", + "min_temp": 16.0, + "max_temp": 30.0, + "target_temp_step": 0.5, + }, + }, + } + climate_confirmed = DomuxInstruction( + "turnOff", "Air Conditioner", "*", "*", "*", "Study", "Ground Floor" + ) + climate_exclusions = ( + "Turn off a device; do not use any air conditioning; turn off a device.", + "Turn off a device; avoid any air conditioning; turn off a device.", + "Turn off a device; without any air conditioning; turn off a device.", + "Turn off a device; do not use any a c; turn off a device.", + "Turn off a device; avoid any A/C; turn off a device.", + ) + for utterance in climate_exclusions: + with self.subTest(climate_exclusion=utterance): + grounded = ground_domux_request( + utterance, + climate_confirmed.to_pipe(), + registry, + ) + self.assertIn("climate.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer="Turn off the Study Air Conditioner.", + confirmed_instruction=climate_confirmed, + ) + self.assertEqual(adapter.sut_calls, []) + + positive_restarts = ( + ( + "I don't want any lamp; turn off Study lamp.", + DomuxInstruction( + "turnOff", "Lamp", "*", "*", "*", "Study", "Ground Floor" + ), + "Turn off Study lamp.", + ), + ( + "I don't want any blind; close Study blind.", + DomuxInstruction( + "turnOff", "Blind", "*", "*", "*", "Study", "Ground Floor" + ), + "Close Study blind.", + ), + ( + "I don't want any air conditioner; turn off Study air conditioner.", + climate_confirmed, + "Turn off Study air conditioner.", + ), + ) + for utterance, confirmed, clarification_answer in positive_restarts: + with self.subTest(generic_device_positive_restart=utterance): + grounded = ground_domux_request( + utterance, + confirmed.to_pipe(), + registry, + ) + self.assertEqual(grounded.negated_entity_ids, ()) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "generic-device-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer=clarification_answer, + confirmed_instruction=confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + + def test_domain_aliases_inside_room_names_do_not_reauthorize_that_domain(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.ac_room", "light", "Light", "AC Room", "Ground Floor"), + EntitySpec("climate.study", "climate", "AC", "Study", "Ground Floor"), + EntitySpec( + "cover.light_room", "cover", "Curtain", "Light Room", "Ground Floor" + ), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + climate_excluded = ground_domux_request( + "I don't want any AC; turn off the light in AC Room.", + "turnOff|Light|*|*|*|AC Room|Ground Floor", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in climate_excluded.candidates), + ("light.ac_room",), + ) + self.assertIn("climate.study", climate_excluded.negated_entity_ids) + self.assertNotIn("light.ac_room", climate_excluded.negated_entity_ids) + + light_excluded = ground_domux_request( + "I don't want any light; close the curtain in Light Room.", + "turnOff|Curtain|*|*|*|Light Room|Ground Floor", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in light_excluded.candidates), + ("cover.light_room",), + ) + self.assertIn("light.study", light_excluded.negated_entity_ids) + self.assertNotIn("cover.light_room", light_excluded.negated_entity_ids) + + def test_domain_alias_inside_room_preserves_the_explicit_room_constraint(self) -> None: + registry = EntityRegistry(( + EntitySpec("climate.ac_room", "climate", "AC", "AC Room", "Ground Floor"), + EntitySpec("climate.study", "climate", "AC", "Study", "Ground Floor"), + EntitySpec( + "light.light_room", "light", "Light", "Light Room", "Ground Floor" + ), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + states = { + "climate.ac_room": { + "entity_id": "climate.ac_room", + "state": "cool", + "attributes": { + "temperature": 24.0, + "hvac_modes": ["off", "cool", "heat"], + "supported_features": 1, + "temperature_unit": "°C", + "min_temp": 16.0, + "max_temp": 30.0, + "target_temp_step": 0.5, + }, + }, + "climate.study": { + "entity_id": "climate.study", + "state": "cool", + "attributes": { + "temperature": 24.0, + "hvac_modes": ["off", "cool", "heat"], + "supported_features": 1, + "temperature_unit": "°C", + "min_temp": 16.0, + "max_temp": 30.0, + "target_temp_step": 0.5, + }, + }, + "light.light_room": { + "entity_id": "light.light_room", + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + }, + "light.study": { + "entity_id": "light.study", + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + }, + } + cases = ( + ( + "I don't want any AC; turn off the AC in AC Room.", + DomuxInstruction( + "turnOff", "AC", "*", "*", "*", "AC Room", "Ground Floor" + ), + "climate.ac_room", + ), + ( + "I don't want any light; turn off the light in Light Room.", + DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Light Room", "Ground Floor" + ), + "light.light_room", + ), + ) + for utterance, confirmed, expected_entity_id in cases: + with self.subTest(spatial_domain_alias=utterance): + grounded = ground_domux_request( + utterance, + confirmed.to_pipe(), + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + (expected_entity_id,), + ) + self.assertNotIn(expected_entity_id, grounded.negated_entity_ids) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "spatial-domain-alias-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer=utterance.split("; ", 1)[1], + confirmed_instruction=confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + expected_entity_id, + ) + + def test_shared_device_labels_preserve_qualified_negative_scope(self) -> None: + registry = EntityRegistry(( + EntitySpec( + "light.living", "light", "Ceiling Light", "Living Room", "Ground Floor" + ), + EntitySpec( + "light.study", "light", "Ceiling Light", "Study", "Ground Floor" + ), + )) + states = { + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in registry.entities + } + confirmed = DomuxInstruction( + "turnOff", "Ceiling Light", "*", "*", "*", "Study", "Ground Floor" + ) + utterances = ( + "Do not use the Living Room Ceiling Light; turn off the Study Ceiling Light.", + "Do not use the Living Room Ceiling Light. Turn off the Study Ceiling Light.", + "Do not turn off the Living Room Ceiling Light; turn off the Study Ceiling Light.", + "Avoid the Living Room Ceiling Light; turn off the Study Ceiling Light.", + "Do not use the light in Living Room; turn off the Study Ceiling Light.", + ) + for utterance in utterances: + with self.subTest(qualified_exclusion=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertEqual(grounded.negated_entity_ids, ("light.living",)) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.study",), + ) + adapter = InMemoryHAAdapter(states) + store = PreparedActionStore( + clock=MutableClock(), + nonce_factory=lambda: "shared-device-restart-nonce", + ) + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=registry, + adapter=adapter, + clarification_answer=( + "Turn off the Study Ceiling Light on Ground Floor." + ), + confirmed_instruction=confirmed, + ) + result = store.commit( + action.confirmation(), + registry=registry, + adapter=adapter, + ) + self.assertEqual((result.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + self.assertEqual( + adapter.sut_calls[0]["data"]["entity_id"], + "light.study", + ) + + def test_target_repair_must_answer_the_unresolved_selector_slot(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("cover.kitchen", "cover", "Curtain", "Kitchen", "Ground Floor"), + )) + grounded = ground_domux_request( + "Turn off the Study or Kitchen light.", + "turnOff|Light|*|*|*|Study|Ground Floor", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("light.study",), + ) + self.assertIn("room", grounded.clarification.unresolved_slots) + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + with self.assertRaisesRegex(GroundingError, "repaired target"): + resolve_clarification_submission( + grounded, + answer="The Ground Floor.", + confirmed_instruction=confirmed, + registry=registry, + ) + resolved = resolve_clarification_submission( + grounded, + answer="The Study light.", + confirmed_instruction=confirmed, + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, "light.study") + + def test_clarification_selector_slots_cannot_form_a_zero_match_target(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("climate.kitchen", "climate", "AC", "Kitchen", "First Floor"), + )) + grounded = ground_domux_request( + "Turn off the device.", + "turnOff|*|*|*|*|*|*", + registry, + ) + with self.assertRaisesRegex(GroundingError, "inconsistent target selectors"): + resolve_clarification_submission( + grounded, + answer="Turn off the Study AC.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ), + registry=registry, + ) + + def test_operation_only_answer_is_allowed_when_original_target_is_fully_bound(self) -> None: + registry = EntityRegistry(( + EntitySpec("climate.lab", "climate", "AC", "Test Lab", "Ground Floor"), + EntitySpec("light.utility", "light", "Utility Light", "Utility Room", "Basement"), + )) + grounded = ground_domux_request( + "Set the Test Lab AC.", + "set|AC|temperature|22|Celsius|Test Lab|Ground Floor", + registry, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("climate.lab",), + ) + self.assertTrue(grounded.clarification.required) + resolved = resolve_clarification_submission( + grounded, + answer="Temperature 22 Celsius.", + confirmed_instruction=DomuxInstruction( + "set", "AC", "temperature", "22", "Celsius", "Test Lab", "Ground Floor" + ), + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, "climate.lab") + + def test_unknown_attribute_position_and_relational_context_require_target_repair(self) -> None: + registry = EntityRegistry(( + EntitySpec("climate.dining", "climate", "AC", "Dining Room", "Ground Floor"), + EntitySpec("climate.lounge", "climate", "AC", "Lounge", "Ground Floor"), + )) + grounded = ground_domux_request( + "Set the downstairs temperature to 24.", + "set|AC|temperature|24|Celsius|Dining Room|Ground Floor", + registry, + ) + self.assertNotIn("unsupported_request_grammar", grounded.clarification.reasons) + self.assertIn("device", grounded.clarification.unresolved_slots) + resolved = resolve_clarification_submission( + grounded, + answer="Set the Dining Room AC on the Ground Floor to 24 Celsius.", + confirmed_instruction=DomuxInstruction( + "set", "AC", "temperature", "24", "Celsius", "Dining Room", "Ground Floor" + ), + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, "climate.dining") + + singleton = EntityRegistry(( + EntitySpec("climate.study", "climate", "AC", "Study", "Ground Floor"), + )) + modified_attribute = ground_domux_request( + "Set the minimum temperature to 20 degrees.", + "set|AC|temperature|20|Celsius|*|*", + singleton, + ) + self.assertIn("attribute", modified_attribute.clarification.unresolved_slots) + with self.assertRaisesRegex(GroundingError, "operation modifier"): + resolve_clarification_submission( + modified_attribute, + answer="The Study AC on the Ground Floor.", + confirmed_instruction=DomuxInstruction( + "set", "AC", "temperature", "20", "Celsius", "Study", "Ground Floor" + ), + registry=singleton, + ) + clarified_attribute = resolve_clarification_submission( + modified_attribute, + answer="Set the Study AC on the Ground Floor temperature to 20 Celsius.", + confirmed_instruction=DomuxInstruction( + "set", "AC", "temperature", "20", "Celsius", "Study", "Ground Floor" + ), + registry=singleton, + ) + self.assertEqual(clarified_attribute.chosen.entity_id, "climate.study") + + lights = EntityRegistry(( + EntitySpec("light.hall", "light", "Light", "Hall", "Ground Floor"), + EntitySpec("light.kitchen", "light", "Light", "Kitchen", "Ground Floor"), + EntitySpec("light.bedroom", "light", "Light", "Bedroom", "First Floor"), + )) + middle = ground_domux_request( + "Turn the one in the middle off.", + "turnOff|Light|*|*|*|Bedroom|First Floor", + lights, + SessionContext(("light.hall", "light.kitchen", "light.bedroom")), + ) + self.assertNotIn("unsupported_request_grammar", middle.clarification.reasons) + self.assertIn("context", middle.clarification.unresolved_slots) + self.assertEqual( + resolve_clarification_submission( + middle, + answer="I mean the Bedroom light on the First Floor.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Bedroom", "First Floor" + ), + registry=lights, + ).chosen.entity_id, + "light.bedroom", + ) + + def test_negative_and_other_selector_phrases_never_direct_execute(self) -> None: + singleton = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + for direction, action in (("increase", "adjustUp"), ("decrease", "adjustDown")): + with self.subTest(direction=direction): + grounded = ground_domux_request( + f"No need to {direction} the Study light brightness.", + f"{action}|Light|brightness|*|*|Study|*", + singleton, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("negative_or_cancelled_intent", grounded.clarification.reasons) + + climate_registry = EntityRegistry(( + EntitySpec("climate.study", "climate", "AC", "Study", "Ground Floor"), + )) + excluded_values = ( + ( + singleton, + "Set the Study light color to no Blue.", + "set|Light|color|Blue|*|Study|*", + "value:blue", + ), + ( + singleton, + "Set the Study light brightness to no 50 percent.", + "set|Light|brightness|50|Percent|Study|*", + "number:50", + ), + ( + climate_registry, + "Set the Study AC to no Cool mode.", + "set|AC|mode|Cool|*|Study|*", + "value:cool", + ), + ( + climate_registry, + "Set the Study AC fan speed to no High level.", + "set|AC|fan speed|High|Level|Study|*", + "value:high", + ), + ( + climate_registry, + "Set the Study AC temperature to no 20 degrees.", + "set|AC|temperature|20|Celsius|Study|*", + "number:20", + ), + ) + for value_registry, utterance, raw_output, token in excluded_values: + with self.subTest(excluded_value=utterance): + grounded = ground_domux_request( + utterance, + raw_output, + value_registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("value", grounded.clarification.unresolved_slots) + self.assertIn(token, grounded.excluded_operation_value_tokens) + + two_lights = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("light.living", "light", "Light", "Living Room", "Ground Floor"), + )) + excluded = ground_domux_request( + "Turn off a light, no Study light.", + "turnOff|Light|*|*|*|*|*", + two_lights, + ) + self.assertTrue(excluded.clarification.required) + self.assertIn("light.study", excluded.negated_entity_ids) + + quantified_command_exclusions = ( + "Anything but turn off Study light.", + "Do anything but turn off Study light.", + "Please do anything but turn off Study light.", + "Turn off a light; anything but turn off Study light.", + "Anything but could you please turn off Study light?", + ) + confirmed_study = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ) + for utterance in quantified_command_exclusions: + with self.subTest(quantified_command_exclusion=utterance): + grounded = ground_domux_request( + utterance, + confirmed_study.to_pipe(), + two_lights, + ) + self.assertIn("light.study", grounded.negated_entity_ids) + adapter = InMemoryHAAdapter({ + entity.entity_id: { + "entity_id": entity.entity_id, + "state": "on", + "attributes": { + "brightness": 128, + "supported_color_modes": ["brightness"], + }, + } + for entity in two_lights.entities + }) + store = PreparedActionStore(clock=MutableClock()) + with self.assertRaises(GroundingError): + store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=two_lights, + adapter=adapter, + clarification_answer="Turn off Study light.", + confirmed_instruction=confirmed_study, + ) + self.assertEqual(adapter.sut_calls, []) + + contrastive = ground_domux_request( + "Turn off not the Study light but the Living Room light.", + "turnOff|Light|*|*|*|Living Room|Ground Floor", + two_lights, + ) + self.assertEqual(contrastive.negated_entity_ids, ("light.study",)) + resolved_contrastive = resolve_clarification_submission( + contrastive, + answer="Not the Study light; use the Living Room light.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Living Room", "Ground Floor" + ), + registry=two_lights, + ) + self.assertEqual(resolved_contrastive.chosen.entity_id, "light.living") + + named_singleton = EntityRegistry(( + EntitySpec( + "light.ceiling", "light", "Ceiling Light", "Study", "Ground Floor" + ), + )) + other = ground_domux_request( + "Turn other Ceiling Light off.", + "turnOff|Ceiling Light|*|*|*|*|*", + named_singleton, + ) + self.assertTrue(other.clarification.required) + self.assertIn("other_reference_requires_selection", other.clarification.reasons) + + for device in ("Lights", "Lamps", "Devices", "ACs"): + with self.subTest(other_device=device): + plural_registry = EntityRegistry(( + EntitySpec( + f"light.{device.casefold()}", + "light", + device, + "Study", + "Ground Floor", + ), + )) + plural_other = ground_domux_request( + f"Turn other {device.casefold()} off.", + f"turnOff|{device}|*|*|*|*|*", + plural_registry, + ) + self.assertTrue(plural_other.clarification.required) + self.assertIn( + "other_reference_requires_selection", + plural_other.clarification.reasons, + ) + + for utterance in ( + "Turn off a light, but not the light—Study light.", + "Turn off a light, but not the light that you and I talked about in Study.", + "Turn off no light.", + ): + with self.subTest(unresolved_exclusion=utterance): + unresolved_exclusion = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + two_lights, + ) + self.assertTrue(unresolved_exclusion.clarification.required) + self.assertTrue( + { + "unsupported_request_grammar", + "negated_selector", + }.intersection(unresolved_exclusion.clarification.reasons) + ) + with self.assertRaisesRegex( + GroundingError, + "new immediate command|explicitly excluded entity|does not select a candidate", + ): + resolve_clarification_submission( + unresolved_exclusion, + answer="Yes.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ), + registry=two_lights, + ) + + def test_unanchored_inventory_words_never_fall_back_to_a_global_singleton(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + EntitySpec("cover.kitchen", "cover", "Curtain", "Kitchen", "Ground Floor"), + )) + utterances = ( + "Turn off the light for Kitchen.", + "For Kitchen, turn off the light.", + "Turn off the light of Kitchen.", + "Turn off the light by Kitchen.", + "Turn off the light about Kitchen.", + "Turn off the light around Kitchen.", + ) + for utterance in utterances: + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("room", grounded.clarification.unresolved_slots) + + def test_command_vocabulary_metadata_collisions_require_explicit_selector_syntax(self) -> None: + collision_registry = EntityRegistry(( + EntitySpec( + "light.brightness", + "light", + "Brightness", + "The", + "First", + ("Mode",), + ), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + cases = ( + ( + "Set the light brightness to 50 percent.", + "set|Light|brightness|50|Percent|*|*", + ), + ("Turn the light off.", "turnOff|Light|*|*|*|*|*"), + ("First, turn the light off.", "turnOff|Light|*|*|*|*|*"), + ) + for utterance, raw_output in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + raw_output, + collision_registry, + ) + self.assertTrue(grounded.clarification.required) + + explicit = ground_domux_request( + "Set the device named Brightness in the room named The to 50 percent brightness.", + "set|Brightness|brightness|50|Percent|The|*", + collision_registry, + ) + self.assertTrue(explicit.clarification.required) + self.assertEqual( + tuple(entity.entity_id for entity in explicit.candidates), + ("light.brightness",), + ) + + affirmative_aliases = EntityRegistry(( + EntitySpec( + "light.living", + "light", + "Light", + "Living Room", + "Ground Floor", + ("Yes", "Proceed"), + ), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + ambiguous = ground_domux_request( + "Turn off the light.", + "turnOff|Light|*|*|*|*|*", + affirmative_aliases, + ) + self.assertEqual(len(ambiguous.candidates), 2) + for answer in ("Yes.", "Proceed."): + with self.subTest(answer=answer), self.assertRaisesRegex( + GroundingError, "does not select" + ): + resolve_clarification_submission( + ambiguous, + answer=answer, + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Living Room", "Ground Floor" + ), + registry=affirmative_aliases, + ) + + ordinal_registry = EntityRegistry(( + EntitySpec("light.first", "light", "Light", "First", "Ground Floor"), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + for utterance in ( + "First, turn the light off.", + "First: turn the light off.", + "Turn the first light off.", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|*|*", + ordinal_registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("room", grounded.clarification.unresolved_slots) + + def test_questions_wait_prefixes_and_action_alternatives_never_execute(self) -> None: + raw = "turnOff|Light|*|*|*|Study|Ground Floor" + for utterance in ( + "Do you want to turn off the Study light?", + "Do I turn off the Study light?", + "Did you turn off the Study light?", + "Do we turn off the Study light?", + "Did we turn off the Study light?", + "Can we turn off the Study light?", + "Could we turn off the Study light?", + "Would we turn off the Study light?", + "Should we turn off the Study light?", + "Do you need to turn off the Study light?", + "Do you have to turn off the Study light?", + "Would you want to turn off the Study light?", + "Can you mean to turn off the Study light?", + "We did turn off the Study light.", + "I did turn off the Study light.", + "You did turn off the Study light.", + "We do turn off the Study light.", + "I do turn off the Study light.", + "You do turn off the Study light.", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertIn("informational_request", grounded.clarification.reasons) + + # These conventional second-person modal forms remain supported as + # polite imperatives; adding a meta verb above changes them to a query. + for utterance in ( + "Can you turn off the Study light?", + "Could you turn off the Study light?", + "Would you turn off the Study light?", + ): + with self.subTest(polite_imperative=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertNotIn("informational_request", grounded.clarification.reasons) + + waited = ground_domux_request( + "Wait, then turn off the Study light.", + raw, + self.registry, + ) + self.assertIn("unsupported_condition_or_time", waited.clarification.reasons) + + for utterance in ( + "From now on, turn the Study light off.", + "Now and then, turn the Study light off.", + ): + with self.subTest(persistent_or_periodic=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertIn( + "unsupported_condition_or_time", + grounded.clarification.reasons, + ) + + alternative = ground_domux_request( + "Turn the Study light on or off; confirm.", + raw, + self.registry, + ) + self.assertIn("action", alternative.clarification.unresolved_slots) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + alternative, + answer="Yes.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Light", "*", "*", "*", "Study", "Ground Floor" + ), + registry=self.registry, + ) + + def test_model_selector_slots_cannot_launder_conditions_into_the_grammar(self) -> None: + modifiers = ( + "subject to nobody being home", + "depending on whether anyone is home", + "so long as the room is empty", + "during dinner", + "upon arrival", + "at dusk", + "momentarily", + "as needed", + ) + for modifier in modifiers: + utterance = f"Turn the Study light off {modifier}." + for slot in ("device", "room", "floor"): + fields = ["turnOff", "Light", "*", "*", "*", "Study", "Ground Floor"] + fields[{"device": 1, "room": 5, "floor": 6}[slot]] = modifier + with self.subTest(modifier=modifier, slot=slot): + grounded = ground_domux_request(utterance, "|".join(fields), self.registry) + self.assertTrue(grounded.clarification.required) + self.assertTrue({ + "unsupported_condition_or_time", "unsupported_request_grammar", + }.intersection(grounded.clarification.reasons)) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer="The Study light on the Ground Floor.", + confirmed_instruction=DomuxInstruction( + "turnOff", "Ceiling Light", "*", "*", "*", + "Study", "Ground Floor", + ), + registry=self.registry, + ) + + def test_selector_slot_text_does_not_change_grammar_support(self) -> None: + utterance = "Turn the Study light off subject to nobody being home." + raw_outputs = ( + "turnOff|Light|*|*|*|Study|Ground Floor", + "turnOff|subject to nobody being home|*|*|*|Study|Ground Floor", + "turnOff|Light|*|*|*|subject to nobody being home|Ground Floor", + "turnOff|Light|*|*|*|Study|subject to nobody being home", + ) + reasons = [] + for raw_output in raw_outputs: + grounded = ground_domux_request(utterance, raw_output, self.registry) + reasons.append("unsupported_request_grammar" in grounded.clarification.reasons) + self.assertEqual(reasons, [True, True, True, True]) + + def test_unique_resolution_rechecks_non_wildcard_room_and_floor(self) -> None: + grounded = ground_domux_request( + "Turn off the Study ceiling light on the Ground Floor.", + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + self.registry, + ) + self.assertFalse(grounded.clarification.required) + for slot, wrong in (("room", "Kitchen"), ("floor", "First Floor")): + source = replace(grounded.source_instructions[0], **{slot: wrong}) + tampered = replace(grounded, source_instructions=(source,)) + with self.subTest(slot=slot), self.assertRaisesRegex( + GroundingError, f"unique request {slot}" + ): + resolve_unique_request(tampered, self.registry) + + def test_target_only_answer_cannot_repair_a_model_missed_operation(self) -> None: + context = SessionContext(("cover.study_curtain",)) + grounded = ground_domux_request( + "Open that one to 60 percent.", + "set|Light|brightness|60|Percent|*|*", + self.registry, + context, + ) + self.assertEqual( + tuple(entity.entity_id for entity in grounded.candidates), + ("cover.study_curtain",), + ) + with self.assertRaisesRegex(GroundingError, "answer-supported patch"): + resolve_clarification_submission( + grounded, + answer="The Study curtain on the Ground Floor.", + confirmed_instruction=DomuxInstruction( + "set", "Curtain", "position", "60", "Percent", "Study", "Ground Floor" + ), + registry=self.registry, + ) + + def test_zero_and_ambiguous_answers_fail(self) -> None: + candidates = self.registry.candidates(parse_domux_output("turnOff|Ceiling Light|*|*|*|*|*")[0]) + for answer in ("", "Kitchen", "Ceiling Light", "9"): + with self.subTest(answer=answer), self.assertRaises(GroundingError): + resolve_clarification(answer, candidates) + + def test_plan_mappings_cover_all_three_domains(self) -> None: + adapter = InMemoryHAAdapter(self.states) + cases = ( + ("turnOff|Ceiling Light|*|*|*|Study|Ground Floor", "light.study_ceiling", "turn_off"), + ("set|Curtain|position|30|Percent|Study|Ground Floor", "cover.study_curtain", "set_cover_position"), + ("set|AC|temperature|23|Celsius|Study|Ground Floor", "climate.study_ac", "set_temperature"), + ) + for raw, entity_id, service in cases: + entity = self.registry.get(entity_id) + plan = build_plan(parse_domux_output(raw)[0], entity, adapter.get_state(entity_id)) + self.assertEqual(plan.service, service) + + def test_light_brightness_zero_uses_home_assistant_turn_off_semantics(self) -> None: + entity = self.registry.get("light.study_ceiling") + adapter = InMemoryHAAdapter(self.states) + cases = ( + "set|Light|brightness|0|Percent|Study|Ground Floor", + "adjustDown|Light|brightness|60|Percent|Study|Ground Floor", + ) + for raw in cases: + with self.subTest(raw=raw): + before = adapter.get_state(entity.entity_id) + before["attributes"]["brightness"] = 153 + before["state"] = "on" + adapter.set_state_for_setup(entity.entity_id, before) + plan = build_plan(parse_domux_output(raw)[0], entity, before) + self.assertEqual(plan.service, "turn_on") + self.assertEqual(plan.service_data["brightness_pct"], 0) + self.assertEqual(plan.expected_projection["state"], "off") + self.assertIsNone(plan.expected_projection["brightness"]) + result = adapter.call_service(plan.domain, plan.service, plan.service_data) + self.assertEqual(controlled_projection(result.after, "light")["state"], "off") + self.assertTrue(projection_matches( + controlled_projection(result.after, "light"), + plan.expected_projection, + )) + + positive = build_plan( + parse_domux_output( + "set|Light|brightness|1|Percent|Study|Ground Floor" + )[0], + entity, + adapter.get_state(entity.entity_id), + ) + self.assertEqual(positive.expected_projection["state"], "on") + self.assertEqual(positive.expected_projection["brightness"], 3) + + def test_home_assistant_integer_coercions_and_optional_cover_position_are_bound(self) -> None: + cover = self.registry.get("cover.study_curtain") + cover_state = json.loads(json.dumps(self.states[cover.entity_id])) + integral = build_plan( + parse_domux_output( + "set|Curtain|position|21|Percent|Study|Ground Floor" + )[0], + cover, + cover_state, + ) + self.assertEqual(integral.service_data["position"], 21) + self.assertIsInstance(integral.service_data["position"], int) + for raw in ( + "set|Curtain|position|20.9|Percent|Study|Ground Floor", + "adjustUp|Curtain|position|10.5|Percent|Study|Ground Floor", + ): + with self.subTest(raw=raw), self.assertRaisesRegex(GroundingError, "integer"): + build_plan(parse_domux_output(raw)[0], cover, cover_state) + + without_position = json.loads(json.dumps(cover_state)) + without_position["attributes"].pop("current_position") + for raw, expected_state in ( + ("turnOn|Curtain|*|*|*|Study|Ground Floor", "open"), + ("turnOff|Curtain|*|*|*|Study|Ground Floor", "closed"), + ): + with self.subTest(raw=raw): + plan = build_plan(parse_domux_output(raw)[0], cover, without_position) + self.assertEqual(plan.expected_projection, { + "entity_id": cover.entity_id, + "state": expected_state, + }) + with self.assertRaisesRegex(GroundingError, "observed current_position"): + build_plan( + parse_domux_output( + "adjustUp|Curtain|position|10|Percent|Study|Ground Floor" + )[0], + cover, + without_position, + ) + + light = self.registry.get("light.study_ceiling") + light_state = json.loads(json.dumps(self.states[light.entity_id])) + light_state["attributes"]["supported_color_modes"] = ["white"] + for raw in ( + "set|Light|brightness|25|Percent|Study|Ground Floor", + "adjustUp|Light|brightness|5|Percent|Study|Ground Floor", + ): + with self.subTest(raw=raw): + plan = build_plan(parse_domux_output(raw)[0], light, light_state) + self.assertEqual(plan.service, "turn_on") + + light_state["attributes"]["supported_color_modes"] = ["color_temp"] + with self.assertRaisesRegex(GroundingError, "integer Kelvin"): + build_plan( + parse_domux_output( + "set|Light|colorTemperature|3000.9|Kelvin|Study|Ground Floor" + )[0], + light, + light_state, + ) + + def test_climate_turn_off_prefers_advertised_off_mode_over_feature_gated_service(self) -> None: + entity = self.registry.get("climate.study_ac") + instruction = parse_domux_output( + "turnOff|AC|*|*|*|Study|Ground Floor" + )[0] + state = json.loads(json.dumps(self.states[entity.entity_id])) + self.assertEqual(state["attributes"]["supported_features"], 9) + plan = build_plan(instruction, entity, state) + self.assertEqual(plan.service, "set_hvac_mode") + self.assertEqual(plan.service_data["hvac_mode"], "off") + adapter = InMemoryHAAdapter({entity.entity_id: state}) + result = adapter.call_service(plan.domain, plan.service, plan.service_data) + self.assertTrue(projection_matches( + controlled_projection(result.after, "climate"), + plan.expected_projection, + )) + + state["attributes"]["hvac_modes"] = ["cool", "heat"] + state["attributes"]["supported_features"] = 128 + feature_plan = build_plan(instruction, entity, state) + self.assertEqual(feature_plan.service, "turn_off") + self.assertNotIn("hvac_mode", feature_plan.service_data) + + state["attributes"]["supported_features"] = 0 + with self.assertRaisesRegex(GroundingError, "turn-off support"): + build_plan(instruction, entity, state) + + def test_plan_rejects_noncanonical_units_and_unused_slots(self) -> None: + adapter = InMemoryHAAdapter(self.states) + entity = self.registry.get("light.study_ceiling") + invalid = ( + "set|Light|brightness|30|Kelvin|Study|Ground Floor", + "turnOff|Light|brightness|30|Percent|Study|Ground Floor", + ) + for raw in invalid: + with self.subTest(raw=raw), self.assertRaises(GroundingError): + build_plan(parse_domux_output(raw)[0], entity, adapter.get_state(entity.entity_id)) + + def test_climate_turn_on_requires_one_unambiguous_active_mode(self) -> None: + adapter = InMemoryHAAdapter(self.states) + off = adapter.get_state("climate.study_ac") + off["state"] = "off" + adapter.set_state_for_setup("climate.study_ac", off) + instruction = parse_domux_output("turnOn|AC|*|*|*|Study|Ground Floor")[0] + with self.assertRaisesRegex(GroundingError, "confirm a mode explicitly"): + build_plan(instruction, self.registry.get("climate.study_ac"), off) + off["attributes"]["hvac_modes"] = ["off", "heat"] + plan = build_plan(instruction, self.registry.get("climate.study_ac"), off) + self.assertEqual(plan.expected_projection["state"], "heat") + receipt = adapter.call_service(plan.domain, plan.service, plan.service_data) + self.assertEqual(receipt.after["state"], "heat") + + def test_climate_preserves_advertised_underscore_enums_and_temperature_step(self) -> None: + state = json.loads(json.dumps(self.states["climate.study_ac"])) + state["attributes"]["hvac_modes"] = ["off", "fan_only", "heat_cool"] + entity = self.registry.get("climate.study_ac") + fan_only = build_plan( + parse_domux_output("set|AC|mode|Fan|*|Study|Ground Floor")[0], entity, state, + ) + self.assertEqual(fan_only.service_data["hvac_mode"], "fan_only") + fan_speed = build_plan( + parse_domux_output("set|AC|fan speed|medium_high|Level|Study|Ground Floor")[0], + entity, + state, + ) + self.assertEqual(fan_speed.service_data["fan_mode"], "medium_high") + with self.assertRaisesRegex(GroundingError, "does not align"): + build_plan( + parse_domux_output("set|AC|temperature|23.3|Celsius|Study|Ground Floor")[0], + entity, + state, + ) + accepted = build_plan( + parse_domux_output("set|AC|temperature|23.5|Celsius|Study|Ground Floor")[0], + entity, + state, + ) + self.assertEqual(accepted.service_data["temperature"], 23.5) + with self.assertRaisesRegex(GroundingError, "does not align"): + build_plan( + parse_domux_output("adjustUp|AC|temperature|0.3|Celsius|Study|Ground Floor")[0], + entity, + state, + ) + adjusted = build_plan( + parse_domux_output("adjustUp|AC|temperature|0.5|Celsius|Study|Ground Floor")[0], + entity, + state, + ) + self.assertEqual(adjusted.service_data["temperature"], 24.5) + + def test_projection_ignores_volatile_home_assistant_fields(self) -> None: + raw = { + "entity_id": "light.study_ceiling", + "state": "on", + "attributes": {"brightness": 153, "friendly_name": "private"}, + "last_changed": "volatile", + "context": {"id": "volatile"}, + } + self.assertEqual( + controlled_projection(raw, "light"), + {"entity_id": "light.study_ceiling", "state": "on", "brightness": 153}, + ) + planned = planning_projection(self.states["light.study_ceiling"], "light") + self.assertIn("supported_color_modes", planned) + + def test_user_words_cannot_be_dropped_or_reversed_by_the_model(self) -> None: + cases = ( + ( + "Turn off the Living Room light on the Ground Floor.", + "turnOn|Light|*|*|*|Living Room|Ground Floor", + "action", + ), + ( + "Open the Study curtain to 20 percent.", + "turnOn|Curtain|*|*|*|Study|Ground Floor", + "value", + ), + ( + "Set the Study AC to 23 Celsius.", + "turnOn|AC|*|*|*|Study|Ground Floor", + "action", + ), + ) + for utterance, raw, missing in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn(missing, grounded.clarification.unresolved_slots) + + def test_negated_target_and_cancelled_answers_fail_closed(self) -> None: + grounded = ground_domux_request( + "Turn off the Living Room light, not the Study light.", + "turnOff|Light|*|*|*|Study|Ground Floor", + self.registry, + ) + confirmed = parse_domux_output( + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor" + )[0] + with self.assertRaisesRegex(GroundingError, "excluded"): + resolve_clarification_submission( + grounded, + answer="The Study light.", + confirmed_instruction=confirmed, + registry=self.registry, + ) + + uncertain = ground_domux_request( + "Maybe turn off the Study light.", + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + self.registry, + ) + for answer in ( + "no thanks", "actually no", "cancel that", "never mind please", "not Study", + "I do not know", "still not sure", "whatever", "maybe", "please ask me later", "banana", + ): + with self.subTest(answer=answer), self.assertRaises(GroundingError): + resolve_clarification_submission( + uncertain, + answer=answer, + confirmed_instruction=confirmed, + registry=self.registry, + ) + + def test_excluded_operation_values_require_a_positive_safe_replacement(self) -> None: + cases = ( + ( + "Set Study light to anything but Blue.", + "set|Light|color|Blue|*|Study|Ground Floor", + ), + ( + "Set Study light to something other than Blue.", + "set|Light|color|Blue|*|Study|Ground Floor", + ), + ( + "Don't use Blue for Study light.", + "set|Light|color|Blue|*|Study|Ground Floor", + ), + ( + "Set the Study AC to anything but Heat mode.", + "set|AC|mode|Heat|*|Study|Ground Floor", + ), + ( + "Set brightness to any value other than 20% for the Study light.", + "set|Light|brightness|20|Percent|Study|Ground Floor", + ), + ( + "Set color to avoid Blue for the Study light.", + "set|Light|color|Blue|*|Study|Ground Floor", + ), + ( + "Make the Study light any color besides Blue.", + "set|Light|color|Blue|*|Study|Ground Floor", + ), + ( + "Make the Study light any color apart from Blue.", + "set|Light|color|Blue|*|Study|Ground Floor", + ), + ) + for utterance, raw in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertIn("excluded_operation_value", grounded.clarification.reasons) + self.assertIn("value", grounded.clarification.unresolved_slots) + for answer in ("Yes.", "The Study light."): + with self.assertRaisesRegex(GroundingError, "excluded"): + resolve_clarification_submission( + grounded, + answer=answer, + confirmed_instruction=parse_domux_output(raw)[0], + registry=self.registry, + ) + + grounded = ground_domux_request(cases[0][0], cases[0][1], self.registry) + safe = parse_domux_output( + "set|Light|color|Red|*|Study|Ground Floor" + )[0] + resolved = resolve_clarification_submission( + grounded, + answer="Use Red instead for the Study light.", + confirmed_instruction=safe, + registry=self.registry, + ) + self.assertEqual(resolved.confirmed_instruction.value, "Red") + + def test_withdrawn_initial_requests_and_deferred_answers_cannot_execute(self) -> None: + raw = "turnOff|Ceiling Light|*|*|*|Study|Ground Floor" + for utterance in ( + "I don't want you to turn off the Study light.", + "No need to turn off the Study light.", + "Turn off the Study light, just kidding.", + "Turn off the Study light—actually, never mind.", + "Please refrain from turning off the Study light.", + "Turn off the Study light, scratch that.", + "Turn off the Study light, I changed my mind.", + "Turn off the Study light, hold on.", + "Dont turn off the Study light.", + "Don’t turn off the Study light.", + "Never ever turn off the Study light.", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertIn("negative_or_cancelled_intent", grounded.clarification.reasons) + with self.assertRaisesRegex(GroundingError, "negated"): + resolve_clarification_submission( + grounded, + answer="Yes.", + confirmed_instruction=parse_domux_output(raw)[0], + registry=self.registry, + ) + + ambiguous = ground_domux_request( + "Turn off the ceiling light.", + "turnOff|Ceiling Light|*|*|*|*|*", + self.registry, + ) + confirmed = parse_domux_output(raw)[0] + for answer in ( + "Study, leave it on.", + "Study, keep it on.", + "Study, not now.", + "Study, wait.", + "Study, please refrain from turning it off.", + "The Study one, but don't do it yet.", + "Study, forget it.", + "Study, I changed my mind.", + "Study, not anymore.", + "Study, skip it.", + "Study, don't.", + "Study, do not.", + "Study, abort it.", + "Study, disregard that.", + "Study, ignore that.", + "Study, leave it alone.", + "Study, no longer.", + "Study, on second thought no.", + "Study, postpone it.", + "Study, defer it.", + "Study, pause.", + "Study, not yet.", + "Study, don't touch it.", + "Study, I don't want that.", + "Study, no thanks.", + "Study, scratch that.", + "Study, nix that.", + "Study, banana.", + "Study, don't go ahead.", + "Study, rather not proceed.", + "Study, do not go ahead.", + "Study, I do not want to proceed.", + "Study, I don't want you to turn it off.", + "Study, not turn it off.", + "Study, I don't need you to turn it off.", + "Study, I withdraw permission to turn it off.", + "Study, I revoke authorization to turn it off.", + "Study, I refuse to let you turn it off.", + "Study, you must not turn it off.", + "Study, you may not turn it off.", + "Study, don't you turn it off.", + "Study, I forbid you to turn it off.", + "Study, under no circumstances turn it off.", + "Use Study. Turn off the light if nobody is home.", + "Use Study. Turn off the light at nine.", + "Use Study. Turn off the light provided nobody is home.", + "Use Study. Turn off the light, scratch that.", + "Use Study, do not confirm.", + "Use Study, don’t execute it.", + "Study, should I turn it off?", + "Study, can I turn it off?", + "Study, would it be safe to turn it off?", + "Study, is it okay to turn it off?", + "Study, do you recommend I turn it off?", + "Study, tell me whether to turn it off.", + "Study, why should I turn it off?", + "Study, are you going to turn it off?", + "Use any device except Study. Turn it off.", + "Not in Study; turn the light off.", + ): + with self.subTest(answer=answer), self.assertRaises(GroundingError): + resolve_clarification_submission( + ambiguous, + answer=answer, + confirmed_instruction=confirmed, + registry=self.registry, + ) + for answer in ("Study.", "The one in the Study.", "Study, turn it off now."): + with self.subTest(answer=answer): + self.assertEqual( + resolve_clarification_submission( + ambiguous, + answer=answer, + confirmed_instruction=confirmed, + registry=self.registry, + ).chosen.entity_id, + "light.study_ceiling", + ) + + def test_clarification_answer_cannot_introduce_exclusions_or_alternatives(self) -> None: + grounded = ground_domux_request( + "Change the Study Ceiling Light color.", + "set|Ceiling Light|color|*|*|Study|Ground Floor", + self.registry, + ) + blue = parse_domux_output( + "set|Ceiling Light|color|Blue|*|Study|Ground Floor" + )[0] + for answer in ( + "Anything besides Blue.", + "Anything apart from Blue.", + "Use either Red or Blue.", + ): + with self.subTest(answer=answer), self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer=answer, + confirmed_instruction=blue, + registry=self.registry, + ) + + def test_clarification_cannot_add_an_operation_that_the_plan_drops(self) -> None: + grounded = ground_domux_request( + "Turn off the Ceiling Light.", + "turnOff|Ceiling Light|*|*|*|*|*", + self.registry, + ) + confirmed = parse_domux_output( + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor" + )[0] + for answer in ("Study, make it Blue.", "Study, set brightness to 50 percent."): + with self.subTest(answer=answer), self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer=answer, + confirmed_instruction=confirmed, + registry=self.registry, + ) + + uncertain = ground_domux_request( + "Maybe turn off the Study light.", + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + self.registry, + ) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + uncertain, + answer="Make it Blue.", + confirmed_instruction=confirmed, + registry=self.registry, + ) + for answer in ( + "Study, maybe", "I do not know, Study?", "Study perhaps", + "Study, but I am not sure", "whatever, Study", + ): + with self.subTest(answer=answer), self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer=answer, + confirmed_instruction=confirmed, + registry=self.registry, + ) + + def test_named_room_is_not_misread_as_an_operational_color_or_mode(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.orange", "light", "Light", "Orange Room", "Ground Floor"), + EntitySpec("climate.heat_room", "climate", "AC", "Heat Room", "Ground Floor"), + )) + grounded = ground_domux_request( + "Turn off the Orange Room light.", + "turnOff|Light|*|*|*|Orange Room|*", + registry, + ) + self.assertFalse(grounded.clarification.required) + heat_room = ground_domux_request( + "Turn off the Heat Room AC.", + "turnOff|AC|*|*|*|Heat Room|*", + registry, + ) + self.assertFalse(heat_room.clarification.required) + named_registry = EntityRegistry(( + EntitySpec("light.blue", "light", "Light", "Blue Room", "Ground Floor"), + EntitySpec("light.study", "light", "Light", "Study", "Ground Floor"), + )) + named = ground_domux_request( + "Turn off the light.", "turnOff|Light|*|*|*|*|*", named_registry, + ) + resolved = resolve_clarification_submission( + named, + answer="Blue Room", + confirmed_instruction=parse_domux_output( + "turnOff|Light|*|*|*|Blue Room|Ground Floor" + )[0], + registry=named_registry, + ) + self.assertEqual(resolved.chosen.entity_id, "light.blue") + + def test_opposing_action_clauses_cannot_collapse_to_one_model_action(self) -> None: + for utterance in ( + "Turn on the Study light then switch that device off.", + "Turn on the Study light, then turn the Study light off.", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOn|Ceiling Light|*|*|*|Study|Ground Floor", + self.registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("action", grounded.clarification.unresolved_slots) + + def test_source_values_ranges_and_multi_operation_text_fail_closed(self) -> None: + cases = ( + ( + "Change the Study light brightness from 50 to 20 percent.", + "set|Light|brightness|50|Percent|Study|*", + "value", + ), + ( + "Set the Study light brightness between 20 and 50 percent.", + "set|Light|brightness|50|Percent|Study|*", + "value", + ), + ( + "Change the Study light color from Red to Blue.", + "set|Light|color|Red|*|Study|*", + "value", + ), + ( + "Use Heat then Cool mode on the Study AC.", + "set|AC|mode|Heat|*|Study|*", + "value", + ), + ( + "Open the Study curtain halfway, then close it.", + "set|Curtain|position|50|Percent|Study|*", + "action", + ), + ( + "Close the Study curtain, then open it to 20 percent.", + "set|Curtain|position|20|Percent|Study|*", + "action", + ), + ( + "Set the Study AC to Cool mode at 20.", + "set|AC|mode|Cool|*|Study|*", + "value", + ), + ) + for utterance, raw, slot in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn(slot, grounded.clarification.unresolved_slots) + + target = ground_domux_request( + "Change the Study light brightness from 50 to 20 percent.", + "set|Light|brightness|20|Percent|Study|*", + self.registry, + ) + color_target = ground_domux_request( + "Change the Study light color from Red to Blue.", + "set|Light|color|Blue|*|Study|*", + self.registry, + ) + self.assertFalse(target.clarification.required) + self.assertFalse(color_target.clarification.required) + + def test_explicit_temperature_unit_cannot_be_silently_changed(self) -> None: + for unit in ("Fahrenheit", "°F", "Kelvin", "K"): + with self.subTest(unit=unit): + grounded = ground_domux_request( + f"Set the Study AC temperature to 20 {unit}.", + "set|AC|temperature|20|Celsius|Study|*", + self.registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("unit", grounded.clarification.unresolved_slots) + + def test_conflicted_action_and_unit_can_converge_after_explicit_clarification(self) -> None: + action_grounded = ground_domux_request( + "Open the Study curtain halfway, then close it.", + "set|Curtain|position|50|Percent|Study|*", + self.registry, + ) + action_resolved = resolve_clarification_submission( + action_grounded, + answer="Close the Study curtain.", + confirmed_instruction=parse_domux_output( + "turnOff|Curtain|*|*|*|Study|Ground Floor" + )[0], + registry=self.registry, + ) + self.assertEqual(action_resolved.confirmed_instruction.action, "turnOff") + + unit_grounded = ground_domux_request( + "Set the Study AC temperature to 20 degrees Fahrenheit.", + "set|AC|temperature|20|Celsius|Study|*", + self.registry, + ) + unit_resolved = resolve_clarification_submission( + unit_grounded, + answer="Use 20 Celsius for the Study AC instead.", + confirmed_instruction=parse_domux_output( + "set|AC|temperature|20|Celsius|Study|Ground Floor" + )[0], + registry=self.registry, + ) + self.assertEqual(unit_resolved.confirmed_instruction.unit, "Celsius") + + def test_compound_effects_and_multiple_attributes_cannot_collapse(self) -> None: + cases = ( + ( + "Turn off the Study light and make it Blue.", + "set|Light|color|Blue|*|Study|*", + "action", + ), + ( + "Turn off the Study light and set brightness to 20 percent.", + "set|Light|brightness|20|Percent|Study|*", + "action", + ), + ( + "Close the Study curtain and set position to 20 percent.", + "set|Curtain|position|20|Percent|Study|*", + "action", + ), + ( + "Turn off the Study AC and use Cool mode.", + "set|AC|mode|Cool|*|Study|*", + "action", + ), + ( + "Turn off the Study light and make it brighter.", + "adjustUp|Light|brightness|10|Percent|Study|*", + "action", + ), + ( + "Close the Study curtain and raise it.", + "adjustUp|Curtain|position|10|Percent|Study|*", + "action", + ), + ( + "Set the Study light brightness and color to Blue.", + "set|Light|color|Blue|*|Study|*", + "attribute", + ), + ( + "Make the Study AC warmer and use Cool mode.", + "set|AC|mode|Cool|*|Study|*", + "attribute", + ), + ( + "Set the Study AC wind speed High and use Cool mode.", + "set|AC|mode|Cool|*|Study|*", + "attribute", + ), + ) + for utterance, raw, slot in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn(slot, grounded.clarification.unresolved_slots) + + compatible = ground_domux_request( + "Turn on the Study light and make it Blue.", + "set|Light|color|Blue|*|Study|*", + self.registry, + ) + self.assertFalse(compatible.clarification.required) + + def test_explicit_multi_targets_cannot_be_truncated_to_one_tuple(self) -> None: + cases = ( + ( + "Turn off the Living Room light and the Study light.", + "turnOff|Light|*|*|*|Living Room|*", + {"light.living_ceiling", "light.study_ceiling"}, + ), + ( + "Turn off the Study light and AC.", + "turnOff|Light|*|*|*|Study|*", + {"light.study_ceiling", "climate.study_ac"}, + ), + ( + "Turn on the Study light and open the curtain.", + "turnOn|Light|*|*|*|Study|*", + {"light.study_ceiling", "cover.study_curtain"}, + ), + ) + for utterance, raw, expected in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertTrue(expected.issubset({item.entity_id for item in grounded.candidates})) + + def test_nested_room_names_select_only_the_longest_explicit_span(self) -> None: + nested = EntityRegistry(( + EntitySpec("light.guest", "light", "Light", "Guest Bedroom", "Ground Floor"), + EntitySpec("light.bedroom", "light", "Light", "Bedroom", "Ground Floor"), + EntitySpec("light.east_hall", "light", "Light", "East Hall", "First Floor"), + EntitySpec("light.hall", "light", "Light", "Hall", "First Floor"), + )) + for room, entity_id in ( + ("Guest Bedroom", "light.guest"), + ("East Hall", "light.east_hall"), + ): + with self.subTest(room=room): + grounded = ground_domux_request( + f"Turn off the {room} light.", + f"turnOff|Light|*|*|*|{room}|*", + nested, + ) + self.assertFalse(grounded.clarification.required) + self.assertEqual(grounded.candidates[0].entity_id, entity_id) + + both = ground_domux_request( + "Turn off the Guest Bedroom light and the Bedroom light.", + "turnOff|Light|*|*|*|Guest Bedroom|*", + nested, + ) + self.assertTrue(both.clarification.required) + self.assertTrue({"light.guest", "light.bedroom"}.issubset( + {entity.entity_id for entity in both.candidates} + )) + + def test_numeric_specific_device_labels_are_not_operation_values(self) -> None: + numeric = EntityRegistry(( + EntitySpec("light.lamp_2", "light", "Lamp 2", "Study", "Ground Floor"), + EntitySpec("cover.curtain_2", "cover", "Curtain 2", "Study", "Ground Floor"), + EntitySpec("climate.ac_2", "climate", "AC 2", "Study", "Ground Floor"), + )) + for utterance, raw in ( + ("Turn off the Study Lamp 2.", "turnOff|Lamp 2|*|*|*|Study|*"), + ("Close the Study Curtain 2.", "turnOff|Curtain 2|*|*|*|Study|*"), + ("Turn off the Study AC 2.", "turnOff|AC 2|*|*|*|Study|*"), + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, numeric) + self.assertFalse(grounded.clarification.required) + + def test_explicit_alternatives_require_value_clarification(self) -> None: + cases = ( + ( + "Set the Study light brightness to 20 or 50 percent.", + "set|Light|brightness|20|Percent|Study|*", + ), + ( + "Set the Study light color to Blue or Red.", + "set|Light|color|Blue|*|Study|*", + ), + ( + "Set the Study AC mode to Heat or Cool.", + "set|AC|mode|Heat|*|Study|*", + ), + ( + "Set the Study light brightness to 20 and 50 percent.", + "set|Light|brightness|20|Percent|Study|*", + ), + ( + "Set the Study light brightness to 20, 50 percent.", + "set|Light|brightness|20|Percent|Study|*", + ), + ( + "Set the Study light color to Blue with a Red accent.", + "set|Light|color|Blue|*|Study|*", + ), + ( + "Set the Study AC mode to Cool with Fan Only.", + "set|AC|mode|Cool|*|Study|*", + ), + ( + "Set the Study light brightness below 20 percent.", + "set|Light|brightness|20|Percent|Study|*", + ), + ( + "Set the Study AC temperature above 20 Celsius.", + "set|AC|temperature|20|Celsius|Study|*", + ), + ) + for utterance, raw in cases: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn("value", grounded.clarification.unresolved_slots) + + def test_fan_speed_is_not_misread_as_an_hvac_mode(self) -> None: + for attribute in ("fan speed", "wind speed"): + with self.subTest(attribute=attribute): + grounded = ground_domux_request( + f"Set the Study AC {attribute} to High.", + f"set|AC|{attribute}|High|Level|Study|*", + self.registry, + ) + self.assertFalse(grounded.clarification.required) + + def test_absolute_and_relative_numeric_actions_cannot_be_interchanged(self) -> None: + rejected = ( + ("Raise the Study light brightness to 20 percent.", "adjustUp|Light|brightness|20|Percent|Study|*"), + ("Lower the Study light brightness to 20 percent.", "adjustDown|Light|brightness|20|Percent|Study|*"), + ("Make the Study AC warmer to 25 Celsius.", "adjustUp|AC|temperature|25|Celsius|Study|*"), + ("Raise the Study curtain to 30 percent.", "adjustUp|Curtain|position|30|Percent|Study|*"), + ("Set the Study light brightness by 20 percent.", "set|Light|brightness|20|Percent|Study|*"), + ("Open the Study curtain by 20 percent.", "set|Curtain|position|20|Percent|Study|*"), + ) + for utterance, raw in rejected: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn("value", grounded.clarification.unresolved_slots) + + accepted = ( + ("Increase the Study light brightness by 20 percent.", "adjustUp|Light|brightness|20|Percent|Study|*"), + ("Make the Study AC 2 degrees warmer.", "adjustUp|AC|temperature|2|Celsius|Study|*"), + ) + for utterance, raw in accepted: + with self.subTest(utterance=utterance): + self.assertFalse(ground_domux_request(utterance, raw, self.registry).clarification.required) + + def test_climate_turn_on_requires_a_resolvable_mode_clarification(self) -> None: + grounded = ground_domux_request( + "Turn on the Study AC.", + "turnOn|AC|*|*|*|Study|*", + self.registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("climate_mode_confirmation_required", grounded.clarification.reasons) + resolved = resolve_clarification_submission( + grounded, + answer="Use Cool mode on the Study AC.", + confirmed_instruction=parse_domux_output( + "set|AC|mode|Cool|*|Study|Ground Floor" + )[0], + registry=self.registry, + ) + plan = build_plan( + resolved.confirmed_instruction, + resolved.chosen, + self.states[resolved.chosen.entity_id], + ) + self.assertEqual((plan.service, plan.service_data["hvac_mode"]), ("set_hvac_mode", "cool")) + + def test_compound_turn_on_must_match_the_service_side_effect(self) -> None: + rejected = ( + ( + "Open the Study curtain to 0 percent.", + "set|Curtain|position|0|Percent|Study|*", + ), + ( + "Turn on the Study AC and set temperature to 20 Celsius.", + "set|AC|temperature|20|Celsius|Study|*", + ), + ( + "Turn on the Study AC and set fan speed to High.", + "set|AC|fan speed|High|Level|Study|*", + ), + ) + for utterance, raw in rejected: + with self.subTest(utterance=utterance): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertTrue(grounded.clarification.required) + self.assertIn("action", grounded.clarification.unresolved_slots) + + accepted = ( + ( + "Open the Study curtain to 20 percent.", + "set|Curtain|position|20|Percent|Study|*", + ), + ( + "Turn on the Study AC and use Cool mode.", + "set|AC|mode|Cool|*|Study|*", + ), + ) + for utterance, raw in accepted: + with self.subTest(utterance=utterance): + self.assertFalse(ground_domux_request(utterance, raw, self.registry).clarification.required) + + def test_conditions_schedules_durations_and_meta_questions_fail_closed(self) -> None: + conditionals = ( + "Turn off the Study light if the room is empty.", + "Turn on the Study light when it gets dark.", + "Turn off the Study light after dinner.", + "Turn off the Study light tomorrow.", + "Turn on the Study light for an hour.", + "Turn off the Study light at noon.", + "Turn off the Study light in five minutes.", + "Turn off the Study light once I leave.", + "Turn off the Study light every night.", + "Turn off the Study light at 6:30.", + "Turn off the Study light provided nobody is home.", + "Turn off the Study light as long as nobody is home.", + "Turn off the Study light in half an hour.", + "Turn off the Study light on Monday.", + "Turn off the Study light at nine.", + ) + for utterance in conditionals: + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, "turnOff|Light|*|*|*|Study|*", self.registry, + ) + self.assertTrue(grounded.clarification.required) + self.assertIn("unsupported_condition_or_time", grounded.clarification.reasons) + with self.assertRaisesRegex(GroundingError, "immediate command"): + resolve_clarification_submission( + grounded, + answer="Yes, the Study light.", + confirmed_instruction=parse_domux_output( + "turnOff|Light|*|*|*|Study|Ground Floor" + )[0], + registry=self.registry, + ) + + for utterance in ( + "Should I turn off the Study light?", + "Tell me how to turn off the Study light.", + "What happens if I turn off the Study light?", + "Can I turn off the Study light?", + "Is it okay to turn off the Study light?", + "Do you recommend I turn off the Study light?", + "Why should I turn off the Study light?", + "Please explain why I should turn off the Study light?", + "Do I need to turn off the Study light?", + "Would it be safe to turn off the Study light?", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, "turnOff|Light|*|*|*|Study|*", self.registry, + ) + self.assertIn("informational_request", grounded.clarification.reasons) + with self.assertRaisesRegex(GroundingError, "informational"): + resolve_clarification_submission( + grounded, + answer="Yes, do it now.", + confirmed_instruction=parse_domux_output( + "turnOff|Light|*|*|*|Study|Ground Floor" + )[0], + registry=self.registry, + ) + + for utterance in ( + "Can you turn off the Study light?", + "Could you turn off the Study light?", + ): + with self.subTest(utterance=utterance): + polite = ground_domux_request( + utterance, + "turnOff|Light|*|*|*|Study|*", + self.registry, + ) + self.assertFalse(polite.clarification.required) + + def test_unconsumed_initial_language_cannot_be_recovered_by_clarification(self) -> None: + modifiers = ( + "subject to nobody being home", + "depending on whether anyone is home", + "so long as the room is empty", + "during dinner", + "upon my arrival", + "at dusk", + "at dawn", + "this Friday", + "next Friday", + "by nine", + "for the next hour", + "momentarily", + "as needed", + "forget it", + "I withdraw that", + "I revoke that", + "I take that back", + "don't bother", + "actually don't", + "use all but Blue", + "use Blue excluding Green", + "use non-Blue", + ) + confirmed = parse_domux_output( + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor" + )[0] + for modifier in modifiers: + utterance = f"Turn off the Study Ceiling Light, {modifier}." + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + self.registry, + ) + self.assertTrue(grounded.clarification.required) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer="The Study Ceiling Light.", + confirmed_instruction=confirmed, + registry=self.registry, + ) + + for utterance in ( + "Turn off the Study Ceiling Light; wait.", + "Turn off the Study Ceiling Light, stop.", + "Turn off the Study Ceiling Light, do not.", + "Turn off the Study Ceiling Light, no.", + "Turn off the Study Ceiling Light, not now.", + "Turn off the Study Ceiling Light; I don't want it.", + "Turn off the Study Ceiling Light; I dont want it.", + "Turn off the Study Ceiling Light; I don't need it.", + "Turn off the Study Ceiling Light; I dont need it.", + "Turn off the Study Ceiling Light; please do not.", + "Turn off the Study Ceiling Light; please no.", + "Turn off the Study Ceiling Light; no please.", + "Turn off the Study Ceiling Light; I mean no.", + "Turn off the Study Ceiling Light; I want no.", + ): + with self.subTest(utterance=utterance): + grounded = ground_domux_request( + utterance, + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + self.registry, + ) + self.assertIn("negative_or_cancelled_intent", grounded.clarification.reasons) + + for utterance in ( + "Turn off the Study Ceiling Light, use the other one.", + "Turn off the Study Ceiling Light, use the other light.", + "Turn off the Study Ceiling Light, not this one.", + "Turn off the Study Ceiling Light, not that one.", + ): + with self.subTest(utterance=utterance): + generic = ground_domux_request( + utterance, + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor", + self.registry, + ) + self.assertIn("unsupported_request_grammar", generic.clarification.reasons) + + def test_model_slots_cannot_launder_unconsumed_request_language(self) -> None: + utterance = ( + "Set the Study Ceiling Light brightness to 50 percent " + "subject to nobody being home." + ) + raws = ( + "set|Ceiling Light|brightness|50|subject to nobody being home|Study|*", + "set|Ceiling Light|subject to nobody being home|50|Percent|Study|*", + ) + confirmed = parse_domux_output( + "set|Ceiling Light|brightness|50|Percent|Study|Ground Floor" + )[0] + for raw in raws: + with self.subTest(raw=raw): + grounded = ground_domux_request(utterance, raw, self.registry) + self.assertIn("unsupported_request_grammar", grounded.clarification.reasons) + with self.assertRaisesRegex(GroundingError, "new immediate command"): + resolve_clarification_submission( + grounded, + answer="Study, confirm 50 percent brightness.", + confirmed_instruction=confirmed, + registry=self.registry, + ) + + def test_displayed_candidate_indices_are_selectors_not_operation_values(self) -> None: + registry = EntityRegistry(( + EntitySpec("light.alpha", "light", "Light", "Alpha", "Ground Floor"), + EntitySpec("light.beta", "light", "Light", "Beta", "Ground Floor"), + EntitySpec("light.gamma", "light", "Light", "Gamma", "Ground Floor"), + )) + grounded = ground_domux_request( + "Turn off the light.", + "turnOff|Light|*|*|*|*|*", + registry, + ) + self.assertEqual(len(grounded.clarification.candidates), 3) + self.assertFalse( + {"action", "attribute", "value", "unit"} + .intersection(grounded.clarification.unresolved_slots) + ) + for index, chosen in enumerate(grounded.candidates, start=1): + with self.subTest(index=index, entity_id=chosen.entity_id): + confirmed = DomuxInstruction( + "turnOff", "Light", "*", "*", "*", chosen.room, chosen.floor, + ) + resolved = resolve_clarification_submission( + grounded, + answer=str(index), + confirmed_instruction=confirmed, + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, chosen.entity_id) + + def test_generic_selector_reversals_fail_closed(self) -> None: + grounded = ground_domux_request( + "Turn off the ceiling light.", + "turnOff|Ceiling Light|*|*|*|*|*", + self.registry, + ) + confirmed = parse_domux_output( + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor" + )[0] + for answer in ( + "Study, not this one.", + "Study, not that one.", + "Study, not this device.", + "Study, not the one I mean.", + "The other one.", + "Use the other one.", + "Leave this one unchanged.", + "Study, not this.", + "I mean the other one.", + ): + with self.subTest(answer=answer), self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer=answer, + confirmed_instruction=confirmed, + registry=self.registry, + ) + + def test_candidate_selector_text_cannot_double_as_operation_authorization(self) -> None: + grounded = ground_domux_request( + "Set the light brightness between 1 and 20 percent.", + "set|Light|brightness|1|Percent|*|*", + self.registry, + ) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + grounded, + answer="2", + confirmed_instruction=parse_domux_output( + "set|Light|brightness|2|Percent|Study|Ground Floor" + )[0], + registry=self.registry, + ) + + numbered = EntityRegistry(( + EntitySpec("light.bedroom_50", "light", "Light", "Bedroom", "Ground Floor"), + EntitySpec("light.study_numbered", "light", "Light", "Study", "Ground Floor"), + )) + numbered_grounded = ground_domux_request( + "Set the light brightness between 20 and 80 percent.", + "set|Light|brightness|20|Percent|*|*", + numbered, + ) + with self.assertRaises(GroundingError): + resolve_clarification_submission( + numbered_grounded, + answer="light.bedroom_50", + confirmed_instruction=parse_domux_output( + "set|Light|brightness|50|Percent|Bedroom|Ground Floor" + )[0], + registry=numbered, + ) + + def test_selector_words_and_numbers_cannot_masquerade_as_operation_values(self) -> None: + numbered = EntityRegistry(( + EntitySpec("light.bedroom_50", "light", "Light", "Bedroom 50", "Ground Floor"), + )) + utterance = "Set the Bedroom 50 light brightness to 20 percent." + wrong = ground_domux_request( + utterance, "set|Light|brightness|50|Percent|Bedroom 50|*", numbered, + ) + right = ground_domux_request( + utterance, "set|Light|brightness|20|Percent|Bedroom 50|*", numbered, + ) + self.assertTrue(wrong.clarification.required) + self.assertIn("value", wrong.clarification.unresolved_slots) + self.assertFalse(right.clarification.required) + + collision_registry = EntityRegistry(( + EntitySpec("light.orange_room", "light", "Light", "Orange Room", "Ground Floor"), + EntitySpec("climate.heat_room", "climate", "AC", "Heat Room", "Ground Floor"), + )) + color_wrong = ground_domux_request( + "Make the Orange Room light brighter.", + "set|Light|color|Orange|*|Orange Room|*", + collision_registry, + ) + mode_wrong = ground_domux_request( + "Make the Heat Room AC warmer.", + "set|AC|mode|Heat|*|Heat Room|*", + collision_registry, + ) + self.assertTrue(color_wrong.clarification.required) + self.assertIn("attribute", color_wrong.clarification.unresolved_slots) + self.assertTrue(mode_wrong.clarification.required) + self.assertIn("attribute", mode_wrong.clarification.unresolved_slots) + + +class FailOnceReadAdapter(InMemoryHAAdapter): + fail_next_read = False + + def get_state(self, entity_id: str) -> dict[str, object]: + if self.fail_next_read: + self.fail_next_read = False + raise AdapterError("injected predispatch read failure") + return super().get_state(entity_id) + + +class ClockAdvancingAdapter(InMemoryHAAdapter): + def __init__(self, states: dict[str, dict[str, object]], clock: MutableClock): + super().__init__(states) + self.clock = clock + self.advance_reads = False + + def get_state(self, entity_id: str) -> dict[str, object]: + value = super().get_state(entity_id) + if self.advance_reads: + self.clock.value += 8 + return value + + +class UnknownOutcomeAdapter(InMemoryHAAdapter): + def call_service(self, domain: str, service: str, data: dict[str, object]) -> ServiceCallResult: + self.sut_calls.append({"kind": "sut", "outcome": "request_error_outcome_unknown"}) + raise ServiceCallError( + "injected transport loss", attempted=True, acknowledged=False, outcome_unknown=True, + ) + + +class WaitFailureAdapter(InMemoryHAAdapter): + def wait_for_projection( + self, entity_id: str, domain: str, expected: dict[str, object], + ) -> dict[str, object]: + del entity_id, domain, expected + raise AdapterError("injected post-ack observation failure") + + +class BlockingAdapter(InMemoryHAAdapter): + def __init__(self, states: dict[str, dict[str, object]]): + super().__init__(states) + self.utility_started = threading.Event() + self.release_utility = threading.Event() + + def call_service(self, domain: str, service: str, data: dict[str, object]) -> ServiceCallResult: + if data.get("entity_id") == "light.utility": + self.utility_started.set() + if not self.release_utility.wait(timeout=2): + raise AssertionError("test did not release the blocked utility call") + return super().call_service(domain, service, data) + + +class PreparedActionTests(unittest.TestCase): + def setUp(self) -> None: + self.registry, self.states = fixture() + self.grounded = ground_domux_request( + "Turn off the Ceiling Light.", + "turnOff|Ceiling Light|*|*|*|*|*", + self.registry, + ) + self.confirmed = parse_domux_output( + "turnOff|Ceiling Light|*|*|*|Study|Ground Floor" + )[0] + self.clock = MutableClock() + self.nonce_index = 0 + + def nonce(self) -> str: + self.nonce_index += 1 + return f"test-nonce-{self.nonce_index}" + + def prepare( + self, + store_type: type[PreparedActionStore] = PreparedActionStore, + *, + adapter: InMemoryHAAdapter | None = None, + grounded=None, + answer: str | None = "Study", + confirmed: DomuxInstruction | None = None, + state_dependencies: tuple[str, ...] = (), + ): + adapter = adapter or InMemoryHAAdapter(self.states) + store = store_type(ttl_seconds=30, clock=self.clock, nonce_factory=self.nonce) + grounded = grounded or self.grounded + action = store.prepare( + actor_id="actor-a", + session_id="session-a", + grounded=grounded, + registry=self.registry, + adapter=adapter, + clarification_answer=answer, + confirmed_instruction=confirmed or self.confirmed, + state_dependencies=state_dependencies, + ) + return adapter, store, action + + def test_clean_commit_changes_only_the_selected_entity(self) -> None: + adapter, store, action = self.prepare() + untouched = adapter.get_state("light.living_ceiling") + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertTrue(result.accepted and result.dispatched and result.acknowledged) + self.assertEqual((result.status, result.after["state"]), ("COMMITTED", "off")) + self.assertEqual(adapter.get_state("light.living_ceiling"), untouched) + self.assertEqual(len(adapter.sut_calls), 1) + + def test_public_handle_confirmation_and_plan_copies_are_immutable(self) -> None: + adapter, store, action = self.prepare() + with self.assertRaises(FrozenInstanceError): + action.entity_id = "light.living_ceiling" # type: ignore[misc] + snapshot = store.snapshot(action.nonce) + snapshot["plan"]["service_data"]["entity_id"] = "light.living_ceiling" + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((result.status, result.after["entity_id"]), ("COMMITTED", "light.study_ceiling")) + tombstone = store.snapshot(action.nonce) + self.assertTrue(tombstone["redacted"]) + retained = json.dumps(tombstone) + self.assertNotIn("Turn off the Ceiling Light", retained) + self.assertNotIn("light.study_ceiling", retained) + self.assertNotIn("actor-a", retained) + + def test_replay_and_two_prepared_nonces_dispatch_at_most_once_each_state(self) -> None: + adapter, store, first = self.prepare() + second = store.prepare( + actor_id="actor-a", session_id="session-a", grounded=self.grounded, + registry=self.registry, adapter=adapter, clarification_answer="Study", + confirmed_instruction=self.confirmed, + ) + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map( + lambda item: store.commit(item.confirmation(), registry=self.registry, adapter=adapter), + (first, second), + )) + self.assertEqual(sum(result.dispatched for result in results), 1) + self.assertEqual({result.reason for result in results}, {"committed", "state_changed"}) + replay = store.commit(first.confirmation(), registry=self.registry, adapter=adapter) + self.assertIn(replay.reason, {"replayed_nonce", "action_not_prepared"}) + self.assertEqual(len(adapter.sut_calls), 1) + + def test_expiry_state_capability_and_candidate_drift_are_zero_dispatch(self) -> None: + adapter, store, action = self.prepare() + self.clock.value = action.expires_at + 1 + self.assertEqual( + store.commit(action.confirmation(), registry=self.registry, adapter=adapter).reason, + "expired", + ) + + self.clock.value = 1000 + adapter, store, action = self.prepare() + adapter.mutate_state_for_setup("light.study_ceiling") + self.assertEqual(store.commit(action.confirmation(), registry=self.registry, adapter=adapter).reason, "state_changed") + + adapter, store, action = self.prepare() + changed_state = adapter.get_state("light.study_ceiling") + changed_state["attributes"]["supported_color_modes"] = ["onoff"] + adapter.set_state_for_setup("light.study_ceiling", changed_state) + self.assertEqual(store.commit(action.confirmation(), registry=self.registry, adapter=adapter).reason, "state_changed") + + adapter, store, action = self.prepare() + changed = self.registry.with_replacement( + EntitySpec("light.study_ceiling", "light", "Ceiling Light", "Library", "Ground Floor") + ) + self.assertEqual(store.commit(action.confirmation(), registry=changed, adapter=adapter).reason, "candidate_set_changed") + + adapter, store, action = self.prepare() + expanded = EntityRegistry((*self.registry.entities, EntitySpec( + "light.bedroom_ceiling", "light", "Ceiling Light", "Bedroom", "Ground Floor", + ))) + self.assertEqual(store.commit(action.confirmation(), registry=expanded, adapter=adapter).reason, "candidate_set_changed") + self.assertEqual(len(adapter.sut_calls), 0) + + def test_confirmation_binds_all_authorization_digests(self) -> None: + mutations = ( + ("actor_id", "actor-b", "actor_mismatch"), + ("session_id", "session-b", "session_mismatch"), + ("request_digest", "0" * 64, "request_mismatch"), + ("clarification_digest", "0" * 64, "clarification_mismatch"), + ("plan_digest", "0" * 64, "plan_mismatch"), + ("candidate_digest", "0" * 64, "confirmation_candidate_mismatch"), + ) + for field, value, reason in mutations: + adapter, store, action = self.prepare() + result = store.commit( + altered_confirmation(action.confirmation(), **{field: value}), + registry=self.registry, + adapter=adapter, + ) + self.assertEqual((result.reason, len(adapter.sut_calls)), (reason, 0)) + + def test_only_declared_or_context_state_is_bound(self) -> None: + adapter, store, action = self.prepare() + adapter.mutate_state_for_setup("light.utility") + self.assertTrue(store.commit(action.confirmation(), registry=self.registry, adapter=adapter).accepted) + + context_grounded = ground_domux_request( + "Turn off that device.", + "turnOff|*|*|*|*|*|*", + self.registry, + SessionContext(("light.study_ceiling", "light.utility")), + ) + adapter, store, action = self.prepare(grounded=context_grounded) + adapter.mutate_state_for_setup("light.utility") + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((result.reason, len(adapter.sut_calls)), ("state_changed", 0)) + + adapter, store, action = self.prepare(state_dependencies=("light.utility",)) + adapter.mutate_state_for_setup("light.utility") + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((result.reason, len(adapter.sut_calls)), ("state_changed", 0)) + + def test_slow_predispatch_reads_cannot_cross_ttl(self) -> None: + adapter = ClockAdvancingAdapter(self.states, self.clock) + adapter, store, action = self.prepare(adapter=adapter) + adapter.advance_reads = True + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((result.reason, len(adapter.sut_calls)), ("expired", 0)) + + def test_abandoned_expired_action_is_redacted_on_purge(self) -> None: + _adapter, store, action = self.prepare() + self.clock.value = action.expires_at + 1 + self.assertEqual(store.purge_expired(), 1) + snapshot = store.snapshot(action.nonce) + self.assertTrue(snapshot["redacted"]) + retained = json.dumps(snapshot) + self.assertNotIn("Ceiling Light", retained) + self.assertNotIn("light.study_ceiling", retained) + + def test_predispatch_read_failure_does_not_consume_nonce(self) -> None: + adapter = FailOnceReadAdapter(self.states) + adapter, store, action = self.prepare(adapter=adapter) + adapter.fail_next_read = True + first = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((first.reason, first.dispatched), ("predispatch_state_read_failed", False)) + second = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((second.status, len(adapter.sut_calls)), ("COMMITTED", 1)) + + def test_dispatch_and_post_ack_unknown_outcomes_are_action_local(self) -> None: + adapter, store, action = self.prepare(adapter=UnknownOutcomeAdapter(self.states)) + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual( + (result.status, result.dispatched, result.acknowledged, result.outcome_unknown), + ("FAILED_DISPATCH", True, False, True), + ) + + adapter, store, action = self.prepare(adapter=WaitFailureAdapter(self.states)) + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual( + (result.status, result.dispatched, result.acknowledged, result.outcome_unknown), + ("FAILED_POSTCONDITION", True, True, True), + ) + + def test_postcondition_failure_is_visible_and_nonce_stays_consumed(self) -> None: + adapter, store, action = self.prepare() + adapter.force_postcondition_mismatch = True + result = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((result.status, result.reason), ("FAILED_POSTCONDITION", "postcondition_mismatch")) + replay = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertEqual((replay.reason, len(adapter.sut_calls)), ("replayed_nonce", 1)) + + def test_dependency_target_race_is_serialized_before_dispatch(self) -> None: + adapter = BlockingAdapter(self.states) + store = PreparedActionStore(ttl_seconds=30, clock=self.clock, nonce_factory=self.nonce) + context_grounded = ground_domux_request( + "Turn off that device.", "turnOff|*|*|*|*|*|*", self.registry, + SessionContext(("light.study_ceiling", "light.utility")), + ) + dependent = store.prepare( + actor_id="actor-a", session_id="session-a", grounded=context_grounded, + registry=self.registry, adapter=adapter, clarification_answer="Study", + confirmed_instruction=self.confirmed, + ) + utility_grounded = ground_domux_request( + "Turn on the Utility Light in the Utility Room on the Ground Floor.", + "turnOn|Utility Light|*|*|*|Utility Room|Ground Floor", + self.registry, + ) + utility = store.prepare( + actor_id="actor-a", session_id="session-a", grounded=utility_grounded, + registry=self.registry, adapter=adapter, + ) + with ThreadPoolExecutor(max_workers=2) as pool: + utility_future = pool.submit( + store.commit, utility.confirmation(), registry=self.registry, adapter=adapter, + ) + self.assertTrue(adapter.utility_started.wait(timeout=1)) + dependent_future = pool.submit( + store.commit, dependent.confirmation(), registry=self.registry, adapter=adapter, + ) + adapter.release_utility.set() + utility_result = utility_future.result(timeout=2) + dependent_result = dependent_future.result(timeout=2) + self.assertEqual(utility_result.status, "COMMITTED") + self.assertEqual((dependent_result.reason, len(adapter.sut_calls)), ("state_changed", 1)) + + def test_b1_binds_plan_and_session_but_deliberately_omits_temporal_guards(self) -> None: + adapter, store, action = self.prepare(ClarifyPrepareStore) + rejected = store.commit( + altered_confirmation(action.confirmation(), session_id="session-b"), + registry=self.registry, + adapter=adapter, + ) + self.assertEqual((rejected.reason, len(adapter.sut_calls)), ("session_mismatch", 0)) + + adapter, store, action = self.prepare(ClarifyPrepareStore) + first = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + second = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertTrue(first.accepted and second.accepted) + self.assertEqual(len(adapter.sut_calls), 2) + + adapter, store, action = self.prepare(ClarifyPrepareStore) + adapter.mutate_state_for_setup("light.study_ceiling") + drift = store.commit(action.confirmation(), registry=self.registry, adapter=adapter) + self.assertTrue(drift.accepted) + self.assertEqual(len(adapter.sut_calls), 1) + + +class _HAHandler(BaseHTTPRequestHandler): + token = "test-token-not-a-real-secret" + state = {"entity_id": "light.demo", "state": "off", "attributes": {"brightness": 0}} + climate_state = { + "entity_id": "climate.demo", + "state": "cool", + "attributes": { + "temperature": 24.0, + "hvac_modes": ["off", "cool"], + "supported_features": 1, + }, + } + config: object = {"unit_system": {"temperature": "°C"}} + get_paths: list[str] = [] + calls: list[dict[str, object]] = [] + post_status: int | None = None + + def log_message(self, _format: str, *_args: object) -> None: + return + + def _json(self, status: int, payload: object) -> None: + encoded = json.dumps(payload).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + def do_GET(self) -> None: # noqa: N802 + if self.headers.get("Authorization") != f"Bearer {self.token}": + self._json(401, {}) + return + type(self).get_paths.append(self.path) + if self.path == "/api/states/light.demo": + self._json(200, self.state) + elif self.path == "/api/states/climate.demo": + self._json(200, self.climate_state) + elif self.path == "/api/config": + self._json(200, self.config) + else: + self._json(404, {}) + + def do_POST(self) -> None: # noqa: N802 + if self.headers.get("Authorization") != f"Bearer {self.token}": + self._json(401, {}) + return + length = int(self.headers.get("Content-Length", "0")) + payload = json.loads(self.rfile.read(length)) + type(self).calls.append(payload) + if type(self).post_status is not None: + self._json(type(self).post_status, {}) + return + if self.path == "/api/services/light/turn_on": + type(self).state = { + "entity_id": "light.demo", "state": "on", "attributes": {"brightness": 0}, + } + self._json(200, [type(self).state]) + else: + self._json(404, {}) + + +class RestAdapterTests(unittest.TestCase): + def setUp(self) -> None: + _HAHandler.state = {"entity_id": "light.demo", "state": "off", "attributes": {"brightness": 0}} + _HAHandler.climate_state = { + "entity_id": "climate.demo", + "state": "cool", + "attributes": { + "temperature": 24.0, + "hvac_modes": ["off", "cool"], + "supported_features": 1, + }, + } + _HAHandler.config = {"unit_system": {"temperature": "°C"}} + _HAHandler.get_paths = [] + _HAHandler.calls = [] + _HAHandler.post_status = None + self.server = ThreadingHTTPServer(("127.0.0.1", 0), _HAHandler) + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + self.thread.start() + + def tearDown(self) -> None: + self.server.shutdown() + self.server.server_close() + self.thread.join(timeout=2) + + def test_official_rest_shape_and_loopback_guard(self) -> None: + port = self.server.server_address[1] + adapter = HomeAssistantRESTAdapter( + f"http://127.0.0.1:{port}", _HAHandler.token, poll_seconds=0.2, + ) + before = adapter.get_state("light.demo") + after = adapter.call_service("light", "turn_on", {"entity_id": "light.demo"}) + self.assertEqual((before["state"], after.after["state"]), ("off", "on")) + self.assertEqual(len(adapter.sut_calls), 1) + with self.assertRaises(ValueError): + HomeAssistantRESTAdapter("https://example.com", "token") + + def test_http_4xx_is_rejected_but_5xx_has_unknown_dispatch_outcome(self) -> None: + port = self.server.server_address[1] + adapter = HomeAssistantRESTAdapter( + f"http://127.0.0.1:{port}", _HAHandler.token, poll_seconds=0.2, + ) + for status, unknown, outcome in ( + (400, False, "request_rejected"), + (500, True, "request_error_outcome_unknown"), + ): + with self.subTest(status=status): + _HAHandler.post_status = status + with self.assertRaises(ServiceCallError) as caught: + adapter.call_service("light", "turn_on", {"entity_id": "light.demo"}) + self.assertTrue(caught.exception.attempted) + self.assertFalse(caught.exception.acknowledged) + self.assertEqual(caught.exception.outcome_unknown, unknown) + self.assertEqual(adapter.sut_calls[-1]["outcome"], outcome) + + def test_climate_temperature_unit_comes_from_official_config(self) -> None: + port = self.server.server_address[1] + adapter = HomeAssistantRESTAdapter( + f"http://127.0.0.1:{port}", _HAHandler.token, poll_seconds=0.2, + ) + state = adapter.get_state("climate.demo") + self.assertEqual(state["attributes"]["temperature_unit"], "°C") + self.assertNotIn("temperature_unit", _HAHandler.climate_state["attributes"]) + self.assertEqual( + _HAHandler.get_paths, + ["/api/states/climate.demo", "/api/config"], + ) + + malformed = ( + [], + {}, + {"unit_system": {}}, + {"unit_system": {"temperature": 123}}, + ) + for config in malformed: + with self.subTest(config=config): + _HAHandler.config = config + with self.assertRaises(AdapterError): + adapter.get_state("climate.demo") + + +if __name__ == "__main__": + unittest.main() diff --git a/cases/domux-clarify-commit/tests/test_dataset.py b/cases/domux-clarify-commit/tests/test_dataset.py new file mode 100644 index 0000000..137213a --- /dev/null +++ b/cases/domux-clarify-commit/tests/test_dataset.py @@ -0,0 +1,168 @@ +from __future__ import annotations + +import hashlib +import json +import re +import sys +import unittest +from collections import Counter +from pathlib import Path + + +CASE_DIR = Path(__file__).resolve().parents[1] +DATASET = CASE_DIR / "data" / "scenarios.jsonl" +FREEZE = CASE_DIR / "data" / "freeze.json" +PROTOCOL = CASE_DIR / "data" / "protocol.json" +SNAPSHOT_MANIFEST = CASE_DIR / "data" / "snapshot_manifest.json" +FIELDS = {"action", "device", "attribute", "value", "unit", "room", "floor"} +sys.path.insert(0, str(CASE_DIR)) + +from clarify_commit import ( # noqa: E402 + DomuxInstruction, + EntityRegistry, + EntitySpec, + SessionContext, + build_plan, + controlled_projection, + ground_domux_request, + projection_matches, + resolve_clarification_submission, +) + + +def sha256(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def canonical(row: object) -> str: + return json.dumps(row, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + + +class DatasetTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.raw_lines = [line for line in DATASET.read_text(encoding="utf-8").splitlines() if line] + cls.rows = [json.loads(line) for line in cls.raw_lines] + cls.freeze = json.loads(FREEZE.read_text(encoding="utf-8")) + cls.protocol = json.loads(PROTOCOL.read_text(encoding="utf-8")) + + def test_file_and_protocol_hashes_match_freeze(self) -> None: + self.assertEqual(sha256(DATASET.read_bytes()), self.freeze["full_sha256"]) + self.assertEqual(sha256(PROTOCOL.read_bytes()), self.freeze["protocol_sha256"]) + self.assertEqual( + sha256(SNAPSHOT_MANIFEST.read_bytes()), self.freeze["snapshot_manifest_sha256"] + ) + self.assertTrue(all(line == canonical(row) for line, row in zip(self.raw_lines, self.rows))) + + def test_split_counts_and_unique_base_ids(self) -> None: + self.assertEqual(len(self.rows), 64) + self.assertEqual(len({row["base_id"] for row in self.rows}), 64) + self.assertEqual(Counter(row["split"] for row in self.rows), {"dev": 16, "eval": 48}) + eval_rows = [row for row in self.rows if row["split"] == "eval"] + self.assertEqual(Counter(row["category"] for row in eval_rows), { + "duplicate_entity": 12, + "missing_slot": 12, + "context_reference": 12, + "negation_correction": 12, + }) + eval_bytes = "".join(canonical(row) + "\n" for row in eval_rows).encode("utf-8") + self.assertEqual(sha256(eval_bytes), self.freeze["evaluation_sha256"]) + + def test_paired_commands_are_distinct_and_not_cross_split_duplicates(self) -> None: + commands: dict[str, str] = {} + for row in self.rows: + self.assertNotEqual(row["clear_command"], row["ambiguous_command"]) + for key in ("clear_command", "ambiguous_command"): + normalized = " ".join(row[key].casefold().split()) + self.assertNotIn(normalized, commands, f"duplicate command across {commands.get(normalized)}") + commands[normalized] = f"{row['base_id']}:{key}" + + def test_inventory_candidates_and_controlled_deltas_are_closed(self) -> None: + for row in self.rows: + with self.subTest(base_id=row["base_id"]): + inventory = {entity["entity_id"]: entity for entity in row["inventory"]} + states = row["initial_states"] + candidates = row["candidate_entity_ids"] + self.assertEqual(set(inventory), set(states)) + self.assertTrue(1 <= len(candidates) <= 3) + self.assertEqual(len(candidates), len(set(candidates))) + self.assertTrue(set(candidates) <= set(inventory)) + self.assertIn(row["expected_target_entity"], candidates) + self.assertIn(row["unrelated_entity_id"], inventory) + self.assertNotIn(row["unrelated_entity_id"], candidates) + self.assertEqual(row["expected_delta"]["before"], states[row["expected_target_entity"]]) + self.assertNotEqual(row["expected_delta"]["before"], row["expected_delta"]["after"]) + self.assertEqual(set(row["confirmed_instruction"]), FIELDS) + self.assertTrue(set(row["state_dependencies"]) <= set(inventory)) + self.assertTrue(all(entity["domain"] in {"light", "cover", "climate"} for entity in inventory.values())) + + def test_expected_target_metadata_matches_confirmed_instruction(self) -> None: + for row in self.rows: + target = next(item for item in row["inventory"] if item["entity_id"] == row["expected_target_entity"]) + plan = row["confirmed_instruction"] + with self.subTest(base_id=row["base_id"]): + self.assertEqual(target["room"], plan["room"]) + self.assertEqual(target["floor"], plan["floor"]) + expected_domain = {"Light": "light", "Curtain": "cover", "AC": "climate"}[plan["device"]] + self.assertEqual(target["domain"], expected_domain) + + def test_context_and_unresolved_correction_strata_are_honest(self) -> None: + eval_rows = [row for row in self.rows if row["split"] == "eval"] + context = [row for row in eval_rows if row["category"] == "context_reference"] + corrections = [row for row in eval_rows if row["category"] == "negation_correction"] + self.assertEqual(len(context), 12) + self.assertTrue(all(row["state_dependencies"] for row in context)) + uncertainty = re.compile( + r"\b(perhaps|confirm|which|not sure|ask|maybe|not decided)\b|do not choose|—or|or did", + re.IGNORECASE, + ) + self.assertTrue(all(uncertainty.search(row["ambiguous_command"]) for row in corrections)) + + def test_no_obvious_secret_private_path_or_high_risk_domain(self) -> None: + text = DATASET.read_text(encoding="utf-8") + banned = ( + r"hf_[A-Za-z0-9]{20,}", + r"(?i)authorization\s*[:=]", + r"-----BEGIN [A-Z ]*PRIVATE KEY-----", + r"/(?:home|data)/[A-Za-z0-9._-]+/", + r"\b(?:10\.|192\.168\.|172\.(?:1[6-9]|2\d|3[01])\.)", + ) + for pattern in banned: + self.assertIsNone(re.search(pattern, text), pattern) + self.assertFalse(any(word in text.casefold() for word in ( + "lock.", "alarm.", "camera.", "gas.", "smoke.", "door.", "medical.", + ))) + + def test_protocol_pilot_is_development_only(self) -> None: + split_by_id = {row["base_id"]: row["split"] for row in self.rows} + pilot_ids = self.protocol["pilot_gate"]["base_ids"] + self.assertEqual(len(pilot_ids), 12) + self.assertEqual(len(set(pilot_ids)), 12) + self.assertTrue(all(split_by_id[base_id] == "dev" for base_id in pilot_ids)) + + def test_all_frozen_rows_round_trip_through_ground_resolve_and_plan(self) -> None: + for row in self.rows: + with self.subTest(base_id=row["base_id"]): + registry = EntityRegistry(EntitySpec(**item) for item in row["inventory"]) + context = SessionContext(tuple(row["session_context"]["last_referenced_entities"])) + confirmed = DomuxInstruction(**row["confirmed_instruction"]) + grounded = ground_domux_request( + row["ambiguous_command"], confirmed.to_pipe(), registry, context, + ) + self.assertTrue(grounded.clarification.required) + resolved = resolve_clarification_submission( + grounded, + answer=row["clarification_answer"], + confirmed_instruction=confirmed, + registry=registry, + ) + self.assertEqual(resolved.chosen.entity_id, row["expected_target_entity"]) + before = row["initial_states"][resolved.chosen.entity_id] + self.assertEqual(before["entity_id"], resolved.chosen.entity_id) + plan = build_plan(confirmed, resolved.chosen, before) + expected = controlled_projection(row["expected_delta"]["after"], resolved.chosen.domain) + self.assertTrue(projection_matches(expected, plan.expected_projection)) + + +if __name__ == "__main__": + unittest.main() diff --git a/cases/domux-clarify-commit/tests/test_evaluate.py b/cases/domux-clarify-commit/tests/test_evaluate.py new file mode 100644 index 0000000..b08fa87 --- /dev/null +++ b/cases/domux-clarify-commit/tests/test_evaluate.py @@ -0,0 +1,427 @@ +from __future__ import annotations + +import hashlib +import io +import json +import sys +import tempfile +import unittest +from contextlib import redirect_stdout +from pathlib import Path + + +CASE_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(CASE_DIR)) + +from evaluate import ( # noqa: E402 + B0, + B1, + B2, + EvaluationInputError, + REGISTERED_SNAPSHOT_FILES, + REGISTERED_SNAPSHOT_MANIFEST_SHA256, + _load_evidence, + _read_jsonl, + _run_b0_trial, + _validate_rows, + _validated_model_run, + analyze_probe, + canonical_json, + evaluate_to_directory, + exact_mcnemar, + holm_adjust, + main, + wilson_interval, +) + + +DATASET = CASE_DIR / "data" / "scenarios.jsonl" +PROTOCOL = CASE_DIR / "data" / "protocol.json" +FREEZE = CASE_DIR / "data" / "freeze.json" + + +def sha256_text(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +def minimal_complete_evidence(rows: list[dict[str, object]]) -> list[dict[str, object]]: + evidence: list[dict[str, object]] = [] + for row in rows: + for variant in ("clear", "ambiguous"): + command = str(row[f"{variant}_command"]) + fields = dict(row["confirmed_instruction"]) + if variant == "ambiguous": + inventory = {item["entity_id"]: item for item in row["inventory"]} + candidates = [inventory[entity_id] for entity_id in row["candidate_entity_ids"]] + domains = {item["domain"] for item in candidates} + if len(domains) > 1: + fields["device"] = "*" + fields["attribute"] = "*" + if len({item["room"] for item in candidates}) > 1: + fields["room"] = "*" + if len({item["floor"] for item in candidates}) > 1: + fields["floor"] = "*" + if len(candidates) == 1 and fields["action"] == "set": + fields["value"] = "*" + raw_output = "|".join( + str(fields[slot]) + for slot in ("action", "device", "attribute", "value", "unit", "room", "floor") + ) + evidence.append({ + "base_id": row["base_id"], + "variant": variant, + "command": command, + "query_sha256": sha256_text(command), + "status": "ok", + "raw_output": raw_output, + "raw_output_sha256": sha256_text(raw_output), + "latency_ms": 1.0, + }) + return evidence + + +def write_frozen_run( + directory: Path, + rows: list[dict[str, object]], + *, + inference_error: tuple[str, str] | None = None, +) -> tuple[Path, Path]: + evidence = minimal_complete_evidence(rows) + if inference_error is not None: + for index, item in enumerate(evidence): + if (item["base_id"], item["variant"]) == inference_error: + evidence[index] = { + **item, + "status": "error", + "raw_output": "", + "raw_output_sha256": sha256_text(""), + "latency_ms": None, + "error_type": "RuntimeError", + } + break + else: + raise AssertionError("requested synthetic inference-error key was not found") + evidence_path = directory / "raw.jsonl" + evidence_path.write_text( + "".join(canonical_json(item) + "\n" for item in evidence), + encoding="utf-8", + ) + + protocol = json.loads(PROTOCOL.read_text(encoding="utf-8")) + model = protocol["model"] + metadata = { + "model_id": model["id"], + "tested_revision": model["revision"], + "precision": model["primary_precision"], + "dataset_sha256": hashlib.sha256(DATASET.read_bytes()).hexdigest(), + "raw_evidence_sha256": hashlib.sha256(evidence_path.read_bytes()).hexdigest(), + "sample_count": 96, + "base_count": 48, + "warmup_count": model["warmup"]["count"], + "generation": { + "do_sample": model["do_sample"], + "temperature": model["temperature"], + "max_new_tokens": model["max_new_tokens"], + "seed": model["seed"], + }, + "offline_inference": True, + "snapshot_revision_verified": True, + "snapshot_manifest_sha256": REGISTERED_SNAPSHOT_MANIFEST_SHA256, + "snapshot_files": REGISTERED_SNAPSHOT_FILES, + "runner_sha256": hashlib.sha256((CASE_DIR / "run_model.py").read_bytes()).hexdigest(), + "grounding_policy_sha256": hashlib.sha256( + (CASE_DIR / "clarify_commit.py").read_bytes() + ).hexdigest(), + "evaluator_sha256": hashlib.sha256((CASE_DIR / "evaluate.py").read_bytes()).hexdigest(), + "run_mode": "formal", + "selective_reruns": 0, + "sample_failures": sum(item["status"] != "ok" for item in evidence), + "runtime": "synthetic-test-runtime", + "gpu": "synthetic-test-gpu", + "visible_gpu_count": 1, + } + metadata_path = directory / "metadata.json" + metadata_path.write_text(json.dumps(metadata, sort_keys=True) + "\n", encoding="utf-8") + return evidence_path, metadata_path + + +class StatisticalTests(unittest.TestCase): + def test_wilson_interval_uses_the_fixed_base_denominator(self) -> None: + interval = wilson_interval(48, 48) + self.assertAlmostEqual(interval["lower"], 0.925899870333882) + self.assertEqual(interval["upper"], 1.0) + zero = wilson_interval(0, 48) + self.assertEqual(zero["lower"], 0.0) + self.assertAlmostEqual(zero["upper"], 0.0741001296661179) + + def test_exact_two_sided_mcnemar_uses_discordant_base_pairs(self) -> None: + result = exact_mcnemar(B0, B1, [False] * 48, [True] * 48) + self.assertEqual(result["paired_bases"], 48) + self.assertEqual(result["b_only_success"], 48) + self.assertEqual(result["discordant_pairs"], 48) + self.assertEqual(result["risk_difference_b_minus_a"], 1.0) + self.assertAlmostEqual(result["exact_two_sided_p"], 2 / (2 ** 48)) + + def test_holm_adjustment_is_monotone_in_sorted_p_values(self) -> None: + comparisons = [ + {"comparison_id": "larger", "exact_two_sided_p": 0.04}, + {"comparison_id": "smaller", "exact_two_sided_p": 0.01}, + ] + adjusted = {item["comparison_id"]: item for item in holm_adjust(comparisons)} + self.assertEqual(adjusted["smaller"]["holm_adjusted_p"], 0.02) + self.assertEqual(adjusted["larger"]["holm_adjusted_p"], 0.04) + self.assertTrue(adjusted["smaller"]["reject_at_0_05"]) + self.assertTrue(adjusted["larger"]["reject_at_0_05"]) + + +class FrozenEvidenceContractTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.rows = _validate_rows(_read_jsonl(DATASET, "dataset")) + + def test_loader_requires_all_48_paired_probes(self) -> None: + evidence = minimal_complete_evidence(self.rows) + indexed = _load_evidence(evidence, self.rows) + self.assertEqual(len(indexed), 96) + with self.assertRaisesRegex(EvaluationInputError, "incomplete"): + _load_evidence(evidence[:-1], self.rows) + + def test_loader_rejects_duplicate_or_edited_frozen_evidence(self) -> None: + evidence = minimal_complete_evidence(self.rows) + with self.assertRaisesRegex(EvaluationInputError, "duplicate"): + _load_evidence([*evidence, dict(evidence[0])], self.rows) + + edited = [dict(item) for item in evidence] + edited[0]["command"] = "post-freeze edit" + with self.assertRaisesRegex(EvaluationInputError, "differs"): + _load_evidence(edited, self.rows) + + bad_hash = [dict(item) for item in evidence] + bad_hash[0]["raw_output_sha256"] = "0" * 64 + with self.assertRaisesRegex(EvaluationInputError, "raw-output hash"): + _load_evidence(bad_hash, self.rows) + + def test_inference_error_is_valid_evidence_not_an_omitted_base(self) -> None: + evidence = minimal_complete_evidence(self.rows) + evidence[0] = { + **evidence[0], + "status": "error", + "raw_output": "", + "raw_output_sha256": sha256_text(""), + "latency_ms": None, + "error_type": "RuntimeError", + } + indexed = _load_evidence(evidence, self.rows) + self.assertEqual(len(indexed), 96) + self.assertEqual(indexed[(str(self.rows[0]["base_id"]), "clear")]["status"], "error") + + def test_metadata_must_bind_the_exact_raw_evidence_hash(self) -> None: + protocol = json.loads(PROTOCOL.read_text(encoding="utf-8")) + model = protocol["model"] + dataset_sha256 = hashlib.sha256(DATASET.read_bytes()).hexdigest() + evidence_sha256 = "a" * 64 + metadata = { + "model_id": model["id"], + "tested_revision": model["revision"], + "precision": model["primary_precision"], + "dataset_sha256": dataset_sha256, + "raw_evidence_sha256": evidence_sha256, + "sample_count": 96, + "base_count": 48, + "warmup_count": model["warmup"]["count"], + "generation": { + "do_sample": model["do_sample"], + "temperature": model["temperature"], + "max_new_tokens": model["max_new_tokens"], + "seed": model["seed"], + }, + "offline_inference": True, + "snapshot_revision_verified": True, + "snapshot_manifest_sha256": REGISTERED_SNAPSHOT_MANIFEST_SHA256, + "snapshot_files": REGISTERED_SNAPSHOT_FILES, + "runner_sha256": hashlib.sha256((CASE_DIR / "run_model.py").read_bytes()).hexdigest(), + "grounding_policy_sha256": hashlib.sha256( + (CASE_DIR / "clarify_commit.py").read_bytes() + ).hexdigest(), + "evaluator_sha256": hashlib.sha256( + (CASE_DIR / "evaluate.py").read_bytes() + ).hexdigest(), + "run_mode": "formal", + "selective_reruns": 0, + "sample_failures": 0, + } + validated = _validated_model_run( + metadata, + protocol, + dataset_sha256, + evidence_sha256, + ) + self.assertTrue(validated["evidence_binding"]["match"]) + self.assertTrue(validated["code_binding"]["match"]) + with self.assertRaisesRegex(EvaluationInputError, "raw_evidence_sha256"): + _validated_model_run(metadata, protocol, dataset_sha256, "b" * 64) + tampered = dict(metadata) + tampered["grounding_policy_sha256"] = "0" * 64 + with self.assertRaisesRegex(EvaluationInputError, "grounding_policy_sha256"): + _validated_model_run(tampered, protocol, dataset_sha256, evidence_sha256) + + def test_runtime_grounding_detects_every_gold_ambiguity_without_using_its_label(self) -> None: + indexed = _load_evidence(minimal_complete_evidence(self.rows), self.rows) + for row in self.rows: + base_id = str(row["base_id"]) + with self.subTest(base_id=base_id): + clear = analyze_probe(row, "clear", indexed[(base_id, "clear")]) + ambiguous = analyze_probe(row, "ambiguous", indexed[(base_id, "ambiguous")]) + self.assertTrue(clear.is_unique) + self.assertTrue(ambiguous.is_clarification) + self.assertIn(row["expected_target_entity"], ambiguous.candidate_ids) + self.assertLessEqual(len(ambiguous.candidate_ids), 3) + + def test_unique_but_non_executable_model_output_abstains(self) -> None: + cases = ( + ("brightness", "150", "Set the {device} in {room} on {floor} to 150 percent brightness."), + ("mode", "Turbo", "Set the {device} in {room} on {floor} to Turbo mode."), + ) + for attribute, value, command_template in cases: + source = next( + row for row in self.rows + if row["confirmed_instruction"]["attribute"] == attribute + ) + row = dict(source) + fields = dict(row["confirmed_instruction"]) + fields["value"] = value + row["clear_command"] = command_template.format( + device=fields["device"], + room=fields["room"], + floor=fields["floor"], + ) + raw_output = "|".join( + str(fields[slot]) + for slot in ("action", "device", "attribute", "value", "unit", "room", "floor") + ) + with self.subTest(attribute=attribute): + outcome = analyze_probe( + row, + "clear", + {"status": "ok", "raw_output": raw_output, "latency_ms": 1.0}, + ) + self.assertEqual(outcome.disposition, "safe_abstain") + self.assertEqual(outcome.reason, "non_executable_instruction") + + def test_b0_records_a_wrong_unique_dispatch_on_an_ambiguous_base(self) -> None: + row = next( + item for item in self.rows + if item["base_id"] == "eval-duplicate_entity-01" + ) + # Construct a unique grounding outcome for the wrong candidate, then + # execute it against the original ambiguous evaluation base. This + # isolates B0's no-clarification failure mode without weakening the + # production source-grounding checks merely to make a model hallucination + # look unique. + grounding_row = dict(row) + grounding_row["ambiguous_command"] = ( + "Turn off the Bedroom light on the Ground Floor." + ) + raw_output = "turnOff|Light|*|*|*|Bedroom|Ground Floor" + outcome = analyze_probe( + grounding_row, + "ambiguous", + {"status": "ok", "raw_output": raw_output, "latency_ms": 1.0}, + ) + self.assertTrue(outcome.is_unique) + + trial = _run_b0_trial(row, outcome) + + self.assertEqual(trial["interpretation"], "b0_unclarified_dispatch_on_gold_ambiguous_probe") + self.assertEqual(trial["sut_calls"], 1) + self.assertTrue(trial["dispatched"]) + self.assertTrue(trial["wrong_target_transition"]) + self.assertFalse(trial["exact_delta_success"]) + self.assertFalse(trial["oracle_pass"]) + + def test_b0_retains_malformed_raw_evidence_when_it_safely_abstains(self) -> None: + row = self.rows[0] + malformed = "not-a-seven-slot-domux-record" + outcome = analyze_probe( + row, + "ambiguous", + {"status": "ok", "raw_output": malformed, "latency_ms": 1.0}, + ) + self.assertEqual(outcome.disposition, "safe_abstain") + + trial = _run_b0_trial(row, outcome) + + self.assertEqual(trial["raw_output"], malformed) + self.assertTrue(trial["safe_abstention"]) + self.assertTrue(trial["oracle_pass"]) + + +class FailureDenominatorAndCliTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.rows = _validate_rows(_read_jsonl(DATASET, "dataset")) + + def test_inference_error_stays_in_all_formal_denominators(self) -> None: + failed_base = str(self.rows[0]["base_id"]) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + evidence, metadata = write_frozen_run( + root, + self.rows, + inference_error=(failed_base, "ambiguous"), + ) + report = evaluate_to_directory( + dataset_path=DATASET, + protocol_path=PROTOCOL, + freeze_path=FREEZE, + evidence_path=evidence, + metadata_path=metadata, + output_dir=root / "direct", + ) + language = report["metrics"]["language"] + self.assertEqual(language["sensitivity"]["denominator"], 48) + self.assertEqual(language["sensitivity"]["successes"], 47) + b2 = report["metrics"]["execution"][B2] + self.assertEqual(b2["ambiguous_clean_exact_delta_success"]["successes"], 47) + self.assertEqual(b2["ambiguous_clean_exact_delta_success"]["denominator"], 48) + for mutation, metric in b2["mutation_oracle_rates"].items(): + expected_denominator = 12 if mutation == "context_state_change" else 48 + self.assertEqual(metric["denominator"], expected_denominator) + self.assertEqual(b2["universal_guard_rate"]["denominator"], 48) + self.assertEqual(b2["universal_guard_rate"]["successes"], 47) + b0 = report["metrics"]["execution"][B0] + self.assertEqual(b0["safe_abstention_rate"]["denominator"], 48) + self.assertEqual(b0["safe_abstention_rate"]["successes"], 47) + self.assertEqual(report["quality_gate"]["result"], "fail") + self.assertTrue((root / "direct" / "report.json").is_file()) + self.assertTrue((root / "direct" / "trials.jsonl").is_file()) + + with redirect_stdout(io.StringIO()): + return_code = main([ + "--from-frozen-evidence", str(evidence), + "--evidence-metadata", str(metadata), + "--dataset", str(DATASET), + "--protocol", str(PROTOCOL), + "--freeze", str(FREEZE), + "--output-dir", str(root / "cli"), + ]) + self.assertEqual(return_code, 1) + self.assertTrue((root / "cli" / "report.json").is_file()) + + def test_formal_evaluation_refuses_missing_metadata(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + evidence, _metadata = write_frozen_run(root, self.rows) + with self.assertRaisesRegex(EvaluationInputError, "requires model-run metadata"): + evaluate_to_directory( + dataset_path=DATASET, + protocol_path=PROTOCOL, + freeze_path=FREEZE, + evidence_path=evidence, + metadata_path=None, + output_dir=root / "result", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/cases/domux-clarify-commit/tests/test_ha_acceptance.py b/cases/domux-clarify-commit/tests/test_ha_acceptance.py new file mode 100644 index 0000000..c50a816 --- /dev/null +++ b/cases/domux-clarify-commit/tests/test_ha_acceptance.py @@ -0,0 +1,1397 @@ +from __future__ import annotations + +import copy +import contextlib +import hashlib +import io +import json +import stat +import subprocess +import sys +import tarfile +import tempfile +import unittest +from dataclasses import replace +from pathlib import Path +from typing import Any +from unittest import mock +from urllib.error import URLError +from urllib.parse import urlparse + + +CASE_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(CASE_DIR)) + +import ha_acceptance as ha # noqa: E402 + + +RUN_ID = "a1b2c3d4e5f6" + + +def completed( + command: list[str], + returncode: int = 0, + *, + stdout: bytes = b"", + stderr: bytes = b"", +) -> subprocess.CompletedProcess[bytes]: + return subprocess.CompletedProcess(command, returncode, stdout, stderr) + + +class FakeDockerRunner: + """Stateful subprocess.run replacement; it never calls Docker.""" + + def __init__(self) -> None: + self.container_name = f"{ha.CONTAINER_PREFIX}-{RUN_ID}" + self.volume_name = f"{ha.VOLUME_PREFIX}-{RUN_ID}" + self.container_exists = False + self.volume_exists = False + self.container_label = RUN_ID + self.volume_label = RUN_ID + self.commands: list[tuple[str, ...]] = [] + self.cp_archive: bytes | None = None + + def _container_inspect(self) -> dict[str, Any]: + return { + "Name": f"/{self.container_name}", + "Config": { + "Image": ha.IMAGE_REFERENCE, + "Labels": {ha.RUN_LABEL: self.container_label}, + }, + "HostConfig": { + "Memory": ha.MEMORY_LIMIT_BYTES, + "NanoCpus": ha.NANO_CPUS, + "NetworkMode": "bridge", + "PidsLimit": ha.PIDS_LIMIT, + "PortBindings": { + f"{ha.CONTAINER_PORT}/tcp": [ + {"HostIp": "127.0.0.1", "HostPort": ""} + ] + }, + "Privileged": False, + "RestartPolicy": {"Name": "no"}, + }, + "Mounts": [ + { + "Destination": "/config", + "Name": self.volume_name, + "RW": True, + "Type": "volume", + } + ], + } + + def __call__( + self, + command: list[str], + *, + input: bytes | None, + stdout: int, + stderr: int, + check: bool, + timeout: int, + ) -> subprocess.CompletedProcess[bytes]: + del stdout, stderr, check, timeout + self.commands.append(tuple(command)) + self.assert_command_shape(command) + arguments = command[1:] + + if arguments[:1] == ["pull"]: + return completed(command, stdout=b"pulled\n") + if arguments[:2] == ["image", "inspect"]: + payload = [ + { + "Architecture": "amd64", + "Config": { + "Labels": {"io.hass.version": ha.HOME_ASSISTANT_VERSION} + }, + "Os": "linux", + "RepoDigests": [ha.IMAGE_REFERENCE], + } + ] + return completed(command, stdout=json.dumps(payload).encode()) + if arguments[:2] == ["volume", "create"]: + self.volume_exists = True + return completed(command, stdout=f"{self.volume_name}\n".encode()) + if arguments[:2] == ["volume", "inspect"]: + if not self.volume_exists: + return completed(command, 1, stderr=b"no such volume") + payload = [ + { + "Labels": {ha.RUN_LABEL: self.volume_label}, + "Name": self.volume_name, + } + ] + return completed(command, stdout=json.dumps(payload).encode()) + if arguments[:1] == ["create"]: + self.container_exists = True + return completed(command, stdout=b"synthetic-container-id\n") + if arguments[:2] == ["container", "inspect"]: + if not self.container_exists: + return completed(command, 1, stderr=b"no such container") + return completed( + command, stdout=json.dumps([self._container_inspect()]).encode() + ) + if arguments[:1] == ["cp"]: + if not self.container_exists or input is None: + return completed(command, 1, stderr=b"copy failed") + self.cp_archive = input + return completed(command) + if arguments[:1] == ["start"]: + return completed(command, stdout=f"{self.container_name}\n".encode()) + if arguments[:1] == ["port"]: + return completed(command, stdout=b"127.0.0.1:34567\n") + if arguments[:2] == ["rm", "--force"]: + self.container_exists = False + return completed(command, stdout=f"{self.container_name}\n".encode()) + if arguments[:2] == ["volume", "rm"]: + if self.container_exists: + return completed(command, 1, stderr=b"volume is in use") + self.volume_exists = False + return completed(command, stdout=f"{self.volume_name}\n".encode()) + raise AssertionError(f"unexpected synthetic Docker command: {arguments}") + + def assert_command_shape(self, command: list[str]) -> None: + if not command or command[0] != "docker": + raise AssertionError("test runner received a non-Docker command") + + +class FakeRestResponse: + def __init__(self, payload: object) -> None: + self.status = 200 + self._raw = json.dumps(payload).encode("utf-8") + + def __enter__(self) -> "FakeRestResponse": + return self + + def __exit__(self, *_args: object) -> None: + return None + + def read(self) -> bytes: + return self._raw + + +class FakeRestBackend: + """Mock HA HTTP transport shared by setup and the real REST adapter.""" + + def __init__(self) -> None: + self.states: dict[str, dict[str, Any]] = { + "light.bed_light": { + "attributes": { + "brightness": 180, + "color_temp_kelvin": 2631, + "max_color_temp_kelvin": 6500, + "min_color_temp_kelvin": 2000, + "supported_color_modes": ["color_temp", "hs"], + }, + "entity_id": "light.bed_light", + "state": "off", + }, + "light.ceiling_lights": { + "attributes": { + "brightness": 180, + "color_temp_kelvin": 2631, + "max_color_temp_kelvin": 6500, + "min_color_temp_kelvin": 2000, + "supported_color_modes": ["color_temp", "hs"], + }, + "entity_id": "light.ceiling_lights", + "state": "on", + }, + "cover.hall_window": { + "attributes": { + "current_position": 10, + "supported_features": 15, + }, + "entity_id": "cover.hall_window", + "state": "open", + }, + "climate.hvac": { + "attributes": { + "fan_mode": "on_high", + "fan_modes": ["on_low", "on_high", "auto_low", "auto_high", "off"], + "hvac_modes": ["off", "heat", "cool", "dry", "fan_only", "auto"], + "max_temp": 35.0, + "min_temp": 7.0, + "supported_features": 385, + "target_temp_step": 1.0, + "temperature": 21.0, + }, + "entity_id": "climate.hvac", + "state": "cool", + }, + } + self.setup_dispatches: list[dict[str, Any]] = [] + self.external_dispatches: list[dict[str, Any]] = [] + self.sut_http_dispatches: list[dict[str, Any]] = [] + self.service_events: list[dict[str, Any]] = [] + + def state(self, entity_id: str) -> dict[str, Any]: + result = json.loads(json.dumps(self.states[entity_id])) + if entity_id.startswith("light.") and result["state"] == "off": + # Pinned HA suppresses active light values while an entity is off. + result["attributes"]["brightness"] = None + result["attributes"]["color_temp_kelvin"] = None + return result + + def dispatch( + self, + domain: str, + service: str, + data: dict[str, Any], + *, + phase: str, + ) -> dict[str, Any]: + entity_id = str(data["entity_id"]) + state = self.states[entity_id] + attributes = state["attributes"] + if domain == "light" and service == "turn_on": + state["state"] = "on" + if "brightness_pct" in data: + attributes["brightness"] = round(float(data["brightness_pct"]) * 255 / 100) + if "color_temp_kelvin" in data: + attributes["color_temp_kelvin"] = round(float(data["color_temp_kelvin"])) + elif domain == "light" and service == "turn_off": + state["state"] = "off" + elif domain == "cover" and service == "set_cover_position": + position = round(float(data["position"])) + attributes["current_position"] = position + state["state"] = "closed" if position == 0 else "open" + elif domain == "climate" and service == "set_hvac_mode": + state["state"] = str(data["hvac_mode"]) + elif domain == "climate" and service == "set_temperature": + attributes["temperature"] = float(data["temperature"]) + else: + raise AssertionError(f"unexpected synthetic service: {domain}.{service}") + event = {"data": dict(data), "domain": domain, "service": service} + self.service_events.append({**event, "phase": phase}) + if phase == "setup": + self.setup_dispatches.append(event) + elif phase == "external_fault_injection": + self.external_dispatches.append(event) + elif phase == "sut": + self.sut_http_dispatches.append(event) + else: + raise AssertionError(f"unexpected synthetic service phase: {phase}") + return self.state(entity_id) + + def open(self, request: Any, *, timeout: float) -> FakeRestResponse: + del timeout + if request.headers.get("Authorization") != "Bearer synthetic-access-secret": + raise AssertionError("real adapter omitted its Bearer token") + path = urlparse(request.full_url).path + if request.method == "GET" and path.startswith("/api/states/"): + entity_id = path.removeprefix("/api/states/") + return FakeRestResponse(self.state(entity_id)) + if request.method == "GET" and path == "/api/config": + return FakeRestResponse({"unit_system": {"temperature": "°C"}}) + if request.method == "POST" and path.startswith("/api/services/"): + domain, service = path.removeprefix("/api/services/").split("/", 1) + data = json.loads(request.data) + self.dispatch(domain, service, data, phase="sut") + return FakeRestResponse([self.state(str(data["entity_id"]))]) + raise AssertionError(f"unexpected real-adapter request: {request.method} {path}") + + def adapter( + self, base_url: str, token: str + ) -> ha.HomeAssistantRESTAdapter: + adapter = ha.HomeAssistantRESTAdapter( + base_url, + token, + timeout_seconds=1, + poll_seconds=0.01, + ) + adapter._opener = self # type: ignore[attr-defined] + return adapter + + +class FakeHomeAssistantApi: + """In-memory onboarding/setup API; SUT dispatch uses the real adapter.""" + + access_token = "synthetic-access-secret" + refresh_token = "synthetic-refresh-secret" + + def __init__( + self, + base_url: str, + backend: FakeRestBackend | None = None, + ) -> None: + self.base_url = base_url + self.backend = backend or FakeRestBackend() + self.steps: set[str] = set() + self.revoked = False + self.direct_service_calls: list[dict[str, object]] = [] + + @staticmethod + def _onboarding(done: bool) -> list[dict[str, Any]]: + return [{"step": step, "done": done} for step in ha.ONBOARDING_STEPS] + + def wait_for_readiness(self) -> ha.HttpResult: + return ha.HttpResult(200, self._onboarding(False)) + + def wait_for_entities(self, entity_ids: Any, token: str) -> None: + self._assert_token(token) + if set(entity_ids) != set(self.backend.states): + raise AssertionError("runner requested unexpected entities") + + def wait_for_projection( + self, + entity_id: str, + token: str, + *, + state: str, + attributes: dict[str, object], + ) -> dict[str, Any]: + self._assert_token(token) + actual = self.backend.state(entity_id) + if actual["state"] != state or any( + not ha._scalar_matches(actual["attributes"].get(key), value) + for key, value in attributes.items() + ): + raise AssertionError("synthetic setup projection does not match") + return actual + + def call_service( + self, + domain: str, + service: str, + data: dict[str, Any], + token: str, + *, + evidence_phase: str, + ) -> ha.HttpResult: + self._assert_token(token) + payload = [ + self.backend.dispatch( + domain, + service, + data, + phase=evidence_phase, + ) + ] + self.direct_service_calls.append( + ha._direct_service_event( + evidence_phase, + domain, + service, + f"/api/services/{domain}/{service}", + 200, + ) + ) + return ha.HttpResult(200, payload) + + def _assert_token(self, token: str | None) -> None: + if token != self.access_token: + raise AssertionError("missing synthetic Bearer token") + + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + form_body: dict[str, str] | None = None, + token: str | None = None, + expected: Any = (200,), + timeout: float = 30, + ) -> ha.HttpResult: + del timeout + status = 200 + payload: Any = {} + if method == "GET" and path == "/api/": + if token is None: + status, payload = 401, {"message": "Unauthorized"} + else: + self._assert_token(token) + payload = {"message": "API running."} + elif method == "POST" and path == "/api/onboarding/users": + if not json_body or set(json_body) != { + "client_id", + "language", + "name", + "password", + "username", + }: + raise AssertionError("invalid user onboarding body") + self.steps.add("user") + payload = {"auth_code": "synthetic-auth-code"} + elif method == "POST" and path == "/auth/token": + if not form_body: + raise AssertionError("missing token form") + if form_body.get("grant_type") == "authorization_code": + payload = { + "access_token": self.access_token, + "expires_in": 1800, + "refresh_token": self.refresh_token, + "token_type": "Bearer", + } + elif form_body.get("grant_type") == "refresh_token": + if form_body.get("refresh_token") != self.refresh_token or not self.revoked: + raise AssertionError("refresh revocation was not tested correctly") + status, payload = 400, {"error": "invalid_grant"} + else: + raise AssertionError("unexpected token grant") + elif method == "POST" and path == "/api/onboarding/core_config": + self._assert_token(token) + self.steps.add("core_config") + elif method == "POST" and path == "/api/onboarding/integration": + self._assert_token(token) + if not json_body or not json_body["redirect_uri"].endswith("/auth-callback"): + raise AssertionError("invalid integration onboarding body") + self.steps.add("integration") + payload = {"auth_code": "synthetic-integration-code"} + elif method == "POST" and path == "/api/onboarding/analytics": + self._assert_token(token) + self.steps.add("analytics") + elif method == "GET" and path == "/api/onboarding": + payload = self._onboarding(set(ha.ONBOARDING_STEPS) == self.steps) + elif method == "GET" and path.startswith("/api/states/"): + self._assert_token(token) + entity_id = path.removeprefix("/api/states/") + payload = self.backend.state(entity_id) + elif method == "GET" and path == "/api/config": + self._assert_token(token) + payload = {"unit_system": {"temperature": "°C"}} + elif method == "POST" and path.startswith("/api/services/"): + self._assert_token(token) + if not json_body: + raise AssertionError("missing service body") + domain, service = path.removeprefix("/api/services/").split("/", 1) + payload = [ + self.backend.dispatch(domain, service, json_body, phase="setup") + ] + elif method == "POST" and path == "/auth/revoke": + if not form_body or form_body.get("token") != self.refresh_token: + raise AssertionError("wrong refresh token was revoked") + self.revoked = True + payload = None + else: + raise AssertionError(f"unexpected synthetic HA request: {method} {path}") + + if status not in set(expected): + raise ha.AcceptanceError(f"{method} {path} returned unexpected HTTP {status}") + return ha.HttpResult(status, payload) + + +class FakeDockerRuntime: + def __init__(self, *, fail: bool = False) -> None: + self.cleaned = False + self.fail = fail + + def prepare(self) -> ha.PreparedRuntime: + if self.fail: + raise ha.AcceptanceError("synthetic prepare failure") + return ha.PreparedRuntime( + base_url="http://127.0.0.1:45678", + image={ + "architecture": "amd64", + "docker_healthcheck": False, + "manifest_digest": ha.IMAGE_DIGEST, + "operating_system": "linux", + "repository": ha.IMAGE_REPOSITORY, + "version": ha.HOME_ASSISTANT_VERSION, + }, + ) + + def cleanup(self) -> None: + self.cleaned = True + + +class RecordedDomuxEvidenceTests(unittest.TestCase): + def test_loader_binds_the_four_exact_v1_command_output_pairs(self) -> None: + records = ha.load_recorded_domux_evidence() + expected = { + ha.DomuxEvidenceKey("eval-duplicate_entity-01", "ambiguous"): ( + "Turn off the light.", + "turnOff|Light|*|*|*|*|*", + 2, + ), + ha.DomuxEvidenceKey("eval-duplicate_entity-02", "clear"): ( + "Set the Curtain in the Hall on the First Floor to 20 percent.", + "set|Curtain|position|20|Percent|Hall|First Floor", + 3, + ), + ha.DomuxEvidenceKey("eval-duplicate_entity-03", "clear"): ( + "Set the AC in the Bedroom on the Second Floor to 22 Celsius.", + "set|AC|temperature|22|Celsius|Bedroom|Second Floor", + 5, + ), + ha.DomuxEvidenceKey("eval-duplicate_entity-04", "clear"): ( + ("Set the Light in the Study on the Ground Floor to 35 percent brightness."), + "set|Light|brightness|35|Percent|Study|Ground Floor", + 7, + ), + } + self.assertEqual(set(records), set(expected)) + for key, (command, raw_output, line_number) in expected.items(): + record = records[key] + self.assertEqual(record.command, command) + self.assertEqual(record.raw_output, raw_output) + self.assertEqual(record.line_number, line_number) + self.assertEqual(record.query_sha256, ha._sha256_text(command)) + self.assertEqual( + record.raw_output_sha256, + ha._sha256_text(raw_output), + ) + self.assertEqual( + record.artifact_sha256, + ha.V1_DOMUX_EVIDENCE_SHA256, + ) + self.assertTrue(record.provenance()["pair_verified"]) + + def test_loader_rejects_a_modified_v1_artifact(self) -> None: + payload = ha.V1_DOMUX_EVIDENCE_PATH.read_bytes().replace( + b"Turn off the light.", + b"Turn off a light.", + 1, + ) + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "domux_raw.jsonl" + path.write_bytes(payload) + with self.assertRaisesRegex( + ha.AcceptanceError, + "artifact digest does not match", + ): + ha.load_recorded_domux_evidence(path) + + def test_injected_v1_record_with_wrong_line_number_is_rejected(self) -> None: + records = ha.load_recorded_domux_evidence() + key = ha.DomuxEvidenceKey("eval-duplicate_entity-01", "ambiguous") + records[key] = replace(records[key], line_number=999) + with self.assertRaisesRegex( + ha.AcceptanceError, + "not bound to verified v1 Domux evidence", + ): + ha._evidence_for_case(records, key) + + def test_loader_checks_field_hashes_after_whole_artifact_hash(self) -> None: + payload = ha.V1_DOMUX_EVIDENCE_PATH.read_bytes().replace( + b"Turn off the light.", + b"Turn off a light.", + 1, + ) + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "domux_raw.jsonl" + path.write_bytes(payload) + with mock.patch.object( + ha, + "V1_DOMUX_EVIDENCE_SHA256", + ha._sha256_bytes(payload), + ): + with self.assertRaisesRegex( + ha.AcceptanceError, + "command/output field digest does not match", + ): + ha.load_recorded_domux_evidence(path) + + def test_evidence_failure_prevents_docker_prepare(self) -> None: + docker = mock.Mock() + + def fail_evidence() -> dict[ha.DomuxEvidenceKey, ha.RecordedDomuxEvidence]: + raise ha.AcceptanceError("synthetic evidence failure") + + with self.assertRaisesRegex(ha.AcceptanceError, "synthetic evidence failure"): + ha.execute_acceptance( + docker, + recorded_evidence_loader=fail_evidence, + ) + docker.prepare.assert_not_called() + docker.cleanup.assert_called_once_with() + + +class FrozenScenarioEvidenceTests(unittest.TestCase): + def test_loader_binds_exact_rows_gold_and_target_semantics(self) -> None: + scenarios = ha.load_frozen_scenario_evidence() + expected = { + "eval-duplicate_entity-01": ( + 17, + "8b8f6498aef6bb4cb7fd5af3ac9adbef3e2cf0013c7a22451fc339e5f85cd871", + "light.eval_de_01_living", + "Turn off the light.", + 3, + ), + "eval-duplicate_entity-02": ( + 18, + "24f7b968a30cba095932b3f5da47ce8f8fc65d521546a830de1b6ea149db7fb4", + "cover.eval_de_02_upstairs_hall", + "Set the Curtain in the Hall on the First Floor to 20 percent.", + 2, + ), + "eval-duplicate_entity-03": ( + 19, + "ef0be54be7aaae140c846076fe9568dbea981fd6a96575b12cb6633057756bdf", + "climate.eval_de_03_bedroom_second", + "Set the AC in the Bedroom on the Second Floor to 22 Celsius.", + 2, + ), + "eval-duplicate_entity-04": ( + 20, + "d9b32826a3e0229d5943e69a8ba3192635194f5ba87120945a6a92274e24c78e", + "light.eval_de_04_study", + "Set the Light in the Study on the Ground Floor to 35 percent brightness.", + 3, + ), + } + self.assertEqual(set(scenarios), set(expected)) + for base_id, (line, row_sha, target, variant_command, count) in expected.items(): + scenario = scenarios[base_id] + self.assertEqual(scenario.line_number, line) + self.assertEqual(scenario.row_sha256, row_sha) + self.assertEqual(scenario.expected_target_entity_id, target) + variant = "ambiguous" if base_id.endswith("01") else "clear" + self.assertEqual(scenario.command_for_variant(variant), variant_command) + self.assertEqual(len(scenario.candidate_entity_ids), count) + self.assertEqual( + scenario.artifact_sha256, + ha.SCENARIO_EVIDENCE_SHA256, + ) + self.assertEqual( + scenario.binding_sha256(), + ha.REQUIRED_FROZEN_SCENARIOS[base_id].binding_sha256, + ) + clarified = scenarios["eval-duplicate_entity-01"] + self.assertEqual( + clarified.clarification_answer, + "The Living Room light on the Ground Floor.", + ) + self.assertEqual( + clarified.confirmed_instruction.to_pipe(), + "turnOff|Light|*|*|*|Living Room|Ground Floor", + ) + self.assertEqual( + clarified.target_inventory_semantics, + { + "aliases": (), + "device": "Light", + "domain": "light", + "floor": "Ground Floor", + "room": "Living Room", + }, + ) + + def test_loader_rejects_modified_scenario_artifact_and_row(self) -> None: + payload = ha.SCENARIO_EVIDENCE_PATH.read_bytes().replace( + b"The Living Room light on the Ground Floor.", + b"The Bedroom light on the Ground Floor.", + 1, + ) + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "scenarios.jsonl" + path.write_bytes(payload) + with self.assertRaisesRegex( + ha.AcceptanceError, + "scenario artifact digest does not match", + ): + ha.load_frozen_scenario_evidence(path) + with mock.patch.object( + ha, + "SCENARIO_EVIDENCE_SHA256", + ha._sha256_bytes(payload), + ): + with self.assertRaisesRegex( + ha.AcceptanceError, + "scenario row digest does not match", + ): + ha.load_frozen_scenario_evidence(path) + + def test_injected_scenario_gold_change_is_rejected_by_binding(self) -> None: + recorded = ha.load_recorded_domux_evidence() + scenarios = ha.load_frozen_scenario_evidence() + base_id = "eval-duplicate_entity-01" + scenarios[base_id] = replace( + scenarios[base_id], + clarification_answer="The Bedroom light on the Ground Floor.", + ) + with self.assertRaisesRegex( + ha.AcceptanceError, + "not bound to a verified frozen scenario", + ): + ha.validate_acceptance_evidence(recorded, scenarios) + + def test_scenario_failure_prevents_docker_prepare(self) -> None: + docker = mock.Mock() + + def fail_scenario() -> dict[str, ha.FrozenScenarioEvidence]: + raise ha.AcceptanceError("synthetic scenario failure") + + with self.assertRaisesRegex(ha.AcceptanceError, "synthetic scenario failure"): + ha.execute_acceptance( + docker, + scenario_evidence_loader=fail_scenario, + ) + docker.prepare.assert_not_called() + docker.cleanup.assert_called_once_with() + + +class PinnedImageAndConfigTests(unittest.TestCase): + def test_official_amd64_image_is_pinned_to_the_required_manifest(self) -> None: + self.assertEqual( + ha.IMAGE_REFERENCE, + "ghcr.io/home-assistant/home-assistant@" + "sha256:8e9751cb66d3ba6624f5360a7d31b0c6821f7f5b3fb8ba0d10d58f0f481c540c", + ) + self.assertEqual(ha.HOME_ASSISTANT_VERSION, "2026.8.3") + self.assertEqual(ha.PLATFORM, "linux/amd64") + + def test_configuration_is_an_in_memory_single_file_archive(self) -> None: + archive_bytes = ha.configuration_archive() + with tarfile.open(fileobj=io.BytesIO(archive_bytes), mode="r:") as archive: + members = archive.getmembers() + self.assertEqual([member.name for member in members], ["configuration.yaml"]) + content = archive.extractfile(members[0]).read().decode("utf-8") # type: ignore[union-attr] + self.assertEqual(content, ha.CONFIGURATION_YAML) + self.assertIn("demo:\n", content) + self.assertIn("onboarding:\n", content) + self.assertNotIn("password", content.lower()) + self.assertNotIn("token", content.lower()) + + +class DockerCliTests(unittest.TestCase): + def test_prepare_and_cleanup_use_one_labelled_container_and_volume(self) -> None: + runner = FakeDockerRunner() + docker = ha.DockerCli(runner=runner, run_id=RUN_ID) + + runtime = docker.prepare() + self.assertEqual(runtime.base_url, "http://127.0.0.1:34567") + self.assertEqual(runtime.image["manifest_digest"], ha.IMAGE_DIGEST) + self.assertTrue(runner.container_exists) + self.assertTrue(runner.volume_exists) + self.assertIsNotNone(runner.cp_archive) + + create = next(command for command in runner.commands if command[1] == "create") + self.assertIn("--restart=no", create) + self.assertIn("--cpus=1.5", create) + self.assertIn("--memory=2g", create) + self.assertIn("--pids-limit=512", create) + self.assertIn("127.0.0.1::8123", create) + self.assertEqual(create[-1], ha.IMAGE_REFERENCE) + self.assertEqual(sum(command[1] == "create" for command in runner.commands), 1) + self.assertEqual( + sum(command[1:3] == ("volume", "create") for command in runner.commands), + 1, + ) + + docker.cleanup() + self.assertFalse(runner.container_exists) + self.assertFalse(runner.volume_exists) + self.assertFalse(any("prune" in command for command in runner.commands)) + + def test_cleanup_refuses_resources_with_a_different_run_label(self) -> None: + runner = FakeDockerRunner() + docker = ha.DockerCli(runner=runner, run_id=RUN_ID) + docker.prepare() + runner.container_label = "0" * 12 + + with self.assertRaisesRegex(ha.AcceptanceError, "failed to clean"): + docker.cleanup() + + self.assertTrue(runner.container_exists) + self.assertTrue(runner.volume_exists) + self.assertFalse( + any(command[1:3] == ("rm", "--force") for command in runner.commands) + ) + + +class HomeAssistantWorkflowTests(unittest.TestCase): + def test_full_mocked_sut_pipeline_is_controlled_one_time_and_redacted(self) -> None: + backend = FakeRestBackend() + initial_light = backend.state("light.bed_light") + self.assertIsNone(initial_light["attributes"]["brightness"]) + self.assertIsNone(initial_light["attributes"]["color_temp_kelvin"]) + self.assertNotIn( + "temperature_unit", + backend.state("climate.hvac")["attributes"], + ) + api = FakeHomeAssistantApi("http://127.0.0.1:45678", backend) + result = ha.exercise_home_assistant( + api, + credential_factory=lambda: ("synthetic-user", "synthetic-password"), + rest_adapter_factory=backend.adapter, + ) + + self.assertEqual(result["auth"]["ttl_seconds"], 1800) + self.assertEqual(result["auth"]["refresh_after_revoke_http"], 400) + setup = result["phases"]["setup"] + sut = result["phases"]["sut"] + self.assertEqual(setup["classification"], "direct_rest_state_normalization") + self.assertFalse(setup["included_in_sut_dispatch_count"]) + self.assertEqual(len(setup["dispatches"]), 5) + self.assertEqual(len(backend.setup_dispatches), 5) + self.assertEqual(len(backend.external_dispatches), 1) + self.assertEqual( + backend.external_dispatches[0], + { + "data": { + "brightness_pct": 25.0, + "entity_id": "light.bed_light", + }, + "domain": "light", + "service": "turn_on", + }, + ) + self.assertEqual(sut["classification"], "clarify_commit_sut") + self.assertEqual(sut["adapter"], "HomeAssistantRESTAdapter") + self.assertEqual(sut["case_count"], 4) + self.assertEqual(sut["successful_transition_count"], 3) + self.assertEqual(sut["rejected_before_dispatch_count"], 1) + self.assertEqual(sut["external_fault_injection_count"], 1) + self.assertEqual(sut["sut_dispatch_total"], 3) + self.assertEqual( + sut["domux_evidence"], + { + "artifact": "evidence/v1/domux_raw.jsonl", + "artifact_sha256": ha.V1_DOMUX_EVIDENCE_SHA256, + "pair_count": 4, + "validation": "whole_artifact_and_per_field_sha256", + }, + ) + self.assertEqual( + sut["scenario_evidence"], + { + "artifact": "data/scenarios.jsonl", + "artifact_sha256": ha.SCENARIO_EVIDENCE_SHA256, + "case_count": 4, + "ha_registry_profile": ( + "semantic_target_mapping_subset_not_full_scenario_inventory" + ), + }, + ) + self.assertEqual(len(backend.sut_http_dispatches), 3) + self.assertEqual( + [item["domain"] for item in sut["cases"]], + ["light", "cover", "climate", "light"], + ) + self.assertEqual( + [item["grounding"]["resolution"] for item in sut["cases"]], + [ + "resolve_clarification_submission", + "resolve_unique_request", + "resolve_unique_request", + "resolve_unique_request", + ], + ) + self.assertEqual( + [ + ( + item["domux_evidence"]["base_id"], + item["domux_evidence"]["variant"], + ) + for item in sut["cases"] + ], + [ + ("eval-duplicate_entity-01", "ambiguous"), + ("eval-duplicate_entity-02", "clear"), + ("eval-duplicate_entity-03", "clear"), + ("eval-duplicate_entity-04", "clear"), + ], + ) + for item in sut["cases"]: + provenance = item["domux_evidence"] + self.assertTrue(provenance["pair_verified"]) + self.assertEqual( + provenance["artifact_sha256"], + ha.V1_DOMUX_EVIDENCE_SHA256, + ) + self.assertEqual( + provenance["validation"], + "whole_artifact_and_per_field_sha256", + ) + self.assertNotIn(str(ha.CASE_DIR), json.dumps(provenance)) + scenario_targets = [ + ( + "light.eval_de_01_living", + "light.ceiling_lights", + 3, + 2, + ), + ( + "cover.eval_de_02_upstairs_hall", + "cover.hall_window", + 2, + 1, + ), + ( + "climate.eval_de_03_bedroom_second", + "climate.hvac", + 2, + 1, + ), + ( + "light.eval_de_04_study", + "light.bed_light", + 3, + 1, + ), + ] + for index, item in enumerate(sut["cases"]): + scenario = item["scenario_provenance"] + scenario_target, ha_target, frozen_count, ha_count = scenario_targets[index] + self.assertEqual(item["ha_registry_profile"], ha.HA_REGISTRY_PROFILE) + self.assertEqual(scenario["ha_registry_profile"], ha.HA_REGISTRY_PROFILE) + self.assertEqual(scenario["source"], "frozen_synthetic_scenario_gold") + self.assertFalse(scenario["post_clarification_model_call"]) + self.assertEqual(scenario["frozen_candidate_count"], frozen_count) + self.assertEqual(scenario["ha_matching_candidate_count"], ha_count) + self.assertEqual( + scenario["inventory_limitation"], + { + "full_scenario_inventory_reproduced": False, + "profile": ha.HA_REGISTRY_PROFILE, + }, + ) + self.assertEqual( + scenario["scenario_target_to_ha_demo_entity"], + { + "ha_demo_entity_id": ha_target, + "scenario_target_entity_id": scenario_target, + "semantic_fields_match": True, + }, + ) + self.assertEqual(scenario["used_for_resolution"], index == 0) + self.assertEqual(len(scenario["row_sha256"]), 64) + self.assertEqual(len(scenario["binding_sha256"]), 64) + self.assertNotIn(str(ha.CASE_DIR), json.dumps(scenario)) + clarified_scenario = sut["cases"][0]["scenario_provenance"] + self.assertEqual( + clarified_scenario["clarification_answer"], + "The Living Room light on the Ground Floor.", + ) + self.assertEqual( + clarified_scenario["confirmed_instruction"], + { + "action": "turnOff", + "attribute": "*", + "device": "Light", + "floor": "Ground Floor", + "room": "Living Room", + "unit": "*", + "value": "*", + }, + ) + self.assertEqual( + [item["service_shape"]["service"] for item in sut["cases"]], + ["turn_off", "set_cover_position", "set_temperature", "turn_on"], + ) + self.assertEqual( + backend.sut_http_dispatches, + [ + { + "data": { + "entity_id": "light.ceiling_lights", + }, + "domain": "light", + "service": "turn_off", + }, + { + "data": { + "entity_id": "cover.hall_window", + "position": 20, + }, + "domain": "cover", + "service": "set_cover_position", + }, + { + "data": { + "entity_id": "climate.hvac", + "temperature": 22.0, + }, + "domain": "climate", + "service": "set_temperature", + }, + ], + ) + self.assertEqual( + [event["phase"] for event in backend.service_events], + ["setup"] * 5 + ["sut"] * 3 + ["external_fault_injection"], + ) + accounting = result["phases"]["service_call_accounting"] + self.assertEqual(accounting["external_fault_injection"], 1) + self.assertEqual(accounting["setup_direct_rest"], 5) + self.assertEqual(accounting["sut_dispatches"], 3) + self.assertEqual(accounting["total"], 9) + self.assertEqual(accounting["direct_rest_events"], api.direct_service_calls) + self.assertEqual( + [event["phase"] for event in accounting["direct_rest_events"]], + ["setup"] * 5 + ["external_fault_injection"], + ) + expected_transitions = { + "recorded_ambiguous_light_off": ( + {"brightness": 178, "state": "on"}, + {"brightness": None, "state": "off"}, + ), + "recorded_unique_cover_position": ( + {"current_position": 80, "state": "open"}, + {"current_position": 20, "state": "open"}, + ), + "recorded_unique_climate_temperature": ( + {"state": "cool", "temperature": 24}, + {"state": "cool", "temperature": 22}, + ), + } + for item in sut["cases"][:3]: + expected_before, expected_after = expected_transitions[item["case"]] + for key, value in expected_before.items(): + self.assertTrue( + ha._scalar_matches(item["controlled_before"][key], value) + ) + for key, value in expected_after.items(): + self.assertTrue( + ha._scalar_matches(item["controlled_after"][key], value) + ) + self.assertTrue(item["postcondition"]["matched_prepared_projection"]) + self.assertTrue(item["postcondition"]["all_registered_entities_exact"]) + self.assertEqual(item["postcondition"]["status"], "COMMITTED") + self.assertFalse(item["replay"]["accepted"]) + self.assertFalse(item["replay"]["dispatched"]) + self.assertEqual(item["replay"]["reason"], "replayed_nonce") + self.assertEqual(item["replay"]["sut_dispatch_delta"], 0) + self.assertEqual(item["outcome"], "COMMITTED") + + drift = sut["cases"][3] + self.assertEqual( + drift["case"], + "recorded_study_light_state_drift_rejected", + ) + self.assertEqual(drift["outcome"], "REJECTED_BEFORE_DISPATCH") + self.assertEqual( + drift["external_mutation"], + { + "classification": "out_of_band_fault_injection", + "data": { + "brightness_pct": 25.0, + "entity_id": "light.bed_light", + }, + "domain": "light", + "http_status": 200, + "included_in_sut_dispatch_count": False, + "observed_path": "/api/states/light.bed_light", + "request_path": "/api/services/light/turn_on", + "service": "turn_on", + "transport": "home_assistant_rest_api", + }, + ) + self.assertEqual( + drift["rejection"], + { + "acknowledged": False, + "accepted": False, + "dispatched": False, + "outcome_unknown": False, + "reason": "state_changed", + "status": "INVALIDATED", + "sut_dispatch_delta": 0, + }, + ) + self.assertEqual( + drift["controlled_before_external_mutation"]["brightness"], + 166, + ) + self.assertEqual( + drift["controlled_after_external_mutation"]["brightness"], + 64, + ) + self.assertNotEqual( + drift["controlled_before_external_mutation"]["entity_id"], + sut["cases"][0]["controlled_after"]["entity_id"], + ) + binding = drift["binding"] + self.assertTrue(binding["matched_before_external_mutation"]) + self.assertTrue(binding["changed_after_external_mutation"]) + self.assertEqual( + binding["prepared_state_digest"], + binding["before_external_mutation_state_digest"], + ) + self.assertNotEqual( + binding["prepared_state_digest"], + binding["after_external_mutation_state_digest"], + ) + for field in ( + "prepared_state_digest", + "before_external_mutation_state_digest", + "after_external_mutation_state_digest", + ): + self.assertEqual(len(binding[field]), 64) + self.assertTrue(set(binding[field]) <= set("0123456789abcdef")) + serialized = json.dumps(result, sort_keys=True) + for private in ( + "synthetic-user", + "synthetic-password", + "synthetic-auth-code", + "synthetic-integration-code", + api.access_token, + api.refresh_token, + "45678", + ): + self.assertNotIn(private, serialized) + + def test_target_drift_injection_must_be_external_and_observed(self) -> None: + for mode, expected_error in ( + ("sut_adapter", "counted as a SUT dispatch"), + ("no_op", "mutated state"), + ): + with self.subTest(mode=mode): + backend = FakeRestBackend() + api = FakeHomeAssistantApi("http://127.0.0.1:45678", backend) + ha.normalize_setup_state(api, api.access_token) + adapter = backend.adapter(api.base_url, api.access_token) + + def mutate(case: ha.TargetDriftCase) -> dict[str, object]: + if mode == "sut_adapter": + adapter.call_service( + case.domain, + case.mutation_service, + case.mutation_service_data, + ) + return { + "classification": "out_of_band_fault_injection", + "data": dict(case.mutation_service_data), + "domain": case.domain, + "http_status": 200, + "included_in_sut_dispatch_count": False, + "observed_path": f"/api/states/{case.entity_id}", + "request_path": ( + f"/api/services/{case.domain}/{case.mutation_service}" + ), + "service": case.mutation_service, + "transport": "home_assistant_rest_api", + } + + with self.assertRaisesRegex(ha.AcceptanceError, expected_error): + ha.run_sut_cases( + adapter, + recorded_evidence=ha.load_recorded_domux_evidence(), + scenario_evidence=ha.load_frozen_scenario_evidence(), + mutate_target=mutate, + ) + + def test_execute_always_cleans_runtime_on_success_and_failure(self) -> None: + backend = FakeRestBackend() + successful = FakeDockerRuntime() + result = ha.execute_acceptance( + successful, + api_factory=lambda base_url: FakeHomeAssistantApi(base_url, backend), + credential_factory=lambda: ("synthetic-user", "synthetic-password"), + rest_adapter_factory=backend.adapter, + ) + self.assertTrue(successful.cleaned) + self.assertEqual(result["status"], "passed") + self.assertEqual(result["schema_version"], 4) + provenance = result["execution_source_bindings"] + self.assertEqual( + set(provenance), + { + "binding_bundle_sha256", + "digest_algorithm", + "host_python", + "inputs", + "module_origins_verified", + "path_base", + "pre_post_execution_match", + "schema_version", + "sources", + }, + ) + self.assertEqual(provenance["schema_version"], 1) + self.assertEqual(provenance["digest_algorithm"], "sha256") + self.assertEqual(provenance["path_base"], "case_directory") + self.assertTrue(provenance["module_origins_verified"]) + self.assertTrue(provenance["pre_post_execution_match"]) + self.assertEqual( + provenance["host_python"], + { + "implementation": sys.implementation.name, + "version": ".".join(str(value) for value in sys.version_info[:3]), + }, + ) + self.assertEqual( + set(provenance["sources"]), + set(ha.EXECUTION_SOURCE_PATHS), + ) + self.assertEqual( + set(provenance["inputs"]), + set(ha.EXECUTION_INPUT_PATHS), + ) + for group, paths in ( + ("inputs", ha.EXECUTION_INPUT_PATHS), + ("sources", ha.EXECUTION_SOURCE_PATHS), + ): + for relative in paths: + payload = (ha.CASE_DIR / relative).read_bytes() + self.assertEqual( + provenance[group][relative], + { + "sha256": hashlib.sha256(payload).hexdigest(), + "size_bytes": len(payload), + }, + ) + bundle_payload = json.dumps( + {"inputs": provenance["inputs"], "sources": provenance["sources"]}, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + self.assertEqual( + provenance["binding_bundle_sha256"], + hashlib.sha256(bundle_payload).hexdigest(), + ) + + failing = FakeDockerRuntime(fail=True) + with self.assertRaisesRegex(ha.AcceptanceError, "prepare failure"): + ha.execute_acceptance( + failing, + api_factory=lambda base_url: FakeHomeAssistantApi(base_url), + ) + self.assertTrue(failing.cleaned) + + def test_production_direct_service_call_records_redacted_success(self) -> None: + observed: dict[str, object] = {} + + def recording_opener(request: Any, *, timeout: float) -> FakeRestResponse: + observed.update({ + "authorization": request.headers.get("Authorization"), + "body": json.loads(request.data), + "method": request.method, + "path": urlparse(request.full_url).path, + "timeout": timeout, + }) + return FakeRestResponse([]) + + api = ha.HomeAssistantApi( + "http://127.0.0.1:45678", + opener=recording_opener, + ) + result = api.call_service( + "light", + "turn_on", + {"brightness_pct": 25, "entity_id": "light.bed_light"}, + "synthetic-access-secret", + evidence_phase="external_fault_injection", + ) + self.assertEqual(result.status, 200) + self.assertEqual(observed["method"], "POST") + self.assertEqual(observed["path"], "/api/services/light/turn_on") + self.assertEqual(observed["authorization"], "Bearer synthetic-access-secret") + self.assertEqual( + observed["body"], + {"brightness_pct": 25, "entity_id": "light.bed_light"}, + ) + self.assertEqual( + api.direct_service_calls, + [{ + "domain": "light", + "http_status": 200, + "phase": "external_fault_injection", + "request_path": "/api/services/light/turn_on", + "service": "turn_on", + }], + ) + self.assertNotIn("synthetic-access-secret", json.dumps(api.direct_service_calls)) + + def test_execute_rejects_execution_file_drift_and_still_cleans(self) -> None: + before = ha.capture_execution_bindings() + after = copy.deepcopy(before) + after["sources"]["ha_acceptance.py"]["sha256"] = "0" * 64 + runtime = FakeDockerRuntime() + backend = FakeRestBackend() + + with mock.patch.object( + ha, + "capture_execution_bindings", + side_effect=(before, after), + ), self.assertRaisesRegex( + ha.AcceptanceError, + "execution sources or inputs changed during acceptance", + ): + ha.execute_acceptance( + runtime, + api_factory=lambda base_url: FakeHomeAssistantApi(base_url, backend), + credential_factory=lambda: ( + "synthetic-user", + "synthetic-password", + ), + rest_adapter_factory=backend.adapter, + ) + self.assertTrue(runtime.cleaned) + + def test_http_transport_failure_does_not_echo_url_or_bearer(self) -> None: + def failing_opener(*_args: Any, **_kwargs: Any) -> Any: + raise URLError("synthetic-secret at http://127.0.0.1:45678") + + api = ha.HomeAssistantApi( + "http://127.0.0.1:45678", opener=failing_opener + ) + with self.assertRaises(ha.AcceptanceError) as caught: + api.request("GET", "/api/", token="synthetic-bearer-secret") + message = str(caught.exception) + self.assertNotIn("synthetic-secret", message) + self.assertNotIn("synthetic-bearer-secret", message) + self.assertNotIn("45678", message) + + +class OutputAndCliTests(unittest.TestCase): + def test_atomic_json_is_canonical_private_and_leaves_no_staging_files(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + output = Path(temporary) / "nested" / "acceptance.json" + payload = ha.atomic_write_json(output, {"z": 2, "a": 1}) + + self.assertEqual(payload, b'{\n "a": 1,\n "z": 2\n}\n') + self.assertEqual(output.read_bytes(), payload) + self.assertEqual(stat.S_IMODE(output.stat().st_mode), 0o600) + self.assertEqual(list(output.parent.glob("*.tmp")), []) + + def test_atomic_json_removes_staging_file_when_publish_fails(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + output = Path(temporary) / "acceptance.json" + with mock.patch.object(ha.os, "replace", side_effect=OSError("synthetic")): + with self.assertRaises(OSError): + ha.atomic_write_json(output, {"status": "passed"}) + self.assertFalse(output.exists()) + self.assertEqual(list(Path(temporary).glob("*.tmp")), []) + + def test_cli_writes_same_redacted_json_to_file_and_stdout(self) -> None: + runtime = FakeDockerRuntime() + backend = FakeRestBackend() + with tempfile.TemporaryDirectory() as temporary: + output = Path(temporary) / "acceptance.json" + stdout = io.StringIO() + stderr = io.StringIO() + with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr): + returncode = ha.main( + ["--output", str(output)], + docker_factory=lambda: runtime, + api_factory=lambda base_url: FakeHomeAssistantApi( + base_url, backend + ), + credential_factory=lambda: ( + "synthetic-user", + "synthetic-password", + ), + rest_adapter_factory=backend.adapter, + ) + self.assertEqual(returncode, 0) + self.assertEqual(stderr.getvalue(), "") + self.assertEqual(stdout.getvalue().encode(), output.read_bytes()) + self.assertNotIn(str(output.parent), stdout.getvalue()) + self.assertNotIn("45678", stdout.getvalue()) + + def test_cli_returns_nonzero_and_does_not_publish_on_failure(self) -> None: + runtime = FakeDockerRuntime(fail=True) + with tempfile.TemporaryDirectory() as temporary: + output = Path(temporary) / "acceptance.json" + stdout = io.StringIO() + stderr = io.StringIO() + with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr): + returncode = ha.main( + ["--output", str(output)], + docker_factory=lambda: runtime, + api_factory=FakeHomeAssistantApi, + ) + self.assertEqual(returncode, 1) + self.assertEqual(stdout.getvalue(), "") + self.assertFalse(output.exists()) + self.assertTrue(runtime.cleaned) + self.assertIn("ha_acceptance failed", stderr.getvalue()) + + +if __name__ == "__main__": + unittest.main() diff --git a/cases/domux-clarify-commit/tests/test_run_model.py b/cases/domux-clarify-commit/tests/test_run_model.py new file mode 100644 index 0000000..0bc1f5d --- /dev/null +++ b/cases/domux-clarify-commit/tests/test_run_model.py @@ -0,0 +1,226 @@ +from __future__ import annotations + +import hashlib +import sys +import tempfile +import unittest +from pathlib import Path + + +CASE_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(CASE_DIR)) + +from run_model import ( # noqa: E402 + MODEL_REVISION, + atomic_write_pair, + canonical_json, + git_blob_sha1, + sha256_file, + sha256_text, + snapshot_manifest, + validate_paths, +) + + +class SnapshotManifestTests(unittest.TestCase): + def _fixture( + self, + root: Path, + ) -> tuple[Path, dict[str, tuple[int, str]], dict[str, bytes]]: + snapshot = root / "snapshot" + snapshot.mkdir() + payloads = { + "config.json": b'{"model":"synthetic"}\n', + "weights.bin": b"small synthetic lfs payload\n", + } + expected_files = { + "config.json": (len(payloads["config.json"]), ""), + "weights.bin": ( + len(payloads["weights.bin"]), + hashlib.sha256(payloads["weights.bin"]).hexdigest(), + ), + } + for name, payload in payloads.items(): + path = snapshot / name + path.write_bytes(payload) + expected_files["config.json"] = ( + len(payloads["config.json"]), + git_blob_sha1(snapshot / "config.json"), + ) + metadata_root = snapshot / ".cache" / "huggingface" / "download" + metadata_root.mkdir(parents=True) + for name, (_size, etag) in expected_files.items(): + (metadata_root / f"{name}.metadata").write_text( + f"{MODEL_REVISION}\n{etag}\nsynthetic-timestamp\n", + encoding="utf-8", + ) + return snapshot, expected_files, payloads + + def test_snapshot_manifest_verifies_revision_etag_size_and_content_hashes(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + snapshot, expected_files, payloads = self._fixture(Path(temporary)) + + entries, manifest_sha256 = snapshot_manifest( + snapshot, + expected_files=expected_files, + expected_manifest_sha256=None, + ) + + self.assertEqual([entry["name"] for entry in entries], sorted(payloads)) + for entry in entries: + name = str(entry["name"]) + expected_size, expected_etag = expected_files[name] + self.assertEqual(entry["size_bytes"], expected_size) + self.assertEqual(entry["hub_etag"], expected_etag) + self.assertEqual(entry["sha256"], sha256_file(snapshot / name)) + self.assertEqual(manifest_sha256, sha256_text(canonical_json(entries))) + + def test_snapshot_manifest_rejects_revision_etag_size_and_hash_mismatches(self) -> None: + cases = ("revision", "etag", "size", "hash") + for mutation in cases: + with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as temporary: + snapshot, expected_files, payloads = self._fixture(Path(temporary)) + if mutation == "revision": + metadata = snapshot / ".cache/huggingface/download/config.json.metadata" + lines = metadata.read_text(encoding="utf-8").splitlines() + lines[0] = "0" * 40 + metadata.write_text("\n".join(lines) + "\n", encoding="utf-8") + message = "revision metadata mismatch" + elif mutation == "etag": + metadata = snapshot / ".cache/huggingface/download/config.json.metadata" + lines = metadata.read_text(encoding="utf-8").splitlines() + lines[1] = "0" * 40 + metadata.write_text("\n".join(lines) + "\n", encoding="utf-8") + message = "Hub etag mismatch" + elif mutation == "size": + (snapshot / "config.json").write_bytes(payloads["config.json"] + b"x") + message = "size mismatch" + else: + original = payloads["weights.bin"] + replacement = bytes([original[0] ^ 1]) + original[1:] + self.assertEqual(len(replacement), len(original)) + (snapshot / "weights.bin").write_bytes(replacement) + message = "LFS digest mismatch" + + with self.assertRaisesRegex(RuntimeError, message): + snapshot_manifest( + snapshot, + expected_files=expected_files, + expected_manifest_sha256=None, + ) + + def test_snapshot_manifest_rejects_missing_extra_and_symlinked_root_files(self) -> None: + for mutation in ("missing", "extra", "symlink"): + with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as temporary: + snapshot, expected_files, _payloads = self._fixture(Path(temporary)) + if mutation == "missing": + (snapshot / "config.json").unlink() + message = "missing=.*config.json" + elif mutation == "extra": + (snapshot / "unregistered.txt").write_text("extra", encoding="utf-8") + message = "extra=.*unregistered.txt" + else: + target = snapshot / "config-target.json" + target.write_bytes((snapshot / "config.json").read_bytes()) + (snapshot / "config.json").unlink() + (snapshot / "config.json").symlink_to(target.name) + expected_files = { + **expected_files, + "config-target.json": ( + target.stat().st_size, + git_blob_sha1(target), + ), + } + metadata = snapshot / ".cache/huggingface/download/config-target.json.metadata" + metadata.write_text( + f"{MODEL_REVISION}\n{expected_files['config-target.json'][1]}\n", + encoding="utf-8", + ) + message = "must not be a symlink" + + with self.assertRaisesRegex(RuntimeError, message): + snapshot_manifest( + snapshot, + expected_files=expected_files, + expected_manifest_sha256=None, + ) + + +class PathValidationTests(unittest.TestCase): + def test_validate_paths_rejects_aliases_and_snapshot_destinations(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + snapshot = root / "snapshot" + snapshot.mkdir() + dataset = root / "dataset.jsonl" + dataset.write_text("{}\n", encoding="utf-8") + output = root / "raw.jsonl" + metadata = root / "metadata.json" + + cases = ( + (dataset, metadata, "output equals dataset"), + (output, output, "output equals metadata"), + (snapshot / "raw.jsonl", metadata, "evidence inside snapshot"), + (output, snapshot / "metadata.json", "metadata inside snapshot"), + ) + for candidate_output, candidate_metadata, label in cases: + with self.subTest(case=label), self.assertRaises(ValueError): + validate_paths( + snapshot, + dataset, + candidate_output, + candidate_metadata, + ) + + +class AtomicWritePairTests(unittest.TestCase): + @staticmethod + def _staging_files(root: Path) -> list[Path]: + return sorted(path for path in root.rglob("*.tmp") if path.is_file()) + + def test_atomic_write_pair_publishes_both_exact_byte_sequences(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + output = root / "evidence" / "raw.jsonl" + metadata = root / "metadata" / "run.json" + raw_bytes = b'{"raw":"evidence"}\n' + metadata_bytes = b'{"raw_evidence_sha256":"synthetic"}\n' + + atomic_write_pair(output, raw_bytes, metadata, metadata_bytes) + + self.assertEqual(output.read_bytes(), raw_bytes) + self.assertEqual(metadata.read_bytes(), metadata_bytes) + self.assertEqual(self._staging_files(root), []) + + def test_atomic_write_pair_never_overwrites_existing_artifacts_or_leaves_staging(self) -> None: + for existing in ("evidence", "metadata", "both"): + with self.subTest(existing=existing), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + output = root / "raw.jsonl" + metadata = root / "metadata.json" + if existing in {"evidence", "both"}: + output.write_bytes(b"existing raw") + if existing in {"metadata", "both"}: + metadata.write_bytes(b"existing metadata") + + with self.assertRaises(FileExistsError): + atomic_write_pair( + output, + b"replacement raw", + metadata, + b"replacement metadata", + ) + + if existing in {"evidence", "both"}: + self.assertEqual(output.read_bytes(), b"existing raw") + else: + self.assertFalse(output.exists()) + if existing in {"metadata", "both"}: + self.assertEqual(metadata.read_bytes(), b"existing metadata") + else: + self.assertFalse(metadata.exists()) + self.assertEqual(self._staging_files(root), []) + + +if __name__ == "__main__": + unittest.main()