Skip to content

Commit 9982abc

Browse files
a7vinxclaude
a7vinx
and
claude
committed
test(protocol): record the fixtures from live sessions
The fixtures were written from the backend's protocol structs because no account was available. Running the recorder against a real session replaced 10 of the 18, and the differences are the point of having recorded them. Four assumptions were wrong. `session:input_state` on an open composer carries no detail; `session:llm_thinking` is often a bare placeholder rather than a tool_call; a `session:history` page returns `messages: null` and omits the cursor once exhausted; `session:state` during a conversation holds "chat". The tests asserted a scenario where they should have asserted a shape — a recording catches whichever instance the session produced, so conditions an ordinary session never reaches are now constructed by the test that needs them. Redaction took three passes, each fault found by reading what it wrote rather than by searching for what leaked. Its phone pattern matched bare digit runs and rewrote message ids into a fake phone number, so it now requires a leading "+" or separators and exempts identifier fields. It scrubbed only phones and emails, while a form is precisely where the user's data lives: the details it asks for are prefilled from their profile and the placeholders are built from those values. And the server writes choice labels in the user's voice — an option reads "I (Name) prefer ...", carrying their name under a key that says nothing about user data. All three are covered now, along with the account and session ids a recording ran under, and a test asserts the shape of a redacted form so the next omission fails rather than ships. The live prompt was a bill negotiation, which the agent spends six minutes thinking about. Asking it to call a friend gets the same coverage in forty seconds. Two observations from live traffic. `session:text` never arrived during a turn in any run: the composer reopens on streaming increments alone and the complete message is the durable record, so the README now says to assemble from parts. And `FormField` carries `description` and `source` on the wire, which the model was dropping. Verified: 95 offline tests and 8 live tests pass, ruff clean, build clean. Eight fixtures remain derived — no ordinary session reaches a restriction, a credit-blocked task, or a rich content document. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 793738a commit 9982abc

17 files changed

Lines changed: 445 additions & 163 deletions

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,11 @@ payloads, and semantics change compatibly or with notice.
5858

5959
Payloads may gain fields at any time — tolerate fields you do not recognise.
6060

61+
A turn commonly delivers `session:text_part` alone: the composer reopens once
62+
the agent has finished speaking, and the complete `session:text` is the durable
63+
record, read back from history. Assemble the parts by `message_id` rather than
64+
waiting for the complete message to arrive live.
65+
6166
## Everything else passes through
6267

6368
The server emits many more events. The SDK delivers every one of them unchanged

‎src/pine_assistant/models/form.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@ class FormField(BaseModel):
1313
name: str
1414
type: str = "text"
1515
label: str | None = None
16+
description: str | None = None
1617
placeholder: str | None = None
18+
source: str | None = None
1719
is_required: bool | None = None
1820
pii_level: str | None = None
1921
prefilled: str | None = None

‎tests/integration/record_fixtures.py‎

Lines changed: 58 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,29 +26,83 @@
2626
from typing import Any
2727

2828
from pine_assistant import AsyncPineAI, S2CEvent, is_supported_event
29+
from tests.protocol.fake import SESSION_ID as PLACEHOLDER_SESSION_ID
2930

3031
FIXTURES = pathlib.Path(__file__).resolve().parents[1] / "protocol" / "fixtures"
31-
DEFAULT_PROMPT = "Help me negotiate my Comcast internet bill down to $50/month."
32+
DEFAULT_PROMPT = "Call my friend and ask what time Saturday's dinner starts."
3233

3334
# Values that identify a person or an account never reach a checked-in fixture.
35+
# The phone pattern requires a leading "+" or separators: Pine's identifiers are
36+
# long digit runs, and a looser pattern rewrites them into a fake phone number.
3437
REDACTIONS = (
35-
(re.compile(r"\+?\d[\d\-\s().]{7,}\d"), "+15555550100"),
38+
(re.compile(r"\+\d[\d\-\s().]{7,}\d"), "+15555550100"),
39+
(re.compile(r"\b\d{3}[-.\s]\d{3}[-.\s]\d{4}\b"), "+15555550100"),
3640
(re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "someone@example.com"),
3741
)
3842

43+
# Fields whose value is the user's own data. A form carries the account details
44+
# it is asking for — names, addresses, account numbers, PINs — and the server
45+
# builds its placeholders from them, so both sides are replaced wholesale rather
46+
# than pattern-matched. A form's submitted `content` goes too — it is a mapping
47+
# of answers, unlike the `content` string on a text or state event.
48+
# The key stays, the shape stays, the value does not.
49+
# `options` belongs here for a reason that is easy to miss: the server composes
50+
# choice labels in the user's voice, so an option reads "I (Charles) prefer ..."
51+
# and carries their name even though nothing about the key says user data.
52+
USER_DATA_KEYS = frozenset({"prefilled", "placeholder", "options"})
53+
USER_DATA_PLACEHOLDER = "[redacted]"
54+
55+
# The account and session a recording ran under are not part of the shape being
56+
# recorded, and a fixture carrying a real session id cannot be replayed into a
57+
# flow test — the client would filter it out as belonging elsewhere.
58+
PLACEHOLDER_USER_ID = "100000000000000001"
59+
60+
# Identifier fields are never redacted — they are opaque numbers, and rewriting
61+
# one destroys the shape the fixture exists to record.
62+
OPAQUE_KEYS = frozenset({
63+
"id", "event_id", "message_id", "session_id", "request_id", "operation_id",
64+
"quoted_message_id", "thinking_id", "turn_id", "revision", "next_message_id",
65+
"max_message_revision", "since_revision", "device_id", "user_id",
66+
})
67+
3968

4069
def redact(value: Any) -> Any:
4170
if isinstance(value, str):
4271
for pattern, replacement in REDACTIONS:
4372
value = pattern.sub(replacement, value)
4473
return value
4574
if isinstance(value, dict):
46-
return {k: redact(v) for k, v in value.items()}
75+
out = {}
76+
for k, v in value.items():
77+
if k in OPAQUE_KEYS:
78+
out[k] = v
79+
elif k in USER_DATA_KEYS and isinstance(v, str) and v:
80+
out[k] = USER_DATA_PLACEHOLDER
81+
elif k in USER_DATA_KEYS and isinstance(v, list):
82+
out[k] = [USER_DATA_PLACEHOLDER for _ in v]
83+
elif k == "content" and isinstance(v, dict) and v:
84+
out[k] = {key: USER_DATA_PLACEHOLDER for key in v}
85+
else:
86+
out[k] = redact(v)
87+
return out
4788
if isinstance(value, list):
4889
return [redact(v) for v in value]
4990
return value
5091

5192

93+
def anonymize(envelope: dict[str, Any]) -> dict[str, Any]:
94+
"""Replace the identities the recording ran under. Redaction cannot reach
95+
them: an account id is an opaque number, exempt from pattern matching so it
96+
does not get rewritten into a fake phone number."""
97+
source = envelope.get("metadata", {}).get("source")
98+
if isinstance(source, dict) and source.get("user_id"):
99+
source["user_id"] = PLACEHOLDER_USER_ID
100+
payload = envelope.get("payload")
101+
if isinstance(payload, dict) and payload.get("session_id"):
102+
payload["session_id"] = PLACEHOLDER_SESSION_ID
103+
return envelope
104+
105+
52106
async def record(prompt: str, raw_dir: pathlib.Path | None) -> dict[str, dict[str, Any]]:
53107
token = os.environ.get("PINE_ACCESS_TOKEN", "")
54108
user_id = os.environ.get("PINE_USER_ID", "")
@@ -104,7 +158,7 @@ def write_fixtures(seen: dict[str, dict[str, Any]]) -> tuple[list[str], list[str
104158
continue
105159
name = event_type.replace("session:", "")
106160
FIXTURES.joinpath(f"{name}.json").write_text(
107-
json.dumps(redact(envelope), indent=2) + "\n"
161+
json.dumps(anonymize(redact(envelope)), indent=2) + "\n"
108162
)
109163
provenance[event_type] = {
110164
"source": "recorded", "derived_from": None, "recorded_at": now,

‎tests/integration/test_live.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121

2222
pytestmark = pytest.mark.skipif(SKIP, reason="PINE_INTEGRATION not set")
2323

24-
PROMPT = "Help me negotiate my Comcast internet bill down to $50/month."
24+
PROMPT = "Call my friend and ask what time Saturday's dinner starts."
2525

2626

2727
def make_client() -> AsyncPineAI:
@@ -78,13 +78,19 @@ async def test_create_list_get_delete(self):
7878

7979
class TestSupportedSurface:
8080
async def test_a_turn_produces_a_substantive_response(self, session):
81-
"""Text, a rich document, or a form — all inside the scope."""
81+
"""Streamed text, a complete message, a rich document, or a form.
82+
83+
A turn often ends on streaming increments alone: the composer reopens
84+
once the agent has finished speaking, and the complete `session:text`
85+
is the durable record, read back from history rather than awaited here.
86+
"""
8287
client, sid = session
8388
events = [e async for e in client.chat(sid, PROMPT)]
8489

8590
types = {e.type for e in events}
8691
assert types & {
8792
S2CEvent.SESSION_TEXT.value,
93+
S2CEvent.SESSION_TEXT_PART.value,
8894
S2CEvent.SESSION_RICH_CONTENT.value,
8995
S2CEvent.SESSION_FORM_TO_USER.value,
9096
}, f"no substantive response; saw {sorted(types)}"

‎tests/protocol/fixtures/form_to_user.json‎

Lines changed: 185 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,198 @@
11
{
22
"metadata": {
3-
"event_id": "00000000-0000-4000-8000-956047064724",
4-
"request_id": "00000000-0000-4000-8000-000000000001",
5-
"timestamp": "2026-08-08T00:00:00Z",
3+
"event_id": "8be552fd-3e06-48ec-95e5-65df7b2bb9fe",
4+
"group_id": "d15ce9f7-f937-4aef-929b-c848a8cc795c",
5+
"is_required_action": true,
6+
"is_volatile": false,
7+
"request_id": "7920bfdc-99e2-4682-a270-2ba0da10a150",
68
"source": {
7-
"role": "agent"
9+
"role": "agent",
10+
"user_id": "100000000000000001"
811
},
9-
"is_volatile": false
12+
"timestamp": "2026-08-08T13:48:53Z"
1013
},
11-
"type": "session:form_to_user",
1214
"payload": {
13-
"session_id": "1900000000000000001",
14-
"message_id": "1900000000000000101",
15-
"type": "session:form_to_user",
1615
"data": {
17-
"message_to_user": "I need your account details.",
1816
"form": {
1917
"fields": [
2018
{
21-
"name": "account_number",
22-
"type": "text",
23-
"label": "Account number",
19+
"description": "The full legal name of the account holder on your Xfinity bill.",
20+
"is_required": true,
21+
"name": "Account Holder Full Name",
22+
"pii_level": "L1",
23+
"placeholder": "[redacted]",
24+
"prefilled": "[redacted]",
25+
"source": "knowledge_base",
26+
"type": "text"
27+
},
28+
{
29+
"description": "Gender of the primary account holder, required by Xfinity customer service representatives for identity authentication during phone calls.",
30+
"is_required": true,
31+
"name": "Legal Gender",
32+
"options": [
33+
"[redacted]",
34+
"[redacted]",
35+
"[redacted]"
36+
],
37+
"pii_level": "L1",
38+
"placeholder": "[redacted]",
39+
"prefilled": "[redacted]",
40+
"source": "knowledge_base",
41+
"type": "radio"
42+
},
43+
{
44+
"description": "The email address associated with your Xfinity account for identity verification.",
45+
"is_required": true,
46+
"name": "Email Address",
47+
"pii_level": "L2",
48+
"placeholder": "[redacted]",
49+
"prefilled": "[redacted]",
50+
"source": "knowledge_base",
51+
"type": "text"
52+
},
53+
{
54+
"description": "The phone number linked to your Xfinity account.",
55+
"is_required": true,
56+
"name": "Phone Number",
57+
"pii_level": "L2",
58+
"placeholder": "[redacted]",
59+
"prefilled": "[redacted]",
60+
"source": "knowledge_base",
61+
"type": "text"
62+
},
63+
{
64+
"description": "Complete service address including street, unit, city, state, and ZIP code where your Xfinity service is installed.",
65+
"is_required": true,
66+
"name": "Service Address",
67+
"pii_level": "L3",
68+
"placeholder": "[redacted]",
69+
"prefilled": "[redacted]",
70+
"source": "knowledge_base",
71+
"type": "text"
72+
},
73+
{
74+
"description": "Your 16-digit Xfinity residential account number (usually starts with 8).",
75+
"is_required": true,
76+
"name": "Xfinity Account Number",
77+
"pii_level": "L2",
78+
"placeholder": "[redacted]",
79+
"prefilled": "[redacted]",
80+
"source": "knowledge_base",
81+
"type": "text"
82+
},
83+
{
84+
"description": "Your 4-digit Xfinity security PIN or account passcode used for phone authentication.",
85+
"is_required": true,
86+
"name": "Xfinity Account Security PIN",
87+
"pii_level": "L3",
88+
"placeholder": "[redacted]",
89+
"prefilled": "[redacted]",
90+
"source": "knowledge_base",
91+
"type": "text"
92+
},
93+
{
94+
"description": "The last 4 digits of the payment card on file or Social Security Number for secondary verification.",
2495
"is_required": true,
25-
"pii_level": "high"
96+
"name": "Last Four Digits of Payment Card or SSN",
97+
"pii_level": "L2",
98+
"placeholder": "[redacted]",
99+
"prefilled": "[redacted]",
100+
"source": "knowledge_base",
101+
"type": "text"
102+
},
103+
{
104+
"description": "Your current Xfinity internet plan speed or bundled packages (e.g., Gigabit Extra 1.2 Gbps, Blast 800 Mbps, or Internet + TV bundle).",
105+
"is_required": true,
106+
"name": "Current Internet Plan or Package",
107+
"pii_level": "L1",
108+
"placeholder": "[redacted]",
109+
"prefilled": "[redacted]",
110+
"source": "knowledge_base",
111+
"type": "text"
112+
},
113+
{
114+
"description": "Sharing a recent PDF bill or screenshot can help identify active line-item fees, equipment rentals, and regional promo codes.",
115+
"is_required": true,
116+
"name": "Bill Sharing Option",
117+
"options": [
118+
"[redacted]",
119+
"[redacted]"
120+
],
121+
"pii_level": "L1",
122+
"placeholder": "[redacted]",
123+
"prefilled": "[redacted]",
124+
"source": "agent",
125+
"type": "radio"
126+
},
127+
{
128+
"description": "Xfinity offers an extra $10/month discount if enrolled in bank account autopay (ACH) versus a credit/debit card.",
129+
"is_required": true,
130+
"name": "Autopay Method Preference",
131+
"options": [
132+
"[redacted]",
133+
"[redacted]",
134+
"[redacted]"
135+
],
136+
"pii_level": "L1",
137+
"placeholder": "[redacted]",
138+
"prefilled": "[redacted]",
139+
"source": "knowledge_base",
140+
"type": "radio"
141+
},
142+
{
143+
"description": "Xfinity rate agreements often come with 1-year or 5-year price guarantee terms.",
144+
"is_required": true,
145+
"name": "Price Guarantee Term Preference",
146+
"options": [
147+
"[redacted]",
148+
"[redacted]",
149+
"[redacted]"
150+
],
151+
"pii_level": "L1",
152+
"placeholder": "[redacted]",
153+
"prefilled": "[redacted]",
154+
"source": "knowledge_base",
155+
"type": "radio"
156+
},
157+
{
158+
"description": "If your current premium tier cannot be reduced to $50/mo, are you open to adjusting your speed tier or removing unneeded add-ons?",
159+
"is_required": true,
160+
"name": "Speed and Package Flexibility",
161+
"options": [
162+
"[redacted]",
163+
"[redacted]",
164+
"[redacted]"
165+
],
166+
"pii_level": "L1",
167+
"placeholder": "[redacted]",
168+
"prefilled": "[redacted]",
169+
"source": "agent",
170+
"type": "radio"
171+
},
172+
{
173+
"description": "If Xfinity retention reps cannot meet $50/mo exactly, select all acceptable backup choices.",
174+
"is_required": true,
175+
"name": "Negotiation Flexibility Alternatives",
176+
"options": [
177+
"[redacted]",
178+
"[redacted]",
179+
"[redacted]",
180+
"[redacted]"
181+
],
182+
"pii_level": "L1",
183+
"placeholder": "[redacted]",
184+
"prefilled": "[redacted]",
185+
"source": "agent",
186+
"type": "multiselect"
26187
}
27-
],
28-
"is_submitted": false
29-
}
30-
}
31-
}
188+
]
189+
},
190+
"message_to_user": "I'm on it! I've sent over a quick form to get your Xfinity account details and preferred autopay options so we can start negotiating that bill down to $50."
191+
},
192+
"message_id": "816306070082297856",
193+
"revision": "3009087",
194+
"session_id": "1900000000000000001",
195+
"type": "session:form_to_user"
196+
},
197+
"type": "session:form_to_user"
32198
}

‎tests/protocol/fixtures/history.json‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,21 @@
11
{
22
"metadata": {
3-
"event_id": "00000000-0000-4000-8000-096445077343",
4-
"request_id": "00000000-0000-4000-8000-000000000001",
5-
"timestamp": "2026-08-08T00:00:00Z",
3+
"event_id": "dc91da89-9959-4e91-994f-93b685600106",
4+
"request_id": "9bc5c248-c2ec-4cc9-92b5-1aeaa9fbf637",
5+
"timestamp": "2026-08-08T13:47:07Z",
66
"source": {
7-
"role": "system"
7+
"role": "system",
8+
"user_id": "100000000000000001"
89
},
910
"is_volatile": false
1011
},
1112
"type": "session:history",
1213
"payload": {
1314
"session_id": "1900000000000000001",
14-
"message_id": "1900000000000000101",
1515
"type": "session:history",
1616
"data": {
17-
"messages": [],
18-
"next_message_id": "1900000000000000090",
19-
"order": "desc"
17+
"messages": null,
18+
"order": "DESC"
2019
}
2120
}
2221
}

0 commit comments

Comments
 (0)