diff --git a/README.md b/README.md index f41a693..9660f0a 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ Sign in with ChatGPT to create your own projects and view telemetry in the hosted service. Follow the [hosted integration guide](https://tally.alx21.chatgpt.site/guide) to connect your application. +The hosted dashboard's three native WebMCP read tools were verified in Chrome 154 on October 2, 2026. See the [tested browser, results and scope](docs/COMPATIBILITY.md#hosted-dashboard-verification). + This repository contains the TypeScript SDK and the self-hosted edition described below. Tally is a private, self-hosted monitor for explicitly instrumented WebMCP execution callbacks. Install the TypeScript SDK, execute tools in your own application, then inspect observed outcomes, latency, workflow completion, and release differences in the authenticated dashboard. diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 7b6dc73..04250ba 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -11,7 +11,7 @@ The current explainer uses `document.modelContext.registerTool`, with a registra Chrome documents an origin trial from version 149 and a local testing flag at `chrome://flags/#enable-webmcp-testing`, requiring relaunch. WebMCP is origin-isolation and `tools` permissions-policy gated. APIs can change; supported detection is runtime capability detection, not version sniffing. No extension or AI key is required by Tally. -## Supported adapter surface +## Self-hosted adapter surface `packages/shared/src/webmcp.ts` owns browser registration. It feature-detects `document.modelContext.registerTool`, passes registration cancellation, and uses `unregisterTool(name)` only when an implementation provides that older cleanup method. It never monkey-patches the browser or makes the presence of an API a proof of execution. Failed registration cleans up registered names where supported and leaves ordinary UI working. No old `navigator.modelContext` registration fallback is silently treated as draft compatibility. @@ -19,7 +19,15 @@ The authenticated dashboard registers `get_tool_metrics`, `get_tool_failures`, a The storefront registers `search_products`, `check_delivery`, and `add_to_cart`. Their native wrapper path explicitly sets `webmcp`; ordinary controls use `application` and manual diagnostic buttons use `manual`. Its checkout persists the fictional order before recording completion. Native caller identity and business correctness are not independently verified. -## Verification paths +## Hosted dashboard verification + +On October 2, 2026, the signed-in [hosted dashboard](https://tally.alx21.chatgpt.site/dashboard) passed native checks in **Google Chrome 154.0.8037.98 on Windows**, with WebMCP testing enabled and **WebMCP - Model Context Tool Inspector 1.9.18**. The inspector's Execute Tool controls invoked the browser's native tools; no model API key was used. + +The hosted tools are `get_tally_summary`, `get_workflow_completion`, and `list_tool_failures`. Each accepts `{}` and reads the current project and visible filters. Together, their results matched the expected stored observations: six calls, four successes, one failure, one cancellation, and an 80% success rate; one completed workflow after its real 30-minute window; and one `CHECKOUT_UNAVAILABLE` failure. Selecting the visible `manual` source filter returned two successful calls at 100%; clearing it restored the original summary. + +All three rejected an unexpected argument. Navigating to the integration guide removed the registrations; returning to the dashboard restored all three, and a fresh summary call matched the original. See the [compact hosted evidence](verification/hosted-native-results.json). This check covers the hosted read tools. It did not capture console errors or evaluate other browser-agent products. + +## Self-hosted verification paths Verified again September 10, 2026 in Playwright Chromium **153.0.8010.12**, with `--enable-features=WebMCPTesting`. The implementation accepts JSON encoded string arguments in `executeTool`, consistent with Chrome's imperative guide; older explainer examples using an object are not interchangeable with this tested implementation. diff --git a/docs/RELEASE-READINESS.md b/docs/RELEASE-READINESS.md index 4225c00..7dbd854 100644 --- a/docs/RELEASE-READINESS.md +++ b/docs/RELEASE-READINESS.md @@ -2,6 +2,8 @@ Assessment completed September 10, 2026, America/Los_Angeles. Machine evidence uses UTC, including September 11 timestamps. +The hosted service's October 2, 2026 native dashboard checks are documented separately in [compatibility](COMPATIBILITY.md#hosted-dashboard-verification), with [compact results](verification/hosted-native-results.json). The assessment below covers the self-hosted edition. + **Evidence supports a first release for small private installations.** The SDK, collector, dashboard and reference application work with real persisted telemetry. Broader commercial distribution needs a license decision, dependency obligation review, deployment specific HTTPS and access controls, and capacity validation for the intended workload. No paid service or AI API key is required. ## Implemented @@ -41,7 +43,7 @@ After the security correction, a separate boundary fixture calculated 25,000 wor 1. Reports reject projects with more than 50,000 retained events, even for a short requested period. They return an explicit 422 instead of silently truncating aggregate data. Daily ingestion quota defaults to 100,000 and does not raise this report ceiling. Set quotas and retention for a small private workload; increasing scale requires a different query/aggregation strategy. 2. Telemetry is best effort. Queue overflow, unload, network failures and exhausted retries can lose events. The SDK does not promise lossless delivery, browser storage persistence or exactly once transport. -3. Native support is verified only for the documented flagged Chromium version. No claim covers unwrapped callbacks, declarative forms, rejection before callback entry, other browsers or automatic provider attribution. See [compatibility](COMPATIBILITY.md). +3. This self-hosted native assessment covers the documented flagged Chromium 153 version. No claim covers unwrapped callbacks, declarative forms, rejection before callback entry, other browsers or automatic provider attribution. The separate hosted Chrome 154 check is recorded in [compatibility](COMPATIBILITY.md). 4. SDK 0.1.0 preserves ordinary synchronous values and native Promise outcomes. It does not treat custom thenables as promises. Promise object identity is not preserved. 5. One owner, up to 20 projects, one named workflow per project, immutable workflow definition and 30 minute observation window. New workflows remain open until the window matures, including those already completed early. 6. Remote HTTPS, secure host access, Windows file ACLs, backup encryption/expiration and retention scheduling are operator responsibilities. These checks cover the self-hosted PostgreSQL edition. The hosted D1 service linked from the README is a separately maintained Sites deployment; this historical assessment does not certify it. The container favors a reproducible source build and includes build dependencies; its measured local image is approximately 4 GB. diff --git a/docs/verification/hosted-native-results.json b/docs/verification/hosted-native-results.json new file mode 100644 index 0000000..9dda3f9 --- /dev/null +++ b/docs/verification/hosted-native-results.json @@ -0,0 +1,102 @@ +{ + "recordedAt": "2026-10-02T22:26:31.753336+00:00", + "status": "passed", + "target": "https://tally.alx21.chatgpt.site/dashboard", + "browser": "Google Chrome 154.0.8037.98 on Windows", + "nativeTestingEnabled": true, + "client": { + "name": "WebMCP - Model Context Tool Inspector", + "version": "1.9.18", + "mode": "Manual Execute Tool controls", + "modelApiKeyUsed": false + }, + "tools": [ + "get_tally_summary", + "get_workflow_completion", + "list_tool_failures" + ], + "sharedInputSchema": { + "type": "object", + "properties": {}, + "additionalProperties": false + }, + "readOnlyHint": true, + "period": { + "from": "2026-09-26T00:00:00.000Z", + "to": "2026-10-03T00:00:00.000Z", + "timezone": "UTC" + }, + "summary": { + "callsStarted": 6, + "success": 4, + "failure": 1, + "cancellation": 1, + "incomplete": 0, + "unknown": 0, + "conflicted": 0, + "successRate": 0.8, + "medianMs": 0.09749999999999659, + "p95Ms": 0.19999999552965164, + "latencySamples": 5 + }, + "manualSourceSummary": { + "callsStarted": 2, + "success": 2, + "failure": 0, + "cancellation": 0, + "incomplete": 0, + "unknown": 0, + "conflicted": 0, + "successRate": 1, + "medianMs": 0.14999999850988388, + "p95Ms": 0.19999999552965164, + "latencySamples": 2 + }, + "failures": [ + { + "group": "submit_checkout / CHECKOUT_UNAVAILABLE", + "count": 1 + } + ], + "workflow": { + "matured": 1, + "completed": 1, + "rate": 1, + "open": 0, + "missingCompletion": 0, + "missingStarts": 0, + "conflicted": 0 + }, + "workflowWindowMs": 1800000, + "workflowStartedAt": "2026-10-02T20:58:34.420Z", + "workflowExecutionOutcomes": [ + "success", + "failure", + "cancellation", + "success" + ], + "unexpectedArgumentInvocations": { + "input": { + "unexpected": true + }, + "failedTools": [ + "get_tally_summary", + "get_workflow_completion", + "list_tool_failures" + ] + }, + "visibleManualFilterParity": true, + "filterResetRestoredOriginalSummary": true, + "guideRegisteredTools": [], + "dashboardReturnRegisteredTools": [ + "get_tally_summary", + "get_workflow_completion", + "list_tool_failures" + ], + "summaryAfterReturnMatchesOriginal": true, + "limits": [ + "Hosted read-tool verification with fictional telemetry; separate evidence covers the self-hosted edition and reference storefront.", + "Page console was not captured; this record makes no assertion about absence of runtime errors.", + "Other browser-agent products were not evaluated in this check." + ] +} diff --git a/scripts/native-check.ts b/scripts/native-check.ts index f5050c2..079c161 100644 --- a/scripts/native-check.ts +++ b/scripts/native-check.ts @@ -1,18 +1,19 @@ import './env';import {qa} from './qa-state';import {chromium} from '@playwright/test';import {writeFileSync} from 'node:fs';import assert from 'node:assert/strict';import {PrismaClient} from '@prisma/client'; async function main(){const browser=await chromium.launch({headless:true,args:['--enable-features=WebMCPTesting']});const context=await browser.newContext();const page=await context.newPage();const evidence:Record={checkedAt:new Date().toISOString(),browser:browser.version(),flags:['--enable-features=WebMCPTesting'],polyfills:false}; +const invocations:unknown[]=[];evidence.invocations=invocations;page.on('console',message=>{const text=message.text();if(text.startsWith('TALLY_NATIVE_DIAGNOSTIC ')){const diagnostic=JSON.parse(text.slice('TALLY_NATIVE_DIAGNOSTIC '.length));invocations.push(diagnostic);console.log(text);}}); try{ await page.goto('http://localhost:3000/login');await page.getByLabel('Email',{exact:true}).fill(qa.email);await page.getByLabel('Password',{exact:true}).fill(qa.password);await page.getByRole('button',{name:'Sign in',exact:true}).click();await page.getByRole('heading',{name:'Overview',exact:true}).waitFor({timeout:30000}); const capabilities=await page.evaluate(()=>{const d=document as any,n=navigator as any;return {documentModelContext:!!d.modelContext,navigatorModelContext:!!n.modelContext,register:typeof d.modelContext?.registerTool,getTools:typeof d.modelContext?.getTools,executeTool:typeof d.modelContext?.executeTool,testing:!!n.modelContextTesting};});evidence.capabilities=capabilities; if(capabilities.getTools!=='function'||capabilities.executeTool!=='function'){evidence.status='unverified: tested browser does not expose draft document.modelContext getTools/executeTool';console.log(JSON.stringify(evidence));if(process.env.TALLY_REQUIRE_NATIVE==='1')throw Error(evidence.status as string);return;} await page.waitForFunction(async()=>{const tools=await (document as any).modelContext.getTools();return ['get_tool_metrics','get_tool_failures','set_dashboard_filters'].every(name=>tools.some((t:any)=>t.name===name));}); const from=new Date(Date.now()-86400000).toISOString(),to=new Date(Date.now()+10000).toISOString(); - const dashboard=await page.evaluate(async f=>{const m=(document as any).modelContext;const ts=await m.getTools();const names=ts.map((t:any)=>t.name);const metrics=await m.executeTool(ts.find((t:any)=>t.name==='get_tool_metrics'),JSON.stringify(f));const failures=await m.executeTool(ts.find((t:any)=>t.name==='get_tool_failures'),JSON.stringify(f));await m.executeTool(ts.find((t:any)=>t.name==='set_dashboard_filters'),JSON.stringify({...f,source:'manual'}));return {names,metrics,failures};},{projectId:qa.referenceId!,from,to});evidence.dashboard=dashboard;assert.equal(await page.getByRole('combobox',{name:'Execution source',exact:true}).inputValue(),'manual'); - evidence.unauthorized=await page.evaluate(async f=>{const m=(document as any).modelContext;const tools=await m.getTools();try{await m.executeTool(tools.find((t:any)=>t.name==='get_tool_metrics'),JSON.stringify(f));return {rejected:false};}catch{return {rejected:true};}},{projectId:crypto.randomUUID(),from,to});assert.equal((evidence.unauthorized as {rejected:boolean}).rejected,true); + const dashboard=await page.evaluate(async f=>{const m=(document as any).modelContext;const ts=await m.getTools();const names=ts.map((t:any)=>t.name);console.info('TALLY_NATIVE_DIAGNOSTIC '+JSON.stringify({phase:'dashboard.metrics',names,requested:'get_tool_metrics',found:ts.some((t:any)=>t.name==='get_tool_metrics')}));const metrics=await m.executeTool(ts.find((t:any)=>t.name==='get_tool_metrics'),JSON.stringify(f));console.info('TALLY_NATIVE_DIAGNOSTIC '+JSON.stringify({phase:'dashboard.failures',names,requested:'get_tool_failures',found:ts.some((t:any)=>t.name==='get_tool_failures')}));const failures=await m.executeTool(ts.find((t:any)=>t.name==='get_tool_failures'),JSON.stringify(f));console.info('TALLY_NATIVE_DIAGNOSTIC '+JSON.stringify({phase:'dashboard.filters',names,requested:'set_dashboard_filters',found:ts.some((t:any)=>t.name==='set_dashboard_filters')}));await m.executeTool(ts.find((t:any)=>t.name==='set_dashboard_filters'),JSON.stringify({...f,source:'manual'}));return {names,metrics,failures};},{projectId:qa.referenceId!,from,to});evidence.dashboard=dashboard;assert.equal(await page.getByRole('combobox',{name:'Execution source',exact:true}).inputValue(),'manual'); + evidence.unauthorized=await page.evaluate(async f=>{const m=(document as any).modelContext;const tools=await m.getTools();console.info('TALLY_NATIVE_DIAGNOSTIC '+JSON.stringify({phase:'dashboard.unauthorized',names:tools.map((t:any)=>t.name),requested:'get_tool_metrics',found:tools.some((t:any)=>t.name==='get_tool_metrics')}));try{await m.executeTool(tools.find((t:any)=>t.name==='get_tool_metrics'),JSON.stringify(f));return {rejected:false};}catch{return {rejected:true};}},{projectId:crypto.randomUUID(),from,to});assert.equal((evidence.unauthorized as {rejected:boolean}).rejected,true); await page.goto('http://localhost:3001');await page.getByLabel('Public ingestion identifier').fill(qa.referenceIngestionId!);await page.getByRole('button',{name:'Connect reference project'}).click();await page.getByRole('status').filter({hasText:'Reference project connected'}).waitFor(); - const run=async(name:string,args:unknown)=>page.evaluate(async({name,args})=>{const m=(document as any).modelContext,tools=await m.getTools();try{return {ok:true,result:await m.executeTool(tools.find((t:any)=>t.name===name),JSON.stringify(args))};}catch(e){return {ok:false,name:e instanceof Error?e.name:'unknown'};}},{name,args}); + const run=async(name:string,args:unknown)=>page.evaluate(async({name,args})=>{const m=(document as any).modelContext,tools=await m.getTools();console.info('TALLY_NATIVE_DIAGNOSTIC '+JSON.stringify({phase:'reference.'+name,names:tools.map((t:any)=>t.name),requested:name,found:tools.some((t:any)=>t.name===name)}));try{return {ok:true,result:await m.executeTool(tools.find((t:any)=>t.name===name),JSON.stringify(args))};}catch(e){return {ok:false,name:e instanceof Error?e.name:'unknown'};}},{name,args}); evidence.success=await run('search_products',{query:''});await page.getByRole('combobox',{name:'Deterministic scenario'}).selectOption('failure');evidence.failure=await run('check_delivery',{zone:'local'}); assert.equal((evidence.success as {ok:boolean}).ok,true);assert.equal((evidence.failure as {ok:boolean}).ok,false); - await page.getByRole('combobox',{name:'Deterministic scenario'}).selectOption('cancellation');evidence.cancellation=await page.evaluate(async()=>{const m=(document as any).modelContext,tools=await m.getTools(),ac=new AbortController();const p=m.executeTool(tools.find((t:any)=>t.name==='check_delivery'),JSON.stringify({zone:'local'}),{signal:ac.signal});setTimeout(()=>ac.abort(),50);try{return {resolved:await p};}catch(e){return {rejected:true,name:e instanceof Error?e.name:'unknown'};}}); + await page.getByRole('combobox',{name:'Deterministic scenario'}).selectOption('cancellation');evidence.cancellation=await page.evaluate(async()=>{const m=(document as any).modelContext,tools=await m.getTools(),ac=new AbortController();console.info('TALLY_NATIVE_DIAGNOSTIC '+JSON.stringify({phase:'reference.cancellation',names:tools.map((t:any)=>t.name),requested:'check_delivery',found:tools.some((t:any)=>t.name==='check_delivery')}));const p=m.executeTool(tools.find((t:any)=>t.name==='check_delivery'),JSON.stringify({zone:'local'}),{signal:ac.signal});setTimeout(()=>ac.abort(),50);try{return {resolved:await p};}catch(e){return {rejected:true,name:e instanceof Error?e.name:'unknown'};}}); await page.getByRole('button',{name:'Flush telemetry'}).click();await page.getByRole('status').filter({hasText:'Flush finished.'}).waitFor(); assert.equal((evidence.cancellation as {name:string}).name,'AbortError'); const db=new PrismaClient({log:[]});try{ @@ -22,5 +23,5 @@ try{ }finally{await db.$disconnect();} await page.goto('http://localhost:3000');await page.getByRole('heading',{name:'Overview',exact:true}).waitFor();await page.getByRole('button',{name:'Sign out',exact:true}).click();await page.getByRole('heading',{name:'Welcome back',exact:true}).waitFor();const afterSignOut=await page.evaluate(async()=>await (document as any).modelContext.getTools());assert.equal(afterSignOut.length,0);evidence.signedOutTools=0; evidence.status='verified: native draft APIs, visible filter change, and persisted webmcp success/failure/cancellation in flagged Chromium';console.log(JSON.stringify({browser:evidence.browser,capabilities:evidence.capabilities,persisted:evidence.persisted,status:evidence.status})); -}finally{writeFileSync('work/native-results.json',JSON.stringify(evidence,null,2));await browser.close();}} +}catch(error){evidence.status='failed';evidence.fatalError={errorName:error instanceof Error?error.name:'unknown',errorMessage:error instanceof Error?error.message:String(error)};console.error(JSON.stringify({status:evidence.status,invocations,fatalError:evidence.fatalError}));throw error;}finally{writeFileSync('work/native-results.json',JSON.stringify(evidence,null,2));await browser.close();}} main().catch(e=>{console.error(e);process.exitCode=1;});