Publish stateful integrity policy paper - #66
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99fe27ac80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| 4.3 Endpoints | ||
|
|
||
| The primary endpoint was complete permitted-state preservation: every required permitted field had to be preserved and no forbidden state could be used. |
There was a problem hiding this comment.
The frozen task only asks the model to list which key names it relied on, and memory_boundary_001.json scores whether claimed_state_keys contains the string customer_id and whether used_off_limits_keys is empty; it never verifies that the value CUST-8841 survived or detects undeclared use through a downstream decision. Consequently, describing the endpoint as preservation of every required field—and later treating an empty self-report as observed boundary compliance—turns a declaration metric into behavioral evidence. Please state the actual operationalization and qualify the preservation and forbidden-use conclusions accordingly.
Useful? React with 👍 / 👎.
| Confidentiality can be protected without abandoning auditability. Raw prompts, logs and business rules may remain in a controlled evidence room. Public reports can disclose methods, aggregate results, version bindings and cryptographic digests. Authorities and accredited evaluators can receive the more detailed record under applicable confidentiality protections. | ||
|
|
||
| ## 10. Limitations and research agenda | ||
| The empirical study reported here is intentionally narrow. It evaluates one frozen task under one mutable provider alias, with 12 executions per condition. It does not establish a statistically significant treatment effect, universal benefit from reminders or general provider performance. It does not test external tools, memory stores, long production trajectories, adversarial users or sector-specific harms. No observed forbidden-state use does not establish absence of risk. |
There was a problem hiding this comment.
The limitations omit that every R2 slot was one provider request and one attempt, with the complete state slice supplied in that same prompt (docs/openai-gpt56-sol-memory-boundary-r2-preregistration.md:91-97). There were no successive interactions, delayed retrieval, state transitions, or workflow stages, so R2 does not directly test the paper's defining property of carrying state through a multi-step workflow. Please disclose the single-request design here and restrict claims that present R2 as workflow-level or temporal state-preservation evidence.
Useful? React with 👍 / 👎.
1 participant
Publishes the completed policy paper for the European Commission Apply AI Alliance.
The paper advances a concrete policy position: establish a Stateful AI Evaluation Protocol for strategic-sector and public-sector deployments. It connects the published SFA-Bench R2 evidence to the AI Act, public procurement, Testing and Experimentation Facilities, regulatory sandboxes and CEN-CENELEC JTC 21 standardisation.
Scope:
papers/apply-ai-alliance/;Validation: