Rewrite v3#23
Open
orual wants to merge 476 commits into
Open
Conversation
…rn_core directly The circular-dependency note claiming pattern_db couldn't depend on pattern_core was wrong — pattern_db already deps pattern_core with the sqlite feature. Drop FilterArgs, GraphDirection, GraphSliceRows, GraphNode; take &TaskFilter / &GraphQuery / &TaskEdgeRef directly. Add a blocks field to TaskFilter so block-scope constraints compose through the same struct. - Delete FilterArgs, GraphDirection, GraphNode, GraphSliceRows from pattern_db::queries::task; remove the incorrect circular-dep note. - list_tasks_filtered now takes &TaskFilter with status: Vec<TaskStatus> (kebab via as_str()), owner: Option<AgentId>, and new blocks field that adds AND block_handle IN (...) dynamically; Some([]) short-circuits to no results, None skips the constraint. - query_task_graph_bfs now takes (&TaskEdgeRef, &GraphQuery) and returns GraphSlice (Vec<TaskEdgeRef> nodes/edges); applies default caps (depth=16, max_nodes=1000) when fields are None. - Add TaskFilter.blocks field with serde attrs matching other Option fields. - Update all callers: queries_task.rs, queries_task_graph.rs, subscriber_task_list_concurrent.rs. - Add 5 new blocks-filter tests; add 4 new TaskFilter.blocks unit tests; add default_query_applies_built_in_caps BFS test. - pattern-db tests: 128 → 139; pattern-core tests: 149 → 154.
…lers
Fills Pattern.Tasks create/update/transition/add_comment dispatch with real
LoroDoc mutations against TaskList blocks. Handlers:
- handle_create mints a snowflake TaskItemId and pushes a new item map
into the `items` movable list.
- handle_update applies a TaskPatch (with Option<Option> double-option
semantics for clearable owner/active_form) and refreshes updated_at.
- handle_transition swaps the status and, on transition to Completed,
stamps completed_at.
- handle_add_comment appends `{author, timestamp, text}` to the item's
comments list.
Each mutation commits the LoroDoc and calls mark_dirty + persist_block so
the subscriber reconciles the SQL index on its own schedule.
TasksHandler drops its `store` field in favour of MemoryHandler's pattern
(take the adapter from cx.user().memory_store() per call, respecting scope
routing). TaskHandlerError is the internal structured error; EffectError is
the opaque dispatch-boundary form.
7 unit tests cover AC4.1 (create), AC4.2 (patch semantics + owner clearing),
AC4.3 (completed_at), AC4.6 (comment ordering + author), AC4.7
(TaskNotFound), plus the NotATaskList schema check.
handle_link / handle_unlink mutate only the source item's `blocks` list — the target's LoroDoc is never touched. This preserves the single-source-of-truth edge model and keeps cross-block links atomic (no two-doc commit coordination). link is idempotent via dedup: repeating link(A, B) does not create a duplicate entry in the canonical .kdl file. Self-edges (link(A, A)) are allowed — graph topology is unconstrained in v1. unlink is a silent no-op when the target edge doesn't exist. 7 unit tests cover AC4.4 (edge append), AC4.5 (unlink), AC4.5b (cross-block atomicity via LoroDoc frontier comparison), AC4.8 (self-edge), plus dedup and TaskNotFound on missing source item.
Fills the Pattern.Tasks list/query dispatch. handle_list_tasks: - Scope: when block is None, enumerates TaskList-schema blocks visible via MemoryStore::list_blocks for the caller. IsolatePolicy routing is handled by MemoryScope wrapping the inner store. - Filter: intersects caller-supplied filter.blocks with the visible set; empty intersection short-circuits to vec![] without touching SQL. - Projection: TaskRow -> TaskView with batched blocker/blocks aggregates (two tuple-IN queries on task_edges, not N+1 lookups). handle_query_graph: - Scope-checks root's block via fetch_task_list. - Delegates to pattern_db::queries::query_task_graph_bfs which now takes &TaskEdgeRef + &GraphQuery directly (Task 1.5's refactor). Adds rusqlite as a direct pattern_runtime dep (bundled-full). Tests seed tasks + task_edges directly to exercise the handler query/projection logic independent of subscriber reconcile (covered separately in pattern_memory). 12 unit tests cover AC5.1 (block scope), AC5.2 (no-block enumeration), AC5.3 (status filter + FTS5 keyword), AC5.4 (forward/reverse direction), AC5.6 (depth=0), AC5.6b + AC5.8 (max_nodes truncation), AC5.7 (cycle termination), blocker/blocks count aggregates, schema enforcement on single-block path.
… + preamble SdkBundle HList grows to 15 handlers; TasksHandler inserted at tag 3, immediately after RecallHandler (storage-adjacent grouping). The eval worker's frunk::hlist! construction updated to match. preamble.rs: adds `import qualified Pattern.Tasks as Tasks` and `import qualified Pattern.Diagnostics as Diagnostics` to the qualified-only block. The type M effect-row synonym now carries Tasks.Tasks (after Recall.Recall) and Diagnostics.Diagnostics (tail). Diagnostics was missing from the preamble pre-existing — agents can now actually call Pattern.Diagnostics.GetDiagnostics from code tool. build_effect_stack_type() updated to produce the matching 15-effect row; its doc-comment and module-level docs refreshed for the new count. Adds tasks_effect_registers_with_eight_methods_at_tag_3 smoke test that asserts Tasks appears at tag 3 with all 8 expected method names. Existing canonical_decl_order + type_m + qualified_imports tests updated for the new row. 1331/1331 tests pass across pattern-runtime/memory/db/core/cli/server.
Phase 4 Task 6. Implements `parse(bytes: &[u8]) -> Result<SkillFile, SkillParseError>` for skill `.md` files: `---\n<yaml>\n---\n\n<body>`. Pipeline: 1. UTF-8 decode (NonUtf8Body on failure). 2. split_frontmatter locates `---` delimiters at line starts, handles CRLF, and rejects mid-line `---foo` as a false-positive close. 3. saphyr::Yaml::load_from_str on the frontmatter span; scan errors carry a SourceSpan built from ScanError::marker().index(). 4. visit_root walks the Mapping, extracting typed fields (name, trust_tier, description, keywords, hooks) and preserving unknown top-level keys into an `extras` LoroValue::Map for round-trip. Converters: - yaml_to_json: saphyr Yaml -> serde_json::Value (hooks payload). - yaml_to_loro: saphyr Yaml -> LoroValue (extras preservation). Both skip non-string map keys (YAML allows complex keys; Pattern.Tasks blocks use only string keys). Trust tier validation uses exact kebab-case match, producing InvalidTrustTier distinct from TypeMismatch so AC7.6 can assert the specific variant. 23 unit tests cover: - AC6.3 (unknown keys -> extras) - AC6.4 (nested hooks preserved through serde_json) - AC6.5 (invalid YAML -> SkillParseError::Yaml with span) - AC6.6 (missing delimiters) - AC6.7 (minimal required fields parses with defaults) - AC7.6 (invalid trust tier -> InvalidTrustTier, not default) Plus edges: CRLF, empty body, non-UTF8, empty name, root-not-mapping, description: null, keyword sequence with non-string entry, nested extras preservation. Also adds crates/pattern_runtime/resources/skills/README.md as the first-party skill resource directory placeholder (referenced by Task 8's assign_trust_tier via CARGO_MANIFEST_DIR). Fixes Task 5's mod.rs layout (clippy::mod-module-files workspace lint): moves markdown_skill/mod.rs -> markdown_skill.rs. 247/247 pattern-memory lib tests pass.
Extend StructuredDocument::render() for BlockSchema::Skill to include name, description, keywords, and body text in the preview string — all four fields are now indexed in memory_blocks_fts via the existing update_block_preview pipeline. The metadata projection is inlined in document.rs using LoroValue reads rather than importing pattern_memory::fs::markdown_skill::project_metadata_from_loro, which would create a circular dependency (pattern_core must never depend on pattern_memory). The inline reads exactly the three fields needed (name, description, keywords_json) and tolerates missing/malformed values with empty-string defaults — no error propagation needed for a display path. Add crates/pattern_memory/tests/skill_fts5.rs with five tests: - fts5_skill_search_by_name - fts5_skill_search_by_description - fts5_skill_search_by_keyword - fts5_skill_search_by_body - fts5_skill_content_snapshot (insta BM25 ordering snapshot)
- Fill `SkillsReq` with 5 variants: List, GetMetadata, Load, Search, GetUsageStats, each with `#[core(module = "Pattern.Skills", name = ...)]` - Create `haskell/Pattern/Skills.hs` mirroring Pattern.Tasks: GADT with 5 constructors + 5 curried helpers (listSkills, getSkillMetadata, loadSkill, searchSkills, getSkillUsageStats) - Add `SkillsHandler::DescribeEffect` impl with full constructor/helper metadata; stub `EffectHandler` returns clear diagnostic error pending Tasks 5-7 - Register SkillsHandler in SdkBundle at tag 4 (after Tasks, per storage-adjacent grouping spec); update eval_worker.rs bundle literal - Update parity table: SkillsReq now has 5 variants; skills_req_variants test exhaustively constructs all 5 - Update bundle tests: 15->16 entries, CANONICAL_EFFECT_ROW gains "Skills", new skills_effect_registers_with_five_methods_at_tag_4 test - Update preamble.rs: add `import qualified Pattern.Skills as Skills`, add Skills.Skills to `type M` effect-row alias and build_effect_stack_type; update tests accordingly
Implements Task 5 of Phase 5 (v3-task-skill-blocks). Replaces the stub EffectHandler<HasCancelState> with a full EffectHandler<SessionContext> implementation (matching TasksHandler's bound, required for db() access). - handle_list: enumerates Skill-schema blocks via list_blocks, projects each LoroDoc into SkillMetadata via project_metadata_from_loro, batch-fetches usage stats via get_usage_stats_batch, merges last_used into SkillInfo. - handle_get_metadata: returns Option<SkillMetadata>; returns None for non-Skill blocks (AC8.3 — not an error). - handle_get_usage_stats: scope-checks the block is a Skill, delegates to get_usage_stats (returns default for new skills with no sqlite row). - handle_search added (Task 6 body) but Load remains stubbed (Task 7). 6 unit tests: list_enumerates_skill_blocks, list_populates_last_used_from_sqlite, get_metadata_returns_typed_frontmatter, get_metadata_on_text_block_returns_none, get_usage_stats_default_for_new_skill, get_usage_stats_after_three_loads.
…ded_event Task 6 of Phase 5 (v3-task-skill-blocks): - handle_search: FTS5 search over Skill blocks, post-filtered by schema, projected to SkillInfo with batch usage-stat join (BM25 order preserved). - render_skill_loaded_event: produces [skill:loaded]/[skill:loaded:end] pseudo-message with name + trust_tier (kebab) + body, wrapped in <system-reminder>. Mirrors render_change_event shape. - 4 search integration tests (MemoryCache + ConstellationDb FTS5 backend): search_matches_skill_name, _description, _body, search_relevance_ranked (insta snapshot of BM25 ordering). - 3 render_skill_loaded_event tests: snapshot, trust_tier kebab variants, structural marker presence. - insta added to pattern-runtime dev-dependencies.
…er + segment-2 pseudo-message pipe Phase 5 Task 7: implements `Pattern.Skills.Load` handler with full segment-2 pseudo-message injection. The plan assumed a sibling-Phase-4 push_pseudo_message helper that didn't exist — Phase 4 only built render_change_event over BlockWrite. Skill-load events don't fit that pipe (BlockWrite is write-specific). Built a parallel pipeline: - pattern_core: TurnOutput.pseudo_messages: Vec<ChatMessage> (skip_serializing_if = Vec::is_empty for clean wire). Future: block writes may converge into this generic pipe. - pattern_runtime/memory/adapter: pending_pseudo_messages buffer, record_pseudo_message + drain_pending_pseudo_messages. - pattern_runtime/memory/turn_history: most_recent_pseudo_messages() returns the immediately-prior turn's queued messages. - pattern_runtime/agent_loop: drains adapter buffer at turn close into TurnOutput.pseudo_messages; passes hist.most_recent_pseudo_messages() into Segment2Pass on next compose cycle. - pattern_provider/compose/passes/segment_2: Segment2Pass::new gains `recent_pseudo_messages: &[ChatMessage]` parameter; appends them to the rendered block-write pseudo-messages. handle_load (sdk/handlers/skills.rs): 1. Fetch block; BlockNotFound on miss (AC8.5). 2. Schema check; SkillError::NotASkill if not Skill (AC8.6). 3. Project SkillMetadata from LoroDoc, read body LoroText. 4. Build pseudo via render_skill_loaded_event(name, tier, body). 5. Push to adapter.record_pseudo_message. 6. record_usage in sqlite inside a transaction. 7. Return Unit. No LoroDoc mutation. No canonical .md write. Tests (8 new + integration test from prior scaffold): - load_missing_block_returns_block_not_found (AC8.5) - load_text_block_returns_not_a_skill (AC8.6) - load_injects_pseudo_message_into_adapter_buffer (AC9.1 unit) - load_drained_pseudo_messages_flow_into_turn_output (AC9.2 structural) - load_updates_use_count_to_five (AC9.3) - load_does_not_modify_lorodoc_body (AC9.3 — 100 loads byte-stable) - load_two_skills_preserves_buffer_order (AC9.4) - load_same_skill_twice_emits_two_markers (AC9.5) - load_does_not_dirty_mount (AC9.6, integration test, pre-existing) 1338/1338 tests passing across pattern-core/-memory/-db/-runtime/-provider. [docs] recover v3-sandbox-io phase plans 03/04/05
Phase 5 Task 9: 4 smoke tests exercising the Tasks + Skills SDK surface end-to-end with real handler dispatch + FTS5 + scope isolation. Tests live in pattern_runtime/tests/ rather than the plan's specified pattern_memory/tests/ location because the smoke test calls handler functions directly (avoiding the Haskell eval path's tidepool-extract requirement) and pattern_memory cannot depend on pattern_runtime (cycle). Tests (AC10.1 / AC10.3 / AC10.5 / AC10.8): 1. smoke_tasks_surface — InMemoryMemoryStore + reconcile_task_list driving create/update/transition/link/list_tasks/query_graph through their internal handler functions. 2. smoke_skills_surface — Arc<MemoryCache> backed by ConstellationDb for real FTS5; exercises list/get_metadata/search/load and asserts pseudo-message injection + content-hash stability across loads (AC9.6 invariant). 3. smoke_cross_schema_fts — Text + TaskList + Skill blocks indexed into memory_blocks_fts; single search hits all three schemas. 4. smoke_scope_enforcement — MemoryScope::Full isolation; persona blocks invisible (even from persona caller), project blocks visible to all callers; persona writes return IsolationDenied. Each test has its own fresh in-memory sqlite + store — no shared state, safe under --test-threads=N (AC10.5 verified). Handler-visibility changes: - handle_create/update/transition/add_comment/link/unlink/list_tasks/ query_graph + TaskHandlerError changed pub(crate) → pub. - handle_list/get_metadata/get_usage_stats/search/load + SkillHandlerError changed pub(crate) → pub. Rationale documented in pattern_runtime/CLAUDE.md: integration tests need direct handler access without the eval-worker round-trip. 1342/1342 tests across pattern-core/-memory/-db/-runtime/-provider.
…ch (atomic refactor; sessions register handlers; daemon-wide stub retired)
…sionConfig + SessionContext; sessions register per-session host_handler at open, unregister on drop; daemon main constructs handler-as-Arc + Clone-for-Router
…ge mirrors TUI AgentMessage shape (batch_id, Recipient, Vec ContentPart) but type-level prevents non-Plugin authorship (plugin self-reports plugin_id and partner_authority, daemon constructs Author Plugin server-side); drop WireHostMessage WireTaskCreate WireTaskTransition WireTaskLink WireTaskQuery WireTaskItem WireSkillInvoke WireSkillInvocation (Task ops are loro delta edits via MemorySync, Skill invocation composes from sync-read plus HostSendMessage); plugin-transport feature gains provider dep (multi-modal types are shared with TUI); host_handler.HostSendMessage wired via agent_registry.route_or_queue; queued follow-up: cross-check plugin_id from connection-authenticated pubkey at handle() time (defense-in-depth on top of pubkey auth at iroh accept)
Lights up MemorySyncProtocol over PLUGIN_MEMORY_SYNC_ALPN. Wire types were already defined in pattern_core; this commit adds the daemon-side handler, the cross-block observer broadcast, and the per-session registration plumbing. pattern_core observer module (new): - OriginTag (plugin_id plus connection_id) for echo-suppression and multi-instance disambiguation. - MemoryEvent (Delta / BlockAvailable / MetadataChanged / BlockGone, each with Option OriginTag). - MemoryObserver wrapping a tokio broadcast Sender (capacity 1024). Lossy on lag - observers self-heal via re-Sync. BlockAddr un-gated: - Moved from traits plugin wire.rs into types memory_types so it is available wherever MemoryStore is, without the plugin-transport feature. traits plugin wire.rs re-exports for back-compat callers. PluginAgentMessage gated: - Was unconditional but referenced Recipient which IS gated behind plugin-transport+provider. Now gated to align the feature boundary. MemoryStore trait grew two default methods: - fn observer returning Option of MemoryObserver ref, default None. - fn push_external_commit(scope, label, update_bytes), default Ok no-op. MemoryCache impl: - observer field initialised in both constructors. memory_observer accessor + impl of trait method returning Some. - push_external_commit looks up block_id from (scope, label), lazy-spawns the per-block subscriber if needed, pushes CommitEvent on its crossbeam. This is how plugin-pushed imports drive the persistence pipeline since loros subscribe_local_update does NOT fire on imports. - spawn_subscriber_for_block grew an observer parameter; threaded through all 5 callers (2 prod, 3 test). subscribe_local_update closure publishes Delta on the observer (origin None) alongside the existing per-block crossbeam push - local edits drive BOTH persistence AND observers via the hybrid pattern. pattern_runtime plugin memory_sync_handler module (new): - MemorySyncApiContext bundles memory_store + observer + session ids. - spawn mirrors host_handler spawn shape. - session_task drives the bidi-stream per Sync request. Enumerates initial watched addrs from SyncRequest Filter/Addrs and emits BlockAvailable for each via export_snapshot. Main loop is tokio select on observer broadcast and plugin rx. Delta ingest applies updates, push_external_commit for persistence, republishes on broadcast with origin Some(self). Subscribe/Unsubscribe live-mutate watched set. Done emits own Done before drop (graceful close). broadcast Lagged logs; plugin self-heals via re-Sync. Session plumbing (pattern_runtime session.rs): - SessionConfig + SessionContext + SessionRegistries grew plugin_memory_sync_handler field alongside plugin_routing_handler. - SessionContext accessor + builder method. - TidepoolSession open spawns memory_sync_handler and registers with the routing handler. SessionContext Drop unregisters. pattern_server (main.rs + server.rs): - gated_memory_sync_arc built parallel to gated_host_arc (same PluginRouteTable, separate per-ALPN routing handler). - Threaded through SessionConfig and SessionRegistries. - Router accept added for PLUGIN_MEMORY_SYNC_ALPN. - 4 test-fixture SessionConfig literals updated. Workspace compiles clean. pattern-server nextest 51/51 green. Still open (queued for stage 7): - Origin plugin_id placeholder. session_task uses a placeholder; session_id disambiguates within-session, real plugin_id from PluginRouteTable lookup at accept time needs threading for multi-instance differentiation. - VV-based resume not implemented (always emits full snapshot). - SnapshotPayload Chunked not emitted (v1 Inline only). - Plugin SDK side, plugin-side MemoryStore impl, and end-to-end test.
New pattern_plugin_sdk module memory_sync_client.rs lays down the plugin half of the MemorySync bidi stream. Pairs with the daemon-side handler landed in commit qtuqozxo. MemorySyncClient struct: - open(plugin_endpoint, daemon_addr, SyncRequest) dials the daemon over PLUGIN_MEMORY_SYNC_ALPN, opens the bidi stream via irpc bidi_streaming. - Receive task drives a local DashMap<BlockAddr, Arc<StructuredDocument>> cache. Handles BlockAvailable (materialize via from_snapshot_with_metadata), Delta (apply_updates on cached doc), MetadataChanged (swap in a new StructuredDocument via from_doc_with_metadata, preserving the loro state via LoroDoc::clone reference clone), BlockGone (cache remove), Done (clean close). - Public API: get_block / has_block / push_delta / subscribe / unsubscribe. - Drop sends Done before teardown so daemon can distinguish graceful close. MemorySyncError covers Open / SnapshotDecode / DeltaApply / StreamClosed / UnsupportedChunked (v1 is Inline-only per protocol doc). Added dashmap to pattern-plugin-sdk Cargo.toml. Still open (stage 7b): - Plugin-side MemoryStore impl that uses MemorySyncClient + host_handler. - subscribe_local_update bridge: when plugin edits local doc, auto-push Delta upstream. Stage 7a leaves push_delta as a manual call. - PluginHandle convenience method that opens MemorySync using the plugin's existing endpoint + daemon addr (stage 7a takes them as constructor args). Still open (stage 7c): - End-to-end integration test exercising host emit -> plugin sees and plugin emit -> host imports + persists + other observers see. pattern-plugin-sdk compiles clean.
…e_local_update Wires the plugin-side half of the echo-suppressed sync loop. When the MemorySyncClient materialises a doc from a BlockAvailable snapshot (or rebuilds one on MetadataChanged), it now also calls subscribe_local_update on the loro doc with a callback that pushes any resulting local-edit update bytes upstream as WireMemoryEdit::Delta. Key shape: - Cache value type changed from Arc<StructuredDocument> to a CachedBlock struct that bundles the doc with the loro Subscription. The Subscription field must be kept alive for the doc's lifetime in the cache; dropping it unsubscribes loro. - subscribe_auto_push helper takes (doc, addr, tx) and registers a callback on doc.inner().subscribe_local_update. The callback constructs a Delta with the addr and update_bytes, then tokio::spawns an async send on tx. Fire-and-forget: if the channel is closed, we log on next call; daemon sees the connection-drop on its end. - Echo suppression is automatic: loro's subscribe_local_update fires ONLY on local edits (commit path via txn.rs), NOT on imports. So when handle_event applies a daemon-pushed Delta via apply_updates, this callback does NOT fire, preventing the plugin from echoing back to the daemon. - Daemon-side echo suppression (OriginTag matching) already handles the symmetric case where the daemon receives a Delta from this plugin. Added loro = 1.10 as a direct dep of pattern-plugin-sdk so we can reference loro::Subscription in the CachedBlock struct. Cleanup: dropped a speculative tx_for_subs field on MemorySyncClient that the compiler flagged as dead code; the actual auto-push tx clone flows through the receive_loop function arg at spawn time, not via Self. pattern-plugin-sdk compiles clean. Still open (stage 7b.2): plugin-side MemoryStore impl bridging the cache (read path via get_block, write path via the auto-push subscription) plus the host_handler client for list/search/create/delete dispatch. Still open (stage 7b.3): PluginHandle convenience method that opens MemorySync using the plugin's existing endpoint + daemon addr (currently MemorySyncClient::open takes them as constructor args). Still open (stage 7c): end-to-end integration test.
… convenience
Plugin-side MemoryStore impl bridging the sync trait surface to async
host RPC via channel + worker dispatcher. Local cache reads
(get_block / get_block_metadata / get_rendered_content) skip the worker;
host_handler RPC calls dispatch via crossbeam + tokio::spawn per request
(concurrent MemoryStore calls run in parallel, not serialized).
Notify primitive in MemorySyncClient: register_block_arrival_waiter lets
create_block + create_or_replace_block synchronously wait for the
daemon's MemorySync to deliver the freshly-created doc's BlockAvailable
via recv_timeout(30s). No sleep-poll. Double-check on registration to
handle the race where BlockAvailable arrives between RPC return and
waiter registration.
7b.3: PluginHandle::open_memory_sync convenience method. Reuses the
plugin's existing endpoint + daemon endpoint addr (both retained on
PluginHandle at register_plugin time) so plugins don't have to
hand-construct either. Usage: handle.open_memory_sync(SyncRequest::Filter(...)).
Protocol additions (also affects daemon-side host_handler.rs):
- MemoryGetSharedBlockArgs grew an explicit requester: Scope field.
Host uses that for the permission check instead of silently
substituting ctx.default_scope - permission gates need the caller's
stated identity.
- MemoryInsertArchival reshaped: was (ArchivalEntry input, () return);
now (MemoryInsertArchivalArgs { scope, content, metadata } input,
SmolStr return). Host generates id + created_at + sets agent_id from
requester scope.
- MemoryListSharedBlocks(Scope) -> Vec<SharedBlockInfo>: new variant.
- MemoryHistoryDepth(BlockAddr) -> UndoRedoDepth: new variant.
UndoRedoDepth got serde::Serialize/Deserialize derives.
- search_archival now respects plugin-supplied scope via WireSearchQuery.scope;
host_handler reads it (was silently using ctx.default_scope).
All 18 trait methods now end-to-end through the bridge:
- create_block, create_or_replace_block, delete_block
- get_block, get_block_metadata, get_rendered_content, list_blocks
- persist_block, update_block_metadata, undo_redo, history_depth
- insert_archival, search_archival, delete_archival
- search, list_shared_blocks, get_shared_block
- commit_write + mark_dirty are no-ops on plugin side (loro auto-handles)
Verified by grep that no "not wired" stub markers remain in the impl.
Added crossbeam-channel = 0.5 to pattern-plugin-sdk Cargo.toml.
Full workspace compiles clean.
Still open:
- 7c: end-to-end test through TidepoolSession + fixture plugin
exercising host emit -> plugin sees, plugin emit -> host imports +
persists + other observers see, echo-suppression assertion.
New `auth::codex_oauth` module: PKCE loopback (port 1455 / 1457 fallback)
+ device-code flow, id_token claim extraction matching codex CLI's
`parse_chatgpt_jwt_claims` (no signature verification — codex's OAuth
path is payload-only base64 decode; TLS to auth.openai.com is the
authentication boundary), token-exchange + refresh with rotation,
`#[derive(Diagnostic)]` errors with typed cause chains
(reqwest::Error, base64::DecodeError, serde_json::Error,
std::io::Error all preserved via #[source]/#[from]).
Public surface: `CodexOAuthConfig`, `LoginFlow::{Loopback,DeviceCode,Auto}`,
`CodexLoginHandle`, `CodexTokenSet`, `IdTokenClaims`, `CodexOAuthError`,
`RefreshFailureKind`, `begin_login`, `complete_login`, `refresh_token`,
`parse_id_token`. Gated under existing `subscription-oauth` feature.
Deps: tiny_http (loopback HTTP listener), open (browser-open helper for
Phase 5 CLI), fs4 (for Phase 2 storage flock). jsonwebtoken not added
— codex's id_token path is payload-only.
Tests (19/19 green): PKCE pinned against RFC 7636 §4.6 vector;
authorize URL params (production config); id_token claim extraction
(namespaced, user_id fallback, missing namespace, malformed JWT,
base64 + JSON decode failures); token-exchange success + 400-error
mapping; refresh classifications (expired/reused/transient) + scope
inclusion + rotation persistence; device-code success + expiry; live
loopback listener via real HTTP GET against OS-assigned port.
…retrofit
New modules:
- auth::file_lock — generic cross-process advisory flock via fs4 +
spawn_blocking. Returns a guard that releases on drop. NOT
feature-gated (used by both subscription-OAuth and non-OAuth code
paths).
- auth::keyring_util — shared keyring helpers (open_entry,
classify_keyring_error) used by both creds_store::keyring and
codex_storage. Single source of truth for "is this keyring error
backend-unreachable or stored-data-corrupt?"
- auth::codex_storage — AuthDotJson / TokenData / AuthMode types
matching codex CLI's schema byte-for-byte (verified against
~/Git_Repos/codex/codex-rs/login/src/auth/storage.rs and token_data.rs).
CodexAuthStore::{new, from_env, load, save, forget} with
keyring-primary + auth.json interop. Pattern never creates the file
— save() writes auth.json only if file_existed was true at load.
Atomic-rename writes with per-call random nonce in temp filename
(pid alone collides across concurrent in-process callers).
Cross-process flock on auth.json.lock spans the full
read-modify-write.
Retrofits:
- creds_store::json_fallback (Anthropic JSON fallback) now uses
file_lock + per-call nonce in the temp filename. The previous
{provider}.json.tmp shared path was a latent race between concurrent
in-process put() calls; this surfaced as a failing test in the new
codex storage suite before the same pattern was applied to Anthropic.
- creds_store::keyring refactored to use keyring_util::open_entry +
classify_keyring_error. Dropped the in-module TapLog trait — the
one-line warn was overkill.
Tests (35 new, 302/302 in crate):
- file_lock: parent-dir creation, concurrent-acquire ordering via
Barrier, guard-drop releases lock.
- codex_storage: pinned-JSON snapshot test guards against drift from
codex's schema; AuthMode wire tags pinned (apikey/chatgpt/
chatgptAuthTokens/agentIdentity); keyring account derivation pinned
(prefix + length + lowercase-hex); 0o600 file mode on Unix;
load/load_file/load_keyring provenance reporting; concurrent
save_file under random-nonce tmp succeeds without races.
Dep tree:
- fs4 moved out of subscription-oauth (consumed by Anthropic path too).
- jsonwebtoken explicitly NOT pulled — codex's OAuth id_token path is
payload-only (verified across token_data.rs + server.rs).
`OpenAiAuthChain` mirrors `AnthropicAuthChain`'s tier-walking shape:
1. Stored OAuth (codex .auth.json / keyring with auth_mode: chatgpt) —
proactive refresh fires when the access_token JWT's exp claim is
within 8s (matches codex's TOKEN_REFRESH_INTERVAL). Refresh is
serialized by an in-process tokio::sync::Mutex AND a cross-process
advisory flock on auth.json.lock. Post-lock re-read dedupes
concurrent refresh attempts to a single network call. Refresh-token
rotation persisted atomically.
2. OPENAI_API_KEY env var.
3. File-embedded API key (codex .auth.json with auth_mode: apikey +
non-null OPENAI_API_KEY).
Constructors: api_key_only(), with_oauth(store, config, http),
oauth_only(...) — mirror AnthropicAuthChain's tier-forcing entry points.
Supporting changes:
- codex_oauth::parse_jwt_expiration — generic JWT exp claim parser used
by the chain to compute proactive-refresh windows.
- codex_storage::StorageMode { KeyringAndFile, FileOnly } — FileOnly
mode for tests so they never touch the developer's real keyring; also
useful for headless environments where no keyring daemon is reachable.
- codex_storage::save_under_lock() + lock() — exposed for the refresh
path where the chain holds the flock through the full read-refresh-write
cycle; calling save() again recursively would deadlock (POSIX flock is
per-open-file-description, same-process re-entry blocks).
Error mapping:
- RefreshFailureKind::{Expired, Exhausted, Revoked} → ProviderError::
NoAuthAvailable (caller should re-prompt login).
- RefreshFailureKind::{Transient, Other} → ProviderError::RefreshFailed
(retry-eligible).
Tests (10, all green; 312/312 in crate):
- Tier ordering: oauth wins over env; env wins over file-embedded;
file-embedded fallback when env absent; no creds → NoAuthAvailable.
- Proactive refresh within 8s buffer triggers + rotates tokens.
- Concurrent resolves dedupe to a single network call (mutex
serialization verified by counting wiremock hits).
- refresh_token_expired classifies as NoAuthAvailable; 503 transient as
RefreshFailed.
- oauth_only/api_key_only modes isolate tiers as advertised.
…ration
Wires the OpenAI provider through `PatternGatewayClient` end-to-end with
proper provider-aware composition all the way down. Phase 4 of the
codex-oauth design plan.
Gateway changes (pattern_provider/src/gateway.rs):
- `chat_url_for` grows arms for `AdapterKind::OpenAI` (Chat Completions
→ `api.openai.com/v1/chat/completions`) and `AdapterKind::OpenAIResp`
(Responses API; api-key tier → `api.openai.com/v1/responses`, OAuth
tier → `chatgpt.com/backend-api/codex/responses`). Signature gains a
`tier: AuthTier` parameter so OpenAIResp can branch on it.
- `auth_headers_for_tier` OAuth-tier arm grows OpenAI-specific logic:
inserts `chatgpt-account-id` (from `ProviderCredential.session_id`,
populated by the codex_oauth JWT decode at login time) and
`originator: pattern` when adapter is OpenAI/OpenAIResp.
- Tier-vs-protocol pre-flight gate in `complete()`: OAuth + OpenAI
(Chat Completions) → `ProviderError::TierMismatch` before any
network call, with a hint mentioning the `openai_resp::` namespace
prefix and the Responses API requirement.
Provider-aware composition (the bigger architectural win):
The compose pipeline in pattern_runtime was hardcoded to Anthropic-
specific defaults — `CacheProfile::default_anthropic_subscriber()`
everywhere + a feature-gated `default_shaper_mode()` constant that only
made sense for Anthropic. For OpenAI traffic this would have shipped
Anthropic prompt-caching markers and the SubscriptionRoutingShape
"You are Claude Code" slot[0] literal — structurally wrong and
honesty-violating.
- `CacheProfile::default_no_cache()` + `CacheProfile::default_for(adapter)`:
Anthropic gets the extended-TTL subscriber profile; OpenAI/Gemini/
others get the no-cache profile (Ephemeral5m, no extended TTL).
- `default_shaper_mode_for(adapter)`: Anthropic gets the feature-gated
default (SubscriptionRoutingShape under subscription-oauth); everyone
else gets HonestPattern.
- `NoOpShaper` now flattens `chat.system_blocks` → `chat.system` (the
flat string field). genai's openai_resp/openai/gemini adapters all
consume `chat.system` and ignore `system_blocks`; without this
flatten, OpenAI requests would arrive with no system prompt at all.
- `SessionContext::provider_kind()` derives `AdapterKind` from the
session's model_id via `AdapterKind::from_model` (single source of
truth; no Pattern-side catalog).
- `compose_request_for_turn`, session/spawn-ephemeral constructors:
call the provider-aware factories instead of hardcoding Anthropic.
Daemon wiring (pattern_server/src/main.rs):
- Registers the OpenAI provider alongside Anthropic in the gateway
builder, with `NoOpShaper` explicitly. Gateway shaper dispatch is
keyed on provider name so the Anthropic shaper is structurally
unable to fire on OpenAI traffic.
- Constructs `OpenAiAuthChain::with_oauth(CodexAuthStore::from_env(), …)`
so daemon sessions see codex-OAuth credentials when present.
- New `ProviderRateLimiter::openai_default()` constructor.
Error type:
- `ProviderError::TierMismatch { model: String, hint: &'static str }`
added to pattern_core. Diagnostic code, help text, doctest.
Bonus bug fix (pattern_runtime/src/sdk/handlers/skills.rs):
The skills FTS5 search handler had been silently returning empty
results since `MemorySearchResult::display_id()` was changed to return
the block's `label` (per `SearchHit::Block { label, .. } => label`).
`handle_search` was still keying its skill-metadata lookup map on
`BlockMetadata.id` (UUID), so every search lookup missed and the
handler returned an empty list even when the underlying FTS5 index
had hits. Fix: key the map on label. The four failing skills tests
+ task_skill_smoke now pass.
Anthropic JsonFallbackStore retrofit (carried in here for completeness):
- `creds_store::json_fallback::put` and `delete` now acquire the
cross-process file lock via `auth::file_lock` (mirrors what codex
storage does). The previous `{provider}.json.tmp` shared path was a
latent in-process race; the random-nonce + flock fix covers both.
Tests (15 new; 1237/1237 across pattern-provider + pattern-server +
pattern-runtime):
- gateway::tests: chat_url for OpenAI api-key + OAuth + base-URL
override (3 tests); auth_headers for OpenAI api-key + OAuth
with/without account_id (3 tests).
- gateway_integration: OAuth + Chat-Completions model →
TierMismatch with proper hint (no network call); OAuth + Responses-
API model passes the tier gate (1 test each).
- shaper::noop::tests: flatten system_blocks → system; drops empty
blocks; prepends to existing chat.system; no-op when only empties (4
tests).
Reactive 401 refresh deferred to a follow-up: the gateway's existing
`open_stream_with_retry` doesn't yet invalidate-and-retry on 401 from
the chatgpt backend. The proactive 8s buffer in `OpenAiAuthChain`
covers the common case; tokens that expire mid-request need the
reactive path, which is a small but real refactor.
Extends the existing `pattern auth {login,status,clear}` group with
OpenAI/codex support. Mirrors the Anthropic surface so the production
binary has one consistent credential-management toolkit.
Subcommand changes:
- `ProviderKind::Openai` variant added.
- `Login` gains `--headless` (forces device-code flow; defaults to
hybrid Auto = PKCE loopback with device-code fallback on bind
failure) and `--codex-home <PATH>` (overrides $CODEX_HOME).
- `Status` and `Clear` gain `--codex-home`.
- `--headless` and `--codex-home` are ignored with a note for
Anthropic/Gemini (they don't apply); kept on the same flag set so
the CLI surface stays uniform.
`pattern auth login --provider openai` flow:
1. Construct CodexAuthStore from $CODEX_HOME (or --codex-home).
2. Call codex_oauth::begin_login(LoginFlow::Auto | DeviceCode).
3. For Loopback: print the URL, attempt to open the browser via the
`open` crate (best-effort; warns on failure but doesn't abort
since the user can copy the URL manually), wait for the callback.
4. For DeviceCode: print the verification URL + user code, poll.
5. Persist via CodexAuthStore::save with file_existed semantics:
keyring always; ~/.codex/.auth.json only if codex CLI already
created it. Preserves any pre-existing OPENAI_API_KEY field and
agent_identity field in the .auth.json so the OAuth path coexists
with codex CLI's other auth modes.
`pattern auth clear --provider openai` calls CodexAuthStore::forget,
which deletes both the keyring entry ("Codex Auth" service) and the
auth.json file (idempotent — missing entries are not errors).
`pattern auth status --provider openai` resolves through
OpenAiAuthChain and prints the active tier (StoredOauth / Pkce /
ApiKey) + token prefix + expiry + account_id, mirroring the Anthropic
status output.
Deps added to pattern_cli: reqwest (for the OpenAiAuthChain http
client and codex_oauth refresh-on-login path); `open` crate (browser
auto-open for the loopback flow).
Tests: existing 345 pattern-cli tests continue to pass. No new unit
tests for the auth subcommand itself — it's thin wiring over
codex_oauth + codex_storage + OpenAiAuthChain, which all have
thorough coverage in their own modules. Manual smoke test of the
real flow happens in Phase 6 (live-fixture capture).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
it's time. There's still more to do, but bc nobody is afaik actively using the current main branch and i want to get the relicense visible, we're going live with it.