Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
226475b
fix(codex-lcm): address review findings
davidrobertson Aug 1, 2026
3e5aa6d
fix(codex-lcm): preserve concurrent raw events
davidrobertson Aug 1, 2026
25247f8
test(codex-lcm): stabilize contract tests
davidrobertson Aug 1, 2026
d78496e
fix(codex-lcm): harden raw log storage contract
davidrobertson Aug 1, 2026
7718996
fix(codex-lcm): restore cross-version raw lock
davidrobertson Aug 1, 2026
1c94ffb
fix(codex-lcm): isolate worker raw locks
davidrobertson Aug 1, 2026
310cd76
fix(codex-lcm): surface raw storage failures
davidrobertson Aug 1, 2026
bfe3081
fix(codex-lcm): publish raw locks atomically
davidrobertson Aug 1, 2026
c6d869e
fix(codex-lcm): reclaim stranded raw lock candidates
davidrobertson Aug 1, 2026
94e1c09
test(storage): characterize facade and raw fallbacks
davidrobertson Aug 1, 2026
b65870f
refactor(storage): extract types and pure helpers
davidrobertson Aug 1, 2026
94ce59a
refactor(storage): extract summary persistence
davidrobertson Aug 1, 2026
6763618
refactor(storage): extract storage queries
davidrobertson Aug 1, 2026
233c73c
refactor(storage): extract context retrieval
davidrobertson Aug 1, 2026
f606878
refactor(storage): extract persistence leaves
davidrobertson Aug 1, 2026
19705d5
refactor(storage): remove no-op catch handling
davidrobertson Aug 1, 2026
c9f395d
refactor(storage): isolate context packing
davidrobertson Aug 1, 2026
6fb0507
docs: add repository agent guidance
davidrobertson Aug 2, 2026
6a1adf1
fix(codex-lcm): address remaining hook failures
davidrobertson Aug 4, 2026
4b69031
fix(codex-lcm): restore recall routing triggers
davidrobertson Aug 4, 2026
5a4f040
fix(codex-lcm): reclaim legacy locks across platforms
davidrobertson Aug 4, 2026
72f50dc
Revert "fix(codex-lcm): reclaim legacy locks across platforms"
davidrobertson Aug 4, 2026
443cc52
fix(codex-lcm): use one cross-platform raw lock
davidrobertson Aug 4, 2026
f665794
fix(codex-lcm): fsync raw files on Windows
davidrobertson Aug 4, 2026
2486064
test(codex-lcm): avoid platform timing assumptions
davidrobertson Aug 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,23 @@ jobs:
- run: npm test
- run: npm run smoke
- run: npm pack --dry-run

cross-platform-lock:
strategy:
matrix:
os:
- macos-latest
- windows-latest
runs-on: ${{ matrix.os }}
defaults:
run:
working-directory: plugins/codex-lcm
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v6
with:
node-version: 22.22.3
cache: npm
cache-dependency-path: plugins/codex-lcm/package-lock.json
- run: npm ci
- run: node --test --test-name-pattern="raw-log workers with the same PID|hook recovers after its lock-owning worker terminates|concurrent single ingest writers append" tests/storage.test.ts tests/hook-cli.test.ts
103 changes: 103 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# PROJECT KNOWLEDGE BASE

Generated: 2026-08-01
Commit: c9f395d
Branch: fix/review-findings

## OVERVIEW

Codex LCM is a local-first Codex plugin that captures sanitized lifecycle events,
stores raw JSONL before indexing, and exposes source-backed recall through MCP.
The repository root is the marketplace, CI, release-documentation, and package
wrapper; the TypeScript product lives under `plugins/codex-lcm`.

## STRUCTURE

```text
./
├── .agents/plugins/marketplace.json # Local marketplace registration
├── .github/workflows/ci.yml # Package quality gate
├── docs/releases/ # Tagged release notes and checklists
└── plugins/codex-lcm/ # Publishable native Codex plugin
├── .codex-plugin/ # Plugin manifest
├── hooks/ # Lifecycle hook registration
├── skills/ # Bundled recall skill
├── src/ # TypeScript implementation
└── tests/ # Node test suite
```

Generated work under `.omo/` and `.superpowers/` is task evidence, not product
source. Do not infer project architecture or contribution rules from it.

## WHERE TO LOOK

| Task | Location | Notes |
|---|---|---|
| Plugin behavior | `plugins/codex-lcm/src/` | See the local `AGENTS.md` for storage invariants |
| Public commands and scripts | `plugins/codex-lcm/package.json` | The root has no package manifest |
| Install wiring | `plugins/codex-lcm/.codex-plugin/plugin.json` | Points to MCP, hooks, and skills |
| MCP registration | `plugins/codex-lcm/.mcp.json` | Starts `node ./bin/codex-lcm mcp` |
| Hook registration | `plugins/codex-lcm/hooks/hooks.codex.json` | Validated by the manifest test |
| Architecture | `plugins/codex-lcm/docs/architecture.md` | Detailed storage and retrieval design |
| Tests | `plugins/codex-lcm/tests/` | See the local test guidance |
| CI | `.github/workflows/ci.yml` | Runs from the plugin package directory |
| Release state | `README.md`, `docs/releases/` | Keep version, tag, and release docs aligned |

## CODE MAP

| Symbol | Location | Role |
|---|---|---|
| `LcmStorage` | `plugins/codex-lcm/src/storage.ts` | Public storage facade |
| `createStorage` | `plugins/codex-lcm/src/storage.ts` | Main construction path and repository-wide hub |
| `main` | `plugins/codex-lcm/src/cli.ts` | CLI dispatcher called by the bin wrapper |
| `startMcpServer` | `plugins/codex-lcm/src/mcp.ts` | JSON-RPC stdio server |
| `callTool` | `plugins/codex-lcm/src/mcp-tools.ts` | MCP tool dispatch |
| `runHook` | `plugins/codex-lcm/src/hook.ts` | Lifecycle-event ingestion |

Centrality was checked with TypeScript language-server references. No codegraph
service is available in this workspace; `createStorage` is the dominant shared
entry point, while the CLI and hook symbols are narrow boundaries.

## CONVENTIONS

- Run Node and npm commands from `plugins/codex-lcm`; root-level npm commands
target no package.
- Keep the plugin dependency-light. It currently has no runtime npm dependencies.
- Treat `.codex-plugin/plugin.json`, `.mcp.json`, and the hook manifest as one
install surface, and update their contract test when wiring changes.
- Keep root documentation focused on installation, releases, and repository
navigation. Put implementation rules in the nearest child `AGENTS.md`.
- Use Node 22.18 or newer locally. CI pins Node 22.22.3.

## ANTI-PATTERNS

- Do not edit `.omo/` evidence as if it were product code.
- Do not copy the full hook or MCP tool lists into contributor guidance; the
manifests, package README, and tests are the source of truth.
- Do not release with a mismatch among `package.json`, the Git tag, root README,
and `docs/releases/`.
- Do not add a root build system for the single nested package.

## COMMANDS

```sh
cd plugins/codex-lcm
npm run typecheck
npm test
npm run smoke
npm pack --dry-run
```

For focused runtime checks, use `node bin/codex-lcm --help`, `doctor --json`,
`health --json`, or `stats --json` from the package directory. Use a temporary
`CODEX_LCM_HOME` for tests and experiments; never point destructive checks at a
user's live store.

## NOTES

- `events.jsonl` is authoritative; SQLite, FTS, summaries, and graph views are
derived and rebuildable.
- Native plugin refreshes may require removing and re-adding the marketplace,
then restarting Codex so the cache, MCP server, hooks, and skill reload.
- Release procedures live in `docs/releases/`; do not encode a second checklist
here.
66 changes: 66 additions & 0 deletions plugins/codex-lcm/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Codex LCM package

This directory is the publishable Codex plugin package. Keep package changes
focused on the native plugin, its command entry point, and the files shipped
with the package. Repository-wide release, marketplace, and navigation rules
live in the parent `AGENTS.md`.

## Package boundary

- Run all package commands from `plugins/codex-lcm`.
- Treat `README.md`, `package.json`, and the package source as the package
contract; update documentation when a public command or install behavior
changes.
- Keep the package dependency-free at runtime. Do not add a production npm
dependency when Node's standard library or existing code is enough.
- Node 22.18 or newer is required. CI currently exercises Node 22.22.3.

## Native install surface

These files form one install contract and must stay aligned:

- `.codex-plugin/plugin.json` declares the plugin, skill directory, MCP file,
and hook file.
- `.mcp.json` starts the local stdio server with `node ./bin/codex-lcm mcp`.
- `hooks/hooks.codex.json` routes supported lifecycle events through the
package entry point and `${PLUGIN_ROOT}`.
- `skills/` contains the bundled `lcm-recall` skill.

When changing any manifest path, command, hook event, or package metadata,
update `tests/plugin-manifest.test.ts` in the same change. Keep hook commands
portable and rooted at `${PLUGIN_ROOT}`; do not hard-code a checkout path or a
different plugin variable.

## TypeScript and modules

- The package uses native ESM (`"type": "module"`).
- Source files are TypeScript executed directly by the supported Node test and
command paths; preserve explicit `.ts` specifiers for local modules, as
enabled by `allowImportingTsExtensions` in `tsconfig.json`.
- Use Node built-ins and the existing package patterns before introducing a
helper or dependency.
- Keep `bin/codex-lcm` as the thin command boundary; put behavior in `src/`.

## Commands

```sh
npm run typecheck
npm test
npm run smoke
npm pack --dry-run
```

Use a temporary `CODEX_LCM_HOME` for local capture, import, or retrieval
checks. `npm pack --dry-run` may use a temporary npm cache when the global
cache is not writable.

## Package anti-patterns

- Do not add a second installer or separate CLI install step; native plugin
installation owns MCP, hooks, and skills.
- Do not duplicate hook or MCP inventories in contributor docs; manifests and
their contract test are the source of truth.
- Do not add a runtime framework, bundler, or root-level package manifest for
this single nested package.
- Do not write generated indexes, smoke-test state, or user data into the
repository. Keep raw storage and temporary checks outside the checkout.
15 changes: 7 additions & 8 deletions plugins/codex-lcm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,27 +67,26 @@ can still be recovered through `lcm_pack_context`.

Standard recall workflow:

- `lcm_grep`: find relevant sessions by searching summary nodes, session summaries, and high-signal events. Set `contentScope: "overflow"` or `"both"` to scan recent sanitized overflow payloads in full, subject to the 8 MiB input safety limit.
- `lcm_grep`: find relevant sessions by searching summary nodes, session summaries, and high-signal events. Set `contentScope: "overflow"` or `"both"` to scan sanitized overflow payloads in full, subject to the 8 MiB input safety limit.
- `lcm_describe`: inspect a session, summary node, or indexed file reference with compact source counts. Overflow results use stable `overflow:<sha256>` IDs; pass `offset` and `maxBytes` to page their content in chunks of at most 512 KiB. Set `includeLineage: true` when exact source ID arrays are needed.
- `lcm_expand`: expand one summary node into bounded source summary nodes and high-signal source events.
- `lcm_expand_query`: answer a focused retrieval need by searching matching summary nodes and recursively expanding their source lineage into bounded evidence. The default budget is 2000 tokens. Use `overview: true` for broad, source-rich lineage views. `sourceLimit` is per matched node/source expansion, and tight budgets reserve room for a focused source-event excerpt when one exists.
- `lcm_pack_context`: pack relevant summary-node context into a model-ready Markdown block.

Diagnostics and lower-level tools:

- `lcm_health`
- `lcm_stats`
- `lcm_context_plan`
- `lcm_current_session`
- `lcm_list_sessions`
- `lcm_usage`
- `lcm_search_sessions`
- `lcm_get_session`
- `lcm_get_session_summary`
- `lcm_get_session_graph`
- `lcm_get_recent_context`
- `lcm_record_note`

Compatibility tools remain callable but are not part of the standard workflow: `lcm_health` (use `lcm_stats`), `lcm_search_sessions` (use `lcm_grep`), and `lcm_get_session_summary` (use `lcm_describe`).

## Installing

Use the root [Installation](../../README.md#installation) section for install,
Expand Down Expand Up @@ -129,7 +128,7 @@ Project and git data are metadata only. Search and retrieval are session-first a

## DAG And Long Sessions

SQLite stores a derived DAG alongside FTS:
`lcm_get_session_graph` derives a bounded DAG on demand from indexed events and summary lineage; it does not persist a duplicate graph projection:

- Session nodes anchor each Codex session.
- Turn nodes group events with the same `turn_id`.
Expand All @@ -138,7 +137,7 @@ SQLite stores a derived DAG alongside FTS:
- Typed edges include `contains`, `next`, `tool_result`, `checkpoint`, and
`summary_source`.

Unknown edge kinds use recursive cycle checks. Known internal edge kinds (`contains`, `next`, `tool_result`, `checkpoint`, and `summary_source`) skip that expensive check because they are derived from append-only session order or summary-source lineage. The graph is derived from raw events and can be rebuilt; `events.jsonl` remains the source of truth.
The graph is deterministic and rebuildable; `events.jsonl` remains the source of truth.

SQLite also stores deterministic extractive summaries in `session_summaries` and
`session_summary_fts`. A summary contains a title, overview, topics, key user
Expand All @@ -156,7 +155,7 @@ The summary-node layer adds a second derived index:

- D0 summary nodes summarize bounded chunks of high-signal events.
- D1 and deeper summary nodes summarize lower-depth summary nodes.
- `summary_source` edges are persisted and connect summary nodes back to their child nodes or raw event nodes.
- `summary_source` edges are derived from stored source IDs when graph slices are read.
- `summary_node_fts` lets retrieval search the summary DAG before falling back to raw event FTS.

This mirrors the lossless-context pattern used by systems such as lossless-claw
Expand All @@ -172,7 +171,7 @@ summary text; empty completion markers are still stored and counted.

Use `lcm_stats` or `node bin/codex-lcm stats --json` to inspect aggregate index
shape without reading raw transcript text. The stats output includes summary
nodes by depth, summary source types, hook-event counts, graph node and edge
nodes by depth, summary source types, hook-event counts, derived graph node and edge
counts, freshness timestamps, max summary depth, `session_summary_count`,
`sessions_with_session_summary`, and the number of sessions with summary nodes.
The legacy `summary_count` field is kept as an alias for session summaries, not
Expand Down
18 changes: 5 additions & 13 deletions plugins/codex-lcm/docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,6 @@ SQLite tables:
- `summary_nodes`
- `summary_node_fts`
- `file_refs`
- `graph_nodes`
- `graph_edges`

Codex LCM creates the index opportunistically during ingestion. If indexing is unavailable, raw-log fallback scans keep health, session lookup, retrieval, and basic search usable.

Expand Down Expand Up @@ -99,8 +97,8 @@ Each summary records:
- source event IDs

`session_summary_fts` lets broad topic queries match these compact clues before
the caller loads raw events. `lcm_get_session_summary` exposes a summary
directly.
the caller loads raw events. `lcm_describe` exposes a session summary directly;
`lcm_get_session_summary` remains as a compatibility tool.

`summary_nodes` stores a multi-depth summary DAG. D0 nodes summarize bounded
chunks of high-signal source events. D1 and deeper nodes summarize lower-depth
Expand Down Expand Up @@ -138,9 +136,9 @@ sample before extracting topics and outcomes. The summary should capture the
initial task, recent drift, and the latest result without scanning an entire
giant transcript on every hook event.

## DAG Index
## Derived DAG

The graph index is derived and deterministic. Raw JSONL remains the source of truth.
Graph slices are derived on demand from indexed events and summary source IDs. No duplicate node/edge projection is persisted; raw JSONL remains the source of truth.

Node kinds:

Expand All @@ -158,13 +156,7 @@ Edge kinds:
- `checkpoint`: session to checkpoint.
- `summary_source`: summary node to source event node or lower-depth summary node.

`summary_source` edges are persisted when summary nodes are rebuilt. Older
indexes without persisted summary-source rows still get a synthesized fallback
when graph slices are read. Unknown edge kinds run a recursive reachability
check from the prospective child to the prospective parent. Known internal edge
kinds (`contains`, `next`, `tool_result`, `checkpoint`, and `summary_source`)
skip that expensive check because they are derived from append-only session
order or summary-source lineage.
All edge kinds are synthesized from append-only session order or summary-source lineage when graph slices are read.

For very long sessions, callers should prefer bounded graph and event access:

Expand Down
Loading