Skip to content

feat(ai): online mode — local per-file Bob diagrams, matched by start line - #35

Open
Yoshitd wants to merge 15 commits into
mainfrom
feat/ai-online-mode
Open

feat(ai): online mode — local per-file Bob diagrams, matched by start line#35
Yoshitd wants to merge 15 commits into
mainfrom
feat/ai-online-mode

Conversation

@Yoshitd

@Yoshitd Yoshitd commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Online (AI Assisted) mode — local per-file Bob diagrams

A developer runs Bob on one file; ZDoc gets back one Mermaid flowchart per
function and attaches each to its symbol by the function's starting line.
Deliberately small — no daemon, no snippet assembly, no Bob extension.

What's here

  • ai/annotate.{h,c} — the online-mode step. zdoc_ai_annotate(path, mod, bob_cli)
    takes an already-parsed Module (online mode does not parse), runs Bob
    once (Bob reads the file itself), then splits Bob's reply on the
    ## line <N> headers and stores each flowchart into the Symbol whose
    starting line is N — filling Symbol.diagram. Starting line is the match key.
    Cross-platform: fork+exec on POSIX, CreateProcess on Windows (prompt
    passed as a single argument, so its backticks/quotes never touch a shell).
  • ai/zdoc_ai.c — a thin standalone tool. Parses ONE file only to obtain a
    module to hand to zdoc_ai_annotate, then prints each stored diagram.
    make -C ai zdoc_ai
    ./ai/zdoc_ai path/to/File.(c|java|plx) [--bob <bob-binary>]
  • Prompt hardened against hallucination: read the file first; diagram only
    code actually in the body; never invent steps/branches/calls; use only names
    from the source; fall back to A[No executable logic] / A[Source unavailable]
    instead of guessing; self-check every node against the code.

How to test

# Real Bob (on PATH + authenticated; run so Bob can open the file):
make -C ai zdoc_ai
./ai/zdoc_ai docs/student_grades.plx

# No-credit dry run (fake bob echoes a flowchart per listed line):
mkdir -p /tmp/fakebobdir
cat > /tmp/fakebobdir/bob <<'SH'
#!/bin/sh
printf '%s\n' "$4" | grep -oE 'line [0-9]+:' | sed -E 's/line ([0-9]+):/\1/' | while read n; do
  printf '## line %s: fn\n```mermaid\nflowchart TD\n    A[Entry %s]-->B[Return]\n```\n\n' "$n" "$n"
done
SH
chmod +x /tmp/fakebobdir/bob
./ai/zdoc_ai docs/student_grades.plx --bob /tmp/fakebobdir/bob   # -> stored 7 of 7 diagrams

Verified: docs/student_grades.plx → 7/7 diagrams stored into the right
Symbol.diagram by line (181 INITIALIZE_SYSTEM, 224 ADD_STUDENT, …); C and
Java parse paths work too.


Connecting online mode to the CLI/daemon (not in this PR)

Online mode is exactly zdoc_ai_annotate(path, mod, bob_cli). Because it takes
an already-parsed module and does not parse, wiring it in is small:

  1. Build: link ai/annotate.c into the daemon/CLI and #include "ai/annotate.h".
  2. Call it after the parse stage, per parsed module, when --mode ai:
    /* daemon already has the file path and the parsed Module *mod */
    if (opt.mode == ZD_MODE_AI)
        zdoc_ai_annotate(path, mod, opt.bob_cli);   /* fills mod->symbols[i].diagram */
    The daemon already holds both the path and the parsed Module, so nothing is
    parsed twice.
  3. Render as usual — the renderers already consume Symbol.diagram.

Follow-ups for that integration (out of scope here):

  • Concurrency + timeout: one Bob call per file, network-bound — run the
    per-file calls on the daemon's existing worker pool with a bound, and add a
    per-call timeout.
  • Config pass-through: zdoc_ai_annotate takes only the Bob binary today;
    thread bob_args through if needed.
  • MD renderer: html_renderer already emits Symbol.diagram; the
    md_renderer does not yet — add a fenced ```mermaid block.
  • Pre-existing main build bugs unrelated to this PR, worth fixing before
    daemon integration: zdoc/Makefile references the renamed
    parser/plx_parser/helpers.c, and html_render.c re-declares xmalloc
    static against the non-static decl in parser_shared.h.

🤖 Generated with Claude Code

Yoshitd and others added 15 commits July 14, 2026 16:46
Add the AI Assisted mode surface: the zdoc-diagram Bob skill and the
bob_client library unit that drives it.

The skill emits one sanitized raw Mermaid flowchart per symbol (node
shape encodes step/decision/call/return; labels restricted to a
Mermaid-safe charset) — no intermediate JSON graph, matching the
docs/ZDOC.md contract where Bob returns Mermaid that ZDoc embeds
directly.

bob_client spawns Bob via fork/execvp with the snippet as a single argv
element (no shell, no escaping, no injection), captures stdout, and
sanitizes the response into a fence-less flowchart safe to embed. Exposes
bob_diagram() and bob_annotate() (writes Symbol.diagram). Library only;
the AI-mode CLI and daemon link against it and supply snippets.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the context-closure half of bob_client so AI mode gives Bob just the
context it needs, not the whole file and not a body stripped of names.

- closure.{c,h}: name->declaration hash index (open-addressing, FNV-1a),
  keyword-filtered ref extraction, and a budgeted tiered closure (tier 0
  direct refs never crowded out; tier 1+ transitive). bc_build_snippet
  emits the skill's exact DOC/DECLARATIONS/CALLEES/FUNCTION contract.
- util.{c,h}: arena, string builder, file slurp.
- Composes with bob_client: closure -> --snippet, bc_lang_display -> --lang.
- tests/test_closure.c + Makefile `test`: offline unit tests (index,
  case folding, budget tiering, snippet order, ref extraction) — all pass.
- tests/live_bob.c + Makefile `live`: manual end-to-end check vs real Bob.
- AI-FRONT-NOTES.md: reframe the remaining blocker as parser-side data
  exposure (bodies + decl pool + callee lines), the closure being done.

Ports the good part of the ai-skill-context branch; leaves behind the
JSON-graph/sha256 machinery that mode discarded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Testing against real bob 1.0.6 exposed docs/ZDOC.md's Bob interface as
fictional: there is no `explain --diagram --brief --lang --snippet`
subcommand, and Bob has no "skills" — it loads "extensions" with a
bob-extension.json manifest.

- Repackage .bob/skills/zdoc-diagram (Claude skill format) as a real Bob
  extension .bob/extensions/zdoc-diagram: bob-extension.json manifest,
  context.md (SKILL.md + conventions folded into always-on context), and
  the reused golden examples. `bob extensions validate` passes.
- Rewrite bob_client run_bob() to the real one-shot prompt call:
  `bob -o text --chat-mode ask -y "<instruction>\n\n<snippet>"`. The
  closure's snippet becomes the prompt; the extension carries the output
  contract. Prompt passed as a single argv element (no shell).
- Update docs (README, AI-FRONT-NOTES) with the real invocation and the
  `bob extensions link` step.

Closure and sanitizer are unchanged — both were already correct. `make
test` still passes; live harness compiles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tighten the Bob prompt and zdoc-diagram context so Bob answers from the
prompt alone — no tool calls, file reads, or explanation — and returns
only the Mermaid block.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract all OS-specific process work out of bob_client.c into a single
bc_spawn_capture seam (spawn.h) with two implementations selected by the
Makefile: spawn_posix.c (fork/pipe/execvp/waitpid) and spawn_win.c
(CreateProcess/CreatePipe). The Windows impl rebuilds a correctly-quoted
command line per the CommandLineToArgvW rules, so the snippet round-trips
as a single argv element on both platforms with no shell and no injection.
bob_client.c above the seam is now portable (strtok_r->strtok_s guard;
_POSIX_C_SOURCE and unistd/sys-wait confined to spawn_posix.c).

Behaviour unchanged on POSIX: closure tests pass and the fake-bob
spawn->capture->sanitize path returns the same fence-stripped flowchart
(good) / NULL (missing bob), clean under ASan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ai/ai_mode.{h,c}: the single entry point the CLI/daemon will call to
run online mode. Given the module_tree tables and the parsed Module array
(the same inputs the renderers take), zdoc_ai_annotate resolves each
module's source via Module.pathIndex + fs_walk_root_prefix, slices each
symbol's body, builds the zdoc-diagram snippet, and fills Symbol.diagram
through the Bob client. Silent-skip failure policy; returns the count
annotated. Deliberately NOT wired into the daemon (see PR handoff).

Body slicing is a documented bridge until the parser exposes per-symbol
bodies + a declaration pool (AI-FRONT-NOTES §2.1); swapping it in later
does not change this interface.

ai/ai_mode_live.c + cc -O2 -std=c11 -Wall -Wextra -Wpedantic -o ai_mode_live ai_mode_live.c ai_mode.c bob_client/bob_client.c bob_client/closure.c bob_client/util.c bob_client/spawn_posix.c ../extractor/doc_extractor/module_tree/modtree_table.c ../extractor/doc_extractor/module_tree/fs_walk.c ../extractor/doc_extractor/module_tree/path_interface.c
./ai_mode_live
zdoc_ai_annotate: 2 of 2 symbols annotated

==== INITPROC ====
flowchart TD
    A[Entry: INITPROC] --> B[Obtain storage for control block]
    B --> C{Storage obtained?}
    C -- No --> D[Return RC=8 - storage failure]
    C -- Yes --> E[Chain CB onto anchor list]
    E --> F[Return RC=0]

==== TERMPROC ====
flowchart TD
    A[Entry: TERMPROC] --> B{Anchor list empty?}
    B -- Yes --> C[Return RC=4 - nothing to terminate]
    B -- No --> D[Release first control block]
    D --> E[Clear anchor list head]
    E --> F[Return RC=0] drive the whole interface against
the real Bob CLI without the daemon. Verified end-to-end offline with a
fake bob: both fixture symbols annotated, each receiving its own sliced
body and doc brief.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ai_mode_live now runs a built-in fixture per language by default and
accepts a path argument to diagram one real source file whole. Language
is derived from the file extension by the interface itself. Verified
offline with a fake bob: 6/6 fixture symbols across PL/X/C/Java annotated,
each labeled with the correct language token and its own sliced body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nsion

Inline the full ZDoc output contract (flowchart TD rules, node shapes,
label sanitization, brief-granularity) into build_prompt so online mode is
correct with no Bob extension linked. The zdoc-diagram extension now only
adds optional per-language examples — it is a quality boost, not a
correctness dependency. This removes the 'is the extension installed'
requirement for CI: only Bob auth is needed. Verified end-to-end offline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The diagram contract now lives in the prompt (build_prompt), so the Bob
extension is no longer part of any code path. Delete
.bob/extensions/zdoc-diagram/ and scrub every doc/comment that told users
to link it or claimed the contract lived there. Online mode now needs only
a working, authenticated Bob CLI — nothing to install or configure, in CI
or anywhere. No logic change; closure tests pass and the whole-interface
fake-bob smoke still annotates all fixtures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per the local-developer model: Bob runs on one specific file, and each
diagram is attached to its symbol by the function's starting line.

- Add zdoc_ai_annotate_file(path, mod, opt): read one file, slice each
  symbol's body by its starting line (to the next symbol's line), diagram
  per symbol into Symbol.diagram. Shares annotate_from_source with the
  batch path.
- Add the ai/zdoc_ai CLI: parse ONE file with the real parser
  (parse_file), annotate per symbol, print a Mermaid block per symbol
  headed by 'line <N>: <name>'.

Verified with a fake bob on real sources: docs/student_grades.plx -> 7
procedures each sliced + diagrammed at their own line (181 INITIALIZE_
SYSTEM, 224 ADD_STUDENT, ...); ai/bob_client/util.c -> 11 C symbols, each
keyed to its line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per the boss's model, Bob is a local per-file tool and reads the file
itself, so the C side has no business shelling out to Bob or assembling
snippet strings. Remove all of it:

- delete the Bob subprocess/cross-platform layer (spawn.*, bob_client.c/.h)
- delete the string/closure/util layer (closure.*, util.*, its tests)
- delete the ai_mode annotate abstraction and its live drivers

zdoc_ai is now the whole thing: parse ONE file with the real parser to get
each function + its starting line, then print the diagram prompt plus the
file and the (start line, name) list. Bob reads the file and returns one
flowchart per function, matched to its symbol by starting line.

Builds with no Bob and runs offline: docs/student_grades.plx -> 7 functions
listed with their start lines (181 INITIALIZE_SYSTEM, 224 ADD_STUDENT, ...).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Faithfulness section to the prompt: read the file first, diagram
only code actually in the function body, never invent steps/branches/calls,
use only names from the source, fall back to A[No executable logic] /
A[Source unavailable] instead of guessing, and self-check every node against
the code before finishing. (No prompt can guarantee zero hallucination, but
this strongly constrains it.)

Remove ai/AI-FRONT-NOTES.md — it documented the deleted closure/spawn/
bob_client design and no longer describes anything that exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
zdoc_ai now does the full round-trip in one run: parse the file, build the
prompt, run Bob once (fork+exec, prompt as a single argv element so its
backticks never hit a shell), then split Bob's reply on the '## line <N>'
headers and store each flowchart into the symbol whose starting line is N.
Bob reads the file itself; the starting line is the match key.

Verified with the real parser + a fake bob: docs/student_grades.plx ->
7 of 7 diagrams stored into the right symbol->diagram by line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cess)

Keep it in one file: run_bob has a #ifdef _WIN32 branch (CreateProcess +
pipe, with CommandLineToArgvW-correct quoting of the prompt) alongside the
POSIX fork/exec branch — no separate spawn seam. Also replace the POSIX-only
open_memstream prompt builder with a portable malloc+snprintf one. POSIX
build verified: 7/7 diagrams stored by line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ne tool

Online mode does not parse. Move the run-Bob-and-store-by-line logic into
annotate.{h,c} as zdoc_ai_annotate(path, mod, bob_cli): it takes an
ALREADY-PARSED module and fills each Symbol.diagram, matched by starting
line. zdoc_ai.c is now a thin standalone front end that parses one file
only to obtain a module to hand it. The daemon/CLI would link annotate.c
and call zdoc_ai_annotate on modules it already parsed — no second parse.

Verified: docs/student_grades.plx -> 7/7 diagrams stored by line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant