Skip to content

Repository files navigation

md2okf

Drop Markdown files into md/, run make wiki, and the Pi coding agent writes an OKF knowledge base into okf/. It takes one source document per run and folds it into the wiki: a page per topic, an index in every directory, links between them, and a log of what each run changed.

OKF, the Open Knowledge Format, is a tree of Markdown files with YAML frontmatter and nothing else. No schema registry, no server, nothing to install. SPEC.md at the repo root is the OKF specification this wiki is built against. The agent reads it at the start of every run, so the spec outranks anything written here.

Path Description
md/ source documents, one Pi run each
okf/ the generated wiki
Makefile every task worth running; make wiki compiles
scripts/ what the Makefile calls — compile, sandbox shell, kit validation
pi/ what the scripts run: the Docker Sandbox kit and the config it carries
SPEC.md the OKF specification the wiki is built against
AGENTS.md instructions for coding agents working on this repo, not for Pi
pdf2md/ optional: converts a PDF into md
web2md/ optional: scrapes a documentation site into md

Compile a wiki

Set up, once

brew install docker/tap/sbx

sbx keeps the OpenRouter key out of the virtual machine. It holds the real string on the host and swaps it into requests at its proxy, so inside the sandbox $OPENROUTER_API_KEY reads proxy-managed. Set it twice:

export OPENROUTER_API_KEY=sk-or-...

echo "$OPENROUTER_API_KEY" | sbx secret set -g openrouter

# And again as a custom secret, to work around a known sbx bug. https://github.com/docker/sbx-releases/issues/25
sbx secret set-custom pi-kit \
  --host openrouter.ai \
  --env OPENROUTER_API_KEY \
  --value "$OPENROUTER_API_KEY"

pi-kit is the kit's name, which comes from pi/spec.yaml.

Run it

Put your Markdown in md/, then:

make wiki

The driver throws the old sandbox away and builds a fresh one, so the current kit and secrets apply. It then runs Pi once for each md/*.md file. okf/ is gitignored apart from okf/.okflintrc.json, so the wiki itself stays out of the repo. md/ is tracked, and ships with one sample document.

What lands in okf/

okf/
├── index.md          # root index, the only one carrying frontmatter
├── log.md            # what each run changed, newest first
├── <page>.md         # a content page at the wiki root
└── <topic>/          # one directory per topic, nested as deep as it needs
    ├── index.md      # a plain link list for this directory
    └── <page>.md     # a content page within the topic

Content pages carry type, title, description and tags in their frontmatter. Slugs are kebab-case. Links are bundle-absolute, so /glossary/verb.md rather than glossary/verb.md. The root index.md names the spec version the agent read. Pages are updated in place, not duplicated, so compiling the same document twice is safe.

Getting Markdown in

md/ wants clean, structured Markdown, and a source document is rarely that. Two helpers produce it. Both are optional, and neither is part of make wiki.

From a PDF. marker converts one with the help of a language model, either a local Ollama model or a cloud model through OpenRouter. Expect to check the output. The step is manual and not wired into makepdf2md/README.md has the commands.

From a website. make scrape walks a documentation site and writes one Markdown document into md/. No model is involved, so the result is deterministic, and the fetched HTML is cached — see web2md/README.md.

How the agent knows what to do

The instructions come in two parts. AGENTS.md holds what every task must respect: the OKF conventions, the directories the agent may write to, and the rule that SPEC.md outranks both. Each task's procedure lives in a skill of its own. There is one today, compile-wiki, and a new task gets a new directory rather than more rules in AGENTS.md.

A skill is a directory holding a SKILL.md — YAML frontmatter with a name and description, then the instructions, plus any scripts it needs. Pi picks skills up from ~/.pi/agent/skills/.

The kit is pi/, and the config it carries lives in pi/files/home/.pi/agent/. That config is copied into the sandbox when the kit is built, not mounted, so an edit reaches Pi on the next fresh sandbox — which make wiki always builds. pi/README.md covers the model and provider settings.

Linting the wiki

okf-lint checks the wiki against the spec. Rules live in okf/.okflintrc.json, tracked and un-ignored by name so it survives the okf/* rule in .gitignore.

The sandbox installs okf-lint at a pinned version, and the compile-wiki skill wraps it in scripts/lint-okf.sh. The agent lints its own output and fixes what the linter reports before it finishes. On the host, make lint-okf runs the same tool through pnpm dlx. It sits outside make lint and outside CI because okf/ is generated.

Development

make lint       # markdownlint, shellcheck, ruff
make test       # pytest, the web2md scraper suite
make validate   # check pi/spec.yaml against the Sandbox Kit schema
make lint-okf   # lint the generated wiki

Touch anything under pi/ or scripts/ and run make validate before you call the job done. It checks the kit spec against the schema bundled in your sbx binary, and needs no Docker, no login and no network. CI runs the same check in its validate-kit job, so catching a break locally saves a red build.

To look inside the sandbox:

./scripts/bash.sh   # reuses the sandbox and whatever a run left behind

Once a sandbox exists, this should print proxy-managed rather than your key:

sbx exec pi-kit -- sh -lc 'echo "$OPENROUTER_API_KEY"'

Python tooling is thin, split across three dependency groups: dev (ruff), test (pytest) and web2md (the scraper's runtime deps). CI installs one group per job with --only-group, so neither the lint job nor the test job pulls in the heavy project dependencies (marker-pdf, torch). The only first-party Python is the web2md scraper.

About

Convert Markdown files to an OKF knowledge base

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages