diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index fab1ef0..a5f39b2 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -118,3 +118,111 @@ See the rules files for detailed coding standards: - `.claude/rules/python.md` - Python coding standards, type hints, async patterns - `.claude/rules/python-testing.md` - Testing conventions, markers, coverage requirements + +%% lat:begin %% +# Before starting work + +- Run `lat search` to find sections relevant to your task. Read them to understand the design intent before writing code. +- Run `lat expand` on user prompts to expand any `[[refs]]` — this resolves section names to file locations and provides context. + +# Post-task checklist (REQUIRED — do not skip) + +After EVERY task, before responding to the user: + +- [ ] Update `lat.md/` if you added or changed any functionality, architecture, tests, or behavior +- [ ] Run `lat check` — all wiki links and code refs must pass +- [ ] Do not skip these steps. Do not consider your task done until both are complete. + +--- + +# What is lat.md? + +This project uses [lat.md](https://www.npmjs.com/package/lat.md) to maintain a structured knowledge graph of its architecture, design decisions, and test specs in the `lat.md/` directory. It is a set of cross-linked markdown files that describe **what** this project does and **why** — the domain concepts, key design decisions, business logic, and test specifications. Use it to ground your work in the actual architecture rather than guessing. + +# Commands + +```bash +lat locate "Section Name" # find a section by name (exact, fuzzy) +lat refs "file#Section" # find what references a section +lat search "natural language" # semantic search across all sections +lat expand "user prompt text" # expand [[refs]] to resolved locations +lat check # validate all links and code refs +``` + +Run `lat --help` when in doubt about available commands or options. + +If `lat search` fails because no API key is configured, explain to the user that semantic search requires a key provided via `LAT_LLM_KEY` (direct value), `LAT_LLM_KEY_FILE` (path to key file), or `LAT_LLM_KEY_HELPER` (command that prints the key). Supported key prefixes: `sk-...` (OpenAI) or `vck_...` (Vercel). If the user doesn't want to set it up, use `lat locate` for direct lookups instead. + +# Syntax primer + +- **Section ids**: `lat.md/path/to/file#Heading#SubHeading` — full form uses project-root-relative path (e.g. `lat.md/tests/search#RAG Replay Tests`). Short form uses bare file name when unique (e.g. `search#RAG Replay Tests`, `cli#search#Indexing`). +- **Wiki links**: `[[target]]` or `[[target|alias]]` — cross-references between sections. Can also reference source code: `[[src/foo.ts#myFunction]]`. +- **Source code links**: Wiki links in `lat.md/` files can reference functions, classes, constants, and methods in TypeScript/JavaScript/Python/Rust/Go/C files. Use the full path: `[[src/config.ts#getConfigDir]]`, `[[src/server.ts#App#listen]]` (class method), `[[lib/utils.py#parse_args]]`, `[[src/lib.rs#Greeter#greet]]` (Rust impl method), `[[src/app.go#Greeter#Greet]]` (Go method), `[[src/app.h#Greeter]]` (C struct). `lat check` validates these exist. +- **Code refs**: `// @lat: [[section-id]]` (JS/TS/Rust/Go/C) or `# @lat: [[section-id]]` (Python) — ties source code to concepts + +# Test specs + +Key tests can be described as sections in `lat.md/` files (e.g. `tests.md`). Add frontmatter to require that every leaf section is referenced by a `// @lat:` or `# @lat:` comment in test code: + +```markdown +--- +lat: + require-code-mention: true +--- +# Tests + +Authentication and authorization test specifications. + +## User login + +Verify credential validation and error handling for the login endpoint. + +### Rejects expired tokens +Tokens past their expiry timestamp are rejected with 401, even if otherwise valid. + +### Handles missing password +Login request without a password field returns 400 with a descriptive error. +``` + +Every section MUST have a description — at least one sentence explaining what the test verifies and why. Empty sections with just a heading are not acceptable. (This is a specific case of the general leading paragraph rule below.) + +Each test in code should reference its spec with exactly one comment placed next to the relevant test — not at the top of the file: + +```python +# @lat: [[tests#User login#Rejects expired tokens]] +def test_rejects_expired_tokens(): + ... + +# @lat: [[tests#User login#Handles missing password]] +def test_handles_missing_password(): + ... +``` + +Do not duplicate refs. One `@lat:` comment per spec section, placed at the test that covers it. `lat check` will flag any spec section not covered by a code reference, and any code reference pointing to a nonexistent section. + +# Section structure + +Every section in `lat.md/` **must** have a leading paragraph — at least one sentence immediately after the heading, before any child headings or other block content. The first paragraph must be ≤250 characters (excluding `[[wiki link]]` content). This paragraph serves as the section's overview and is used in search results, command output, and RAG context — keeping it concise guarantees the section's essence is always captured. + +```markdown +# Good Section + +Brief overview of what this section documents and why it matters. + +More detail can go in subsequent paragraphs, code blocks, or lists. + +## Child heading + +Details about this child topic. +``` + +```markdown +# Bad Section + +## Child heading + +Details about this child topic. +``` + +The second example is invalid because `Bad Section` has no leading paragraph. `lat check` validates this rule and reports errors for missing or overly long leading paragraphs. +%% lat:end %% diff --git a/.claude/settings.json b/.claude/settings.json index 1e5b3bf..219a118 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -99,7 +99,13 @@ "Bash(uv run ruff format:*)", "Bash(uv run ruff check:*)", "Bash(uv run pyright:*)", - "Bash(python -m pytest:*)" + "Bash(python -m pytest:*)", + "mcp__lat__lat_locate", + "mcp__lat__lat_section", + "mcp__lat__lat_search", + "mcp__lat__lat_expand", + "mcp__lat__lat_check", + "mcp__lat__lat_refs" ] }, "enableAllProjectMcpServers": false, diff --git a/.coderabbit.yaml b/.coderabbit.yaml index d8e59ab..3da9ffb 100644 --- a/.coderabbit.yaml +++ b/.coderabbit.yaml @@ -192,9 +192,13 @@ knowledge_base: code_guidelines: enabled: true filePatterns: + - ".claude/rules/general.mdc" + - ".claude/rules/python.mdc" + - ".claude/rules/python-testing.mdc" - ".cursor/rules/general.mdc" - ".cursor/rules/python.mdc" - ".cursor/rules/python-testing.mdc" + - "lat.md/**/*.md" learnings: scope: auto issues: diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index f80c4a6..82eec35 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -21,9 +21,6 @@ }, "ghcr.io/devcontainers/features/node:1": { // Required for local Context7 MCP server "version": "lts" - }, - "ghcr.io/devcontainers-community/features/deno:1": { // Required for https://github.com/pydantic/mcp-run-python - "version": "2.5.6" } }, // Port forwarding for local services @@ -141,4 +138,4 @@ // ] // Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root. // "remoteUser": "root" -} \ No newline at end of file +} diff --git a/.github/workflows/lat-check.yaml b/.github/workflows/lat-check.yaml index cd618a4..ab1e2b4 100644 --- a/.github/workflows/lat-check.yaml +++ b/.github/workflows/lat-check.yaml @@ -14,12 +14,6 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout the repository - uses: actions/checkout@v4.2.0 - - name: Set up Node.js - uses: actions/setup-node@v4 - with: - node-version: "22" - - name: Install lat.md - run: npm install -g lat.md + uses: actions/checkout@v4 - name: Run lat check - run: lat check + uses: lars20070/lat-check-action@v1 diff --git a/.mcp.json.example b/.mcp.json.example index 4058abb..4380ee0 100644 --- a/.mcp.json.example +++ b/.mcp.json.example @@ -18,9 +18,8 @@ }, "lat": { "type": "stdio", - "command": "node", + "command": "lat", "args": [ - "/path/to/lat.md/dist/src/cli/index.js", "mcp" ] } diff --git a/.vscode/extensions.json b/.vscode/extensions.json index 4c3e60e..843c9b7 100644 --- a/.vscode/extensions.json +++ b/.vscode/extensions.json @@ -23,6 +23,8 @@ "ms-vscode.vscode-copilot-data-analysis", "AutomataLabs.copilot-mcp", "CodeRabbit.coderabbit-vscode", + // Claude + "anthropic.claude-code", // spell checker "streetsidesoftware.code-spell-checker", // logging diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 9c015aa..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1,107 +0,0 @@ -%% lat:begin %% -# Before starting work - -- Run `lat search` to find sections relevant to your task. Read them to understand the design intent before writing code. -- Run `lat expand` on user prompts to expand any `[[refs]]` — this resolves section names to file locations and provides context. - -# Post-task checklist (REQUIRED — do not skip) - -After EVERY task, before responding to the user: - -- [ ] Update `lat.md/` if you added or changed any functionality, architecture, tests, or behavior -- [ ] Run `lat check` — all wiki links and code refs must pass -- [ ] Do not skip these steps. Do not consider your task done until both are complete. - ---- - -# What is lat.md? - -This project uses [lat.md](https://www.npmjs.com/package/lat.md) to maintain a structured knowledge graph of its architecture, design decisions, and test specs in the `lat.md/` directory. It is a set of cross-linked markdown files that describe **what** this project does and **why** — the domain concepts, key design decisions, business logic, and test specifications. Use it to ground your work in the actual architecture rather than guessing. - -# Commands - -```bash -lat locate "Section Name" # find a section by name (exact, fuzzy) -lat refs "file#Section" # find what references a section -lat search "natural language" # semantic search across all sections -lat expand "user prompt text" # expand [[refs]] to resolved locations -lat check # validate all links and code refs -``` - -Run `lat --help` when in doubt about available commands or options. - -If `lat search` fails because no API key is configured, explain to the user that semantic search requires a key provided via `LAT_LLM_KEY` (direct value), `LAT_LLM_KEY_FILE` (path to key file), or `LAT_LLM_KEY_HELPER` (command that prints the key). Supported key prefixes: `sk-...` (OpenAI) or `vck_...` (Vercel). If the user doesn't want to set it up, use `lat locate` for direct lookups instead. - -# Syntax primer - -- **Section ids**: `lat.md/path/to/file#Heading#SubHeading` — full form uses project-root-relative path (e.g. `lat.md/tests/search#RAG Replay Tests`). Short form uses bare file name when unique (e.g. `search#RAG Replay Tests`, `cli#search#Indexing`). -- **Wiki links**: `[[target]]` or `[[target|alias]]` — cross-references between sections. Can also reference source code: `[[src/foo.ts#myFunction]]`. -- **Source code links**: Wiki links in `lat.md/` files can reference functions, classes, constants, and methods in TypeScript/JavaScript/Python/Rust/Go/C files. Use the full path: `[[src/config.ts#getConfigDir]]`, `[[src/server.ts#App#listen]]` (class method), `[[lib/utils.py#parse_args]]`, `[[src/lib.rs#Greeter#greet]]` (Rust impl method), `[[src/app.go#Greeter#Greet]]` (Go method), `[[src/app.h#Greeter]]` (C struct). `lat check` validates these exist. -- **Code refs**: `// @lat: [[section-id]]` (JS/TS/Rust/Go/C) or `# @lat: [[section-id]]` (Python) — ties source code to concepts - -# Test specs - -Key tests can be described as sections in `lat.md/` files (e.g. `tests.md`). Add frontmatter to require that every leaf section is referenced by a `// @lat:` or `# @lat:` comment in test code: - -```markdown ---- -lat: - require-code-mention: true ---- -# Tests - -Authentication and authorization test specifications. - -## User login - -Verify credential validation and error handling for the login endpoint. - -### Rejects expired tokens -Tokens past their expiry timestamp are rejected with 401, even if otherwise valid. - -### Handles missing password -Login request without a password field returns 400 with a descriptive error. -``` - -Every section MUST have a description — at least one sentence explaining what the test verifies and why. Empty sections with just a heading are not acceptable. (This is a specific case of the general leading paragraph rule below.) - -Each test in code should reference its spec with exactly one comment placed next to the relevant test — not at the top of the file: - -```python -# @lat: [[tests#User login#Rejects expired tokens]] -def test_rejects_expired_tokens(): - ... - -# @lat: [[tests#User login#Handles missing password]] -def test_handles_missing_password(): - ... -``` - -Do not duplicate refs. One `@lat:` comment per spec section, placed at the test that covers it. `lat check` will flag any spec section not covered by a code reference, and any code reference pointing to a nonexistent section. - -# Section structure - -Every section in `lat.md/` **must** have a leading paragraph — at least one sentence immediately after the heading, before any child headings or other block content. The first paragraph must be ≤250 characters (excluding `[[wiki link]]` content). This paragraph serves as the section's overview and is used in search results, command output, and RAG context — keeping it concise guarantees the section's essence is always captured. - -```markdown -# Good Section - -Brief overview of what this section documents and why it matters. - -More detail can go in subsequent paragraphs, code blocks, or lists. - -## Child heading - -Details about this child topic. -``` - -```markdown -# Bad Section - -## Child heading - -Details about this child topic. -``` - -The second example is invalid because `Bad Section` has no leading paragraph. `lat check` validates this rule and reports errors for missing or overly long leading paragraphs. -%% lat:end %% diff --git a/pyproject.toml b/pyproject.toml index afbfa44..e16b613 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "pytest-assay" -version = "0.1.5" +version = "0.1.6" description = "Evaluation framework for Pydantic AI agents" keywords = [ "pytest", "pytest-plugin", "pydantic", "pydantic-ai", diff --git a/tests/_ollama.py b/tests/_ollama.py index fbd8ef5..aef4659 100644 --- a/tests/_ollama.py +++ b/tests/_ollama.py @@ -1,5 +1,7 @@ #!/usr/bin/env python3 """Shared Ollama constants for test configuration.""" -OLLAMA_BASE_URL = "http://localhost:11434" +import os + +OLLAMA_BASE_URL = os.environ.get("OLLAMA_HOST", "http://localhost:11434") OLLAMA_MODEL = "qwen2.5:14b" diff --git a/uv.lock b/uv.lock index 85314c8..df6e1e0 100644 --- a/uv.lock +++ b/uv.lock @@ -3183,7 +3183,7 @@ wheels = [ [[package]] name = "pytest-assay" -version = "0.1.5" +version = "0.1.6" source = { editable = "." } dependencies = [ { name = "choix" },