From 82691138099125e012b097ee8bf41902b97b1100 Mon Sep 17 00:00:00 2001 From: MgnMtn Date: Fri, 10 Jul 2026 10:41:00 +0100 Subject: [PATCH] docs: documentation for user agents --- AGENTS.md | 32 ++++++++ CLAUDE.md | 35 ++++++++ docs/ARCHITECTURE.md | 103 ++++++++++++++++++++++++ docs/CONFIG.md | 56 +++++++++++++ docs/DEVELOPMENT.md | 93 ++++++++++++++++++++++ docs/EXTENDING.md | 85 ++++++++++++++++++++ docs/MODULE_REFERENCE.md | 168 +++++++++++++++++++++++++++++++++++++++ docs/README.md | 25 ++++++ 8 files changed, 597 insertions(+) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md create mode 100644 docs/ARCHITECTURE.md create mode 100644 docs/CONFIG.md create mode 100644 docs/DEVELOPMENT.md create mode 100644 docs/EXTENDING.md create mode 100644 docs/MODULE_REFERENCE.md create mode 100644 docs/README.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..5e264f0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,32 @@ +# AGENTS.md + +Guidance for AI coding agents working in this repository. + +*multiplex* is a modular framework for prototyping LLM-based mutation testing. One run mutates a single Java method (located via tree-sitter), generates mutants with an LLM, splices each mutant back into the source file, and runs the target project's tests to see which mutants are killed. + +## Documentation map + +Task-scoped docs live in `docs/` — start at [docs/README.md](docs/README.md) and load only the file relevant to your task instead of reading source: + +- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — pipeline, approaches, execution backends, output artifacts. +- [docs/MODULE_REFERENCE.md](docs/MODULE_REFERENCE.md) — every function's signature, behavior, and side effects. +- [docs/CONFIG.md](docs/CONFIG.md) — full `config.yml` schema. +- [docs/EXTENDING.md](docs/EXTENDING.md) — checklists for adding approaches, execution backends, or LLM providers. +- [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) — setup, conventions, CI, known bugs/gotchas. + +## Commands + +```bash +uv sync -p 3.13 # install deps (pin 3.13: tiktoken fails to build on 3.14) +uv run pytest tests # all tests — must run from repo root (relative resource paths) +uv run pytest tests/checks/test_compiles.py::test_name # one test +uv run ruff check . # lint (matches CI) +uv run ./multiplex ./path/to/config.yml # run the tool +``` + +## Critical rules + +- **Import convention**: modules inside `multiplex/` import each other **without** the package prefix (`from model import Model`); tests import **with** it (`from multiplex.checks... import ...`). Match the file you are editing. Rationale in `docs/DEVELOPMENT.md`. +- Adding an approach touches three places (package + prompt keys + dispatch); all `system_prompts` config keys are read unconditionally at startup. See `docs/EXTENDING.md`. +- tree-sitter versions are pinned; do not bump them casually. +- Check `docs/DEVELOPMENT.md` § Known issues before building on `execute/maven.py` (broken) or the STPA mutant loop (off-by-one); `execute/defects4j.py` is the reference execution backend. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..bf61101 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,35 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Overview + +*multiplex* is a modular framework for prototyping and comparing LLM-based mutation testing approaches. One run mutates a single Java method (located via tree-sitter), generates mutants with an LLM, injects each mutant back into the source file, and runs the target project's tests to determine which mutants are killed. + +## Documentation map + +Detailed docs live in `docs/` — load only the file relevant to the task instead of reading source: + +- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — end-to-end pipeline, approaches, execution backends, output artifacts. +- [docs/MODULE_REFERENCE.md](docs/MODULE_REFERENCE.md) — every function's signature, behavior, and side effects. +- [docs/CONFIG.md](docs/CONFIG.md) — full `config.yml` schema. +- [docs/EXTENDING.md](docs/EXTENDING.md) — checklists for adding approaches, execution backends, or LLM providers. +- [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) — setup, conventions, CI, and the current list of known bugs/gotchas. + +## Commands + +```bash +uv sync -p 3.13 # install deps (pin 3.13: tiktoken fails to build on 3.14) +uv run pytest tests # all tests — must run from repo root (relative resource paths) +uv run pytest tests/checks/test_compiles.py::test_name # one test +uv run ruff check . # lint +uv run ./multiplex ./path/to/config.yml # run the tool +``` + +## Critical conventions + +- **Import convention**: the tool runs as a directory (`uv run ./multiplex`), so `multiplex/` itself is on `sys.path`. Modules inside `multiplex/` import each other **without** the package prefix (`from model import Model`, `from util.io import ...`); tests import **with** it (`from multiplex.checks... import ...`). Match the style of the file you are editing. +- Pipeline entry point and dispatch: `multiplex/__main__.py`. Adding an approach touches three places (new `approach//controller.py` package, prompt keys in config **and** the `prompts` dict in `__main__.py`, dispatch branch) — see docs/EXTENDING.md. +- All `system_prompts` config keys are read unconditionally at startup; a missing key is an immediate KeyError. +- `/output/` is wiped at the start of every run; the target source file is edited in place and restored from a `.orig` backup. +- Before relying on `execute/maven.py` or the STPA mutant loop, check docs/DEVELOPMENT.md § Known issues — both have verified bugs (Maven backend is broken; use `execute/defects4j.py` as reference). diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..04573d6 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,103 @@ +# Architecture + +*multiplex* is a modular framework for prototyping LLM-based mutation testing. +One run mutates **one Java method** in a target project and evaluates the +generated mutants against that project's test suite. + +## Pipeline + +Orchestrated top-to-bottom by `multiplex/__main__.py`: + +``` +config.yml + │ + ▼ +1. Load YAML config; read all 9 system_prompts keys into a dict + │ + ▼ +2. Back up target source file to .orig + Delete and recreate /output/ ← destroys previous results + │ + ▼ +3. util/extract_method.py (tree-sitter) + Find method by name + start line + → writes output/original_method.java + → returns (start_byte, end_byte) offsets into the source file + │ + ▼ +4. approach//controller.main(model, output_path, prompts) + One or more LLM calls (via model.Model / LiteLLM) + → writes mutant files to output/-mutants/mutant_N.java + │ + ▼ +5. execute/{maven,defects4j}.run_mutants(...) + For each mutant file: + a. restore source file from .orig backup + b. util/rewrite_method.py splices mutant text into the file + at the saved byte offsets + c. checks/compilable.py — tree-sitter parse-error scan + checks/syntactic_equivalence.py — AST equality vs original (ignores comments) + d. run the project's tests (Defects4J backend only; see Status below) + → writes output/-mutants/mutant_summary.csv (Defects4J) + │ + ▼ +6. Restore original source file from .orig backup +``` + +Key mechanism: the method's **byte offsets** captured in step 3 are reused in +step 5b to splice each mutant into a pristine copy of the file. Anything that +changes the file between those steps invalidates the offsets. + +## Mutant-generation approaches (`multiplex/approach/`) + +Each approach is a package with a `controller.py` exposing +`main(model, output_dir, prompts)`. Dispatch is an if/elif chain in +`__main__.py` keyed on `mutation.approach`. + +| Approach | Strategy | LLM calls | +|----------|----------|-----------| +| `basic` | Single system prompt, ask for a mutant 10 times | 10 | +| `hazop` | Chain: describe method line-by-line → mutate descriptions using HAZOP guidewords (NO/MORE/LESS/...) → implement each deviation | 2 + 1 per deviation | +| `stpa` | Chain: describe control flow as GraphViz DOT → identify Unsafe Control Actions (UCAs) → implement each UCA | 2 + 1 per UCA | +| `mutahunter` | Single call with AST + line-numbered source; LLM returns YAML of line-level mutations spliced in locally (prompts not bundled — licensing) | 1 | +| `llmorpheus` | tree-sitter query finds mutation sites (conditions, loop headers, call args), each replaced by ``; LLM proposes 3 replacements per site | 1 per placeholder | + +Intermediate artifacts are files in `output/` — approaches communicate between +their own steps via files, not in-memory state (see artifact table below). + +## Execution backends (`multiplex/execute/`) + +Selected by `project.runtool`: + +- `d4j` → `defects4j.py` — the complete backend. Baselines the unmutated + project first (aborts if its tests fail), times the baseline run, then runs + each compilable mutant's tests with a timeout of 5× baseline. A surviving + mutant is one whose test run reports `Failing tests: 0`. Requires the + `defects4j` CLI and a `JDK_11` env var (see DEVELOPMENT.md). +- `mvn` → `maven.py` — **currently broken/incomplete**; do not rely on it + (details in DEVELOPMENT.md § Known issues). + +## Output artifacts + +Everything lands under `/output/` (wiped at the start of each run): + +| Artifact | Written by | +|----------|-----------| +| `original_method.java` | extract_method (step 3); read by every approach | +| `hazop-descriptions.txt`, `hazop-mutated-descriptions.txt` | hazop chain steps | +| `control_diagram.txt`, `ucas.csv` | stpa chain steps | +| `placeholders/{N_placeholder.java, N_orig.java, placeholders.json}` | llmorpheus site finder | +| `-mutants/mutant_N.java` | every approach's final step | +| `-mutants/mutant_summary.csv` | defects4j backend; columns `MUTANT, EQUIVALENCE, COMPILABLE, SURVIVES` | +| `-test/_test.txt` | defects4j backend; per-mutant test output | + +## LLM access (`multiplex/model.py`) + +All LLM traffic goes through `Model.make_request(messages)`, a thin wrapper +over LiteLLM `completion()`. Model/endpoint come from config; the API key is +read from the environment variable *named* by `llm.token_env_var` (never stored +in config). Azure endpoints get `AZURE_AI_API_BASE`/`AZURE_AI_API_KEY` set +specially. To support another provider, replace or extend `model.py`. + +LLM responses are treated as fenced code: approaches strip a leading +```` ```java ```` fence and keep everything before the next ```` ``` ```` fence. diff --git a/docs/CONFIG.md b/docs/CONFIG.md new file mode 100644 index 0000000..30a35da --- /dev/null +++ b/docs/CONFIG.md @@ -0,0 +1,56 @@ +# Configuration Reference + +*multiplex* takes a single YAML config file: +`uv run ./multiplex ./path/to/config.yml`. +Template: [`examples/config.yml`](../examples/config.yml). + +All keys below are read unconditionally by `multiplex/__main__.py` — a missing +key raises `KeyError` at startup, even for approaches you are not running. + +## `project` + +| Key | Type | Meaning | +|-----|------|---------| +| `projectroot` | path | Root of the Java project under test. `output/` is created (and **wiped every run**) inside it. | +| `filename` | path | The `.java` source file containing the method under test. Backed up to `.orig` during the run. | +| `method` | string | Name of the method (or constructor) to mutate. | +| `line` | int | 1-based line number of the method **name identifier** in `filename` (not annotations above it). Both `method` and `line` must match for extraction to succeed; disambiguates overloads. | +| `runtool` | `mvn` \| `d4j` | Execution backend. `d4j` (Defects4J) is the complete one; `mvn` is currently broken (DEVELOPMENT.md § Known issues). | + +## `mutation` + +| Key | Type | Meaning | +|-----|------|---------| +| `approach` | `basic` \| `hazop` \| `stpa` \| `mutahunter` \| `llmorpheus` | Mutant-generation approach. Any other value prints "Invalid approach" and continues to the execution phase (which then fails on the missing mutants dir). | + +## `llm` + +| Key | Type | Meaning | +|-----|------|---------| +| `model` | string | LiteLLM model string, e.g. `ollama/gpt-oss:20b`. | +| `endpoint` | URL | LLM API endpoint, e.g. `http://127.0.0.1:11434`. If the string contains `azure`, Azure env vars are set automatically. | +| `token_env_var` | string or empty | **Name** of the environment variable holding the API key — never the key itself. Leave empty for keyless local endpoints (e.g. Ollama). The named var must exist or startup fails with KeyError. | + +Note: the top-level README shows this key as `tokenenvvar`; the code and +example config use `token_env_var`. `token_env_var` is correct. + +## `system_prompts` + +All nine keys are required (multi-line YAML strings, typically `|` blocks). +User prompts are constructed in code; only system prompts live in config. + +| Key | Used by | +|-----|---------| +| `basic_generate_mutants` | basic | +| `hazop_describe_process` | hazop step 1 (line-by-line description) | +| `hazop_identify_deviations` | hazop step 2 (guideword mutation; must instruct CSV output `number, original_rule, GUIDEWORD, changed_rule`) | +| `hazop_implement_deviations` | hazop step 3 (code generation) | +| `stpa_describe_control_flow` | stpa step 1 (GraphViz DOT control diagram) | +| `stpa_identify_ucas` | stpa step 2 (numbered UCA list) | +| `stpa_implement_ucas` | stpa step 3 (code generation) | +| `mutahunter_generate_mutants` | mutahunter — **not bundled**; users must paste Mutahunter's own prompt text (licensing restriction). Response must be YAML with `mutants: [{mutated_code, line_number}]`. | +| `llmorpheus_system` | llmorpheus (per-placeholder replacement generation) | + +Output-format expectations matter: downstream parsing is brittle string/CSV/ +YAML/regex handling (see MODULE_REFERENCE.md), so prompts must pin the exact +output format the next step expects. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md new file mode 100644 index 0000000..7b1ac03 --- /dev/null +++ b/docs/DEVELOPMENT.md @@ -0,0 +1,93 @@ +# Development Guide + +## Setup and commands + +Dependency management is [uv](https://github.com/astral-sh/uv); Python >= 3.13. + +```bash +uv sync -p 3.13 # install deps incl. dev group (pytest, ruff, pylint) +uv run pytest tests # all tests — run from the REPO ROOT +uv run pytest tests/checks/test_compiles.py # one file +uv run pytest tests/checks/test_compiles.py::test_name # one test +uv run ruff check . # lint (CI uses --output-format=github) +uv run ./multiplex ./path/to/config.yml # run the tool +``` + +- **Pin Python 3.13** (as CI does): `requires-python` allows >= 3.13, but + litellm's `tiktoken` dependency fails to build on 3.14. +- Tests must run from the repo root: they reference `tests/resources/...` + relatively. +- Running the tool needs a reachable LLM endpoint and a target Java project; + `d4j` runs additionally need the `defects4j` CLI (expected under + `./defects4j/framework/bin`) and a `JDK_11` env var pointing at a JDK 11 home. + +CI (GitHub Actions, on push/PR to `main`): `test.yml` runs +`uv run pytest tests` on Python 3.13; `ruff.yml` runs `uv run ruff check`. + +## Import convention (critical) + +The tool is executed as a directory (`uv run ./multiplex`), which puts +`multiplex/` itself on `sys.path`: + +- **Inside `multiplex/`**: flat imports, no package prefix — + `from model import Model`, `from util.io import write_to_file`, + `import approach.basic.controller as basic`. +- **Inside `tests/`**: package-prefixed imports — + `from multiplex.checks.syntactic_equivalence import check_mutant_equivalent`. + +Using the wrong style in either place breaks at import time. This also means +`multiplex/` modules are not importable as `multiplex.x` **and** `x` in the +same process; keep the two worlds separate. + +## Conventions + +- tree-sitter is the universal Java analysis tool (extraction, compilability, + equivalence, placeholder finding). Versions are pinned + (`tree-sitter==0.23.2`, `tree-sitter-java==0.23.5`); the code depends on that + API — do not bump casually. +- Approaches communicate between their own steps via files in `output/`, not + in-memory state; artifact names are prefixed with the approach name. +- LLM code responses are unfenced by stripping a leading ```` ```java ```` + fence and truncating at the next ```` ``` ```` fence + (`removeprefix` + `split`, see EXTENDING.md for the exact two lines). +- Mutant files are named `mutant_.java` in `output/-mutants/`. +- Status/progress reporting is `print()`-based throughout (no logging config, + one `logging.warning` in extract_method). + +## Known issues and gotchas (verified 2026-07) + +Bugs — fix or work around, don't replicate: + +1. **`execute/maven.py` is broken**: `run_mutants` calls `rewrite_method` with + 5 args (signature takes 4 → TypeError), never calls its own + `_execute_mutant` (so `mvn test` never runs), and passes a directory to + `check_mutant_compilable` (which parses a single file). Use + `execute/defects4j.py` as the reference implementation. +2. **STPA off-by-one**: `approach/stpa/code_generator.py` loops + `range(0, ucas_count - 1)`, silently skipping the last UCA. +3. **Method-not-found crash**: `extract_method_from_file` returns `None` on a + miss, but `__main__.py` unpacks it as a tuple → opaque TypeError. Symptom: + check `project.method`/`project.line` in the config (`line` must be the + line of the method-name identifier). +4. **README key mismatch**: README shows `llm.tokenenvvar`; the code reads + `llm.token_env_var` (the example config is correct). +5. **`tests/execute/test_defects4j.py` is an empty stub** — the execute layer + has zero test coverage. + +Behavioral gotchas — by design (or at least current design), be aware: + +- `/output/` is **deleted without prompting** at every run start + (the confirmation prompt in `__main__.py` is commented out, answer + hard-coded to `"y"`). +- The target source file is mutated **in place** during execution; it is + backed up to `.orig` and restored at run end, but a crash mid-run + can leave a mutant in the working tree (the `.orig` file remains for manual + restore). +- "Compilable" means *parses without tree-sitter ERROR nodes* — no compiler + runs; type errors and unresolved symbols count as compilable. +- An invalid `mutation.approach` value only prints "Invalid approach" and + falls through to the execution phase. +- All nine `system_prompts` keys are read at startup regardless of the chosen + approach — a missing key is a KeyError before anything runs. +- Mutahunter prompts are not bundled (licensing); the user pastes them into + the config. diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md new file mode 100644 index 0000000..dbe1494 --- /dev/null +++ b/docs/EXTENDING.md @@ -0,0 +1,85 @@ +# Extending multiplex + +Checklists for adding modules. Follow the import convention: inside +`multiplex/`, import siblings **without** the `multiplex.` prefix +(DEVELOPMENT.md § Import convention). + +## Add a mutant-generation approach + +Use `approach/basic/` as the single-prompt template or `approach/hazop/` as +the multi-prompt-chain template. + +1. **Create the package** `multiplex/approach//` with `__init__.py` and + `controller.py` exposing: + ```python + def main(model, output_dir, prompts): + ... + ``` + - `model` is a `model.Model`; call `model.make_request(messages)` with + OpenAI-style message dicts. + - Read the method under test via + `from approach.util import get_method_under_test`. + - Pass intermediate results between steps as files in `output_dir` + (convention: prefix them with your approach name). + - **Required output**: write final mutants to + `Path(output_dir, "-mutants/") / f"mutant_{count}.java"`. Each file + must contain the complete replacement method body (it is spliced verbatim + over the original method's byte range). Strip LLM code fences: + ```python + mutant = mutant.removeprefix("```java") + mutant = mutant.split("```", 1)[0] + ``` +2. **Register the prompt key(s)**: add `_...` entries to the `prompts` + dict in `multiplex/__main__.py` *and* to the `system_prompts` section of + every config file (including `examples/config.yml`). All keys in that dict + are read unconditionally — omitting one breaks **every** run with KeyError, + not just runs of your approach. +3. **Register the dispatch branch**: add an `elif config['mutation']['approach'] + == "":` branch in `multiplex/__main__.py` calling + `.main(model, output_path, prompts)`, plus the corresponding + `import approach..controller as `. +4. The `-mutants` directory name must equal the `mutation.approach` config + value — the execution backends reconstruct the path as + `output/-mutants/`. + +## Add an execution/evaluation backend + +Model on `multiplex/execute/defects4j.py` (not `maven.py`, which is broken). + +1. Create `multiplex/execute/.py` exposing: + ```python + def run_mutants(project_root, original_file, output_path, + method_start_byte, method_end_byte, duplicate, approach): + ``` +2. The expected loop, per mutant file in `output/-mutants/`: + - restore the pristine source: `shutil.copy2(duplicate, original_file)` + (guard on `duplicate.exists()`); + - `rewrite_method(original_file, method_start_byte, method_end_byte, + mutant_path)` — note: **4 arguments**; + - `check_mutant_equivalent(mutant_path, output/original_method.java)` and + `check_mutant_compilable(original_file)` from `checks/`; + - if compilable, run the project's tests with your build tool and decide + survived/killed; + - collect rows and finish with `write_mutant_summary(mutants_dir, rows)` + (header row: `["MUTANT", "EQUIVALENCE", "COMPILABLE", "SURVIVES"]`). +3. Register it in `multiplex/__main__.py` under a new `project.runtool` value. +4. Useful pattern from defects4j: run the unmutated project first as a + baseline (abort if it fails), and use a multiple of its wall-clock time as + the per-mutant test timeout to catch infinite-loop mutants. + +## Add / change the LLM provider + +`multiplex/model.py` wraps LiteLLM, so most providers work by editing only the +config (`llm.model`, `llm.endpoint`, `llm.token_env_var`). If a provider needs +extra setup (custom env vars, headers), extend `Model.__init__` — see the +existing Azure special-case. Keep the `make_request(messages) -> str` interface +unchanged; every approach depends on it. + +## Testing your module + +Put tests under `tests//test_.py` with fixture files in +`tests/resources/`. Import production code **with** the package prefix +(`from multiplex.checks.compilable import ...`). Run from the repo root: +`uv run pytest tests`. LLM calls are not mocked anywhere yet — keep pure logic +(parsing, splicing, checks) in functions separate from `model.make_request` +call sites so it is testable without an LLM. diff --git a/docs/MODULE_REFERENCE.md b/docs/MODULE_REFERENCE.md new file mode 100644 index 0000000..10370f7 --- /dev/null +++ b/docs/MODULE_REFERENCE.md @@ -0,0 +1,168 @@ +# Module Reference + +Function-level reference for every module. Paths are relative to `multiplex/`. +Signatures are exact; use this instead of opening source files. + +Note on imports: inside `multiplex/` modules import each other without the +package prefix (e.g. `from util.io import write_to_file`). See DEVELOPMENT.md. + +## model.py + +```python +class Model: + def __init__(self, model=None, endpoint=None, api_key_var=None) + def current_model(self) -> str + def make_request(self, messages) -> str # returns response.choices[0].message.content +``` + +- `api_key_var` is the **name** of an env var; its value is read from + `os.environ` (KeyError if the named var is unset). +- If `"azure" in endpoint`, sets `AZURE_AI_API_BASE` and `AZURE_AI_API_KEY`. +- `messages` is a LiteLLM/OpenAI-style list of `{"role": ..., "content": ...}`. + +## util/io.py + +```python +write_to_file(output_file_path, output) # plain text write (mode "w") +write_ucas(ucas_output_path, ucas_output) # writes str line-by-line +write_mutant_summary(mutants_dir, mutants) # writes mutants_dir/mutant_summary.csv from list-of-rows +read_input_to_str(input_file_path) -> str +read_ucas(input_file_path) -> (list[str], int) # (lines, line count) +read_hazop_mutations(input_file_path) -> (list[str], int) # 4th CSV column of each line +reset_source_code(duplicate_file_path, filename) +``` + +- `reset_source_code`: if `.orig` exists, moves it over `filename` + (restore); then always re-copies `filename` → `.orig`. Called at run start + and end by `__main__.py`. + +## util/extract_method.py + +```python +extract_method_from_file(file_path, method_name, output_dir, start_line) + -> (start_byte, end_byte) | None +``` + +- tree-sitter walk for a `method_declaration`/`constructor_declaration` whose + identifier text == `method_name` **and** whose identifier is on line + `start_line` (1-based). Both must match — `start_line` is the line of the + method *name*, not of annotations above it. +- Writes the method source to `output_dir/original_method.java`. +- Returns `None` if not found (logs a warning). Caller in `__main__.py` + unpacks the tuple directly, so a miss crashes with TypeError. + +## util/rewrite_method.py + +```python +rewrite_method(orig_path, start_byte, end_byte, mutant_file_path) +``` + +- Reads the pristine file from `orig_path + ".orig"` (the backup), splices the + mutant file's stripped text over bytes `[start_byte:end_byte)`, writes the + result to `orig_path`. Byte offsets come from `extract_method_from_file`. + +## util/parser.py + +```python +parse_output(input) -> dict | None # strips ```yaml fences, yaml.safe_load; None on parse error +add_mutant_to_method(numbered_src, mutant, line_number) -> str +``` + +- `add_mutant_to_method` takes source whose lines are prefixed `"N: "`, + replaces line `line_number` with `mutant` (preserving indentation), and + returns un-numbered source. Used by the mutahunter approach. + +## checks/compilable.py + +```python +check_mutant_compilable(filename) -> bool +``` + +- **Not a real compile.** tree-sitter parses the file and returns False if any + `ERROR` or missing node exists. Catches syntax errors only — type errors, + missing symbols, etc. pass. + +## checks/syntactic_equivalence.py + +```python +check_mutant_equivalent(mutant_filename, original_filename) -> bool +``` + +- Serializes both files' tree-sitter ASTs to strings, dropping comment nodes; + True iff the strings are identical. Detects mutants that only change + comments/formatting. Prints a `SequenceMatcher` similarity ratio as a side + effect (not used in the decision). + +## approach/util.py + +```python +get_method_under_test(output_dir) -> str # reads output_dir/original_method.java; FileNotFoundError if absent +``` + +## approach/*/controller.py (all five) + +```python +main(model, output_dir, prompts) # prompts: the full dict built in __main__.py +``` + +Each controller just calls its package's step functions in order with the +relevant `prompts[...]` key(s): + +- `basic`: `code_generator.generate_code` — 10 requests, each response saved as + `basic-mutants/mutant_N.java`. +- `hazop`: `method_explainer.describe_method` → `hazop-descriptions.txt`; + `description_mutator.mutate_descriptions` → `hazop-mutated-descriptions.txt` + (CSV rows `number, original_rule, GUIDEWORD, changed_rule`); + `code_generator.generate_code` → one mutant per changed rule. +- `stpa`: `control_diagram.create_control_diagram` → `control_diagram.txt` (DOT); + `uca_generator.generate_ucas` → `ucas.csv`; + `code_generator.generate_code` → one mutant per UCA + (loop is `range(0, ucas_count - 1)`: the **last UCA is skipped** — see + DEVELOPMENT.md § Known issues). +- `mutahunter`: `code_generator.generate_code` — one request containing the + method's AST plus line-numbered source; expects YAML back with + `mutants: [{mutated_code, line_number}, ...]`, spliced locally via + `util.parser.add_mutant_to_method`. The user-prompt template is intentionally + gutted (licensing) — users must fill in Mutahunter's own prompt text. +- `llmorpheus`: `placeholders.create_placeholders` — tree-sitter query captures + if/while/do/switch conditions, for-loop parts, enhanced-for parts, loop + headers, and method-call names/receivers/args; each capture produces + `placeholders/N_placeholder.java` (method with `` substituted), + `placeholders/N_orig.java` (original fragment), and an entry in + `placeholders/placeholders.json` (`{tag, start_byte, end_byte, text}`). + `code_generator.generate_code` — one request per placeholder asking for 3 + replacements ("Option 1/2/3" fenced blocks, extracted with the regex + ```` \n```\n(.*?)\n``` ```` under `re.DOTALL`); each replacement spliced into + the method at the placeholder's byte offsets → + `llmorpheus-mutants/mutant_N.java`. + +## execute/defects4j.py + +```python +run_mutants(project_root, original_file, output_path, + method_start_byte, method_end_byte, duplicate, approach) +``` + +- `_execute(project_root, file=None, approach=None, timer=None) -> bool`: + deletes `.classes_instrumented`/`target`, runs `defects4j test -w + ` with `PATH += ./defects4j/framework/bin` and + `JAVA_HOME = $JDK_11`; True iff output contains `Failing tests: 0`. + With `timer` set it enforces a subprocess timeout and writes test output to + `output/-test/_test.txt`. Kills stray Java processes after. +- `run_mutants` baselines the original project (raises IOError if its tests + fail), sets per-mutant timeout = 5× baseline duration, then per mutant: + restore from backup → equivalence check → rewrite → compilable check → + (if compilable) run tests. Appends `[name, equivalent, compilable, survives]` + rows and writes `mutant_summary.csv`. + +## execute/maven.py + +```python +run_mutants(project_root, filename, output_path, + method_start_byte, method_end_byte, duplicate, approach) +``` + +- **Broken — do not use as reference.** Calls `rewrite_method` with a stale + 5-argument signature (TypeError), never invokes its own `_execute_mutant` + (`mvn clean test`), and calls `check_mutant_compilable` on a directory. + Model a fix on `execute/defects4j.py`. See DEVELOPMENT.md § Known issues. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..b7a24b9 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,25 @@ +# docs/ — Agent Documentation Index + +Documentation for humans and AI agents working on *multiplex*. Each file is +self-contained: **load only the file(s) relevant to your task** rather than +reading source code or every doc. + +| File | Read it when you need to... | +|------|-----------------------------| +| [ARCHITECTURE.md](ARCHITECTURE.md) | Understand the pipeline end-to-end: data flow, output artifacts, how mutants are injected and evaluated. | +| [MODULE_REFERENCE.md](MODULE_REFERENCE.md) | Call or modify an existing function — every public function's signature, inputs, outputs, and side effects, without opening source files. | +| [CONFIG.md](CONFIG.md) | Read, write, or validate a `config.yml` — full schema, every key. | +| [EXTENDING.md](EXTENDING.md) | Add a new mutant-generation approach, execution backend, or LLM provider — exact checklists of files to touch. | +| [DEVELOPMENT.md](DEVELOPMENT.md) | Set up, run commands, run tests, follow conventions, or avoid known gotchas and open bugs. | + +Quick orientation (enough for trivial tasks, no further reading needed): + +- *multiplex* mutates **one Java method** per run: extract it with tree-sitter, + ask an LLM to generate mutants, splice each mutant back into the source file, + run the project's tests, and record which mutants survive. +- Entry point: `multiplex/__main__.py`, run as `uv run ./multiplex ./config.yml`. +- Python >= 3.13 (pin 3.13 — see DEVELOPMENT.md), deps via `uv`, lint via `ruff`, + tests via `pytest` from the repo root. +- Code inside `multiplex/` imports siblings **without** the `multiplex.` prefix + (`from model import Model`); tests import **with** it + (`from multiplex.checks... import ...`). See DEVELOPMENT.md before editing imports.