Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Copilot Instructions for multiplex

*multiplex* is a modular framework for prototyping LLM-based mutation testing. One run mutates a single Java method (located via tree-sitter), generates mutants with an LLM, splices each mutant back into the source file, and runs the target project's tests to see which mutants are killed.
*multiplex* is a modular framework for prototyping LLM-based mutation testing. One run mutates a single method/function — Java or Python, set by `project.language` and located via tree-sitter generates mutants with an LLM, splices each mutant back into the source file, and runs the target project's tests to see which mutants are killed.

## Read the docs before reading source

Expand All @@ -26,7 +26,8 @@ uv run ./multiplex ./path/to/config.yml # run the tool

- **Import convention**: the tool runs as a directory (`uv run ./multiplex`), so `multiplex/` itself is on `sys.path`. Modules inside `multiplex/` import each other **without** the package prefix (`from model import Model`, `from util.io import ...`); tests import **with** it (`from multiplex.checks... import ...`). Match the style of the file being edited.
- Adding a mutant-generation approach touches three places: a new `multiplex/approach/<name>/` package with `controller.py`, an `APPROACH_PROMPT_KEYS` entry in `multiplex/prompts.py`, and a dispatch branch in `multiplex/__main__.py`. Only the selected approach's `system_prompts` keys are required; `resolve_prompts` validates the approach and its keys up front, raising `SystemExit` before any destructive step.
- Mutant files must be complete replacement methods written to `output/<approach>-mutants/mutant_N.java`; they are spliced verbatim over the original method's byte range.
- tree-sitter versions are pinned (`tree-sitter==0.23.2`, `tree-sitter-java==0.23.5`); do not bump them casually — the parsing code depends on that API.
- A runnable end-to-end example lives in `examples/` (self-contained Maven project, `basic` approach, `mvn` backend): `uv run multiplex ./examples/config.yml`.
- Mutant files must be complete replacement methods written to `output/<approach>-mutants/mutant_N.<ext>` (`.java`/`.py`, from the `LanguageSpec`); they are spliced verbatim over the original method's byte range.
- The source language is set by `project.language` (default `java`); a `languages.LanguageSpec` (grammar, node types, extension, fence, prompt noun) is resolved once in `__main__.py` and threaded into `extract_method`, each approach's `main`, the checks, and the execution backend. Adding a language = a `_REGISTRY` entry in `multiplex/languages/__init__.py` (+ prompts, example, and an llmorpheus query). See `docs/EXTENDING.md`.
- tree-sitter versions are pinned (`tree-sitter==0.23.2`, `tree-sitter-java==0.23.5`, `tree-sitter-python` 0.23.x); do not bump them casually — the parsing code depends on that API.
- Runnable end-to-end examples live in `examples/`: Java (`config-java.yml`, Maven project, `mvn` backend) and Python (`config-python.yml`, pytest project, `pytest` backend). `uv run ./multiplex ./examples/config-java.yml`.
- Keep pure logic (parsing, splicing, checks) separate from `model.make_request` call sites — LLM calls are not mocked in tests.
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Guidance for AI coding agents working in this repository.

*multiplex* is a modular framework for prototyping LLM-based mutation testing. One run mutates a single Java method (located via tree-sitter), generates mutants with an LLM, splices each mutant back into the source file, and runs the target project's tests to see which mutants are killed.
*multiplex* is a modular framework for prototyping LLM-based mutation testing. One run mutates a single method/function — Java or Python, set by `project.language` and located via tree-sitter generates mutants with an LLM, splices each mutant back into the source file, and runs the target project's tests to see which mutants are killed.

## Documentation map

Expand All @@ -11,7 +11,7 @@ Task-scoped docs live in `docs/` — start at [docs/README.md](docs/README.md) a
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — pipeline, approaches, execution backends, output artifacts.
- [docs/MODULE_REFERENCE.md](docs/MODULE_REFERENCE.md) — every function's signature, behavior, and side effects.
- [docs/CONFIG.md](docs/CONFIG.md) — full `config.yml` schema.
- [docs/EXTENDING.md](docs/EXTENDING.md) — checklists for adding approaches, execution backends, or LLM providers.
- [docs/EXTENDING.md](docs/EXTENDING.md) — checklists for adding approaches, execution backends, languages, or LLM providers.
- [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) — setup, conventions, CI, known bugs/gotchas.

## Commands
Expand All @@ -30,4 +30,4 @@ uv run ./multiplex ./path/to/config.yml # run the tool
- Pipeline entry point and dispatch: `multiplex/__main__.py`. Adding an approach touches three places (new `approach/<name>/controller.py` package + `APPROACH_PROMPT_KEYS` entry in `multiplex/prompts.py` + dispatch branch). Only the selected approach's `system_prompts` keys are required; `resolve_prompts` validates them (and the approach name) up front, raising `SystemExit` before any destructive step. See `docs/EXTENDING.md`.
- `<projectroot>/output/` is wiped at the start of every run; the target source file is edited in place and restored from a `.orig` backup.
- tree-sitter versions are pinned; do not bump them casually.
- A runnable example lives in `examples/`: `uv run multiplex ./examples/config.yml` (self-contained Maven project, needs `mvn` + a JDK + a running Ollama). See `docs/DEVELOPMENT.md` § Example.
- Runnable examples live in `examples/`: `uv run ./multiplex ./examples/config-java.yml` (Java, Maven projectneeds `mvn` + a JDK) and `uv run ./multiplex ./examples/config-python.yml` (Python, pytest backend). Both need a running Ollama. See `docs/DEVELOPMENT.md` § Example.
23 changes: 15 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,20 +69,21 @@ Once configured and modules are set up, users can run *multiplex* using the foll
uv run /path/to/multiplex ./path/to/config.yml
```

#### Try the bundled example
A self-contained example (a small Maven project mutated with the `basic`
approach) is included. With `mvn` + a JDK on your `PATH` and a running Ollama
(`ollama pull gpt-oss:20b`, or edit `llm.model` in the config), run from the
repo root:
#### Try the bundled examples
Self-contained examples are included for both supported languages, each mutating
a small project with the `basic` approach. With a running Ollama
(`ollama pull gpt-oss:20b`, or edit `llm.model` in the config), run from the repo
root:
```bash
uv run multiplex ./examples/config.yml
uv run ./multiplex ./examples/config-java.yml # Java (needs mvn + a JDK on PATH)
uv run ./multiplex ./examples/config-python.yml # Python (uses pytest via uv run)
```
Results are written to `examples/project/example/output/basic-mutants/`
Results are written to the project's `output/<approach>-mutants/`
(`mutant_summary.csv`). See [`examples/README.md`](examples/README.md) for
details.

## 🧩 Existing Modules
*multiplex* currently includes five mutant generation modules and two execution and evaluation modules:
*multiplex* currently includes five mutant generation modules and three execution and evaluation modules:

**Mutant Generation Modules:**
- HAZOP
Expand All @@ -94,6 +95,12 @@ details.
**Execution and Evaluation Modules**
- Maven
- Defects4J (enables users to target specific bugs in the [Defects4J](https://github.com/rjust/defects4j) dataset)
- Pytest (for Python projects)

**Languages:** Java (default) and Python, selected per run with `project.language`
in the config. All five generation modules and the equivalence/compilable checks
are language-aware. See [`docs/EXTENDING.md`](docs/EXTENDING.md) § Add a language
for how to add another.

## 🏗️ Adding Modules

Expand Down
57 changes: 41 additions & 16 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# Architecture

*multiplex* is a modular framework for prototyping LLM-based mutation testing.
One run mutates **one Java method** in a target project and evaluates the
generated mutants against that project's test suite.
One run mutates **one method/function** in a target project and evaluates the
generated mutants against that project's test suite. The source language is set
by `project.language` (`java` — the default — or `python`); see the language
registry below.

## Pipeline

Expand All @@ -12,25 +14,26 @@ Orchestrated top-to-bottom by `multiplex/__main__.py`:
config.yml
1. Load YAML config; read all 9 system_prompts keys into a dict
1. Load YAML config; resolve the approach's system_prompts and the language
(languages.get_language(project.language), default "java")
2. Back up target source file to <filename>.orig
Delete and recreate <projectroot>/output/ ← destroys previous results
3. util/extract_method.py (tree-sitter)
Find method by name + start line
→ writes output/original_method.java
3. util/extract_method.py (tree-sitter, grammar from the LanguageSpec)
Find method/function by name + start line
→ writes output/original_method.<ext> (.java / .py)
→ returns (start_byte, end_byte) offsets into the source file
4. approach/<name>/controller.main(model, output_path, prompts)
4. approach/<name>/controller.main(model, output_path, prompts, language)
One or more LLM calls (via model.Model / LiteLLM)
→ writes mutant files to output/<approach>-mutants/mutant_N.java
→ writes mutant files to output/<approach>-mutants/mutant_N.<ext>
5. execute/{maven,defects4j}.run_mutants(...)
5. execute/{maven,defects4j,pytest_runner}.run_mutants(...)
For each mutant file:
a. restore source file from .orig backup
b. util/rewrite_method.py splices mutant text into the file
Expand All @@ -51,7 +54,7 @@ changes the file between those steps invalidates the offsets.
## Mutant-generation approaches (`multiplex/approach/`)

Each approach is a package with a `controller.py` exposing
`main(model, output_dir, prompts)`. Dispatch is an if/elif chain in
`main(model, output_dir, prompts, language)`. Dispatch is an if/elif chain in
`__main__.py` keyed on `mutation.approach`.

| Approach | Strategy | LLM calls |
Expand All @@ -62,6 +65,11 @@ Each approach is a package with a `controller.py` exposing
| `mutahunter` | Single call with AST + line-numbered source; LLM returns YAML of line-level mutations spliced in locally (prompts not bundled — licensing) | 1 |
| `llmorpheus` | tree-sitter query finds mutation sites (conditions, loop headers, call args), each replaced by `<PLACEHOLDER>`; LLM proposes 3 replacements per site | 1 per placeholder |

All five approaches are language-aware via the `language` argument (file
extension, code fence, prompt noun). llmorpheus additionally selects its
mutation-site query per language from `MUTATION_QUERIES` in
`approach/llmorpheus/placeholders.py`.

Intermediate artifacts are files in `output/` — approaches communicate between
their own steps via files, not in-memory state (see artifact table below).

Expand All @@ -78,21 +86,37 @@ Selected by `project.runtool`:
`mvn -f <project_root> clean test`; a mutant survives if the build passes
(`BUILD SUCCESS`). Baselines the original first (aborts if its tests fail),
then per mutant does the same equivalence → rewrite → compilable → test →
summary flow as the Defects4J backend. Used by the runnable example under
summary flow as the Defects4J backend. Used by the runnable Java example under
`examples/` (see DEVELOPMENT.md § Example).
- `pytest` → `pytest_runner.py` — self-contained backend for Python projects.
Runs `sys.executable -m pytest -q <project_root>` (pytest under multiplex's own
interpreter); a mutant survives if pytest exits 0 (all tests pass). Same
baseline → per-mutant flow as the Maven backend. Drives the runnable Python
example (`examples/config-python.yml`). Named `pytest_runner` so it does not
shadow the installed `pytest` package.

## Language registry (`multiplex/languages/`)

`get_language(project.language)` returns a `LanguageSpec` (grammar, definition
node types, comment node types, extension, code-fence tag, prompt noun,
mutahunter label). It is resolved once in `__main__.py` and threaded into
`extract_method`, every approach's `main`, the checks, and the execution
backend — so the pipeline is language-agnostic and adding a language is a
registry entry plus prompts and an example (see docs/EXTENDING.md § Add a
language). `project.language` defaults to `java`.

## Output artifacts

Everything lands under `<projectroot>/output/` (wiped at the start of each run):

| Artifact | Written by |
|----------|-----------|
| `original_method.java` | extract_method (step 3); read by every approach |
| `original_method.<ext>` | extract_method (step 3); read by every approach (`.java`/`.py`) |
| `hazop-descriptions.txt`, `hazop-mutated-descriptions.txt` | hazop chain steps |
| `control_diagram.txt`, `ucas.csv` | stpa chain steps |
| `placeholders/{N_placeholder.java, N_orig.java, placeholders.json}` | llmorpheus site finder |
| `<approach>-mutants/mutant_N.java` | every approach's final step |
| `<approach>-mutants/mutant_summary.csv` | defects4j backend; columns `MUTANT, EQUIVALENCE, COMPILABLE, SURVIVES` |
| `placeholders/{N_placeholder.<ext>, N_orig.<ext>, placeholders.json}` | llmorpheus site finder |
| `<approach>-mutants/mutant_N.<ext>` | every approach's final step |
| `<approach>-mutants/mutant_summary.csv` | maven/defects4j/pytest backends; columns `MUTANT, EQUIVALENCE, COMPILABLE, SURVIVES` |
| `<approach>-test/<mutant>_test.txt` | defects4j backend; per-mutant test output |

## LLM access (`multiplex/model.py`)
Expand All @@ -104,4 +128,5 @@ in config). Azure endpoints get `AZURE_AI_API_BASE`/`AZURE_AI_API_KEY` set
specially. To support another provider, replace or extend `model.py`.

LLM responses are treated as fenced code: approaches strip a leading
```` ```java ```` fence and keep everything before the next ```` ``` ```` fence.
```` ```<lang> ```` fence (the tag is `language.fence`, e.g. `java`/`python`) and
keep everything before the next ```` ``` ```` fence.
14 changes: 8 additions & 6 deletions docs/CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

*multiplex* takes a single YAML config file:
`uv run ./multiplex ./path/to/config.yml`.
Template: [`examples/config.yml`](../examples/config.yml).
Templates: [`examples/config-java.yml`](../examples/config-java.yml),
[`examples/config-python.yml`](../examples/config-python.yml).

Only the keys the selected `mutation.approach` needs are required. `project`,
`mutation`, and `llm` keys are always required; `system_prompts` keys are
Expand All @@ -13,11 +14,12 @@ startup with an actionable `SystemExit`, before any output is wiped.

| Key | Type | Meaning |
|-----|------|---------|
| `projectroot` | path | Root of the Java project under test. `output/` is created (and **wiped every run**) inside it. |
| `filename` | path | The `.java` source file containing the method under test. Backed up to `<filename>.orig` during the run. |
| `method` | string | Name of the method (or constructor) to mutate. |
| `line` | int | 1-based line number of the method **name identifier** in `filename` (not annotations above it). Both `method` and `line` must match for extraction to succeed; disambiguates overloads. |
| `runtool` | `mvn` \| `d4j` | Execution backend. `mvn` runs a plain Maven project (`mvn clean test`) and is used by the runnable `examples/` setup; `d4j` targets a Defects4J checkout (needs the `defects4j` CLI + `JDK_11`). |
| `language` | `java` \| `python` | Source language of the file under test. **Optional; defaults to `java`** (existing configs need no change). Selects the tree-sitter grammar, the definition node types extracted, the artifact/mutant file extension, and the code-fence/prompt wording. An unknown value fails fast at startup with a `SystemExit`. |
| `projectroot` | path | Root of the project under test. `output/` is created (and **wiped every run**) inside it. |
| `filename` | path | The source file containing the method/function under test (`.java` or `.py`). Backed up to `<filename>.orig` during the run. |
| `method` | string | Name of the method/constructor (Java) or function (Python) to mutate. |
| `line` | int | 1-based line number of the method/function **name identifier** in `filename` (not annotations/decorators above it). Both `method` and `line` must match for extraction to succeed; disambiguates overloads. |
| `runtool` | `mvn` \| `d4j` \| `pytest` | Execution backend. `mvn` runs a plain Maven project (`mvn clean test`); `d4j` targets a Defects4J checkout (needs the `defects4j` CLI + `JDK_11`); `pytest` runs pytest under multiplex's own interpreter (`sys.executable -m pytest <projectroot>`) — a mutant survives if pytest exits 0. `mvn` drives the Java `examples/` setup and `pytest` the Python one. |

## `mutation`

Expand Down
Loading
Loading