Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
176 changes: 176 additions & 0 deletions .claude/skills/add-language/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
---
name: add-language
description: Add a new source/target language (lexer, parser, emitter) to Praxly2 and wire it through the Universal AST, translator, and editor UI. Use when adding or removing a language, adding a target the translator can emit, or debugging why a newly added language does not appear in the pickers.
---

# Adding a language to Praxly2

Every language in Praxly2 is three files plus a handful of registrations. The
compiler work is the easy part to get right; the registrations are where people
lose time, because missing one fails silently in a different layer.

Read [`specs/`](../../../specs/) for the language you are adding **before**
writing the lexer — it is the authority on what the language accepts, and it
must be updated in the same change if you alter behaviour.
[`docs/ADDING_A_LANGUAGE.md`](../../../docs/ADDING_A_LANGUAGE.md) has longer
code examples; this skill is the integration path.

## The one invariant

**Parsers may only produce node types already defined in `src/language/ast.ts`.**
There is no such thing as a language-specific AST node. If your language has a
construct the Universal AST can't express, either desugar it into existing nodes
in the parser, or add the node to the AST and update _every_ emitter,
`interpreter.ts`, `visitor.ts`, and `translator.ts` — see "Adding an AST node"
at the bottom.

## 1. Write the language module

```
src/language/<lang>/
lexer.ts class <Lang>Lexer { tokenize(): Token[] }
parser.ts class <Lang>Parser { parse(): Program }
emitter.ts class <Lang>Emitter extends ASTVisitor
```

`src/language/java/` is the most complete reference. Blocks
(`src/language/blocks/`) is deliberately _not_ a model to copy — its source is
Blockly workspace JSON, so it uses `toAst.ts`/`fromAst.ts` instead of a
lexer/parser/emitter and bypasses the emitter dispatch entirely.

**Lexer**

- Import `Token` from `'../lexer'`.
- Terminate the stream with `{ type: 'EOF', value: '', start: this.pos }`.
- Match multi-character operators (`==`, `!=`, `<=`, `>=`) **before** their
single-character prefixes, or `==` lexes as two `=`.
- Skip whitespace and comments without emitting tokens. Comments that must
survive into translations are handled by `src/language/comments.ts`, not here.
- Python's lexer is the odd one out: it converts indentation into virtual `{}`
and `;` tokens so the parser can treat it as brace-delimited. Copy that
approach only if your language is indentation-sensitive.

**Parser**

- Import `generateId` from `'../ast'`. **Every node needs `id: generateId()`** —
the debugger and source maps are keyed on it, and a missing id silently
disables line highlighting for that construct.
- Recursive descent, lowest precedence at the top of the call tree:
`assignment → logicalOr → logicalAnd → equality → comparison → term → factor
→ unary → postfix → primary`.
- Normalise operators to Universal AST spelling: logical operators become
`'and'`/`'or'`/`'not'`; not-equal becomes `'!='`. Emitters translate back out.
- Use `synchronize()` to skip to the next statement after a parse error.

**Emitter**

- Extend `ASTVisitor` from `'../visitor'` and implement all 21 abstract
`visit*` methods (`tsc` will list any you miss — that is the checklist).
- Build output with `this.emit(line, nodeId?)`, `this.indent()`, `this.dedent()`.
- Pass the node id to `emit()` for any statement you want to be debuggable; that
is what populates the `SourceMap` the debugger highlights from.
- Use `this.escapeString(value, quote?)` for string literals so backslashes,
quotes, and newlines re-parse instead of breaking the output across lines.
- Guard expression precedence with the `Precedence` constants from `visitor.ts`.

## 2. Register the language (6 places)

Miss one and the failure looks unrelated, so work down the list.

| # | File | Change | Symptom if missed |
| --- | ----------------------------------------- | --------------------------------------------------------- | -------------------------------------------------------------- |
| 1 | `src/language/visitor.ts` | add to the `TargetLanguage` union | `tsc` error in `translator.ts` |
| 2 | `src/language/translator.ts` | import the emitter + add a `case` in `translateWithMap()` | runtime `Unsupported target language` |
| 3 | `src/components/LanguageSelector.tsx` | add to `SupportedLang` **and** `LANG_LABELS` | `tsc` error (`LANG_LABELS` is an exhaustive `Record`) |
| 4 | `src/components/editor/SourcePane.tsx` | add to `SOURCE_OPTIONS` | language can't be picked as a _source_ |
| 5 | `src/components/editor/AddPanelStrip.tsx` | add to `PANEL_LANGS` | no icon in the left rail, can't open as a _translation panel_ |
| 6 | `src/utils/editorUtils.ts` | add a `case` to `getCodeMirrorExtensions()` | no syntax highlighting (falls through to `[]`, which is valid) |

`LANG_LABELS` being a `Record<SupportedLang, string>` is deliberate: adding to
the union without adding a label is a compile error, which is the one
registration you cannot forget.

Then wire parsing:

**`src/hooks/useCodeParsing.ts`** — add the import and a `case` in `parseCode()`:

```typescript
case '<lang>':
tokens = new <Lang>Lexer(input).tokenize();
return new <Lang>Parser(tokens).parse();
```

`getTranslation()` in the same hook routes through `Translator` and needs no
change once step 2 is done.

Two allow-lists also gate embed links, and are easy to overlook because they are
plain string arrays rather than typed unions:

- `VALID_TARGET_LANGS` in `src/hooks/useEmbedLinkImport.ts` — `?targetLang=` on
an editor link
- `VALID_TO_LANGS` in `src/pages/EmbedPage.tsx` — `?to=` on an embed link

## 3. Syntax highlighting (optional)

CodeMirror ships grammars for Java, Python, and JavaScript. CSP and Praxis use
Lezer grammars instead:

- `src/language/<lang>/<lang>.grammar` — compiled to `.grammar.js` by the Vite
plugin (`resolve.extensions` in `vite.config.js` lists `.grammar`)
- `src/language/<lang>/lezer.ts` — wraps the compiled grammar as an extension
- return it from `getCodeMirrorExtensions()`

Returning `[]` is fine while you get the compiler working; the pane renders as
plain text.

## 4. Demo program and tests

- Add `examples/demo.<ext>` exercising every node your parser can produce, and
register it in `src/utils/demoPrograms.ts` (`DEMO_PROGRAMS`). See
`examples/README.md` for the coverage policy. `tests/round-trip.test.ts` picks
it up and translates it to every target, asserting output equivalence — this
is the strongest signal that your emitter is faithful.
- Add `tests/<lang>.test.ts`. See the **add-tests** skill.
- Optionally add entries to `EXAMPLE_PROGRAMS` in `src/utils/sampleCodes.ts`
(note its `ExampleLanguage` type is narrower than `SupportedLang` — text
languages only).

## Adding an AST node

Only if the Universal AST genuinely can't express the construct. Update, in
order, and let `tsc` drive you:

1. `src/language/ast.ts` — the node type
2. `src/language/visitor.ts` — `abstract visitX()` + the dispatch `case`
3. every emitter under `src/language/*/emitter.ts` (5 of them)
4. `src/language/interpreter.ts` — execution
5. `src/language/translator.ts` — recurse into the body in `analyzeBlock`, or
type inference silently skips anything nested inside your node
6. `src/language/blocks/fromAst.ts` + `toAst.ts` if it should be representable
as a block

## Verify

```bash
npx tsc --noEmit # exhaustive-switch errors are your checklist
npm run test:run # unit + round-trip
npm run test-browser # Selenium csv/ suite — needs Chrome; not part of test:run
```

Do **not** edit `csv/praxly.test.csv`; it is the original regression corpus.

## Checklist

- [ ] `lexer.ts` — operators longest-match-first, ends with `EOF`
- [ ] `parser.ts` — correct precedence, `generateId()` on every node
- [ ] `emitter.ts` — all 21 `visit*` implemented, `emit(line, nodeId)` for statements
- [ ] `TargetLanguage` + `translator.ts` case
- [ ] `SupportedLang` + `LANG_LABELS`
- [ ] `SOURCE_OPTIONS` + `PANEL_LANGS`
- [ ] `getCodeMirrorExtensions()` case
- [ ] `useCodeParsing.ts` case
- [ ] `VALID_TARGET_LANGS` + `VALID_TO_LANGS` if embed links should carry it
- [ ] `examples/demo.<ext>` + `DEMO_PROGRAMS`
- [ ] `tests/<lang>.test.ts`
- [ ] `specs/<lang>.md` written or updated
- [ ] `tsc`, `test:run`, `test-browser` all green
187 changes: 187 additions & 0 deletions .claude/skills/add-tests/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
---
name: add-tests
description: Write Vitest tests for Praxly2 language features, hooks, and stores — lexer, parser, interpreter, translator, round-trip, and debugger layers. Use when adding a language feature, fixing a parser/interpreter/emitter bug, adding a UI hook, or deciding which of the existing test files a new case belongs in.
---

# Testing Praxly2

Unit tests live in `tests/` and run under Vitest with **no DOM environment** —
they exercise the compiler pipeline and plain TypeScript modules directly.

```bash
npm run test:run # single run
npm run test # watch mode
npm run test-browser # Selenium csv/ suite — separate, needs Chrome
```

## Hard rule

**Never edit `csv/praxly.test.csv`.** It is the original regression corpus, run
in a real browser by `npm run test-browser` and _not_ by `npm run test:run`.
After any parser, interpreter, or emitter change, run the browser suite too —
`test:run` passing is not sufficient evidence for those layers.

## Where a test goes

| File | Holds |
| ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tests/<lang>.test.ts` | Everything specific to one language: lexer, parser, interpreter, and that language's emitter output. One per language (`python`, `java`, `javascript`, `csp`, `praxis`). |
| `tests/round-trip.test.ts` | Cross-language fidelity: translates each `examples/demo.*` to every target, re-parses, re-runs, and compares output. |
| `tests/examples.test.ts` | Guards that every `examples/demo.*` still parses, runs without error, and translates to all targets without throwing. |
| `tests/control-flow.test.ts`, `blank-lines.test.ts`, `comments.test.ts`, `blocks.test.ts`, `debugger.test.ts`, `placeholder.test.ts` | Cross-cutting behaviour that isn't owned by one language. |
| `tests/bugfixes.test.ts` | Regression cases for specific fixed bugs. Add a comment naming the original failure. |
| `tests/chatStore.test.ts` | Non-compiler modules (zustand stores, plain utils). |

A new _language feature_ goes in `tests/<lang>.test.ts`. A new _AST node_ or
interpreter behaviour usually needs a case in the relevant language file **and**
a demo update so `round-trip.test.ts` covers it in every target.

## The four APIs you will assert against

Get these right — they are the most common source of broken test scaffolding.

```typescript
// 1. Lexer → Token[]
const tokens = new CSPLexer('x <- 5').tokenize();

// 2. Parser → Program (Universal AST)
const program = new CSPParser(tokens).parse();

// 3. Interpreter → string[] ← an ARRAY OF LINES, not a string, not { output }
const lines = new Interpreter().interpret(program, source);

// 4. Translator → string
const code = new Translator().translate(program, 'java');
```

`interpret()` takes the original source as its second argument (used for error
locations) and returns one entry per output line. `DISPLAY`/`print` without a
newline buffers into the current line, so two statements can land in a single
array entry — assert with `toEqual([...])` on the whole array when spacing
matters.

## Patterns

**Lexer** — assert on token presence, not position, so unrelated token changes
don't break the test:

```typescript
it('should tokenize assignment operator', () => {
const tokens = new CSPLexer('x <- 5').tokenize();
expect(tokens).toContainEqual(expect.objectContaining({ type: 'OPERATOR', value: '<-' }));
});

it('should skip comments', () => {
const tokens = new CSPLexer('x <- 5 // comment\ny <- 10').tokenize();
const identifiers = tokens.filter((t) => t.type === 'IDENTIFIER').map((t) => t.value);
expect(identifiers).toEqual(['x', 'y']);
});
```

**Parser** — `toMatchObject` on the shape, so you assert the fields you care
about and ignore `id` (which is regenerated every parse):

```typescript
it('parses REPEAT UNTIL as a negated While', () => {
const program = new CSPParser(
new CSPLexer('REPEAT UNTIL(x > 5) { x <- x + 1 }').tokenize()
).parse();
expect(program.body[0]).toMatchObject({
type: 'While',
condition: { type: 'UnaryExpression', operator: 'not' },
});
});
```

**Interpreter** — assert the line array. Note that CSP's `DISPLAY` appends a
space rather than a newline, so three iterations land in **one** array entry:

```typescript
it('runs REPEAT n TIMES', () => {
const source = 'i <- 0\nREPEAT 3 TIMES\n{\n i <- i + 1\n DISPLAY(i)\n}';
const program = new CSPParser(new CSPLexer(source).tokenize()).parse();
expect(new Interpreter().interpret(program, source)).toEqual(['1 2 3 ']);
});
```

Write the source in the syntax the language actually has — check `specs/<lang>.md`
first. A construct the parser doesn't recognise usually yields an **empty
program**, so the test "passes" against `[]` while proving nothing. If an
interpreter assertion comes back empty, suspect the syntax before the
interpreter.

**Translator** — check for the meaningful snippet, never whole-program equality
(formatting is not a stable contract):

```typescript
it('translates if/else to Java', () => {
const source = 'IF (x > 0) { DISPLAY("pos") } ELSE { DISPLAY("neg") }';
const program = new CSPParser(new CSPLexer(source).tokenize()).parse();
const java = new Translator().translate(program, 'java');
expect(java).toContain('if (x > 0)');
expect(java).toContain('else');
});
```

**Round-trip** — the strongest assertion available, because every language runs
on the same interpreter: translate, re-parse, re-run, compare output. Prefer
this over eyeballing emitted text when you change an emitter.

```typescript
it('CSP → Python preserves behaviour', () => {
const source = 'x <- 5\nDISPLAY(x)';
const program = new CSPParser(new CSPLexer(source).tokenize()).parse();
const python = new Translator().translate(program, 'python');
const reparsed = new PythonParser(new PythonLexer(python).tokenize()).parse();
expect(new Interpreter().interpret(reparsed, python)).toEqual(
new Interpreter().interpret(program, source)
);
});
```

**Debugger** — drive `Debugger` directly; `step()` returns a `DebugStep` with
`variables`, `callStack`, cumulative `output`, and `isComplete`. See the `parse`
helper at the top of `tests/debugger.test.ts` for the established shape.

**Hooks and components** — there is no DOM environment configured, so test the
extracted logic rather than rendering. The refactor deliberately put page
behaviour in hooks and pure helpers for this reason; the pure ones are directly
testable with no setup:

- `src/utils/panelLayout.ts` — `toggleStack`, `reorderPanel`, `swapStacked`, `inSameColumn`
- `src/utils/languageSwitch.ts` — `planLanguageSwitch` returns a
`'empty' | 'translated' | 'untranslatable'` plan
- `src/utils/aiPanelContext.ts` — `buildAiPanelContext`
- `src/utils/debugHandlers.ts` — `computeMultiplePanelHighlighting`

`tests/chatStore.test.ts` shows the pattern for a module that needs
`localStorage`: stub it on `globalThis` before importing the store. If you ever
do need to render a component, add `environment: 'jsdom'` and `@testing-library/react`
first — neither is currently a dependency.

## What to cover, by change type

**New statement node** — lexer tokens; parser node shape; interpreter execution
(including the loop/branch boundary); each emitter's output; then add it to the
language's `examples/demo.*` so round-trip covers all five targets.

**New expression type** — tokenisation; expression-tree shape including
precedence against neighbouring operators; evaluated value; one emitter's
output, with parenthesisation checked.

**New built-in** — parser recognises it as a `CallExpression`; interpreter
returns the right value (and mutates in place where the spec says so); each
emitter maps it to the target's equivalent. Cross-language differences belong in
`specs/stdlib.md`.

**Bug fix** — add the failing input to `tests/bugfixes.test.ts` with a one-line
comment naming what used to break.

## Conventions

- `describe` per layer (`'CSP Lexer'`, `'CSP Parser'`, …), `it` per case.
- Small and focused beats one big integration test — a failure should name the
layer that broke.
- Cover the edge cases the parser actually branches on: empty body, nested
loops, chained calls, trailing `else if`.
- Never assert on `id` values; they are freshly generated per parse.
Loading
Loading