Generate syntax highlighting from Macaulay2 itself

[d-torrance]

Note

The code and this description were written by Claude Opus 5. I reviewed
the result, but please read it with that in mind.

The two new wiki pages — Macaulay2 language grammar and SimpleDoc
language grammar — describe both languages this extension highlights, and
the interpreter can report its own operator tables through getParsing. This
uses both instead of maintaining that information by hand.

One caveat up front: operator precedence does not help a TextMate grammar.
It is a per-line regex scanner with no expression parser. What getParsing
actually buys is completeness, correct alternation ordering, and a
machine-checkable classification that fails loudly when Macaulay2 changes.

Generation

generate-grammar.m2 writes syntaxes/m2-tokens.json (builtin symbols, the
105 symbolic operators classified by getParsing, and the SimpleDoc section
keywords), then scripts/generate-syntax.js expands the .in templates.

JSON templates are expanded structurally — parse, substitute into string
values, re-encode — rather than as text, so escaping is correct by
construction. Doing it textually is a trap: a placeholder sits inside a JSON
string, so an operator alternation's backslashes need JSON escaping on top of
their regex escaping, and a bare \. yields a grammar that will not parse.

Delimiters ((, ), [, ], {, }, ,, ;, <|, |>) are keywords
too and are excluded, or every parenthesis would be scoped as an operator.

Grammar

• Operators go from three patterns to all of them, split into assignment
  and everything else. Alternations are sorted longest-first because Oniguruma
  is leftmost-first, not longest-match. Operators that are a strict prefix of
  one in the other group get a generated negative lookahead — without it
  == was scoped as two assignment =.
• Numbers are highlighted at all for the first time, including 0x/0o/
  0b and the p/e suffixes. The float pattern refuses a trailing dot, so
  1..5 is not lexed as the float 1. followed by .5.
• Word keywords become keyword.control, distinct from types and functions.
• /// is split three ways: doc/multidoc/document get the new
  SimpleDoc grammar, TEST gets real Macaulay2 highlighting, and a bare ///
  is a raw string with no inner patterns. Previously all three leaked code
  scopes into raw text — which is why a -- inside a raw string was painted
  as a comment.

SimpleDoc

A new text.macaulay2.simpledoc grammar. TextMate cannot count indentation in
general, but begin/while rules can backreference the indent captured by
begin, which is exactly the off-side rule (VS Code's own Markdown grammar
uses the same trick). Each section body is then highlighted by its own body
form: Macaulay2 code, verbatim, prose with @…@ escapes, item heads, or menu
entries.

Only whole-line -- is a comment, matching SimpleDoc, so a trailing
-- note in prose is no longer greyed out.

Which body form each keyword takes cannot come from Macaulay2 — the tables
map a keyword to a handler function, and the body form is decided inside that
function. So it is hard-coded, and checked against the keyword names M2
reports: a keyword added upstream now fails the build instead of silently
going unhighlighted.

Known limits: detection must be on the same line as the /// (the tokenizer
sees one line at a time), and the "indentation floor only ever drops" quirk is
not reproduced.

Formatter and language configuration

The formatter's operator list is generated from the same data. That drops
⊘= (U+2298), which is not a Macaulay2 operator, and adds the real ⊠=
(U+22A0) and <-. A test had been pinning the bug, so it changes here too.

wordPattern now accepts ' and $, and deliberately excludes _: it is a
binary operator, so foo_bar is three tokens and should not select as one
word.

Tests and CI

36 → 60 tests. Every pattern must compile as a JavaScript RegExp, because
the webview REPL re-compiles them in the browser; the committed grammars must
match their templates (pure JS, no M2 needed); and real tokenization pins the
scopes.

Macaulay2 is now installed in CI. The runtime test at
extension.test.ts calls this.skip() when M2 is missing, so it had been
silently skipping on every run. CI also checks that regenerating is a no-op.

Three bugs were caught only by tokenizing for real, and were invisible from
reading the JSON: the == split above, TEST /// never closing because
applyEndPatternLast let the nested rule reopen instead of close, and while
patterns consuming the indent they matched, which silently disabled every
^-anchored rule.

Also

Deletes the stale, unreferenced macaulay2.tmLanguage.yaml and a dead
repository.storage entry whose regex ((\)) never compiled.

Worth a look

• CI uses ppa:macaulay2/macaulay2, so the generated files are kept at the
  released symbol set. isSkewAffineRing exists only in development builds
  and has been removed from the committed output by hand; regenerating against
  a development Macaulay2 will add it back, and the up-to-date check will then
  fail until it reaches a release.
• package.json still maps .d/.dd to this grammar. Those are M2's
  interpreter sources in a different language, and sharper operator scoping
  will make them look worse — probably its own issue.

🤖 Generated with Claude Code