Fix: ChoiceRule.Synthesize stack-overflows on a self-referential choice with a parantheses alternative

[ChrisH07]

ChoiceRule.Synthesize stack-overflows on a self-referential choice with a parantheses alternative

Summary

When a grammar rule is a self-referential ChoiceRule (i.e. one of its alternatives is,
directly or indirectly, itself) and one of the alternatives is an untyped parantheses rule
wrapping that same choice, calling Parser.Update(element) (or any other path that invokes
Rule.Synthesize for that choice without an existing, matching parse tree) recurses forever
and crashes the process with a stack overflow. It reproduces with a 6-line grammar and
2-level type hierarchy — nothing exotic, no arithmetic precedence, no ambiguity beyond
"this thing could always be wrapped in redundant parens."

This matters for any grammar modeling a recursive boolean/arithmetic expression language with
optional parenthesization (a very common shape — see UVL.anytext's Constraint rule in this
same test suite, which has the identical shape). It's specifically triggered by the model → text
(Synthesize) direction; parsing (Match) is unaffected.

Minimal reproduction

Recursion.anytext:

grammar Recursion (rec)
root Container

Container:
  'value' value=Expr;

Expr:
  ParenExpr | Atom;

parantheses ParenExpr:
  '(' <nsp> Expr <nsp> ')';

Atom:
  name=Identifier;

terminal Identifier:
  /[a-zA-Z_][a-zA-Z0-9_]*/;

Program.cs:

using Repro.Grammar;
using Repro.Metamodel.Recursion;

var grammar = new RecursionGrammar();
var parser = grammar.CreateParser();

var container = parser.Initialize(new[] { "value x" }) as Container;
Console.WriteLine($"Parsed: {(container?.Value as Atom)?.Name}");

((Atom)container!.Value).Name = "y";
Console.WriteLine("Calling parser.Update(container)...");
var edits = parser.Update(container); // <-- crashes here

Console.WriteLine($"Edit count: {edits.Count}"); // never reached

Expected

Parsed: x
Calling parser.Update(container)...
Edit count: 1

Actual

Unhandled stack overflow (process terminates, cannot be caught) with a stack trace that
alternates indefinitely between ChoiceRule.Synthesize and SequenceRule.Synthesize/
SynthesizeCore:

   at NMF.AnyText.Rules.ChoiceRule.Synthesize(...)
   at NMF.AnyText.Rules.SequenceRule.SynthesizeCore(...)
   at NMF.AnyText.Rules.SequenceRule.Synthesize(...)
   at NMF.AnyText.Rules.ChoiceRule.Synthesize(...)
   at NMF.AnyText.Rules.SequenceRule.SynthesizeCore(...)
   at NMF.AnyText.Rules.SequenceRule.Synthesize(...)
   ... (repeats until the stack is exhausted)
   at NMF.AnyText.Rules.QuoteRule.Synthesize(...)
   at NMF.AnyText.Model.AssignRule`2.Synthesize(...)
   at NMF.AnyText.Rules.SequenceRule.SynthesizeParseObject(...)
   at NMF.AnyText.Model.ElementRule`1.Synthesize(...)
   at NMF.AnyText.Rules.Rule+<SynthesizeChanges>d__57.MoveNext()
   at NMF.AnyText.Parser.Update(IEnumerable`1 updates)
   at NMF.AnyText.Parser.Update(Object updatedElement, String[] updatedFeatures)
   at NMF.AnyText.Parser.Update(Object updatedElement)

Note: the recursive choice must be reached through an assigned/containment feature
(AssignRule → QuoteRule → ChoiceRule, as in Container.Value) — I could not reproduce it
by calling Update directly on a root-rule element of the same shape, which apparently takes a
different code path (SynthesizeParseObject reusing the existing parse tree structurally rather
than freely re-deriving alternatives from scratch).

Root cause

Two things compound:

1. The paren-wrapping alternative always "can synthesize" any element the choice can
represent at all.
ParenExpr is a parantheses rule (ParanthesesRule : SequenceRule,
AnyText.Core/Model/ParanthesesRule.cs) with no ModelElementRule<T> type gate and no
metaclass of its own — it's fully transparent (ParanthesesRuleApplication.GetValue just
returns Inner[1].GetValue(...)). So
ParanthesesRule.CanSynthesize
→ SequenceRule.CanSynthesize
(SequenceRule.cs:337-340)
→ delegates straight to the enclosing choice's CanSynthesize for the same semantic element.
Because ParenExpr is declared before Atom in Expr's alternative list, ChoiceRule.Synthesize
picks it first (ChoiceRule.cs:190-199,
Array.Find returns the first match).

2. Nothing stops the choice from picking that same alternative again on the way back in.
SynthesisPlan is supposed to guard against this: ChoiceRule.CanSynthesize calls
synthesisPlan.BlockRecursion(this, semanticElement)
(ChoiceRule.cs:182-187), which
writes false into SynthesisPlan._decisions[(rule, semanticObject)]
(SynthesisPlan.cs). But that dictionary is only
ever read from SynthesisPlan.CanSynthesize(Rule, object, ParseContext)
(SynthesisPlan.cs:20-28) — and that
method is only called from ChoiceRule's own alternative loop
(Array.Exists(Alternatives, r => synthesisPlan.CanSynthesize(r.Rule, ...))). Every other
combinator — SequenceRule.CanSynthesize
(SequenceRule.cs:337-340),
OneOrMoreRule, ZeroOrOneRule, QuoteRule — calls the inner rule's CanSynthesize/Synthesize
directly, bypassing SynthesisPlan entirely. So when ParenExpr's sequence reaches back into
Expr (the same ChoiceRule) for its middle element, ChoiceRule.CanSynthesize is re-entered as
a raw call: it writes BlockRecursion again (a no-op re-write of an entry nobody reads back) and
evaluates all alternatives from scratch — there's no check at entry for "is (this, semanticElement) already in progress?" It picks ParenExpr again. Forever.

ChoiceRule.Synthesize (the actual tree-construction method, not just the CanSynthesize probe)
makes it unconditional, because unlike CanSynthesize it doesn't even accept a SynthesisPlan
parameter — it allocates a fresh one on every call
(ChoiceRule.cs:190-192):

public override RuleApplication Synthesize(object semanticElement, ParsePosition position, ParseContext context)
{
    var synthesisPlan = new SynthesisPlan();   // <-- no history from the caller
    var alternative = Array.Find(Alternatives, a => synthesisPlan.CanSynthesize(a.Rule, semanticElement, context));
    ...
}

So even the partial protection CanSynthesize has (within a single top-level probe) is thrown
away at every recursive Synthesize step. Net effect: Expr → ParenExpr → Expr → ParenExpr → …,
printing ((((((...x...))))) with no depth bound until the stack overflows.

Originally suggested fix

• ChoiceRule.Synthesize should accept/thread a SynthesisPlan (mirroring CanSynthesize's
  signature) instead of creating a new one on every call.
• ChoiceRule.CanSynthesize (and Synthesize, once it takes a plan) should check whether
  (this, semanticElement) already has an in-progress/blocked entry in the plan before
  evaluating alternatives, and short-circuit if so, rather than only ever writing to that entry
  for other call sites to observe.

Either change alone would fix this reproduction; both together would close the gap for
SequenceRule/OneOrMoreRule/etc. as well, since they'd now be threading a plan that the
ChoiceRule they eventually reach will actually consult on re-entry.

Not what was actually implemented — see below. Threading SynthesisPlan through
Rule.Synthesize would mean changing an abstract method signature implemented by 22 other rule
types across the codebase (SequenceRule, LiteralRule, EnumRule, AssignRule, QuoteRule,
OneOrMoreRule, ZeroOrOneRule, ParanthesesRule, etc.), which is a much larger and riskier
change than this bug warrants.

Fix (implemented)

Instead of threading a SynthesisPlan everywhere, ChoiceRule now carries its own thread-local
reentrancy guard: a per-instance ThreadLocal<HashSet<object>> of semantic elements currently
being synthesized by this choice rule, checked at the top of both CanSynthesize and
Synthesize and cleared in a finally block. If a semantic element is already in progress for
this rule (i.e. we've re-entered ChoiceRule.CanSynthesize/Synthesize for the same
(this, element) pair via a SequenceRule/ParanthesesRule/etc. bypassing SynthesisPlan), the
second attempt fails immediately instead of re-evaluating alternatives and recursing again.

This is confined entirely to ChoiceRule.cs — no signature changes, no other rule types touched.
Uses ReferenceEqualityComparer since semantic elements are model instances; thread-local (rather
than a plain instance field) because Rule instances represent grammar structure and may be
shared across concurrently-running parsers (e.g. multiple documents in AnyText.Lsp).

Verification

• This repro: Parser.Update(container) on the mutated Atom no longer stack-overflows; produces
  Edit count: 1 as expected.
• Full AnyText.Tests suite: 214/214 passing, no regressions.

Workaround

List typed/terminal alternatives before any transparent parantheses alternative in a
self-referential choice:

Expr:
  Atom | ParenExpr;

This isn't a complete fix on its own (an actual instance of a different recursive alternative
that's also representable via parens could still hit the same gap), but for grammars dominated by
leaf/typed alternatives (identifiers, literals, comparisons) it avoids the crash in practice.

---

Summary by MergeMonkey

• Reference Updates:
  • Adds XML documentation explaining the thread-local recursion guard mechanism
• What's New:
  • Adds thread-local tracking to prevent stack overflow in self-referential ChoiceRule synthesis
• Improvements:
  • Fixes infinite recursion when a ChoiceRule contains a self-referential alternative with parentheses
• Refactors:
  • Updates history/changelog with the fix entry

[mergemonkeyhq]

⚡ Risk Assessment — NEEDS-TESTING · ~5 min review

Focus areas: ChoiceRule synthesis logic · ThreadLocal recursion tracking · Changelog update

Assessment: Adds recursion guard to prevent stack overflow in self-referential grammar rules

Walkthrough

When a grammar rule contains a self-referential ChoiceRule with a parentheses alternative, the synthesis process enters infinite recursion. The fix introduces a ThreadLocal that tracks which semantic elements are currently being synthesized. If an element is already in progress, CanSynthesize returns false and Synthesize returns a FailedRuleApplication, breaking the recursion cycle.

Changes

Files	Summary
ChoiceRule recursion guard AnyText/AnyText.Core/Rules/ChoiceRule.cs	Adds ThreadLocal<HashSet> to track in-progress synthesis, preventing infinite recursion in self-referential choice rules during CanSynthesize and Synthesize operations
Changelog update AnyText/AnyText.history	Adds history entry documenting the stack overflow fix for self-referential ChoiceRule with parentheses alternative

Sequence Diagram

sequenceDiagram
    participant Caller
    participant ChoiceRule
    participant InProgressSet as _synthesisInProgress
    participant Alternatives
    Caller->>ChoiceRule: CanSynthesize(element)
    alt element in InProgressSet
        ChoiceRule->>Caller: return false
    else element not in set
        ChoiceRule->>InProgressSet: Add(element)
        loop For each alternative
            ChoiceRule->>Alternatives: CanSynthesize(element)
        end
        ChoiceRule->>InProgressSet: Remove(element)
        ChoiceRule->>Caller: return result
    end

Loading

Dig Deeper With Commands

• /review <file-path> <function-optional>
• /chat <file-path> "<question>"
• /roast <file-path>

Runs only when explicitly triggered.

[georghinkel]

My first idea was that the grammar is wrong, because AnyText uses the same syntax like EBNF but the grammar is in fact a PEG instead of a CFG, meaning that the order of a choice is important. However, the parser could clearly differentiate between an opening parenthesis and an atom, rendering the grammar correct and so yes, this is a bug. At very least, AnyText should warn people about this.

The downside of the solution with the thread-local variable is memory usage, but given that you clean the hashset after use, it is only transient and hence acceptable.

So thank you very much :)