Skip to content

lowering: a self tail call becomes a loop - #39

Merged
simontreanor merged 1 commit into
mainfrom
feat/self-tail-call
Jul 31, 2026
Merged

lowering: a self tail call becomes a loop#39
simontreanor merged 1 commit into
mainfrom
feat/self-tail-call

Conversation

@simontreanor

Copy link
Copy Markdown
Owner

Dogfooding finding #6 (see #34), implementing the option you picked: lowering-only, no language change.

CPython has no TCE and caps the stack near 1000 frames, so an interactive loop written as recursion walks a stack it has no reason to build. The dogfooded game called setRecursionLimit 20000 at startup for exactly this.

A direct, saturated self tail call now lowers to:

def turn(state, deck):          # let turn state deck =
    while True:                 #   if over state then state
        if over(state):         #   else turn (step state) (rest deck)
            return state
        state, deck = (step(state), rest(deck))
        continue

Several parameters rebind simultaneously from one tuple, so an argument still reads the previous iteration's values, exactly as the call it replaces evaluated its arguments before entering.

Why it is small. The pass runs on the lowered body rather than the Pyfun AST. lower_return emits a Return only in tail position, so "tail call" is just "a Return whose value calls our own name at our own arity", and the tail-position walk is a walk over statements that already exist. No new notion of tail position was needed.

The four preconditions, each falling back to the ordinary recursive def (always correct, merely stack-bound):

  1. the body must not rebind the function's own name (a shadowing let would make the loop jump where the program meant to call);
  2. it must not be a generator or coroutine (return in a generator raises StopIteration with a value; an async tail call is an await);
  3. the walk does not descend into a for (a continue there belongs to the for) or a try (looping inside it would put every later iteration under a handler that covered one call);
  4. no nested function may mention a name this frame binds.

Number 4 is the one that makes this sound, and it is worth the review time. The loop reuses one cell per parameter where recursion gave each frame its own, so a closure that outlives its iteration would see the final value rather than the one it was made with. let collect n acc = … collect (n - 1) (List.concat acc [fun k -> n]) must therefore stay recursive, and there is an end-to-end test asserting the closures still yield [3, 2, 1] rather than trusting the reasoning.

Merge-order note: this branch adds PyStmt::UnpackAssign to the Python IR, and so does #38 (which needs it for destructuring parameters). Both are off main rather than stacked, so whichever merges second will hit a small conflict: an enum variant and its emitter arm, added identically in both.

Tests: 8 in tests/compile.rs (loop shape, simultaneous rebinding, a tail call in a match arm, non-tail left alone, mutual recursion left alone, the closure guard, plus two e2e: 50k/100k-deep recursion that only completes as a loop, and the closure semantics above). Full suite, clippy and fmt clean. Documented as DESIGN §5.4 with mechanics in INTERNALS.

CPython has no tail-call elimination and caps the stack near 1000 frames,
so a function that drives an unbounded loop by calling itself in tail
position walks a stack it has no reason to build. The dogfooded game called
setRecursionLimit 20000 at startup for exactly this: every turn and every
rejected input was a frame that never returned until the game ended.

A direct, saturated self tail call now lowers to `while True:` with the
parameters rebound and a `continue`. Several parameters rebind
simultaneously from one tuple, so an argument still reads the previous
iteration's values, exactly as the call it replaces evaluated its arguments
before entering the function. The language does not change: the recursive
definition is still the only way to write it, and this is a property of the
emitted code.

The pass runs on the *lowered* body rather than the Pyfun AST, which is
what keeps it small: `lower_return` emits a Return only in tail position,
so a tail call is just a Return whose value calls our own name at our own
arity, and the tail-position walk is a walk over statements that already
exist.

Four preconditions, each falling back to the ordinary recursive def, which
is always correct and merely stack-bound: the body must not rebind the
function's own name; it must not be a generator or coroutine; the walk does
not descend into a `for` (a `continue` there belongs to the `for`) or a
`try` (looping inside it would put every later iteration under a handler
that covered one call); and no nested function may mention a name this
frame binds. The last one is the subtle one, and it is why the rewrite is
sound: the loop reuses one cell per parameter where recursion gave each
frame its own, so a closure outliving its iteration would otherwise see the
final value rather than the one it was made with. An end-to-end test pins
that behaviour rather than trusting the reasoning.

Mutual recursion and general TCO stay out: both need a trampoline, which
costs the readable output lowering exists to protect.
@simontreanor
simontreanor force-pushed the feat/self-tail-call branch from 8d62a37 to 89e6ef9 Compare July 31, 2026 11:24
@simontreanor
simontreanor merged commit 82fd069 into main Jul 31, 2026
11 checks passed
@simontreanor
simontreanor deleted the feat/self-tail-call branch July 31, 2026 11:27
simontreanor added a commit that referenced this pull request Jul 31, 2026
Two dogfooding reports from real programs, and the standard-library sweep
they triggered.

Language:

* a `type` declaration can name an imported type, bare or module-qualified
  (#36) — the one gap that changed a program's architecture rather than its
  phrasing, forcing two modules into one file
* field access resolves from the base's type when it is known, so two
  records may share a field name without prefixes (#37)
* parameters destructure: tuples (#38), records (#40), and `_`
* a direct self tail call lowers to a loop, so an interactive turn loop no
  longer walks the stack (#39, #41)

Standard library — about 115 new members, taking every module to the F#
core set: List (#42), Seq (#44), Set and Map (#46), String (#47), Option
and Result (#48), then a member-by-member FSharp.Core audit (#51). Every
built-in member now carries a one-line description and its complexity in
hover and completion (#43, #49), enforced by tests.

Fixes:

* `pyfun run` on a single file gives the program its own stdin, so an
  interactive program is runnable by the command whose job is running
  programs (#35)
* a partially applied lambda closes over its argument instead of being
  wrapped, so `List.map ((+) 2)` emits `lambda b: 2 + b` (#52)
* every multi-argument callback's scheme put the effect variable on the
  wrong arrows, so `List.fold` could never accept an effectful folder (#51)
* `Seq.empty` lowered to a bare `iter()`, a TypeError (#51)

One source-incompatible change, which is why this is 0.4.0 and not 0.3.1:
a dotted `extern` target whose module prefix cannot be decided from the
text is now a compile error naming the `extern import` to add (#50).
`sys.stdout.flush` used to emit `import sys.stdout` and fail at runtime;
declaring `extern import sys` fixes it.
simontreanor added a commit that referenced this pull request Aug 2, 2026
The course was written against a smaller language and a smaller library, so
parts of it now describe a compiler that no longer behaves that way, and parts
of it leave a learner unprepared for an error the compiler will actually raise.

Wrong, and verified against the 0.5.0 binary:

- Lesson 8's "a `let` cannot destructure a constructor" demo quoted an error
  that does not fire. `let Some x = Some 1` parses as a *function* named `Some`
  with parameter `x`, so it type-checks; the demo needs `let (Some x) = …`, and
  the parenthesized form is now shown with the reason the brackets matter.
- Lesson 17 said a recursive function always shares Python's stack. A saturated
  self tail call has lowered to `while True` since #39, so the lesson now shows
  a hundred thousand levels of `countDown` running, and keeps the stack warning
  for the shape that still has one (`fact`, whose call sits under a `*`).
- Four quoted typed-hole notes listed suggestions the stdlib sweep changed.
  Lesson 9's was worse than stale: it taught "the compiler names your answer"
  above a note where `String.upper` no longer appears, having been pushed past
  the six-fit cap by the new `string -> string` members. That exercise now
  normalizes case with `String.lower`, which the note does name, and the lesson
  says plainly that the list is a shortlist.
- Lesson 18 called `async` the third built-in builder; it is the fourth.
- Lesson 6 quoted emitted Python without the `_pf_t0` temporary a record update
  actually emits.
- Lesson 5 defined `sign`, which shadows the prelude's `sign` and gives a
  student the wrong hover.

Missing, in the order a learner hits them:

- `extern import` was taught nowhere, yet since #50 a target like
  `sys.stdout.flush` is a hard compile error asking for exactly that line.
  Lesson 12 now walks the error and the fix.
- Instance-access externs (`= .with_name`) were referenced by lesson 23 as
  something lesson 12 covers. Lesson 12 did not. Now it does, and the
  cross-reference is true.
- Lesson 15 showed only values crossing a module boundary. Types cross too, and
  since the dogfooding fix a record may name another module's type, which is
  what lets a program split along its data.
- `Seq` was used in lesson 13 before ever being introduced, and laziness was the
  one collection idea the course did not teach. Lesson 7 introduces it, along
  with the two conventions the sweep settled: total functions, and `Option` from
  any accessor that can come up empty.
- `Format` was invisible outside quoted hole notes, and `input` was untaught, so
  "how do I read from the user" had no answer on the site.

One compiler fix came out of writing that up: `Format.thousands` and
`Format.grouped` had each other's hover documentation (`thousands` formats a
float with grouping, `grouped` an integer), and the padding pair's docs omitted
their fill argument.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant