lowering: a self tail call becomes a loop - #39
Merged
Conversation
CPython has no tail-call elimination and caps the stack near 1000 frames, so a function that drives an unbounded loop by calling itself in tail position walks a stack it has no reason to build. The dogfooded game called setRecursionLimit 20000 at startup for exactly this: every turn and every rejected input was a frame that never returned until the game ended. A direct, saturated self tail call now lowers to `while True:` with the parameters rebound and a `continue`. Several parameters rebind simultaneously from one tuple, so an argument still reads the previous iteration's values, exactly as the call it replaces evaluated its arguments before entering the function. The language does not change: the recursive definition is still the only way to write it, and this is a property of the emitted code. The pass runs on the *lowered* body rather than the Pyfun AST, which is what keeps it small: `lower_return` emits a Return only in tail position, so a tail call is just a Return whose value calls our own name at our own arity, and the tail-position walk is a walk over statements that already exist. Four preconditions, each falling back to the ordinary recursive def, which is always correct and merely stack-bound: the body must not rebind the function's own name; it must not be a generator or coroutine; the walk does not descend into a `for` (a `continue` there belongs to the `for`) or a `try` (looping inside it would put every later iteration under a handler that covered one call); and no nested function may mention a name this frame binds. The last one is the subtle one, and it is why the rewrite is sound: the loop reuses one cell per parameter where recursion gave each frame its own, so a closure outliving its iteration would otherwise see the final value rather than the one it was made with. An end-to-end test pins that behaviour rather than trusting the reasoning. Mutual recursion and general TCO stay out: both need a trampoline, which costs the readable output lowering exists to protect.
simontreanor
force-pushed
the
feat/self-tail-call
branch
from
July 31, 2026 11:24
8d62a37 to
89e6ef9
Compare
This was referenced Jul 31, 2026
Merged
simontreanor
added a commit
that referenced
this pull request
Jul 31, 2026
Two dogfooding reports from real programs, and the standard-library sweep they triggered. Language: * a `type` declaration can name an imported type, bare or module-qualified (#36) — the one gap that changed a program's architecture rather than its phrasing, forcing two modules into one file * field access resolves from the base's type when it is known, so two records may share a field name without prefixes (#37) * parameters destructure: tuples (#38), records (#40), and `_` * a direct self tail call lowers to a loop, so an interactive turn loop no longer walks the stack (#39, #41) Standard library — about 115 new members, taking every module to the F# core set: List (#42), Seq (#44), Set and Map (#46), String (#47), Option and Result (#48), then a member-by-member FSharp.Core audit (#51). Every built-in member now carries a one-line description and its complexity in hover and completion (#43, #49), enforced by tests. Fixes: * `pyfun run` on a single file gives the program its own stdin, so an interactive program is runnable by the command whose job is running programs (#35) * a partially applied lambda closes over its argument instead of being wrapped, so `List.map ((+) 2)` emits `lambda b: 2 + b` (#52) * every multi-argument callback's scheme put the effect variable on the wrong arrows, so `List.fold` could never accept an effectful folder (#51) * `Seq.empty` lowered to a bare `iter()`, a TypeError (#51) One source-incompatible change, which is why this is 0.4.0 and not 0.3.1: a dotted `extern` target whose module prefix cannot be decided from the text is now a compile error naming the `extern import` to add (#50). `sys.stdout.flush` used to emit `import sys.stdout` and fail at runtime; declaring `extern import sys` fixes it.
simontreanor
added a commit
that referenced
this pull request
Aug 2, 2026
The course was written against a smaller language and a smaller library, so parts of it now describe a compiler that no longer behaves that way, and parts of it leave a learner unprepared for an error the compiler will actually raise. Wrong, and verified against the 0.5.0 binary: - Lesson 8's "a `let` cannot destructure a constructor" demo quoted an error that does not fire. `let Some x = Some 1` parses as a *function* named `Some` with parameter `x`, so it type-checks; the demo needs `let (Some x) = …`, and the parenthesized form is now shown with the reason the brackets matter. - Lesson 17 said a recursive function always shares Python's stack. A saturated self tail call has lowered to `while True` since #39, so the lesson now shows a hundred thousand levels of `countDown` running, and keeps the stack warning for the shape that still has one (`fact`, whose call sits under a `*`). - Four quoted typed-hole notes listed suggestions the stdlib sweep changed. Lesson 9's was worse than stale: it taught "the compiler names your answer" above a note where `String.upper` no longer appears, having been pushed past the six-fit cap by the new `string -> string` members. That exercise now normalizes case with `String.lower`, which the note does name, and the lesson says plainly that the list is a shortlist. - Lesson 18 called `async` the third built-in builder; it is the fourth. - Lesson 6 quoted emitted Python without the `_pf_t0` temporary a record update actually emits. - Lesson 5 defined `sign`, which shadows the prelude's `sign` and gives a student the wrong hover. Missing, in the order a learner hits them: - `extern import` was taught nowhere, yet since #50 a target like `sys.stdout.flush` is a hard compile error asking for exactly that line. Lesson 12 now walks the error and the fix. - Instance-access externs (`= .with_name`) were referenced by lesson 23 as something lesson 12 covers. Lesson 12 did not. Now it does, and the cross-reference is true. - Lesson 15 showed only values crossing a module boundary. Types cross too, and since the dogfooding fix a record may name another module's type, which is what lets a program split along its data. - `Seq` was used in lesson 13 before ever being introduced, and laziness was the one collection idea the course did not teach. Lesson 7 introduces it, along with the two conventions the sweep settled: total functions, and `Option` from any accessor that can come up empty. - `Format` was invisible outside quoted hole notes, and `input` was untaught, so "how do I read from the user" had no answer on the site. One compiler fix came out of writing that up: `Format.thousands` and `Format.grouped` had each other's hover documentation (`thousands` formats a float with grouping, `grouped` an integer), and the padding pair's docs omitted their fill argument.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dogfooding finding #6 (see #34), implementing the option you picked: lowering-only, no language change.
CPython has no TCE and caps the stack near 1000 frames, so an interactive loop written as recursion walks a stack it has no reason to build. The dogfooded game called
setRecursionLimit 20000at startup for exactly this.A direct, saturated self tail call now lowers to:
Several parameters rebind simultaneously from one tuple, so an argument still reads the previous iteration's values, exactly as the call it replaces evaluated its arguments before entering.
Why it is small. The pass runs on the lowered body rather than the Pyfun AST.
lower_returnemits aReturnonly in tail position, so "tail call" is just "aReturnwhose value calls our own name at our own arity", and the tail-position walk is a walk over statements that already exist. No new notion of tail position was needed.The four preconditions, each falling back to the ordinary recursive def (always correct, merely stack-bound):
letwould make the loop jump where the program meant to call);returnin a generator raisesStopIterationwith a value; an async tail call is anawait);for(acontinuethere belongs to thefor) or atry(looping inside it would put every later iteration under a handler that covered one call);Number 4 is the one that makes this sound, and it is worth the review time. The loop reuses one cell per parameter where recursion gave each frame its own, so a closure that outlives its iteration would see the final value rather than the one it was made with.
let collect n acc = … collect (n - 1) (List.concat acc [fun k -> n])must therefore stay recursive, and there is an end-to-end test asserting the closures still yield[3, 2, 1]rather than trusting the reasoning.Merge-order note: this branch adds
PyStmt::UnpackAssignto the Python IR, and so does #38 (which needs it for destructuring parameters). Both are offmainrather than stacked, so whichever merges second will hit a small conflict: an enum variant and its emitter arm, added identically in both.Tests: 8 in
tests/compile.rs(loop shape, simultaneous rebinding, a tail call in a match arm, non-tail left alone, mutual recursion left alone, the closure guard, plus two e2e: 50k/100k-deep recursion that only completes as a loop, and the closure semantics above). Full suite, clippy and fmt clean. Documented as DESIGN §5.4 with mechanics in INTERNALS.