fix: parsing while statement

Author: Joel Abrahams

CLAUDE.md

@@ -55,11 +55,10 @@ The pipeline for a single file is: `String` → **parse** → `AST` → **check*
 has its own README worth reading before making non-trivial changes there (`src/parse/README.md`,
 `src/check/README.md`, `src/generate/README.md`).
 
-- **`src/parse`**: `lex/` tokenizes source into a `Vec<Token>`, handling indentation by emitting synthetic
-  `Indent`/`Dedent` tokens that the parser uses to delimit blocks. The parser (`expression.rs`, `statement.rs`,
+- **`src/parse`**: `lex/` tokenizes source into a `Vec<Token>`. The parser (`expression.rs`, `statement.rs`,
   `class.rs`, `definition.rs`, `control_flow_*.rs`, `collection.rs`, `call.rs`, `operation.rs`, `ty.rs`, etc.)
   walks tokens via a `TokenIterator` and builds an `AST` (`parse/ast`), where each node carries a `Position` for
-  error reporting. Errors here mean the token stream doesn't conform to the grammar.
+  error reporting.
 
 - **`src/check`**: the type checker, and where most language semantics live. Three phases:
   1. **Context building** (`check/context`): scans all ASTs (including implicit/explicit imports and built-in
@@ -102,6 +101,54 @@ has its own README worth reading before making non-trivial changes there (`src/p
   fixture path resolution, randomized temp output dirs, and the Python-AST-diff assertion logic used by
   `test_directory`/`test_directory_args`.
 
+## Block syntax (post indent/dedent removal)
+
+`for`/`while`/`with` bodies always require an explicit `do ... end` block — there is no single-statement
+shorthand for these three (`for a in b do c` is a parse error; it must be `for a in b do c end`). `if`/`then`/
+`else` branches are the exception: each branch is parsed as one `parse_expr_or_stmt`, which accepts either a
+bare single statement/expression or an explicit `do ... end` block (`if a then do ... end else c` is valid).
+A leading newline before a statement/expression is insignificant whitespace and is skipped (see
+`parse_expression`'s and `parse_expr_or_stmt`'s `eat_while(&Token::NL)`) — but a *trailing* newline is still
+usually required as a statement separator inside a block, so constructs that need to look past it for an
+optional following keyword (e.g. `parse_if` scanning past newlines for a possible `else`) must use a
+lookahead-with-rollback helper (`LexIterator::peek_if_skipping`) rather than unconditionally consuming the
+newline, or they'll break "no `else`, followed by more statements in the same block".
+
+The call-site "handle" construct for a call that may raise is `<expr> ! where <case> ... end`, e.g.
+`f(10) ! where err: MyErr => do ... end end` — the `!` marks the call as fallible and must be consumed before
+looking for `where` (`parse_expr_or_stmt` in `expr_or_stmt.rs`).
+
+`type X: Parent when <cond>` (single-line) / `type X: Parent when\n <cond>\n...\nend` (multi-line, terminated
+by `end`) is the *conditional type alias* form (produces `Node::TypeAlias`, binds `self` to `Parent` while
+checking the conditions). `type X where <defs> end` is a different form — an interface/type body of field and
+function *signatures* (produces `Node::TypeDef`, does **not** bind `self`). These two are easy to conflate
+(`when` vs `where`) since both start with `type X: Parent`; picking the wrong one either fails to parse or fails
+type-checking with a confusing "Undefined variable: self".
+
+## Known incomplete work (branch `feat-remove-indent-dedent`, as of 2026-08-24)
+
+This branch is mid-refactor from indentation-based blocks to the `do`/`end` scheme above, and several
+`tests/resource/valid/**` fixtures were rewritten ahead of the features they exercise:
+
+- **`trait` is unimplemented.** It's a real, documented keyword (see the README's "traits" section and
+  `docs/spec/trait-def` in `docs/spec/grammar.md`) but the lexer/parser has zero support for it today. A few
+  fixtures (`tests/resource/valid/class/parent.mamba`, `multiple_parent.mamba`,
+  `fun_with_body_in_interface.mamba`, `class_super_one_line_init.mamba`, and transitively `types.mamba` via a
+  dropped parent class) were rewritten to use `trait` and no longer parse. Fixing these needs either
+  implementing `trait` as a real parser+checker+codegen feature, or reverting them to the `class`/`type`-based
+  syntax their paired `.py` reference files still expect.
+- **Class-body statements/field-initializers that depend on constructor state are never moved into a generated
+  `__init__`.** E.g. `class X(a: Float) where\n def y: Y := Y(a)\nend` (a bare, non-`def` constructor arg used
+  in a field initializer) or a bare executable statement in a class body (e.g. a `print(...)` call) — Python
+  reference fixtures expect these to be hoisted into `__init__` (with a `None` placeholder left at class level
+  for fields), but `src/generate/convert/class.rs`'s `extract_class`/`init` only handles parent-`__init__`
+  calls and auto-generated `self.field = arg` assignments for constructor args, not general relocation. This
+  needs a free-variable analysis over `Core` expressions to detect which class-body statements reference
+  constructor-only names. The type checker itself does correctly resolve these now (see `constrain_class_body`
+  in `src/check/constrain/generate/class.rs`, which binds non-`def` constructor args into the class body's
+  environment) — it's specifically the codegen relocation that's missing, so affected programs type-check but
+  transpile to Python that references undefined names.
+
 ## Documentation
 
 `docs/` contains the (partially outdated, per its own README) language specification and philosophy docs,

src/parse/README.md

@@ -13,11 +13,8 @@ Implementation wise, an `AST` is where a `Node` are stored.
 
 ## Lex
 
-The lexer converts a string to a vector of `Token`s. This is in part to simplify the parsing stage.
-
-The lexer also deals with the logic necessary to detect indentations. It produces special tokens `Indent` and `Dedent`,
-which the next stage uses to identify the start and end of blocks.
-
+The lexer converts a string to a vector of `Token`s.
+This is in part to simplify the parsing stage.
 We get an error if we encounter an illegal character
 
 ## Parse

src/parse/class.rs

@@ -32,7 +32,7 @@ pub fn parse_class(it: &mut LexIterator) -> ParseResult {
 
     let mut parents = vec![];
     if it.eat_if(&Token::DoublePoint).is_some() {
-        it.peek_while_not_token(&Token::Where, &mut |it, lex| match lex.token {
+        it.peek_while_not_tokens(&[Token::NL, Token::Where], &mut |it, lex| match lex.token {
             Token::Id(_) | Token::LRBrack => {
                 parents.push(*it.parse(&parse_parent, "parents", start)?);
                 it.eat_if(&Token::Comma);

src/parse/control_flow_stmt.rs

@@ -1,6 +1,5 @@
 use crate::parse::ast::{Node, AST};
 use crate::parse::block::parse_block;
-use crate::parse::expr_or_stmt::parse_expr_or_stmt;
 use crate::parse::iterator::LexIterator;
 use crate::parse::lex::token::Token;
 use crate::parse::operation::parse_expression;
@@ -34,9 +33,8 @@ pub fn parse_cntrl_flow_stmt(it: &mut LexIterator) -> ParseResult {
 fn parse_while(it: &mut LexIterator) -> ParseResult {
     let start = it.start_pos("while statement")?;
     it.eat(&Token::While, "while statement")?;
-    let cond = it.parse(&parse_expression, "while statement", start)?;
-    it.eat(&Token::Do, "while")?;
-    let body = it.parse(&parse_expr_or_stmt, "while statement", start)?;
+    let cond = it.parse(&parse_expression, "while statement condition", start)?;
+    let body = it.parse(&parse_block, "while statement block", start)?;
 
     let node = Node::While {
         cond,
@@ -282,7 +280,7 @@ mod test {
 
     #[test]
     fn while_verify() {
-        let source = String::from("while a do d");
+        let source = String::from("while a do d end");
         let statements = parse_direct(&source).unwrap();
 
         let (cond, body) = match &statements.first().expect("script empty.").node {
@@ -296,8 +294,14 @@ mod test {
                 lit: String::from("a")
             }
         );
+
+        let Node::Block { statements } = body.node else {
+            panic!("expected block: {body:#?}");
+        };
+
+        assert_eq!(statements.len(), 1);
         assert_eq!(
-            body.node,
+            statements[0].node,
             Node::Id {
                 lit: String::from("d")
             }

src/parse/expr_or_stmt.rs

@@ -9,6 +9,8 @@ use crate::parse::statement::{is_start_statement, parse_reassignment, parse_stat
 use crate::parse::Lex;
 
 pub fn parse_expr_or_stmt(it: &mut LexIterator) -> ParseResult {
+    it.eat_while(&Token::NL);
+
     let expr_or_stmt = it.peek_or_err(
         &|it, lex| match &lex.token {
             Token::Do => it.parse(&parse_block, "expression or statement", lex.pos),
@@ -19,9 +21,11 @@ pub fn parse_expr_or_stmt(it: &mut LexIterator) -> ParseResult {
         "expression or statement",
     )?;
 
-    // if expression/statement followed by newline and indent, we are dealing with a handle block
+    // a call that may raise is marked with '!'; if followed by 'where' we are dealing with a handle block
+    it.eat_if(&Token::Raise);
+
     if it.peek_if(&|lex: &Lex| lex.token == Token::Where) {
-        it.eat(&Token::NL, "internal error in parsing call")?; // peek covers this
+        it.eat(&Token::Where, "handle expression")?;
 
         // parse handle cases if indentation block after
         let cases = it.parse_vec(&parse_match_cases, "handle cases", expr_or_stmt.pos)?;

src/parse/result.rs

@@ -91,7 +91,7 @@ pub fn expected_one_of(tokens: &[Token], actual: &Lex, parsing: &str) -> ParseEr
 
 pub fn expected(expected: &Token, actual: &Lex, parsing: &str) -> ParseErr {
     let msg = format!(
-        "Expected {}{expected:?} token while parsing {}{parsing}, but found '{}' ({})",
+        "Expected {} token while parsing {}{parsing}, but found '{}' ({})",
         an_or_a(expected),
         an_or_a(parsing),
         actual.token,

src/parse/statement.rs

@@ -1,9 +1,9 @@
 use crate::parse::ast::node_op::NodeOp;
 use crate::parse::ast::Node;
 use crate::parse::ast::AST;
+use crate::parse::block::parse_block;
 use crate::parse::control_flow_stmt::parse_cntrl_flow_stmt;
 use crate::parse::definition::parse_definition;
-use crate::parse::expr_or_stmt::parse_expr_or_stmt;
 use crate::parse::iterator::LexIterator;
 use crate::parse::lex::token::{Lex, Token};
 use crate::parse::operation::parse_expression;
@@ -174,8 +174,7 @@ pub fn parse_with(it: &mut LexIterator) -> ParseResult {
         None
     };
 
-    it.eat(&Token::Do, "with")?;
-    let expr = it.parse(&parse_expr_or_stmt, "with", start)?;
+    let expr = it.parse(&parse_block, "with", start)?;
 
     let node = Node::With {
         resource,

tests/resource/invalid/type/collection/set_conflicting_collection_types.mamba

@@ -2,3 +2,4 @@ def x := {"one", 51}
 
 for item in x do
     print(item + 2)
+end

tests/resource/invalid/type/control_flow/for_non_iterable.mamba

@@ -1,2 +1,3 @@
 for i in 10 do
     print(i)
+end

tests/resource/invalid/type/control_flow/for_over_union_one_not_minus.mamba

@@ -2,3 +2,4 @@ def x: {Range, String} := "MyString"
 
 for i in x do
     print(i - 1)
+end