Experiment: Compile down to machine instructions

Cargo.toml

@@ -13,18 +13,23 @@ keywords = ["python", "transpiler", "compiler"]
 categories = ["parsing"]
 
 [dependencies]
-ansi_term = "0.12.1"
-assert_cmd = "0.10"
-clap = { version = "4.5.40", features = ["derive"] }
-glob = "0.3.0"
-itertools = "0.8.2"
-log = "0.4"
-loggerv = "0.7"
-pathdiff = "0.1.0"
-permutate = "0.3.2"
-python-parser = "0.1.0"
+ansi_term = "0.12.1"   # Colors in the terminal
+clap = { version = "4.5.40", features = ["derive"] } # Compiler frontend
+cranelift-codegen = { version = "0.114.0", features = ["all-native-arch"] } # Machine code IR, instruction selection and register allocation
+cranelift-frontend = "0.114.0" # Build Cranelift IR function bodies (SSA construction)
+cranelift-module = "0.114.0" # Function/data declaration shared across Cranelift backends
+cranelift-native = "0.114.0" # Detect the host target when --target is not given
+cranelift-object = "0.114.0" # Emit a native object file from compiled Cranelift IR
+glob = "0.3.0"          # Make traversing directories (Mamba projects) easier
+itertools = "0.8.2"     # Tools to make iterating over collections easier
+log = "0.4"             # Log frontend
+loggerv = "0.7"         # Logger which accepts -vvv flag
+pathdiff = "0.1.0"      # Traverse directories (Mamba projects)
+python-parser = "0.1.0" # Parse Python, both in tests and to generate Mamba IR
+target-lexicon = "0.12.16" # Parse a --target triple for the Cranelift backend
+tempfile = "3.1.0"      # Stage object files before linking them into a --bin executable
 
 [dev-dependencies]
-test-case = "3.3.1"
-# dependencies: test -> test util -> mamba
+assert_cmd = "0.10"     # Assert commands work
+test-case = "3.3.1"     # Parameterize tests
 tests_util = { path = "./tests_util" }

README.md

@@ -750,34 +750,45 @@ end recover do
 end
 ```
 
+## 💽 Machine Output
+
+There is an experimental feature where we output a very small subset of the language to machine code.
+This is more of a 'fun' feature meant to explore a bit how compiler backends work to an extent.
+We mostly limit this to simple arithmetic for now.
+
+To use, us either the `--bin` flag to produce a binary, or `--asm` to print AT&T style assembly to stdout.
+We aim to make sure that:
+
+1. The output is identical to running and checking the output of the resulting Python (see `./tests/execution.rs`).
+2. That compilation works as is identical on the latest Windows, Linux, and Mac OS.
+   We verify this by making use of GitHub agents which run the test suite on each, see `./github/workflows/test.yml`.
+
+In general, we aim to stay within the Rust ecosystem as much as possible.
+We prefer writing our own boilerplate, or depending on rust crates, over depending on native C.
+The reasoning is that we want to reduce external dependencies, and more importantly, that this arguably improves the educational value this crate provides (for the author).
+Having to (re)-implement difficult compilation problems which have been solved in the past (and there are _many_, including edge cases) increases our exposure to them.
+
 ## 💻 The Command Line Interface
 
 ```
-USAGE:
-    mamba.exe [FLAGS] [OPTIONS]
-
-FLAGS:
-    -a, --annotate          Enable type annotation of the output source.
-                            Currently still buggy feature.
-    -d, --debug             Add line numbers to log statements
-    -h, --help              Prints help information
-    -l, --level             Print log level
-        --no-module-path    Disable the module path in the log statements
-        --no-color          Disable colorized output
-    -v                      Set level of verbosity
-                            - v   : info, error, warning printed to stderr (Default)
-                            - vv  : debug messages are printed
-                            - vvv : trace messages are printed
-    -V, --version           Prints version information
-
-OPTIONS:
-    -i, --input <INPUT>      Input file or directory.
-                             If file, file taken as input.
-                             If directory, recursively search all sub-directories for *.mamba files.
-                             If no input given, current directory used as input directory.
-    -o, --output <OUTPUT>    Output directory to store Python files.
-                             Output directory structure reflects input directory structure.
-                             If no output given, 'target' directory created in current directory.
+Transpile Mamba to Python code, compile it to a native binary, or print its assembly.
+
+Usage: mamba [OPTIONS]
+
+Options:
+  -i, --input <INPUT>    Input file or directory. If file, file taken as input. If directory, recursively search all sub-directories for *.mamba files. If no input given, current directory used as input directory
+  -o, --output <OUTPUT>  Output location. With `--python` (the default): output directory to store Python files, structured to reflect the input directory; if not given, a 'target' directory is created in the current directory. With `--bin`: path of the linked executable to produce; if not given, 'a.out' is created in the current directory. Ignored with `--asm`, which always prints to stdout instead of writing a file
+      --python           Output Python source (the default)
+      --bin              Compile and link a native executable via the Cranelift backend, instead of outputting Python source. Only a small subset of the language is currently supported:literals, arithmetic and comparison operators, if/else, top-level function definitions and calls, and `print`
+      --asm              Compile via the Cranelift backend and print the resulting disassembly to stdout, instead of outputting Python source or linking an executable. No file is written -- pipe stdout (e.g. `> out.s`) if you want to save it. Same language subset as `--bin` (see its help). Printed in AT&T syntax (`movq %rsp, %rbp`, source before destination) -- Cranelift's own disassembler doesn't support switching to Intel syntax
+      --target <TARGET>  Target triple to pass to Cranelift, e.g. `x86_64-unknown-linux-gnu` (only meaningful with `--bin`/`--asm`; defaults to the host triple)
+  -v...                  Set level of verbosity: - `-v`   : info, error, warning printed to stderr (default) - `-vv`  : debug messages are printed - `-vvv` : trace messages are printed
+  -d, --debug            Add line numbers to log statements
+      --no-module-path   Disable the module path in the log statements
+      --no-color         Disable colorized output
+  -l, --level            Print log level
+  -a, --annotate         Enable type annotation of the output source. Currently still buggy feature
+  -h, --help             Print help (see more with '--help')
 ```
 
 You can type `mamba -help` for a message containing roughly the above information.

flake.nix

@@ -39,7 +39,7 @@
             git      # Version control tool
             less     # Used under the hood by git
             more     # Nice to have next to 'less'
-
+            
             clang                    # C++ tooling
             llvmPackages.bintools    #
             llvmPackages_latest.llvm # LLVM build tools (also provides llvm-cov/llvm-profdata, see LLVM_COV below)

src/backend/cranelift/README.md

@@ -0,0 +1,66 @@
+<p align="center">
+    <img src="../../image/logo.svg" height="150" alt="Mamba logo"/>
+</p>
+
+# Cranelift
+
+Compiles a checked `ASTTy` directly to native machine code via [Cranelift](https://cranelift.dev/), instead of transpiling to Python source.
+Unlike the Python backend, there is no intermediate `PythonCore`-style tree:
+lowering walks the `ASTTy` once and emits Cranelift IR straight into a `cranelift_object::ObjectModule` via imperative builder calls, which Cranelift itself then turns into machine code.
+
+Three public entry points, all in `mod.rs`, mirroring the Python backend's `write_output`/`gen`/`gen_arguments` shape:
+
+- `write_output` -- compiles and links an executable, written to disk (the `--bin` CLI flag).
+- `print_asm` -- compiles and prints the disassembly to stdout instead, no file written (the `--asm` CLI flag).
+- `compile` / `disassemble` -- the single-file entry points those two build on: `compile` returns object bytes,
+  `disassemble` returns disassembly text (see "Assembly output" below).
+
+The general idea is that we are able to leverage the type checker so that we _know_ what the type of each node at compile time.
+This means that we offer the flexibility of not having to exhaustively define types everywhere.
+The type checker still verifies correctness and gives this information to us so that we are able to produce machine code.
+Else, without knowing the type in advance, we would not be able to produce machine code except in the most trivial cases.
+
+## Supported language subset
+
+Only a small slice of Mamba compiles down to machine code,
+enforced by simply erroring (`BackendErr::unimplemented`) on anything else:
+
+- `Int`, `Bool`, `Float` primitives -- no collections, strings (beyond a `print` argument), classes, or traits.
+- Arithmetic (`+ - * /`) and comparison (`< <= > >= == !=`) operators, over `Int` or `Float` -- `operation.rs`'s
+  `lower_arith`/`lower_cmp` check the *operand's* resolved type (not just that it's some supported primitive) to
+  pick `iadd`/`fadd` and friends, `icmp`/`fcmp`, since Cranelift has no single opcode for both.
+- `if`/`else`, both as a statement and in a function's tail (return) position.
+- `for <id> in <a> .. <b>` / `..=` loops over `Int` ranges -- not arbitrary collections, since collections aren't supported at all.
+- Plain (`:=`) reassignment of an already-declared variable -- not compound assignment (`+=` and friends).
+- Top-level function definitions and calls, including forward references within the same file.
+- `print`, lowered directly to libc `puts` (string literal) or `printf` (an `Int`/`Bool` value). A `Float` value is
+  rejected -- `printf`'s `%lld` would read the raw float bits as an integer, and a `%f`-style call needs SysV
+  variadic-call ABI plumbing (setting `%al` to the vector-register count) this backend doesn't have yet.
+
+Every other top-level statement in a file is collected into a synthetic `main`, since machine code needs an explicit
+entry point the way a `.mamba` file's top-to-bottom script execution doesn't.
+
+## Layout
+
+- `convert/` -- the lowering itself, split by AST category (`definition.rs`, `control_flow.rs`, `call.rs`, `operation.rs`, plus a shared `common.rs`), the same way `backend::python::convert` is.
+  `mod.rs` holds the entry point (`lower_program`) and the three dispatchers a Mamba node can be lowered as:
+   a statement (`lower_stmt`), the tail of a function body (`lower_tail`), or a value-producing expression (`lower_expr`).
+- `primitive.rs` -- resolves a checked `Name` to the one Cranelift `Type` it supports (`Int`/`Bool`/`Float`),
+  the same role `backend::python::name` plays for Python's richer type surface.
+- `link.rs` -- shells out to the system `cc` to link object files into an executable,
+  the same approach `rustc` itself uses rather than reimplementing a linker.
+- `result.rs` -- `BackendErr`/`BackendResult`, mirroring `backend::python::result`.
+
+## Assembly output
+
+`disassemble` asks Cranelift to compute disassembly text (`Context::set_disasm` + `CompiledCode::vcode`) while lowering,
+gated behind a `want_asm: bool` threaded through `convert::lower_program` so `compile` (the `--bin` path) never pays for it.
+It's printed in AT&T syntax (source operand before destination, e.g. `movq %rsp, %rbp`) as that's what Cranelift's own disassembler always produces;
+real Intel-syntax output would mean re-disassembling the emitted machine code with an external disassembler (e.g. capstone) instead.
+To keep things simple and to keep external dependencies to a minimum we opt not to do that.
+
+This is instructions only, not a full disassembly of the object, it doesn't cover the data section.
+A string literal (e.g. a `print("...")` argument) is emitted as a separate anonymous data blob,
+so it never appears in the output;
+The instructions that reference it only show an opaque symbol (e.g. `load_ext_name userextname0+0, %rdi`).
+This is similar to how import like `puts` shows up as a bare symbol rather than "the puts function".