Kilo/electric bear mdr

README.md

@@ -19,9 +19,14 @@ pip install frscript
 - **Command line launcher** (`fr`)
 - **Compiled language** - Fast by design
 - **File and Socket I/O** - Low-level file operations and sockets.
+- **HTTP library** - A working HTTP/1.1 server/client library in `examples/http/http_lib.fr`, running on all three backends.
+- **Fr OS target** - `fr native ... --target fr-os` emits a static Fr OS userspace ELF using the Fr OS syscall ABI and user ISA profile.
 - **Multiprocessing**. - Easy threading with fork() and wait().
 - **Python integration** - You can use any Python libraries with Frscript.
-- **Aggressive optimization** - Bytecode-level optimizations. Faster than python.
+- **Aggressive optimization** - Purity call-graph analysis, interprocedural ranges, call-site specialization, pure constant-call folding, LICM, SROA/escape analysis, pure-function lookup tables, inline list fast paths and assembly peepholes.
+- **Deterministic memory** - Reference-counted native structs/lists/sets with function-scope cleanup, plus SROA and stack allocation for non-escaping values.
+- **Modern memory model** - Reference-semantics structs/lists/sets, scope-based reference counting, automatic stack allocation for non-escaping values, and a free-list native allocator.
+- **Opt-in pointers** - `int*`, `&value`, `*p`, pointer fields and pointer parameters, lowered directly to addresses in native code.
 - **Stack-based VM** - Fast and memory efficient.
 - **Readable bytecode** - Optimize hot code manually
 - **WASM Backend** - Run sandboxed code in web browsers.

TODO.md

@@ -1,117 +1,53 @@
-
-# TODO: Native no-libc runtime - Remaining Work
-
-## Current State
-
-- Native tests passing: 141/247
-- Hello world binary: 222 bytes (no libc)
-- Index error tests pass with ?line,0:message format
-- Basic arithmetic, function calls, list bounds checking work
-
-## Remaining Test Failures (106 tests)
-
-### 1. Assertion Failures (2 tests)
-
-- `assertions/assert_fail.fr` - Output "" != expected "Assertion failed"
-- `assertions/assert_output.fr` - Output "" != expected "hello"
-- **Fix**: runtime_assert in fr_rt.c outputs empty instead of message. Need to fix fr_fputs usage.
-
-### 2. Control Flow - Loops (8 tests)
-
-- `control_flow/for_in_list.fr` - Output "" != expected "1\n2\n3"
-- `control_flow/for_in_string.fr` - in function `_fr_main_fall_1`
-- `control_flow/for_in_empty_string.fr` - in function `_fr_main_fall_1`
-- `control_flow/for_range.fr` - Output "" != expected "0\n1\n2\n3\n4"
-- `control_flow/for_range_start.fr` - Output "" != expected "2\n3\n4"
-- `control_flow/for_range_step.fr` - Output "" != expected "0\n2\n4\n6\n8"
-- `control_flow/for_range_step_negative.fr` - Output "" != expected "10\n7\n4\n1"
-- `control_flow/for_range_step_odd.fr` - Output "" != expected "1\n3\n5\n7\n9"
-- `control_flow/break_for.fr` - Output "" != expected "0\n1\n2\n3\n4"
-- `control_flow/continue_for.fr` - Output "" != expected "0\n1\n3\n4"
-- `control_flow/break_while.fr` - Output "" != expected "0\n1\n2"
-- **Fix**: Loop iteration over lists/strings/ranges may be missing runtime support or codegen has issues with loop constructors.
-
-### 3. Struct Allocation (12 tests)
-
-- `data_structures/struct_basic.fr` - undefined reference to `_fr_heap_ptr`
-- `data_structures/struct_field_assign.fr`
-- `data_structures/struct_field_cache.fr`
-- `data_structures/struct_get_local.fr`
-- `data_structures/struct_mixed_types.fr`
-- `data_structures/struct_multi_field_read.fr`
-- `data_structures/struct_nested.fr`
-- `data_structures/struct_return.fr`
-- `data_structures/struct_string_field.fr`
-- `data_structures/struct_in_list.fr`
-- `data_structures/struct_as_param.fr`
-- **Fix**: Codegen emits `_fr_heap_ptr` global but fr_rt.c uses mmap-based heap with `heap_base`/`heap_ptr` locals. Need to either:
-  - Add `_fr_heap_ptr` global to fr_rt.c and update init_heap, OR
-  - Change codegen to call `runtime_alloc_struct(size)` instead of direct heap access
-
-### 4. List Operations (8 tests)
-
-- `data_types/list_append.fr` - Output "[list]" != expected "[1, 2, 3, 4]"
-- `data_types/list_empty.fr` - Output "[list]" != expected "[]"
-- `data_types/list_assignment.fr` - Output "" != expected "10"
-- `data_types/list_indexing.fr` - Output "" != expected "1"
-- `data_types/list_literal.fr` - Output "[list]" != expected "[1, 2, 3]"
-- `data_types/list_len.fr` - Output "" != expected "3"
-- `data_types/list_method_syntax.fr` - Output "" != expected "5"
-- `data_types/list_pop.fr` - in function `_fr_main_main_entry`
-- `data_types/list_pop_last.fr` - in function `_fr_main_main_entry`
-- `data_types/list_pop_remaining.fr` - in function `_fr_main_main_entry`
-- **Fix**: runtime_list_to_str returns "[list]" stub. Need proper list formatting. Also need to verify list append/set operations work.
-
-### 5. Set Operations (3 tests)
-
-- `data_types/set_add_duplicate.fr` - Output "{set}" != expected "{1, 2, 3}"
-- `data_types/set_empty.fr` - Output "{set}" != expected "{}"
-- `data_types/set_string.fr` - Output "{set}" != expected "{hello, world}"
-- **Fix**: runtime_set_to_str returns "{set}" stub. Need proper set formatting with sorted elements.
-
-### 6. String Operations (1 test)
-
-- `control_flow/switch_string.fr` - in function `_fr_main_main_entry`
-- **Fix**: Missing runtime_str_to_lower or similar string case conversion for switch matching.
-
-### 7. Float Formatting (3 tests - already fixed, need verification)
-
-- `data_types/float_multiplication.fr` - Output "7.50" != expected "7.5"
-- `data_types/float_division.fr` - Output "3.50" != expected "3.5"
-- `data_types/float_negative.fr` - Output "-3.50" != expected "-3.5"
-- `data_types/float_addition.fr` - Output "3.50" != expected "3.5"
-- **Fix**: Already modified runtime_float_to_str to strip trailing zeros. Need to re-test.
-
-### 8. Misc (1 test)
-
-- `misc/wait_basic.fr` - in function `_fr_main_main_entry`
-- **Fix**: Missing fork/wait syscalls or runtime support.
-
-### 9. Input Function
-
-- Need to implement runtime_input_str or similar if any tests use input()
-
-## Implementation Order
-
-1. Fix struct allocation (_fr_heap_ptr issue) - unblocks 12 tests
-2. Fix list_to_str - unblocks 8 tests
-3. Fix set_to_str - unblocks 3 tests
-4. Fix assert output - unblocks 2 tests
-5. Fix loop iteration - unblocks 10 tests
-6. Fix float formatting verify - 3-4 tests
-7. Fix switch_string - 1 test
-8. Fix wait_basic - 1 test
-9. Fix any remaining individual tests
-
-## Files to Modify
-
-- `runtime/fr_rt.c` - Add missing runtime functions
-- `src/optimizer/codegen.py` - Fix struct alloc to use runtime call or add _fr_heap_ptr
-- `src/run_single_test.py` - Already updated, may need minor fixes
-- `src/cli.py` - Already updated
+# Roadmap
+
+## Completed
+
+- Native no-libc runtime: **254/254 native tests pass**.
+- Python VM: **325/325 tests pass**.
+- C VM: **325/325 tests pass**.
+- Freestanding native runtime:
+  - real mmap-backed allocator with segregated free lists, `free`, `realloc`
+  - `printf` implementation (`%d`, `%ld`, `%s`, `%f`, `%g`, `%x`, `%c`)
+  - lists/sets/strings/math/file I/O/process/sleep/input
+  - `_setjmp`/`longjmp` based exception handling
+- Memory model:
+  - reference semantics for structs, lists and sets in all three runtimes
+  - C VM reference counting for mutable aggregates
+  - Python VM garbage collection
+  - native SROA + escape analysis + stack allocation for non-escaping structs
+- Language-level pointers:
+  - `int*`, `*int`, `int**` types and struct pointer fields
+  - `&lvalue`, `*p`, `*p = value`, `**pp = value`
+  - pointers to locals, globals, struct fields and list elements
+  - automatic dereference for `p.field` and chained `p.next.value`
+  - global variables and pointers to globals in all runtimes
+- Compiler analysis:
+  - call-graph side-effect/purity analysis
+  - interprocedural integer/boolean range analysis
+  - natural-loop induction-variable bounds
+  - branch-sensitive per-call-site specialization with constants and ranges
+  - dead branch / unreachable block removal
+  - linear block merging and dead function elimination
+  - purity-aware DCE/CSE
+  - compile-time pure-function simulation with embedded lookup tables
+  - tunable budgets for every speculative analysis (`OptimizerConfig`)
+- Native reference counting:
+  - function allocation scopes
+  - retain on global/list/field/pointer/return escapes
+  - release on overwrite and scope exit
+  - safe stack/string pointer handling via object magic/range checks
+
+## Next
+
+- Collect reference-count cycles (managed object cycles are currently leaked,
+  like most RC systems).
+- Pointer lifetimes beyond a frame could become a compile-time error rather than
+  a documented C-style rule.
+- Wasm backend needs the new pointer/global opcodes; it is currently not part of
+  the validated test matrix.
 
 ## Notes
 
-- Wasm backend has 208 failures but is not the current focus
-- C VM has 11 failures (separate from native)
-- Python VM has 2 failures (separate from native)
+- `int *args` is still varargs; use `int* p` or `*int p` for pointer params.
+- Pointers into a list element are invalidated when the list grows.
+- Escaping native heap objects live until process exit.

bench/README.md

@@ -0,0 +1,46 @@
+# Benchmarks
+
+The benchmarks are ordinary Fr programs and are not part of the test suite.
+
+## Vector reductions
+
+`vector_bench.fr` builds 100,000-element integer and float lists and runs
+integer sum, float sum, and float sum-of-squares reductions. The native backend
+recognizes these loops and emits AVX2 vector loops with a scalar remainder,
+while the Python/C VMs run the same source as scalar loops.
+
+```sh
+python3 src/cli.py compile bench/vector_bench.fr -o /tmp/vec.bc
+python3 src/cli.py native /tmp/vec.bc -o /tmp/vec -O3
+time /tmp/vec
+```
+
+## N-body
+
+`nbody.fr` is the correctness-sized run (1,000 steps). `nbody_bench.fr` builds
+128 bodies and runs 1,000 steps; it combines nested loops, struct field traffic,
+list indexing/set, floating-point math, calls and reference-counted heap
+objects. `memory_bench.fr` holds 250,000 reference-counted structs in a list and
+walks them, giving a reproducible allocation/RSS measurement.
+
+```sh
+python3 src/cli.py compile bench/nbody_bench.fr -o /tmp/nbody.bc
+python3 src/cli.py native /tmp/nbody.bc -o /tmp/nbody -O3
+time /tmp/nbody
+/usr/bin/time -f 'rss_kb=%M elapsed=%e' /tmp/nbody
+```
+
+Memory benchmark:
+
+```sh
+python3 src/cli.py compile bench/memory_bench.fr -o /tmp/mem.bc
+python3 src/cli.py native /tmp/mem.bc -o /tmp/mem -O3
+/usr/bin/time -f 'rss_kb=%M elapsed=%e' /tmp/mem
+```
+
+The compiler optimizations validated against this target include scope elision,
+LICM, compare/branch fusion, LEA instruction selection, inline list length/list
+get fast paths, XMM liveness preservation across calls, pure constant-call
+folding, and assembly peepholes. The full test suite (which exercises unrelated
+integer, string, struct, list, error and HTTP workloads) is run after every
+optimization change.

bench/memory_bench.fr

@@ -0,0 +1,26 @@
+// Memory/allocation benchmark: hold 250k reference-counted structs in a list.
+
+struct Node {
+    int a
+    int b
+    int c
+    int d
+}
+
+void main() {
+    list nodes = []
+    int i = 0
+    while (i < 250000) {
+        nodes.append(Node(i, i + 1, i + 2, i + 3))
+        i = i + 1
+    }
+
+    int total = 0
+    int j = 0
+    while (j < len(nodes)) {
+        Node n = nodes[j]
+        total = total + n.a + n.d
+        j = j + 1
+    }
+    println(str(total))
+}

bench/nbody.fr

@@ -0,0 +1,81 @@
+struct Body {
+    float x
+    float y
+    float z
+    float vx
+    float vy
+    float vz
+    float mass
+}
+
+float energy(list bodies) {
+    float e = 0.0
+    int i = 0
+    while (i < len(bodies)) {
+        Body a = bodies[i]
+        e = e + 0.5 * a.mass * (a.vx * a.vx + a.vy * a.vy + a.vz * a.vz)
+        int j = i + 1
+        while (j < len(bodies)) {
+            Body b = bodies[j]
+            float dx = a.x - b.x
+            float dy = a.y - b.y
+            float dz = a.z - b.z
+            float d = sqrt(dx * dx + dy * dy + dz * dz)
+            e = e - (a.mass * b.mass) / d
+            j = j + 1
+        }
+        i = i + 1
+    }
+    return e
+}
+
+void advance(list bodies, float dt, int steps) {
+    int s = 0
+    while (s < steps) {
+        int i = 0
+        while (i < len(bodies)) {
+            Body a = bodies[i]
+            int j = i + 1
+            while (j < len(bodies)) {
+                Body b = bodies[j]
+                float dx = a.x - b.x
+                float dy = a.y - b.y
+                float dz = a.z - b.z
+                float d2 = dx * dx + dy * dy + dz * dz + 0.01
+                float inv = 1.0 / (d2 * sqrt(d2))
+                a.vx = a.vx - dx * b.mass * inv
+                a.vy = a.vy - dy * b.mass * inv
+                a.vz = a.vz - dz * b.mass * inv
+                b.vx = b.vx + dx * a.mass * inv
+                b.vy = b.vy + dy * a.mass * inv
+                b.vz = b.vz + dz * a.mass * inv
+                bodies[j] = b
+                j = j + 1
+            }
+            bodies[i] = a
+            i = i + 1
+        }
+        int k = 0
+        while (k < len(bodies)) {
+            Body p = bodies[k]
+            p.x = p.x + dt * p.vx
+            p.y = p.y + dt * p.vy
+            p.z = p.z + dt * p.vz
+            bodies[k] = p
+            k = k + 1
+        }
+        s = s + 1
+    }
+}
+
+void main() {
+    list bodies = [
+        Body(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 39.47),
+        Body(4.84, -1.16, -0.10, 0.60, 2.81, -0.02, 0.037),
+        Body(8.34, 4.12, -0.40, -1.01, 1.85, 0.01, 0.011),
+        Body(12.89, -15.10, -0.20, 1.08, 0.44, -0.01, 0.001),
+        Body(15.37, -25.91, 0.10, 0.97, 0.58, 0.00, 0.0005)
+    ]
+    advance(bodies, 0.01, 1000)
+    println(str(energy(bodies)))
+}