diff --git a/docs/guides/ci-validation.md b/docs/guides/ci-validation.md index 07bc502..fc3d292 100644 --- a/docs/guides/ci-validation.md +++ b/docs/guides/ci-validation.md @@ -1,57 +1,225 @@ -# Validating firmware in CI +# Firmware integration testing in CI -RP2040Sharp is designed to validate compiler/firmware output (for example, the -[PyMCU](https://docs.pymcu.org) compiler) in CI **without flaky or hanging builds**: +RP2040Sharp's headline use case: run your **real firmware** in CI and assert on what it +actually does — toggles a pin, prints over UART, echoes bytes — **without flaky or hanging +builds**. It's how [PyMCU](https://docs.pymcu.org) validates the firmware its compiler +produces on every push. -- Runs are **bounded** — a wedged program fails with a reason instead of stalling the job. -- The clock is driven by executed cycles, so results are **deterministic** and reproducible. +Why it works well in CI: -There are two ways to use it in a pipeline. +- **Deterministic** — time is driven by executed CPU cycles, never wall-clock, so a run is + reproducible across machines and runners. +- **Never hangs** — execution is bounded; wedged or crashed firmware fails a test with a + reason instead of stalling the job until the runner times out. +- **Fast & headless** — no hardware, no USB, no flashing; MicroPython boots in seconds, + bare-metal firmware in milliseconds. -## In a .NET test project (recommended) +## Set up a test project -Use the TestKit directly from xUnit/NUnit. This is what PyMCU's integration suite does: +The TestKit ships on NuGet. Any .NET test runner works (this guide uses NUnit, like +PyMCU; xUnit is identical in spirit). + +```bash +dotnet add package RP2040Sharp.TestKit +``` + +```xml + + + + + +``` + +Two namespaces give you everything: + +```csharp +using RP2040.TestKit.Boards; // PicoSimulation +using RP2040.TestKit.Extensions; // .Should() for Gpio pins, UART, CPU +``` + +## The basic shape of a test ```csharp -[Fact] -public void Blink_firmware_reports_pass() +[TestFixture] +public class BlinkTests { - using var pico = new PicoSimulation(); - pico.LoadFlash(RP2040Machine.Uf2ToFlash(File.ReadAllBytes("blink.uf2"))!); + private static byte[] _firmware = null!; + + // Build (or load) the firmware once for the whole fixture. + [OneTimeSetUp] + public void Build() => _firmware = MyCompiler.Build("blink"); // or File.ReadAllBytes(...) + + // A fresh machine per test keeps tests independent. + private static PicoSimulation Sim() + { + var pico = new PicoSimulation(withUsbCdc: false); // bare-metal: no USB host + pico.LoadFlash(_firmware); + return pico; + } + + [Test] + public void Led_is_high_after_boot() + { + using var pico = Sim(); + pico.RunMilliseconds(5); + pico.Gpio[25].Should().BeHigh(); + } +} +``` + +```{tip} +For **bare-metal** firmware that doesn't use USB, construct `new PicoSimulation(withUsbCdc: false)`. +Attaching a USB host makes the device think a host is present (and, for CircuitPython, mounts +the filesystem read-only). For MicroPython/CircuitPython REPL tests, leave it on (the default). +``` - var result = pico.RunUntilHalt(pico.Uart0, "PASS"); +`LoadFlash` takes a flat flash image (`byte[]`). For UF2 files, convert first: - result.Succeeded.Should().BeTrue($"firmware halted with {result.Outcome}"); - pico.Cpu.Should().NotHaveFaulted(); +```csharp +pico.LoadFlash(RP2040Machine.Uf2ToFlash(File.ReadAllBytes("firmware.uf2"))!); +``` + +## Asserting on behavior + +### GPIO — the blink test + +```csharp +[Test] +public void Led_toggles_over_time() +{ + using var pico = Sim(); + bool sawHigh = false, sawLow = false; + + // Sample across more than one blink period. + for (int i = 0; i < 120 && !(sawHigh && sawLow); i++) + { + pico.RunMilliseconds(20); + if (pico.Gpio[25].OutputValue) sawHigh = true; else sawLow = true; + } + + sawHigh.Should().BeTrue("the LED should be driven high during a blink"); + sawLow.Should().BeTrue("the LED should be driven low during a blink"); } ``` +Pin assertions: `Should().BeHigh()`, `BeLow()`, `BeOutput()`, `BeInput()`; raw state via +`Gpio[n].OutputValue` / `Gpio[n].DigitalValue`. You can also drive inputs from the test: +`pico.Sio.SetGpioExternalIn(5, high: true)`. + +### UART — banners and round-trips + +```csharp +[Test] +public void Boot_prints_banner() +{ + using var pico = Sim(); + pico.RunUntilOutput(pico.Uart0, "ECHO", timeoutMs: 20_000).Should().BeTrue(); + pico.Uart0.Should().Contain("ECHO"); +} + +[Test] +public void Echoes_a_byte() +{ + using var pico = Sim(); + pico.RunUntilOutput(pico.Uart0, "ECHO", timeoutMs: 20_000); + var before = pico.Uart0.ByteCount; + + pico.Uart0.InjectByte(0x41); // 'A' — drive the device's RX line + pico.RunUntilOutput(pico.Uart0, _ => pico.Uart0.ByteCount > before, timeoutMs: 5_000) + .Should().BeTrue("the firmware should echo the injected byte"); + + pico.Uart0.Bytes[^1].Should().Be(0x41); +} +``` + +`RunUntilOutput` runs in batches until the text appears (or a predicate over the captured +text/state is true), or the timeout elapses — returning `bool`. UART probe surface: +`.Text`, `.Bytes`, `.ByteCount`, `.Contain(...)`, `.InjectByte(...)`. + +### Pass/fail firmware and crash detection + +If your firmware prints a result and you want a single bounded check that never hangs, use +`RunUntilHalt` — it returns *why* it stopped: + +```csharp +var result = pico.RunUntilHalt(pico.Uart0, "PASS", maxInstructions: 5_000_000); + +result.Succeeded.Should().BeTrue($"firmware halted with {result.Outcome}"); // PredicateMet / LockedUp / BudgetReached +pico.Cpu.Should().NotHaveFaulted(); +``` + +See [Firmware testing with the TestKit](testkit.md) for the full assertion set +(`NotBeLockedUp`, `BeInThreadMode`, `HaveExecutedAtMost`, …). + +## Compiling firmware on the fly (with caching) + +If your suite *compiles* firmware (a compiler's own tests, like PyMCU), build each program +**once per session** and cache it — compilation, not emulation, is the slow part. A small +helper does the job: + +```csharp +public static class Firmware +{ + private static readonly ConcurrentDictionary> Cache = new(); + private static readonly SemaphoreSlim Gate = new(Math.Clamp(Environment.ProcessorCount, 2, 8)); + + public static byte[] Build(string name) => + Cache.GetOrAdd(name, _ => new Lazy(() => + { + Gate.Wait(); // bound parallel compiler invocations + try { return Compile(name); } // shell out to your build tool → return the .bin + finally { Gate.Release(); } + })).Value; +} +``` + +- `Lazy` ensures each program compiles exactly once even under parallel test runs. +- The `SemaphoreSlim` keeps a fixture-heavy suite from spawning one compiler per core. +- Call it from `[OneTimeSetUp]`, never per test. + +## Run it in GitHub Actions + +No special setup — it's just `dotnet test`: + +```yaml +name: Integration tests +on: [push, pull_request] +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-dotnet@v4 + with: { dotnet-version: '10.0.x' } + - run: dotnet test -c Release +``` + +Because runs are deterministic and bounded, the job is stable: it won't flake on timing and +won't hang on broken firmware. + ## Headless runner CLI -For pipelines that just need an exit code (no C# harness), use the `rp2040sharp` runner: +For pipelines that don't host C# — e.g. a build matrix that only needs an exit code — use +the `rp2040sharp` runner instead: ```bash dotnet run --project src/RP2040Sharp.Runner -c Release -- \ firmware.uf2 --expect-text "PASS" --channel uart --max-instructions 5000000 ``` -Exit codes: - -| Code | Meaning | +| Exit | Meaning | |---|---| | `0` | expected text found | -| `1` | text not found within the instruction budget | -| `2` | the firmware crashed (CPU lockup) | -| `64` | usage error | -| `66` | image file not found | - -Options: +| `1` | text not found within the budget | +| `2` | firmware crashed (CPU lockup) | +| `64` / `66` | usage error / image not found | | Option | Default | Description | |---|---|---| | `--expect-text ` | — | Pass only if `` appears in serial output | | `--channel uart\|usb` | `uart` | Serial channel to watch | | `--max-instructions ` | `500000000` | Hard execution budget | -| `--quiet` | off | Do not echo serial output to stdout | +| `--quiet` | off | Don't echo serial output to stdout | -Serial output goes to **stdout**; the run summary goes to **stderr**. +Serial output goes to **stdout**; the run summary to **stderr**. diff --git a/docs/index.md b/docs/index.md index f50fb8f..fd0d9af 100644 --- a/docs/index.md +++ b/docs/index.md @@ -5,25 +5,31 @@ microcontroller, written entirely in modern **C# (.NET 10)**. It runs real, unmo RP2040 firmware — including **MicroPython** — and reaches an interactive REPL in ~3–4 s of simulated time (~460 MIPS on Apple Silicon). -It is a C# port and re-imagination of [rp2040js](https://github.com/wokwi/rp2040js) by -Uri Shaked, built for the .NET ecosystem with a focus on speed, type safety, and use as a -**deterministic firmware testkit** — for example, validating the output of the -[PyMCU](https://docs.pymcu.org) compiler in CI. +Its headline use case is **firmware integration testing in CI/CD**: run your real +firmware headlessly and assert on what it actually does — toggles a pin, prints over UART, +echoes bytes — with runs that are **deterministic** and **never hang**. It's how the +[PyMCU](https://docs.pymcu.org) compiler validates its output on every push. It is a C# +port and re-imagination of [rp2040js](https://github.com/wokwi/rp2040js) by Uri Shaked. ```bash -dotnet add package RP2040Sharp -dotnet add package RP2040Sharp.TestKit # fluent harness for firmware tests +dotnet add package RP2040Sharp.TestKit # fluent firmware-testing harness ``` ```csharp -using RP2040.Peripherals; +using RP2040.TestKit.Boards; // PicoSimulation +using RP2040.TestKit.Extensions; // .Should() for pins, UART, CPU -var machine = new RP2040Machine(); -machine.LoadFlash(File.ReadAllBytes("firmware.bin")); -machine.Uart0.OnByteTransmit += b => Console.Write((char)b); -machine.Run(125_000); // 1 ms at 125 MHz +using var pico = new PicoSimulation(withUsbCdc: false); +pico.LoadFlash(File.ReadAllBytes("firmware.bin")); + +// Run real firmware and assert on its behavior — bounded, deterministic, headless. +pico.RunUntilOutput(pico.Uart0, "ready", timeoutMs: 5_000).Should().BeTrue(); +pico.Gpio[25].Should().BeHigh(); +pico.Cpu.Should().NotHaveFaulted(); ``` +→ Start with the **[firmware integration testing guide](guides/ci-validation.md)**. + --- ## Why RP2040Sharp @@ -31,9 +37,12 @@ machine.Run(125_000); // 1 ms at 125 MHz ::::{grid} 1 2 2 3 :gutter: 3 -:::{grid-item-card} Real firmware, unmodified -Boots the real RP2040 B1 BootROM and runs stock MicroPython/CircuitPython UF2 images — -no patches, no shims. +:::{grid-item-card} Firmware testing in CI +:link: guides/ci-validation +:link-type: doc + +A fluent TestKit to drive real firmware and assert on GPIO, UART, and CPU state — +`dotnet test`, no hardware. ::: :::{grid-item-card} Deterministic by design @@ -46,6 +55,11 @@ Bounded execution: wedged or crashed firmware fails a test with a diagnostic rea instead of stalling the build. ::: +:::{grid-item-card} Real firmware, unmodified +Boots the real RP2040 B1 BootROM and runs stock MicroPython/CircuitPython UF2 images — +no patches, no shims. +::: + :::{grid-item-card} GDB debugging Attach `arm-none-eabi-gdb` over `target remote :3333` — registers, memory, single-step, breakpoints.