feat: gVisor DST integration with syscall-level fault injection

.gitignore

@@ -39,3 +39,4 @@ bloodhound-verify/
 *.raw
 *.img
 crates/*/target/
+gvisor/

CLAUDE.md

@@ -304,9 +304,11 @@ ADRs in `docs/adr/` are **living decision logs** that track architectural decisi
 | `002-actor-based-design.md` | Actor hierarchy, message types, coordination changes |
 | `003-qemu-hypervisor-integration.md` | QEMU patches, bloodhound-ctrl protocol changes |
 | `004-fault-injection-strategy.md` | Fault types, BUGGIFY usage, probability handling |
-| `005-execution-modes.md` | Harness vs async-VM mode, VmTrait changes |
+| `005-execution-modes.md` | Harness vs async-VM vs gVisor DST mode changes |
 | `006-property-checking-system.md` | Property checks, triggers, StateQueryClient, PropertyExecutor |
 | `007-container-translator-caching.md` | Container auto-translation, ImageCache, digest-based caching |
+| `008-oci-image-integration.md` | OCI image support, container image handling |
+| `009-gvisor-dst-integration.md` | gVisor fork, syscall faults, VirtualClocks, FD filtering |
 
 ### Creating a New ADR
 
@@ -316,13 +318,33 @@ When making a decision that doesn't fit existing ADRs:
 3. Add initial Decision Log entry
 4. Update `docs/adr/README.md` index
 
-## Two Modes of Operation
+## Three Modes of Operation
 
 1. **Harness Mode** (`--actor-mode`): Simulated VMs, fast, no QEMU required. Use for development and CI.
 
 2. **Async-VM Mode** (`--async-vm --actor-mode`): Actual QEMU VMs with deterministic execution. Use for final validation.
 
-Both modes use the same `SimulationCoordinator` - the only difference is whether VMs are simulated or real.
+3. **gVisor DST Mode**: Modified gVisor runtime with virtual time and syscall-level fault injection. Use for container-native testing without VM overhead.
+
+Harness and Async-VM modes use the same `SimulationCoordinator`. gVisor DST mode uses a separate implementation in the gVisor fork at `./gvisor/`.
+
+### gVisor DST Mode Details
+
+```
+Docker → containerd → runsc-dst (gVisor fork)
+                          ↓
+                    DST Coordinator
+                    ├── VirtualClocks (deterministic time)
+                    ├── FaultInjector (syscall-level faults)
+                    └── SnapshotTree (state management)
+```
+
+Key files in gVisor fork:
+- `pkg/sentry/dst/bloodhound.go` - Core DST coordinator
+- `pkg/sentry/time/virtual_clocks.go` - Virtual time implementation
+- `runsc/config/dst.go` - DST configuration
+- `runsc/boot/dst.go` - DST RPC handlers
+- `runsc/boot/loader.go` - DST initialization (lines 610-650)
 
 ## Common Pitfalls
 
@@ -344,6 +366,81 @@ Watch out for:
 - Not running tests with multiple seeds
 - Not documenting what's tested vs untested
 
+## Systematic Debugging Approach
+
+When debugging complex multi-process systems like gVisor DST integration, follow this systematic approach to avoid circular debugging:
+
+### 1. Add Logging at Boundaries First
+
+Before changing logic, add logging at data flow boundaries:
+```go
+// Log what values are being SET
+log.Warningf("SetProbabilities: DiskWrite=%.4f DiskRead=%.4f", probs.DiskWriteFailure, probs.DiskReadFailure)
+
+// Log what values are being READ/USED
+log.Warningf("ShouldInjectFault: %s prob=%.4f", faultType, probability)
+```
+
+### 2. Trace Data Flow Through Process Boundaries
+
+In multi-process systems, config values often get lost at process boundaries:
+```
+Parent Process (runsc) → [flags/config] → Child Process (sandbox/boot)
+```
+
+Key questions:
+- Is the value being set in the config struct?
+- Is ToFlags() propagating the value to child processes?
+- Is the child parsing the flag correctly?
+
+### 3. Binary Search the Pipeline
+
+When a value shows 0 but should be non-zero:
+1. Log at the SOURCE (config parsing)
+2. Log at the DESTINATION (where value is used)
+3. If source is correct but destination is wrong, binary search the middle
+
+### 4. Document Each Finding
+
+Track what you learn:
+```
+Config shows: DiskRead=0.2 ✓
+SetProbabilities receives: DiskRead=0.0 ✗
+→ Problem is between config and SetProbabilities
+→ Check ToFlags() propagation
+```
+
+### 5. Example: gVisor DST Flag Propagation Bug
+
+**Symptom**: `DiskReadFailure` was 0 despite config having 0.2
+
+**Debug process**:
+1. Added logging to `SetProbabilities` → saw `DiskRead=0.0`
+2. Verified daemon.json had `--dst-fault-disk-read=0.2` ✓
+3. Checked `config/dst.go` flag registration ✓
+4. Checked `config/flags.go` `ToFlags()` → **missing `FaultDiskRead` propagation**
+
+**Root cause**: `ToFlags()` propagated `FaultDiskWrite` to child processes but not `FaultDiskRead`
+
+**Fix**: Added missing line in `ToFlags()`:
+```go
+if c.DST.FaultDiskRead > 0 {
+    rv = append(rv, fmt.Sprintf("--dst-fault-disk-read=%f", c.DST.FaultDiskRead))
+}
+```
+
+### 6. gVisor-Specific: Process Hierarchy
+
+```
+docker run → containerd → runsc create/start
+                              ↓
+                         gofer process (file system)
+                              ↓
+                         sandbox process (kernel) ← DST runs here
+```
+
+Config must flow through ALL of these. Check `ToFlags()` for any new config fields.
+
 ## Platform Notes
 
 ### Linux (Primary Platform)
@@ -378,4 +475,4 @@ The goal is to build reliable software, not to impress with clever code.
 
 ---
 
-*Last updated: v0.2.0*
+*Last updated: v0.3.0 - Added gVisor DST mode and systematic debugging guide*

Cargo.toml

@@ -63,6 +63,10 @@ rust-embed = "8.0"
 mime_guess = "2.0"
 futures = "0.3"
 
+# Firecracker/Cloud Hypervisor REST API client
+hyper = { version = "0.14", features = ["client", "http1", "stream"] }
+hyperlocal = "0.8"
+
 [dev-dependencies]
 assert_cmd = "2.0"
 predicates = "3.0"