diff --git a/.claude/commands/model-zoo-example-akida2.md b/.claude/commands/model-zoo-example-akida2.md new file mode 100644 index 0000000..532c092 --- /dev/null +++ b/.claude/commands/model-zoo-example-akida2.md @@ -0,0 +1,429 @@ +--- +description: Generate a complete brainchip_devhub Akida 2 model zoo example from a source example +--- + +# Akida 2 Model Zoo Example Generator + +Generate a self-contained **Akida 2** model zoo example in `brainchip_devhub`, ported from a +source example. This skill targets Akida 2 only; for Akida 1 use the separate +`model-zoo-example` skill. + +The canonical reference is `akida2/model_zoo/vww/` — every generated file follows the +structure of its VWW counterpart. Read the reference files (Step 2) and mirror them. + +## Usage + +``` +/model-zoo-example-akida2 +``` + +- `` — subdirectory name under `akida_models/scripts/` (e.g. `kws`, `mnist`, `face`). + +Parse this from `$ARGUMENTS`. Set: +- `NAME` = the example name +- `SOURCE_DIR` = `akida_models/scripts//` +- `TARGET_DIR` = `brainchip_devhub/akida2/model_zoo//` + +Resolve the `brainchip_devhub` and `akida_models` repo locations before doing anything else, +in this order: +1. Environment variables `BRAINCHIP_DEVHUB` / `AKIDA_MODELS`, if set. +2. Paths passed as `--devhub ` / `--akida-models ` in `$ARGUMENTS`. +3. Sibling / ancestor directories of the current working directory (if invoked inside a + `brainchip_devhub` checkout, look for a sibling `akida_models`). + +If a repo cannot be found, stop and ask rather than guessing. Do not hardcode any absolute +user path. + +> **Prerequisite:** `akida2/model_zoo/vww/` must exist in the checkout as the reference +> example. If it is not present, stop and confirm with the user — do not generate against a +> missing reference. + +--- + +## Step 1 — Read source scripts + +Read every `.py` and `.sh` file under `SOURCE_DIR = akida_models/scripts//`. For each +file, identify: + +- **Model file**: the script that builds and returns the model. Note where the model-building + code is and what its builder function is called, and whether it builds from scratch or from + pretrained weights (this only affects the `_untrained` save-path suffix). You do **not** need + to dissect the architecture (pretrained URL, `include_top`, custom head layers, etc.) — in + 3a this file's model definition is lifted essentially verbatim, not reconstructed. The source + is authoritative for the architecture. + +- **Data file**: the script that loads the dataset. Record the dataset name, directory + structure (does it have `train/`+`val/` subdirs, a TFDS dataset, or something else?), + input resolution, any augmentation applied, and whether data is uint8 or float. + +- **Training file**: the script that trains the model. Record the loss function, optimizer, + number of epochs, learning rate schedule, and any callbacks. + +- **Pipeline structure**: read the source `.sh` file as the authoritative pipeline + reference. Record: the number of training/fine-tuning epochs, the LR used, whether the + training action is full training from scratch or fine-tuning from a pretrained starting + point, any multi-phase training. The reference `akida2/model_zoo/vww/` provides structural + patterns only; all epoch counts, LR values, and training modes come from the source. + +- **Quantization**: look in `.sh` files for the quantization command for reference. Note the + bit-widths the source uses. For **Akida 2** targets this skill quantizes with `quantizeml` + (not `cnn2snn quantize`): an 8-bit `i8/w8/a8` variant (no QAT) and a 4-bit `i8/w4/a4` + variant (QAT only). The input layer weights are always 8-bit. See 3f for the exact pipeline + — the source is reference-only for quantization; the v2 scheme above is fixed. + +- **Any notebooks** in `SOURCE_DIR` — read their content if they exist. + +--- + +## Step 2 — Read the reference example + +Read every file in `akida2/model_zoo/vww/` — it is the pattern each generated file follows: + +- `vww_model.py`, `vww_data.py`, `vww_train.py`, `vww_eval.py`, `vww_benchmark.py` +- `vww_train.sh`, `update_readme.py`, `colab_setup.py` +- `vww_notebook_training.ipynb`, `vww_notebook_benchmark.ipynb`, `vww_notebook.py` +- `docs/README.md.template`, `docs/metrics.json` + +Also read the top-level `pyproject.toml` to see which packages are already available +(`quantizeml`, `cnn2snn`, `akida_models`, …) versus needing a note. + +Akida 2 status note: the reference hardware is an **FPGA at 25 MHz**; AKD2500 production +silicon does not exist yet. The software pipeline (train → quantize → convert → +software-backend eval → sparsity) is fully real, but the hardware benchmark is FPGA-based and +**latency-only** (the power path is still under development). Do not fabricate power numbers +or a production clock. + +--- + +## Step 3 — Generate target files + +Create `TARGET_DIR` and the files below. For each, the VWW file is the direct structural +template — preserve every pattern and only substitute `NAME`-specific content. + +### 3a. `_model.py` + +The model definition is **ported from the source example, not designed or derived**. The +source normally includes the model-building code — lift it rather than reasoning about the +architecture. + +- Locate the model-definition code in the source (the function/section that builds and + returns the Keras model) and copy it into `_model.py` essentially verbatim. +- Adapt only the mechanical wrappers, never the architecture: + - imports, + - the builder function name → `build__model(...)`, + - the `set_akida_version(AkidaVersion.v2)` context, + - a `-s/--savepath` CLI (default `./models/_[_untrained].h5`; use + the `_untrained` suffix if the source builds from scratch, omit it if the source starts + from pretrained weights) and `model.save(..., include_optimizer=False)`, + - `--seed` + `set_random_seed(args.seed)` if the source doesn't already seed. +- **Preserve whatever the source does** — pretrained backbone vs. `include_top`, custom head, + `input_scaling`, `rescale`, layer names, input dtype. Do not add, remove, or "improve" + layers. The source is authoritative. Input handling (e.g. uint8 input with on-graph + scaling) is carried over exactly as the source has it. +- The `akida_models` factories are version-aware inside the `set_akida_version(AkidaVersion.v2)` + context, so that context is the whole version switch — there is no per-layer + v2-compatibility work. + +`akida2/model_zoo/vww/vww_model.py` shows what a lifted-and-adapted result looks like, but the +source example — not this reference — dictates the architecture. + +### 3b. `_data.py` + +Follow the reference `vww_data.py` structure. Key rules: +- Expose exactly two public functions: `get_data(...)` and `get_samples(data_path, + input_shape, num_samples=1024)`. `get_samples` **always** returns a `np.ndarray` of + `dtype=uint8` (required by the benchmark utilities), regardless of dataset type. +- `get_data`'s **signature and return arity follow the dataset**, not a fixed shape: + - Directory dataset → `get_data(data_path, input_shape, batch_size, seed=42)` returning + `(train_dataset, val_dataset)`. + - TFDS dataset → `get_data(data_path, input_shape, batch_size, dtype=tf.uint8, seed=42)` + returning `(train_dataset, val_dataset, test_dataset)`. + Match whichever the source/dataset naturally provides, and make sure `_train.py` / + `_eval.py` unpack the matching number of values. Include a `seed=` parameter and call + `set_random_seed(seed)`. +- Adapt the internals to the dataset **type** discovered in Step 1. The two real cases: + - **Directory dataset** (`train/`+`val/` subdirs): `ImageDataGenerator` + + `flow_from_directory` with `class_mode='sparse'`. + - **TFDS dataset**: `tfds.load(name, split=[...], as_supervised=True, data_dir=data_path)`. + Split strings define the partition (e.g. `['train[:80%]','train[80%:90%]','train[90%:]']`); + map a `resize_and_cast` that resizes to `input_shape` and casts to `dtype` (uint8 by + default); augment only the train split; call `tfds.disable_progress_bar()`. If the source + overrides the dataset download URL to a BrainChip mirror + (`..._dataset_builder._URL = "https://data.brainchip.com/dataset-mirror/..."`), replicate + that override. + - (No `.npz` branch — if a genuinely different source format appears, adapt from the closest + reference.) +- Preserve augmentation from the source where it applies (spatial augmentation for images; + none for spectrograms/1D signals unless the source does it). In the TFDS case, augment only + the training dataset and cast back to the model's input dtype afterward. +- `get_samples()` mechanics follow the dataset type: directory examples glob/sample image + files; TFDS examples do `tfds.load(split=...).take(num_samples)`, resize, and stack. Both + return uint8 arrays. +- Default `data_path` should be `./data/`. + +### 3c. `_train.py` + +Follow the reference `vww_train.py` structure. Key rules: +- The training function is named `train_(model, train_ds, val_ds, epochs, + learning_rate, regularization=None, seed=42)`. +- Use `SparseCategoricalCrossentropy(from_logits=True)` for sparse integer labels. Use + `CategoricalCrossentropy` only if the source uses one-hot labels. +- Optimizer is `Adam` (legacy: `tf_keras.optimizers.legacy.Adam`). +- **LR schedule: match the source — do not mandate one.** (The v1 references use different + schedules — `CosineDecay` with warmup, or exponential decay via a `get_custom_scheduler()` + helper — so there is no single correct one.) Pick whichever fits the source's intent and + wrap it in a scheduler callback. Do NOT invent a step-decay schedule. +- **Do not add a `RestoreBest` callback** — the references don't use one (one even has it as a + dead import; don't propagate that). +- The optional activity-regularization path (`-reg` adding `L1L2` on `ReLU` layers to increase + sparsity) is real — carry it over as in the reference. +- Carry over `tf.config.experimental.enable_op_determinism()` at import time if the + source/reference does (small throughput cost, aids reproducibility). +- CLI: `-l`, `-s`, `-d`, `-b`, `-e`, `-lr`, `-reg`, `--seed`, with epoch/LR defaults from the + source. +- Unpack `get_data(...)` with the arity the data module provides (2 or 3 values — see 3b). +- `_train.py` is also used for the 4-bit QAT fine-tuning step (see 3f); the training + loop is the same, it just loads a quantized model to fine-tune. `load_quantized_model` loads + both float and quantizeml-quantized `.h5` files. + +### 3d. `_eval.py` + +Copy the reference `vww_eval.py` and substitute only: +- `vww_data` → `_data`. +- The default `--data` path. +- The `evaluate_akida_model` function is identical; copy verbatim. +- `--save-metrics` uses the **variant-keyed** scheme — copy it verbatim from + `akida2/model_zoo/vww/vww_eval.py`. It writes `float_acc`/`params` for the float model and + `_quant_acc` (from `.h5`) / `_akida_acc` (from `.fbz`) per stored variant, + with the variant inferred from the filename (`i8_w8_a8` → `w8a8`; `i8_w4_a4` → `w4a4_qat`). + +### 3e. `_benchmark.py` + +Copy the reference `vww_benchmark.py` and substitute only: +- `from vww_data import get_samples` → `from _data import get_samples`. +- The default `--data` path. +- All `brainchip_utils` imports, benchmark calls, and plotting calls are identical. + +Akida 2 benchmark specifics (already in the reference — preserve them): +- `MEASURED_CLOCK = 25e6` (FPGA). +- A **projected latency** at a higher clock: `projected_ms = mean_inf_clk / PROJECTED_CLOCK * + 1000` (cycle count is clock-independent, so this is exact). `PROJECTED_CLOCK` is a + **provisional placeholder** (reference uses 100 MHz with a `# TODO: confirm target clock`) — + do not present it as final. +- **Latency-only**: no power measurement (the FPGA power path is WIP); no power columns/keys. +- `--save-metrics` writes variant-keyed metrics: `_sparsity`, and per map-mode + `__{nps,passes,cycles,latency_ms,projected_ms}`, with `` ∈ + {`w8a8`, `w4a4_qat`} inferred from the `.fbz` filename. + +### 3f. `_train.sh` + +The pipeline. Follow `akida2/model_zoo/vww/vww_train.sh`. Epoch counts and LR values come +from the source (read in Step 1); the reference provides structural shape only. + +Data-path forwarding at the top: +```bash +DATADIR="${1:-}" +DATA_ARG=${DATADIR:+-d "$DATADIR"} +``` + +Quantization uses **`quantizeml`** (not `cnn2snn quantize`). The **input layer weights are +always 8-bit** (`-i 8`), regardless of the target precision. Two quantized variants are +produced: + +| Variant | quantize command | QAT? | +|---|---|---| +| 8-bit | `quantizeml quantize -m .h5 -i 8 -w 8 -a 8 -s _i8_w8_a8.h5` | No — 8-bit PTQ is accurate enough | +| 4-bit | `quantizeml quantize -m .h5 -i 8 -w 4 -a 4 -s _i8_w4_a4_pretmp.h5`, then QAT-fine-tune | Yes — **QAT only** | + +Pipeline order: +1. Build untrained/starting model: `python _model.py -s models/_[_untrained].h5` +2. Float-train → `models/_.h5`, then `python _eval.py -l ....h5 $DATA_ARG` +3. **8-bit**: `quantizeml quantize ... -i 8 -w 8 -a 8 -s ...i8_w8_a8.h5` → eval `.h5` → + `cnn2snn convert -m ...i8_w8_a8.h5` → eval `.fbz` → `python _benchmark.py -l ...i8_w8_a8.fbz $DATA_ARG` +4. **4-bit (QAT only)**: `quantizeml quantize ... -i 8 -w 4 -a 4 -s ...i8_w4_a4_pretmp.h5` → + `python _train.py -l ...i8_w4_a4_pretmp.h5 -s ...i8_w4_a4_qat.h5 -e -lr $DATA_ARG` + → eval `.h5` → `cnn2snn convert -m ...i8_w4_a4_qat.h5` → eval `.fbz` → benchmark `.fbz` → + `rm -f models/__i8_w4_a4_pretmp.h5` + +Key points: +- **4-bit PTQ accuracy is poor** (as seen on Akida 1). Do not store, evaluate, convert, or + record metrics for the 4-bit PTQ model — it is a throwaway on the way to QAT. Write it to a + clearly-temporary `_pretmp.h5` name and `rm` it at the end. +- `quantizeml` has **no `convert` subcommand**. Conversion to `.fbz` is always + `cnn2snn convert -m .h5` (it accepts quantizeml-quantized models). Output filename is + `.fbz`. +- `quantizeml quantize` calibrates during quantization; with no `-sa samples.npz` it uses + random calibration samples. **Note:** quantizeml warns that random calibration is inaccurate + for per-axis activation quantization — for accurate results pass real calibration samples + (`-sa`) or set `QuantizationParams.per_tensor_activations=True`. Flag this as a per-example + decision. + +Model naming (keep bit-widths explicit; 8-bit needs no suffix, 4-bit always carries `_qat`): +- Float: `__untrained.h5` → `_.h5` +- 8-bit: `__i8_w8_a8.h5` → `__i8_w8_a8.fbz` +- 4-bit throwaway: `__i8_w4_a4_pretmp.h5` (deleted at pipeline end) +- 4-bit QAT: `__i8_w4_a4_qat.h5` → `__i8_w4_a4_qat.fbz` + +Float-training and QAT epoch/LR values are provisional if the source doesn't specify them — +mark them confirm-on-first-run. + +### 3g. `update_readme.py` + +Copy verbatim from `akida2/model_zoo/vww/update_readme.py`. No example-specific content. + +### 3h. `docs/README.md.template` + +Follow `akida2/model_zoo/vww/docs/README.md.template` exactly (same sections, same order), +rewriting content for this dataset/model. Sections in order: + +1. Logo image line — copy verbatim. +2. `# ` — readable title. +3. `## Model Card` — the model card and benchmark tables: + - **Model card**: rows-per-variant. One row per stored quantized variant (8-bit, + 4-bit QAT), columns `Variant | Weights/Acts | QAT | Quantized acc. | Akida acc. | Sparsity`. + The 8-bit row shows `-` in the QAT column; the 4-bit row shows `yes`. Float accuracy + + params are stated once above the table. + - **Benchmark table**: the two stored variants × {Minimal, AllNps} = 4 rows, latency-only, + with `Latency @ 25 MHz` and `Projected @ MHz (provisional)` columns. No power columns. + - A short architecture description. +4. `## Requirements` — copy from the reference; add example-specific deps if any. +5. `## Dataset` — describe the dataset from Step 1. +6. `## Dataset setup` — download/obtain instructions; include a URL + wget/extract if known, + else a ``. +7. `## Pipeline` — copy the pipeline table from the reference, adapting the quantization rows + to this example. +8. `## Usage` → two subsections: + - `### Notebook` — link the two notebooks (`_notebook_training.ipynb` and + `_notebook_benchmark.ipynb`) with a short description of each, and place the + **Colab badge** immediately after the training-notebook description (see 3m). Add the + hardware-benchmark note after the benchmark-notebook description. + - `### Script` — the `bash _train.sh [DATADIR]` intro, adapting filenames. +9. `## Contributing and Maintenance` — copy from the reference, substituting `` and the + per-variant `--save-metrics` command list (float + 8-bit + 4-bit-QAT). + +### 3i. `docs/metrics.json` + +Every `{key}` in the template must appear here or `update_readme.py` crashes. Take the key +set from `akida2/model_zoo/vww/docs/metrics.json` and adapt to any keys you introduced. The +reliable method: regex-extract all `{...}` placeholders from the template you wrote and emit +exactly that set, each `"TBD"`. + +**Verify the three-way bijection**: the template placeholders, the metrics.json keys, and the +union of keys the eval + benchmark `--save-metrics` blocks write must be **equal** — no extras +on any side. Check programmatically (regex the template, compare to the script-written set), +not by eye. Then run `update_readme.py` — it must render with no `KeyError`. + +### 3j. `README.md` + +Generate by running `update_readme.py`. The result has `"TBD"` wherever metrics go — correct +and expected until training runs complete. + +### 3l. `colab_setup.py` + +A module with a single `setup()` function that makes the "Open in Colab" badge work; it is a +no-op on local runs. Follow `akida2/model_zoo/vww/colab_setup.py`: +- Module constants: `REPO_URL`, `REPO_DIR = 'brainchip_devhub'`, + `EXAMPLE_SUBDIR = 'akida2/model_zoo/'`, and (for a directory dataset) `DATA_DIR` + + `DATASET_URL`. +- `setup()` returns immediately with a friendly message if `'google.colab' not in sys.modules` + — this is what makes it safe to leave the notebook's first cell in permanently. +- On Colab: clone the repo with `GIT_LFS_SKIP_SMUDGE=1` (pretrained weights aren't needed for + the default train-from-scratch path), `os.chdir` into `EXAMPLE_SUBDIR`, put the repo root and + the example dir on `sys.path`, then `pip install -q akida_models==1.14.0 tf_keras quantizeml` + (add any extra imports the notebook/modules need — check by grep; e.g. `pooch` for some + examples, not VWW). +- **Dataset handling depends on the data type** (from Step 1): + - **Directory dataset** (VWW): include a `wget` + `tar -xzf` block guarded by + `if not os.path.exists(DATA_DIR)` (needs a public `DATASET_URL`). + - **TFDS dataset**: omit the download block entirely — the notebook's data cell auto-downloads + via `tfds.load`. +- End with a note to restart the runtime if TensorFlow was just (re)installed. + +### 3m. Notebooks + +Generate two notebooks plus a Jupytext mirror, following the `akida2/model_zoo/vww/` +notebooks: + +- **`_notebook_training.ipynb`** — the training walkthrough. Cell structure: + 1. Markdown: logo (absolute `raw.githubusercontent.com` URL) + title + overview. + 2. Code: the Colab-only setup cell — `if 'google.colab' in sys.modules:` → `wget` + `colab_setup.py` (from `akida2/model_zoo//`) if absent → `import colab_setup; + colab_setup.setup()`. + 3. Setup (imports, `DATA_PATH`, `MODELS_DIR`, `SEED`, `RUN_FLOAT_TRAINING = True`, + `enable_op_determinism()`). + 4. Dataset (`get_data`, unpacking the right arity), Model (`build__model`), Float + training (train from scratch), evaluate float. + 5. **Quantization with `quantizeml`** — two variants: 8-bit + `QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)` → `quantize` + → eval; and 4-bit `(…weight_bits=4, activation_bits=4)` → `quantize` → **QAT fine-tune + via `train_`** → eval. (Do not use `cnn2snn.quantize` — that is Akida 1.) + 6. Conversion: `cnn2snn.convert` for both variants → `.fbz`. + 7. Akida software-backend eval for both variants; activation sparsity for both. + 8. Summary table (float vs 8-bit vs 4-bit-QAT). + Train from scratch by default (`RUN_FLOAT_TRAINING = True`); do not add a pretrained-load + fast path unless committed pretrained models exist. +- **`_notebook_benchmark.ipynb`** — accuracy + hardware benchmark. Device-guarded + (`get_akida_device` → `None` when absent → skip latency, still compute sparsity on the + software backend). Latency-only, `MEASURED_CLOCK = 25e6` + provisional `PROJECTED_CLOCK`. + This notebook is **not** Colab-fied (needs hardware) — no setup cell, no badge. +- **`_notebook.py`** — the Jupytext `py:percent` mirror of the training notebook. + Generate with `jupytext --to py:percent --opt comment_magics=false` (so the `!wget` magic + stays uncommented), then remove the `comment_magics: false` line from the header for + consistency. Verify it round-trips back to the `.ipynb` with no cell mismatches. + +Validate every generated notebook with `nbformat.validate` and give each cell an `id`. + +### 3n. Directory stubs + +Create `models/` and `data/` each containing a `.gitignore` that ignores everything except +itself (copy verbatim from the reference — it is not a `.gitkeep`): +``` +# Git to Ignore everything in this directory +* +# Except this .gitignore file +!.gitignore +``` + +--- + +## Step 4 — Report + +Summarise: +1. Files created (paths). +2. TODOs left for the user (dataset URL/hash, the provisional epoch/LR values, the + `quantizeml` calibration-samples decision, the provisional projected clock). +3. Verification commands: + ```bash + cd TARGET_DIR + python -c "import ast; [ast.parse(open(f).read()) for f in ['_model.py','_data.py','_train.py','_eval.py','_benchmark.py']]" + bash -n _train.sh + python update_readme.py # must render with no KeyError + python -c "import nbformat; [nbformat.validate(nbformat.read(f, as_version=4)) for f in ['_notebook_training.ipynb','_notebook_benchmark.ipynb']]" + ``` + Also confirm the `--save-metrics` key set matches the template exactly (extract `{...}` + placeholders from the template, compare against the union of keys the eval + benchmark + scripts write — the two sets must be equal). +4. A reminder that the pipeline was never executed — all accuracy/latency numbers in + `docs/metrics.json` and the README are `"TBD"` until the user runs the real pipeline and + the `--save-metrics` maintenance commands. + +--- + +## Key invariants + +- `get_samples()` must always return `np.ndarray` of `dtype=uint8` — required by + `per_layer_benchmark` and `full_model_benchmark`. +- The model definition is **ported from the source example, not designed or reconstructed**. + Preserve the source's architecture exactly (backbone, head, `input_scaling`, layer names, + input dtype); adapt only mechanical wrappers. The source is authoritative. +- The **input layer weights are always 8-bit** (`-i 8`), regardless of the rest of the + model's target precision. +- Quantization is `quantizeml quantize`: an 8-bit `i8/w8/a8` variant (no QAT — PTQ is accurate + enough) and a 4-bit `i8/w4/a4` variant (QAT only — 4-bit PTQ accuracy is poor, so the PTQ + model is a throwaway, never stored/evaluated). Both convert to `.fbz` with the same + `cnn2snn convert` — there is no `quantizeml convert`. +- Do not fabricate Akida 2 hardware numbers: AKD2500 production silicon does not exist yet; + the reference platform is a 25 MHz FPGA, benchmarking is latency-only (no power) with a + clearly-provisional projected clock. The software pipeline is fully real regardless. +- The metrics.json / template / `--save-metrics` key sets must be in exact three-way bijection + (variant-prefixed keys). Verify programmatically, not by eye. diff --git a/akida2/model_zoo/vww/README.md b/akida2/model_zoo/vww/README.md new file mode 100644 index 0000000..e245f51 --- /dev/null +++ b/akida2/model_zoo/vww/README.md @@ -0,0 +1,239 @@ +BrainChip Dev Hub + +# Visual Wake Words (VWW) — Akida 2 + +## Model Card + +Float accuracy: **TBD**  |  Parameters: **TBD** + +The quantized variants below all share the same float backbone. On Akida 2 the +model is quantized with **`quantizeml`**: 8-bit weights and activations need no +quantization-aware training (QAT), while a lower-precision 4-bit variant (4-bit +weights and activations, 8-bit input layer) uses QAT to recover accuracy — 4-bit +PTQ accuracy is poor, so only the QAT result is reported. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VariantWeights / ActsQATQuantized acc.Akida acc.Sparsity
8-bitw8 / a8-TBDTBDTBD
4-bitw4 / a4yesTBDTBDTBD
+ +**Akida 2 hardware benchmark (FPGA @ 25 MHz)** + +Latency is measured on the Akida 2 FPGA reference platform, which runs at +**25 MHz**. A projected latency at a higher target clock is also shown to +indicate expected performance on faster silicon. The cycle count is fixed for a +given model and mapping regardless of clock rate, so the projection is an exact +rescale of the measured cycles. + +> **Note:** the projected clock is **provisional** — it is a placeholder pending +> confirmation of the target Akida 2 silicon clock. Power measurement on the FPGA +> platform is still under development, so only latency is reported at this time. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VariantMappingNPsPassesCyclesLatency @ 25 MHz (ms)Projected @ 100 MHz (ms) (provisional)
8-bitMinimalTBDTBDTBDTBDTBD
AllNPsTBDTBDTBDTBDTBD
4-bit (QAT)MinimalTBDTBDTBDTBDTBD
AllNPsTBDTBDTBDTBDTBD
+ +The model is a standard **AkidaNet** (from `akida_models`) with +width multiplier **alpha = 0.25** and input resolution **96 × 96**, built for +Akida 2. + +## Requirements + +For environment requirements and setup, see the [Requirements](../../../README.md#requirements) +section of the top-level README. + +## Dataset + +Visual Wake Words is a binary image classification benchmark, specifically +designed to target edge deployment on resource-constrained devices. It is +derived from the MS-COCO 2014 dataset. Each image is labelled **person** +or **non-person** based on whether a person occupies at least 2% of the frame. +Images are resized to **96 × 96 RGB**. The dataset contains approximately +115k training images and 8k validation images. + +Reference: Chowdhery et al., *Visual Wake Words Dataset* (2019), +[arXiv:1906.05721](https://arxiv.org/abs/1906.05721). + +## Dataset setup + +The dataset can be downloaded from the SiLabs ML benchmarks mirror: + +```bash +wget https://www.silabs.com/public/files/github/machine_learning/benchmarks/datasets/vw_coco2014_96.tar.gz +tar -xzf vw_coco2014_96.tar.gz +``` + +The scripts default to looking for the data at `./data/vw_coco2014_96`. If you +want to store the dataset on a dedicated data drive, you can pass the path +explicitly to each script (see `--data` / `-d` in the individual scripts). +Alternatively, it may be more convenient to keep the dataset in its preferred +location and create a symbolic link from the default path (one-off step): + +```bash +ln -s /path/to/your/data/vw_coco2014_96 ./data/vw_coco2014_96 +``` + +This way the scripts work out of the box without any extra arguments. + +## Pipeline + +Training produces a float model, then quantizes it with `quantizeml` into +several variants, each converted to Akida format: + +| Stage | Description | +|---|---| +| Full-precision | Float32 training from scratch | +| 8-bit quantization | `quantizeml quantize` to 8-bit weights and activations (8-bit input); no QAT required | +| 4-bit quantization | `quantizeml quantize` to 4-bit weights and activations (8-bit input), with QAT fine-tuning — 4-bit PTQ accuracy is poor so only the QAT model is kept | +| Conversion to Akida | Automated conversion of each quantized model to Akida 2 format with `cnn2snn convert` | + +## Reference Models + +Pretrained models are made available here, within the `pretrained_models/` +folder. However, those are handled using the `git-lfs` package (git large +file storage). For those to be downloaded with the repo, you will need to +set up `git-lfs`. For further instructions, see the +[Trained models](../../../README.md#trained-models) section of the top-level README. + +## Usage + +### Notebook + +Two notebooks are provided that walk through a) preparation of a trained Akida-compatible model and +b) evaluation and benchmarking of that model on Akida. + +[vww_notebook_training.ipynb](vww_notebook_training.ipynb) walks through the +complete training pipeline end-to-end. It is written to expose and explain the Akida-specific +aspects of the workflow: how the model is constructed for Akida 2 compatibility, +what the quantization constraints mean in practice, and what the conversion +step does. Start here if you want to understand *why* the pipeline is structured +the way it is. + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Brainchip-Inc/brainchip_devhub/blob/main/akida2/model_zoo/vww/vww_notebook_training.ipynb) + +[vww_notebook_benchmark.ipynb](vww_notebook_benchmark.ipynb) walks through +evaluation of model accuracy on Akida and, if a hardware device is available, covers benchmarking +of model latency. + +> **Note:** the hardware benchmark section requires a physical Akida 2 FPGA +> platform with a connected board. + +### Script + +For straightforward reproduction of the training and evaluation results, run +the full pipeline in one shot: + +```bash +bash vww_train.sh [DATADIR] +``` + +The optional `DATADIR` argument overrides the default dataset location +(`./data/vw_coco2014_96`). + +## Contributing and Maintenance + +This README is autogenerated from `docs/README.md.template` +so that the accuracy and hardware benchmark values are written directly +by the code (via the `metrics.json` file, also in the docs folder). + +When the associated model or training pipeline is modified to improve +performance, you should rerun the evaluations of the float and quantized +model versions, plus the hardware benchmark, including the +`--save-metrics` argument, and then regenerate the README from the template +using `update_readme.py`: +```bash +# Float model +python vww_eval.py -l pretrained_models/akidanet_vww.h5 --save-metrics + +# 8-bit variant +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w8_a8.h5 --save-metrics +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w8_a8.fbz --save-metrics +python vww_benchmark.py -l pretrained_models/akidanet_vww_i8_w8_a8.fbz --save-metrics + +# 4-bit variant (QAT) +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w4_a4_qat.h5 --save-metrics +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w4_a4_qat.fbz --save-metrics +python vww_benchmark.py -l pretrained_models/akidanet_vww_i8_w4_a4_qat.fbz --save-metrics + +python update_readme.py +``` +Then commit the changed files (template, metrics and updated README). + +Likewise, if you want to edit the contents of this README, you should +not edit it directly, but instead edit `docs/README.md.template` and +then regenerate the README using +``` bash +python update_readme.py +``` diff --git a/akida2/model_zoo/vww/colab_setup.py b/akida2/model_zoo/vww/colab_setup.py new file mode 100644 index 0000000..bfd9248 --- /dev/null +++ b/akida2/model_zoo/vww/colab_setup.py @@ -0,0 +1,75 @@ +"""Colab setup for the VWW (Akida 2) training notebook. + +IF YOU ARE RUNNING THIS LOCALLY: you can ignore this file completely. +It exists solely to make the "Open in Colab" badge work, and does nothing +on a normal local run. It is not part of the VWW model/training code +(see vww_data.py, vww_model.py, vww_train.py for that). + +If you ARE on Colab, this is the file that gets you running: + - Clones this repo (skipping Git LFS smudge, since pretrained weights + aren't needed for the default training path) + - Points Python at the right folders (repo root for brainchip_utils, + this example's folder for vww_data / vww_model / vww_train) + - Installs akida_models, tf_keras, quantizeml + - Downloads and extracts the VWW dataset (derived from MS-COCO 2014, + hosted by Silicon Labs) if it isn't already present + +Called from the notebook's first cell like: + import colab_setup + colab_setup.setup() +""" +import os +import subprocess +import sys + +REPO_URL = 'https://github.com/Brainchip-Inc/brainchip_devhub.git' +REPO_DIR = 'brainchip_devhub' +EXAMPLE_SUBDIR = 'akida2/model_zoo/vww' + +DATA_DIR = './data/vw_coco2014_96' +DATASET_URL = ('https://www.silabs.com/public/files/github/machine_learning/' + 'benchmarks/datasets/vw_coco2014_96.tar.gz') + + +def _run(cmd): + print(f'$ {cmd}') + subprocess.run(cmd, shell=True, check=True) + + +def setup(): + """Set up a fresh Colab session to run this notebook. No-op if not on Colab.""" + if 'google.colab' not in sys.modules: + print('Not running on Colab \u2014 nothing to do. (This step only matters ' + 'for Colab; local runs already have everything they need.)') + return + + if not os.path.exists(REPO_DIR): + # Skip Git LFS smudge: pretrained weights aren't needed for the default + # (RUN_FLOAT_TRAINING = True) path, so avoid downloading them here. + os.environ['GIT_LFS_SKIP_SMUDGE'] = '1' + _run(f'git clone --depth 1 {REPO_URL} {REPO_DIR}') + + os.chdir(os.path.join(REPO_DIR, EXAMPLE_SUBDIR)) + + # Repo root on sys.path for `brainchip_utils`; example folder for the + # local vww_data / vww_model / vww_train modules imported later in + # the notebook. + repo_root = os.path.abspath(os.path.join(os.getcwd(), '..', '..', '..')) + sys.path.insert(0, repo_root) + sys.path.insert(0, os.getcwd()) + + # Akida 2 quantization uses quantizeml (installed as a dependency of + # akida_models / cnn2snn, but pinned here for a clean Colab environment). + _run('pip install -q akida_models==1.14.0 tf_keras quantizeml') + + if not os.path.exists(DATA_DIR): + os.makedirs('./data', exist_ok=True) + _run(f'wget -q {DATASET_URL}') + _run('tar -xzf vw_coco2014_96.tar.gz -C ./data') + print('Dataset downloaded and extracted to', DATA_DIR) + else: + print('Dataset already present at', DATA_DIR) + + print(f'Colab setup complete. Working directory: {os.getcwd()}') + print('If TensorFlow was just installed/upgraded, restart the runtime ' + '(Runtime > Restart session) and re-run this cell before continuing.') diff --git a/akida2/model_zoo/vww/data/.gitignore b/akida2/model_zoo/vww/data/.gitignore new file mode 100644 index 0000000..647613e --- /dev/null +++ b/akida2/model_zoo/vww/data/.gitignore @@ -0,0 +1,4 @@ +# Git to Ignore everything in this directory +* +# Except this .gitignore file +!.gitignore \ No newline at end of file diff --git a/akida2/model_zoo/vww/docs/README.md.template b/akida2/model_zoo/vww/docs/README.md.template new file mode 100644 index 0000000..8513652 --- /dev/null +++ b/akida2/model_zoo/vww/docs/README.md.template @@ -0,0 +1,239 @@ +BrainChip Dev Hub + +# Visual Wake Words (VWW) — Akida 2 + +## Model Card + +Float accuracy: **{float_acc}**  |  Parameters: **{params}** + +The quantized variants below all share the same float backbone. On Akida 2 the +model is quantized with **`quantizeml`**: 8-bit weights and activations need no +quantization-aware training (QAT), while a lower-precision 4-bit variant (4-bit +weights and activations, 8-bit input layer) uses QAT to recover accuracy — 4-bit +PTQ accuracy is poor, so only the QAT result is reported. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VariantWeights / ActsQATQuantized acc.Akida acc.Sparsity
8-bitw8 / a8-{w8a8_quant_acc}{w8a8_akida_acc}{w8a8_sparsity}
4-bitw4 / a4yes{w4a4_qat_quant_acc}{w4a4_qat_akida_acc}{w4a4_qat_sparsity}
+ +**Akida 2 hardware benchmark (FPGA @ 25 MHz)** + +Latency is measured on the Akida 2 FPGA reference platform, which runs at +**25 MHz**. A projected latency at a higher target clock is also shown to +indicate expected performance on faster silicon. The cycle count is fixed for a +given model and mapping regardless of clock rate, so the projection is an exact +rescale of the measured cycles. + +> **Note:** the projected clock is **provisional** — it is a placeholder pending +> confirmation of the target Akida 2 silicon clock. Power measurement on the FPGA +> platform is still under development, so only latency is reported at this time. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VariantMappingNPsPassesCyclesLatency @ 25 MHz (ms)Projected @ 100 MHz (ms) (provisional)
8-bitMinimal{w8a8_minimal_nps}{w8a8_minimal_passes}{w8a8_minimal_cycles}{w8a8_minimal_latency_ms}{w8a8_minimal_projected_ms}
AllNPs{w8a8_allnps_nps}{w8a8_allnps_passes}{w8a8_allnps_cycles}{w8a8_allnps_latency_ms}{w8a8_allnps_projected_ms}
4-bit (QAT)Minimal{w4a4_qat_minimal_nps}{w4a4_qat_minimal_passes}{w4a4_qat_minimal_cycles}{w4a4_qat_minimal_latency_ms}{w4a4_qat_minimal_projected_ms}
AllNPs{w4a4_qat_allnps_nps}{w4a4_qat_allnps_passes}{w4a4_qat_allnps_cycles}{w4a4_qat_allnps_latency_ms}{w4a4_qat_allnps_projected_ms}
+ +The model is a standard **AkidaNet** (from `akida_models`) with +width multiplier **alpha = 0.25** and input resolution **96 × 96**, built for +Akida 2. + +## Requirements + +For environment requirements and setup, see the [Requirements](../../../README.md#requirements) +section of the top-level README. + +## Dataset + +Visual Wake Words is a binary image classification benchmark, specifically +designed to target edge deployment on resource-constrained devices. It is +derived from the MS-COCO 2014 dataset. Each image is labelled **person** +or **non-person** based on whether a person occupies at least 2% of the frame. +Images are resized to **96 × 96 RGB**. The dataset contains approximately +115k training images and 8k validation images. + +Reference: Chowdhery et al., *Visual Wake Words Dataset* (2019), +[arXiv:1906.05721](https://arxiv.org/abs/1906.05721). + +## Dataset setup + +The dataset can be downloaded from the SiLabs ML benchmarks mirror: + +```bash +wget https://www.silabs.com/public/files/github/machine_learning/benchmarks/datasets/vw_coco2014_96.tar.gz +tar -xzf vw_coco2014_96.tar.gz +``` + +The scripts default to looking for the data at `./data/vw_coco2014_96`. If you +want to store the dataset on a dedicated data drive, you can pass the path +explicitly to each script (see `--data` / `-d` in the individual scripts). +Alternatively, it may be more convenient to keep the dataset in its preferred +location and create a symbolic link from the default path (one-off step): + +```bash +ln -s /path/to/your/data/vw_coco2014_96 ./data/vw_coco2014_96 +``` + +This way the scripts work out of the box without any extra arguments. + +## Pipeline + +Training produces a float model, then quantizes it with `quantizeml` into +several variants, each converted to Akida format: + +| Stage | Description | +|---|---| +| Full-precision | Float32 training from scratch | +| 8-bit quantization | `quantizeml quantize` to 8-bit weights and activations (8-bit input); no QAT required | +| 4-bit quantization | `quantizeml quantize` to 4-bit weights and activations (8-bit input), with QAT fine-tuning — 4-bit PTQ accuracy is poor so only the QAT model is kept | +| Conversion to Akida | Automated conversion of each quantized model to Akida 2 format with `cnn2snn convert` | + +## Reference Models + +Pretrained models are made available here, within the `pretrained_models/` +folder. However, those are handled using the `git-lfs` package (git large +file storage). For those to be downloaded with the repo, you will need to +set up `git-lfs`. For further instructions, see the +[Trained models](../../../README.md#trained-models) section of the top-level README. + +## Usage + +### Notebook + +Two notebooks are provided that walk through a) preparation of a trained Akida-compatible model and +b) evaluation and benchmarking of that model on Akida. + +[vww_notebook_training.ipynb](vww_notebook_training.ipynb) walks through the +complete training pipeline end-to-end. It is written to expose and explain the Akida-specific +aspects of the workflow: how the model is constructed for Akida 2 compatibility, +what the quantization constraints mean in practice, and what the conversion +step does. Start here if you want to understand *why* the pipeline is structured +the way it is. + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Brainchip-Inc/brainchip_devhub/blob/main/akida2/model_zoo/vww/vww_notebook_training.ipynb) + +[vww_notebook_benchmark.ipynb](vww_notebook_benchmark.ipynb) walks through +evaluation of model accuracy on Akida and, if a hardware device is available, covers benchmarking +of model latency. + +> **Note:** the hardware benchmark section requires a physical Akida 2 FPGA +> platform with a connected board. + +### Script + +For straightforward reproduction of the training and evaluation results, run +the full pipeline in one shot: + +```bash +bash vww_train.sh [DATADIR] +``` + +The optional `DATADIR` argument overrides the default dataset location +(`./data/vw_coco2014_96`). + +## Contributing and Maintenance + +This README is autogenerated from `docs/README.md.template` +so that the accuracy and hardware benchmark values are written directly +by the code (via the `metrics.json` file, also in the docs folder). + +When the associated model or training pipeline is modified to improve +performance, you should rerun the evaluations of the float and quantized +model versions, plus the hardware benchmark, including the +`--save-metrics` argument, and then regenerate the README from the template +using `update_readme.py`: +```bash +# Float model +python vww_eval.py -l pretrained_models/akidanet_vww.h5 --save-metrics + +# 8-bit variant +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w8_a8.h5 --save-metrics +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w8_a8.fbz --save-metrics +python vww_benchmark.py -l pretrained_models/akidanet_vww_i8_w8_a8.fbz --save-metrics + +# 4-bit variant (QAT) +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w4_a4_qat.h5 --save-metrics +python vww_eval.py -l pretrained_models/akidanet_vww_i8_w4_a4_qat.fbz --save-metrics +python vww_benchmark.py -l pretrained_models/akidanet_vww_i8_w4_a4_qat.fbz --save-metrics + +python update_readme.py +``` +Then commit the changed files (template, metrics and updated README). + +Likewise, if you want to edit the contents of this README, you should +not edit it directly, but instead edit `docs/README.md.template` and +then regenerate the README using +``` bash +python update_readme.py +``` diff --git a/akida2/model_zoo/vww/docs/metrics.json b/akida2/model_zoo/vww/docs/metrics.json new file mode 100644 index 0000000..d21cbf2 --- /dev/null +++ b/akida2/model_zoo/vww/docs/metrics.json @@ -0,0 +1,30 @@ +{ + "float_acc": "TBD", + "params": "TBD", + "w4a4_qat_akida_acc": "TBD", + "w4a4_qat_allnps_cycles": "TBD", + "w4a4_qat_allnps_latency_ms": "TBD", + "w4a4_qat_allnps_nps": "TBD", + "w4a4_qat_allnps_passes": "TBD", + "w4a4_qat_allnps_projected_ms": "TBD", + "w4a4_qat_minimal_cycles": "TBD", + "w4a4_qat_minimal_latency_ms": "TBD", + "w4a4_qat_minimal_nps": "TBD", + "w4a4_qat_minimal_passes": "TBD", + "w4a4_qat_minimal_projected_ms": "TBD", + "w4a4_qat_quant_acc": "TBD", + "w4a4_qat_sparsity": "TBD", + "w8a8_akida_acc": "TBD", + "w8a8_allnps_cycles": "TBD", + "w8a8_allnps_latency_ms": "TBD", + "w8a8_allnps_nps": "TBD", + "w8a8_allnps_passes": "TBD", + "w8a8_allnps_projected_ms": "TBD", + "w8a8_minimal_cycles": "TBD", + "w8a8_minimal_latency_ms": "TBD", + "w8a8_minimal_nps": "TBD", + "w8a8_minimal_passes": "TBD", + "w8a8_minimal_projected_ms": "TBD", + "w8a8_quant_acc": "TBD", + "w8a8_sparsity": "TBD" +} diff --git a/akida2/model_zoo/vww/models/.gitignore b/akida2/model_zoo/vww/models/.gitignore new file mode 100644 index 0000000..647613e --- /dev/null +++ b/akida2/model_zoo/vww/models/.gitignore @@ -0,0 +1,4 @@ +# Git to Ignore everything in this directory +* +# Except this .gitignore file +!.gitignore \ No newline at end of file diff --git a/akida2/model_zoo/vww/pretrained_models/.gitignore b/akida2/model_zoo/vww/pretrained_models/.gitignore new file mode 100644 index 0000000..647613e --- /dev/null +++ b/akida2/model_zoo/vww/pretrained_models/.gitignore @@ -0,0 +1,4 @@ +# Git to Ignore everything in this directory +* +# Except this .gitignore file +!.gitignore \ No newline at end of file diff --git a/akida2/model_zoo/vww/update_readme.py b/akida2/model_zoo/vww/update_readme.py new file mode 100644 index 0000000..82e42d9 --- /dev/null +++ b/akida2/model_zoo/vww/update_readme.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License +"""Regenerate README.md from README.md.template + metrics.json.""" +import json +import pathlib + +here = pathlib.Path(__file__).parent +metrics = json.loads((here / "docs" / "metrics.json").read_text()) +template = (here / "docs" / "README.md.template").read_text() +(here / "README.md").write_text(template.format_map(metrics)) +print("README.md updated.") diff --git a/akida2/model_zoo/vww/vww_benchmark.py b/akida2/model_zoo/vww/vww_benchmark.py new file mode 100644 index 0000000..9fab059 --- /dev/null +++ b/akida2/model_zoo/vww/vww_benchmark.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License +""" +VWW hardware benchmark for Akida 2. + +Runs a latency benchmark on an Akida VWW model, prints a summary, checks +per-layer activation sparsity, and generates summary plots. + +Differences from the Akida 1 benchmark: + * The reference Akida 2 hardware is an FPGA running at 25 MHz, so the + measured clock frequency is 25e6 (not 400 MHz as on AKD1500). + * Because the FPGA runs at a low clock, we also report a PROJECTED latency + at a higher target clock. Cycle count is fixed for a given model + mapping + regardless of clock speed, so projected_latency_ms = cycles / target_clock. + * Power measurement is NOT performed. The FPGA power-measurement path is + still under development, so this script is latency-only for now (the + power columns are omitted from the metrics and README table). + +Example +------- + python vww_benchmark.py -l models/akidanet_vww_i8_w8_a8.fbz +""" +import argparse +import json +import pathlib +import sys +import time + +import numpy as np +import akida +from akida_models.sparsity import compute_sparsity + +from vww_data import get_samples +from brainchip_utils.hardware_utils import (get_mapping_stats, get_akida_device, + per_layer_benchmark, full_model_benchmark) +from brainchip_utils.plot_utils import (plot_full_model_results, plot_per_layer_results, + pretty_print_sparsity) + +# Measured clock: Akida 2 reference hardware is an FPGA clocked at 25 MHz. +MEASURED_CLOCK = 25e6 # 25 MHz FPGA + +# Projected clock: latency is also projected to a higher target clock to +# indicate expected performance on faster (e.g. ASIC) hardware. Cycle count is +# clock-independent, so the projection is exact given the target frequency. +# TODO: confirm the intended projection target clock for Akida 2 silicon. +# 100 MHz is a provisional placeholder only. +PROJECTED_CLOCK = 100e6 # provisional + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description='Hardware latency benchmark for an Akida 2 VWW model') + parser.add_argument('-l', '--loadmodel', required=True, + help='Model to load (.fbz akida model)') + parser.add_argument('-d', '--data', default='./data/vw_coco2014_96', + help='VWW dataset root (contains train/ and val/ subdirs)') + parser.add_argument('--save-metrics', action='store_true', + help='Write benchmark values to metrics.json') + args = parser.parse_args() + + NUM_SAMPLES = 1000 + + # ------------------------------------------------------------------------- + # Model + # ------------------------------------------------------------------------- + ak_model = akida.Model(args.loadmodel) + imsize = tuple(ak_model.input_shape) + + # ------------------------------------------------------------------------- + # Device + # ------------------------------------------------------------------------- + device = get_akida_device(target_version=ak_model.ip_version) + if device is None: + print("No compatible Akida hardware device found. Skipping benchmarking") + sys.exit(0) + + # ------------------------------------------------------------------------- + # Sample + # ------------------------------------------------------------------------- + # Processing in Akida is activity dependent (because it exploits sparsity) + # and that activity is dependent on the input. That makes it imperative to + # use real inputs when benchmarking Akida, rather than synthetic random + # samples. + samples = get_samples(args.data, imsize, num_samples=NUM_SAMPLES) + + # ------------------------------------------------------------------------- + # Full-model benchmark (latency only) + # ------------------------------------------------------------------------- + # NOTE: power measurement is intentionally not requested here -- the FPGA + # power path is still under development, so we report latency only. When a + # power path exists, this is where full_model_benchmark's power fields would + # be surfaced (as in the Akida 1 example). + map_modes = ['Minimal', 'AllNps'] + full_results = dict() + for mm in map_modes: + map_mode = getattr(akida.MapMode, mm) + print(f'\nRunning full-model benchmark (MapMode={mm})...') + res = full_model_benchmark(ak_model, device, samples, + map_mode=map_mode, + clock_freq=MEASURED_CLOCK) + + # Projected latency at a higher target clock. Cycle count (mean_inf_clk) + # is clock-independent, so this is an exact rescale, not an estimate of + # host overhead. + res['projected_clk_ms'] = res['mean_inf_clk'] / PROJECTED_CLOCK * 1000 + full_results[mm] = res + + # Re-map without hw_only to populate ak_model.sequences for stats + ak_model.map(device, mode=map_mode) + num_nps, num_passes, num_sequences = get_mapping_stats(ak_model) + full_results[mm]['num_nps'] = num_nps + full_results[mm]['num_passes'] = num_passes + print(f' Mapping: {num_nps} NP(s), {num_passes} pass(es), {num_sequences} sequence(s)') + print(f' Measured latency @ {MEASURED_CLOCK/1e6:.0f} MHz: ' + f'{res["mean_clk_ms"]:.3f} ms') + print(f' Projected latency @ {PROJECTED_CLOCK/1e6:.0f} MHz: ' + f'{res["projected_clk_ms"]:.3f} ms') + if num_sequences > 1: + print('WARNING: note, model not completely mapped to hardware') + + # ------------------------------------------------------------------------- + # Per-layer benchmark. Minimal mapping mode, batch-size 1 + # ------------------------------------------------------------------------- + ak_model.map(device, mode=akida.MapMode.Minimal, hw_only=True) + ak_model.summary() + + # Check sparsity per-layer + sparsity_dict = compute_sparsity(ak_model, samples=samples) + pretty_print_sparsity(sparsity_dict) + + print(f'Running per-layer benchmark ({NUM_SAMPLES} samples)...') + per_layer_results = per_layer_benchmark(ak_model, device, samples, + repeats=NUM_SAMPLES, + clock_freq=MEASURED_CLOCK) + + # ------------------------------------------------------------------------- + # Plots + # ------------------------------------------------------------------------- + # Map without hw_only so ak_model.sequences is available for plot_mapping + ak_model.map(device, mode=akida.MapMode.Minimal) + perlayer_savepath = 'benchmark_results_layers.png' + if args.save_metrics: + perlayer_savepath = pathlib.Path(__file__).parent / 'docs' / ('ref_' + perlayer_savepath) + plot_per_layer_results(per_layer_results, ak_model, sparsity_dict, + model_name=args.loadmodel, + savepath=perlayer_savepath) + print('\nPer-layer results plot saved to ' + str(perlayer_savepath)) + + full_savepath = 'benchmark_results_full.png' + if args.save_metrics: + full_savepath = pathlib.Path(__file__).parent / 'docs' / ('ref_' + full_savepath) + plot_full_model_results(full_results, ak_model, device, + model_name=args.loadmodel, + savepath=full_savepath) + print('Full model results plot saved to ' + str(full_savepath)) + + if args.save_metrics: + # Updates the stored metrics used to generate the README performance + # tables. For code maintenance only, run against the pretrained models. + # + # Keys are variant-prefixed to match the multi-variant v2 README table. + # The variant is inferred from the loaded .fbz filename (same scheme as + # vww_eval.py). Power keys are intentionally absent: benchmarking is + # latency-only until the FPGA power path exists. + stem = pathlib.Path(args.loadmodel).stem + if 'i8_w8_a8' in stem: + variant = 'w8a8' + elif 'i8_w4_a4' in stem: + variant = 'w4a4_qat' + else: + variant = 'w8a8' # fallback; benchmarking is only meaningful for a quantized .fbz + + metrics_path = pathlib.Path(__file__).parent / 'docs' / 'metrics.json' + metrics = json.loads(metrics_path.read_text()) if metrics_path.exists() else {} + metrics[f'{variant}_sparsity'] = f'{np.mean(list(sparsity_dict.values())) * 100:.2f}%' + for mm, res in full_results.items(): + mode = mm.lower() + metrics[f'{variant}_{mode}_nps'] = str(res['num_nps']) + metrics[f'{variant}_{mode}_passes'] = str(res['num_passes']) + metrics[f'{variant}_{mode}_cycles'] = f'{res["mean_inf_clk"]:.0f}' + metrics[f'{variant}_{mode}_latency_ms'] = f'{res["mean_clk_ms"]:.3f}' + metrics[f'{variant}_{mode}_projected_ms'] = f'{res["projected_clk_ms"]:.3f}' + metrics_path.write_text(json.dumps(metrics, indent=4) + '\n') + print(f'Metrics saved to {metrics_path}') diff --git a/akida2/model_zoo/vww/vww_data.py b/akida2/model_zoo/vww/vww_data.py new file mode 100644 index 0000000..b001c84 --- /dev/null +++ b/akida2/model_zoo/vww/vww_data.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License + +import numpy as np +import tensorflow as tf +from tf_keras.preprocessing.image import ImageDataGenerator +from tf_keras.utils import set_random_seed + +# Define the base directory for the VWW dataset + +def get_data(data_path, input_shape, batch_size, seed=42): + """ Loads VWW data. + + Args: + data_path (str): path to data + input_shape (tuple): input image shape (height, width, channels) + batch_size (int): the batch size + dtype (tf.dtypes.DType, optional): input data type. Defaults to tf.uint8. + + Returns: + tf_keras.data.Dataset, tf_keras.data.Dataset: training dataset, validation dataset + """ + set_random_seed(seed) + + # Set aside .1 split for validation + validation_split = 0.1 + + # Create a data generator with data augmentation and load files from + # directory + train_datagen = ImageDataGenerator( + rotation_range = 10, + width_shift_range = 0.05, + height_shift_range = 0.05, + zoom_range = 0.1, + horizontal_flip = True, + validation_split = validation_split) + + train_generator = train_datagen.flow_from_directory( + data_path, + target_size=input_shape[:2], + batch_size=batch_size, + subset = 'training', + color_mode='rgb', + class_mode = 'sparse', + shuffle=True) + + val_datagen = ImageDataGenerator( + validation_split = validation_split) + + val_generator = val_datagen.flow_from_directory( + data_path, + target_size=input_shape[:2], + batch_size=batch_size, + subset = 'validation', + color_mode='rgb', + class_mode = 'sparse', + shuffle=False) + + return train_generator, val_generator + +def get_samples(data_path, input_shape, num_samples=1024): + """ Loads image samples from the train split as a numpy array. + + No augmentation is applied; images are only resized to input_shape. + Suitable for model calibration and testing. + + Args: + data_path (str): path to data + input_shape (tuple): input image shape (height, width, channels) + num_samples (int): number of samples to return. Defaults to 1024. + + Returns: + np.ndarray: array of shape (num_samples, height, width, channels), dtype uint8 + """ + generator = ImageDataGenerator().flow_from_directory( + data_path, + target_size=input_shape[:2], + batch_size=num_samples, + shuffle=False) + + images, _ = next(generator) + return images[:num_samples].astype(np.uint8) diff --git a/akida2/model_zoo/vww/vww_eval.py b/akida2/model_zoo/vww/vww_eval.py new file mode 100644 index 0000000..75e9bd6 --- /dev/null +++ b/akida2/model_zoo/vww/vww_eval.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License +""" +VWW evaluation for tf_keras or akida models. +Example +------- + python eval.py -d /data/vww_coco2014_96/ -l akidanet_vww.h5 +""" +import argparse +import json +import pathlib +import numpy as np +import tensorflow as tf + +from tqdm import tqdm + +import akida + +from cnn2snn import load_quantized_model + +from vww_data import get_data +from brainchip_utils.hardware_utils import get_akida_device + +tf.config.experimental.enable_op_determinism() + +# --------------------------------------------------------------------------- +# Evaluation on Akida +# --------------------------------------------------------------------------- +def evaluate_akida_model(akida_model, val_dataset): + """Run inference with an Akida model and return (predictions, labels).""" + device = get_akida_device(target_version = akida_model.ip_version) + if device is not None: + akida_model.map(device, mode=akida.MapMode.Minimal) + print('Running inference on Akida hardware device') + akida_model.summary() + + labels_all = None + logits_all = None + + # Akida can't directly digest the tensorflow dataset, we need to + # manually iterate over the dataset to deliver inputs as numpy arrays. + # val_dataset is a Keras DirectoryIterator, which cycles indefinitely, + # so we must limit iteration to a single epoch (len(val_dataset) batches). + num_batches = len(val_dataset) + for _ in tqdm(range(num_batches), desc="Evaluating on Akida"): + batch, label_batch = next(val_dataset) + if not isinstance(batch, np.ndarray): + batch = batch.numpy() + + # Inference on Akida + logits_batch = akida_model.predict(batch.astype(np.uint8)) + logits_batch = logits_batch.squeeze(axis=(1, 2)) # (B, 1, 1, C) -> (B, C) + + if labels_all is None: + labels_all = label_batch + logits_all = logits_batch + else: + labels_all = np.concatenate([labels_all, label_batch]) + logits_all = np.concatenate([logits_all, logits_batch]) + + preds = np.argmax(logits_all, axis=1) + accuracy = np.mean(np.equal(np.array(preds), np.array(labels_all))) + print(f'Akida accuracy: {accuracy:.4f}') + return preds, labels_all + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-l', '--loadmodel', required=True, + help='Model to load (.h5 tf_keras or .fbz akida model)') + parser.add_argument('-d', '--data', default='./data/vw_coco2014_96', + help='VWW dataset root (contains train/ and val/ subdirs)') + parser.add_argument('--save-metrics', action='store_true', + help='Write accuracy (and param count for .h5) to metrics.json') + args = parser.parse_args() + + + # --------------------------------------------------------------------------- + # Model + # --------------------------------------------------------------------------- + if args.loadmodel.endswith('.h5'): + model = load_quantized_model(args.loadmodel) + model.compile(metrics=['accuracy']) + isakida = False + imsize = model.input_shape[1:] + elif args.loadmodel.endswith('.fbz'): + model = akida.Model(args.loadmodel) + isakida = True + imsize = tuple(model.input_shape) + + # --------------------------------------------------------------------------- + # Data loading + # --------------------------------------------------------------------------- + train_ds, val_ds = get_data(args.data, imsize, batch_size=32) + + # --------------------------------------------------------------------------- + # Evaluation + # --------------------------------------------------------------------------- + if isakida: + preds, labels = evaluate_akida_model(model, val_ds) + accuracy = float(np.mean(np.equal(preds, labels))) + else: + _, accuracy = model.evaluate(val_ds, verbose=0) + print(f'Validation accuracy: {accuracy:.4f}') + + # --------------------------------------------------------------------------- + # Persist metrics + # --------------------------------------------------------------------------- + if args.save_metrics: + # Updates the stored metrics used to generate the README performance + # tables. For code maintenance only, run against the pretrained models. + # + # The Akida 2 VWW example has two quantized variants, disambiguated by + # filename. Each variant contributes a "quantized" accuracy (from its + # .h5) and an "akida" accuracy (from its .fbz): + # akidanet_vww.h5 -> float_acc, params + # akidanet_vww_i8_w8_a8.{h5,fbz} -> w8a8_quant_acc / w8a8_akida_acc + # akidanet_vww_i8_w4_a4_qat.{h5,fbz} -> w4a4_qat_quant_acc / w4a4_qat_akida_acc + # (The 4-bit PTQ model is a throwaway on the way to QAT -- it is never + # evaluated or stored, so it has no metrics.) + metrics_path = pathlib.Path(__file__).parent / 'docs' / 'metrics.json' + metrics = json.loads(metrics_path.read_text()) if metrics_path.exists() else {} + acc_str = f'{accuracy * 100:.2f}%' + stem = pathlib.Path(args.loadmodel).stem + + # Determine the variant prefix from the filename. + if 'i8_w8_a8' in stem: + variant = 'w8a8' + elif 'i8_w4_a4' in stem: + variant = 'w4a4_qat' + else: + variant = None # float model + + if variant is None: + # Float model: record float accuracy and parameter count. + metrics['float_acc'] = acc_str + metrics['params'] = f'{model.count_params():,}' + elif isakida: + metrics[f'{variant}_akida_acc'] = acc_str + else: + metrics[f'{variant}_quant_acc'] = acc_str + metrics_path.write_text(json.dumps(metrics, indent=4) + '\n') + print(f'Metrics saved to {metrics_path}') diff --git a/akida2/model_zoo/vww/vww_model.py b/akida2/model_zoo/vww/vww_model.py new file mode 100644 index 0000000..6401885 --- /dev/null +++ b/akida2/model_zoo/vww/vww_model.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License +""" +Create a model for the VWW dataset targeting the Akida 2 platform. + +This model is based on the AkidaNet architecture with a small width +multiplier (alpha=0.25, adequate for this binary task) at 96x96 RGB input. +Unlike the Akida 1 VWW example -- which fine-tunes an ImageNet-pretrained +backbone -- this Akida 2 model is built directly from the `akidanet_imagenet` +factory with `include_top=True` and a 2-class head (person / non-person), +returning output logits for training without a softmax. + +The model expects uint8 inputs in the [0, 255] range: input scaling +(divide by 255) is included in the model via the Rescaling layer, so the +preprocessing pipeline should NOT apply any additional normalization. + +Structure (matches the reference float model akidanet_vww.h5): + input (uint8, 96x96x3) + -> rescaling (scale=1/255, offset=0) + -> conv_0..conv_3 + -> dw/pw_separable_4 .. dw/pw_separable_13 + -> global_avg pooling + -> dropout (1e-3) + -> classifier (Dense, 2 units, linear) + +Usage: + python vww_model.py [-s OUTPUT_PATH] +""" + +import argparse + +from tf_keras.utils import set_random_seed +from akida_models.imagenet import akidanet_imagenet +from cnn2snn import set_akida_version, AkidaVersion + + +def build_vww_model(seed=42): + set_random_seed(seed) + + classes = 2 + # akidanet_imagenet is version-aware: under AkidaVersion.v2 it builds a + # model using v2-compatible layer/activation variants. include_top=True + # appends the standard global_avg -> dropout(1e-3) -> Dense(classes) head. + # input_scaling=(255, 0) bakes the /255 rescaling into the model so that + # uint8 inputs can be fed directly (the factory default is (128, -1), + # which would be WRONG here -- it must be passed explicitly). + with set_akida_version(AkidaVersion.v2): + model = akidanet_imagenet( + input_shape=(96, 96, 3), + alpha=0.25, + classes=classes, + include_top=True, + input_scaling=(255, 0), + ) + + return model + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description='Build the AkidaNet-VWW model for Akida 2') + parser.add_argument("-s", + "--savepath", + type=str, + default='./models/akidanet_vww_untrained.h5', + help="Save model with the specified path + name") + parser.add_argument('--seed', type=int, default=42, + help='Random seed for reproducibility') + args = parser.parse_args() + + model = build_vww_model(seed=args.seed) + model.summary() + model.save(args.savepath, include_optimizer=False) + print(f'Model saved to {args.savepath}') diff --git a/akida2/model_zoo/vww/vww_notebook.py b/akida2/model_zoo/vww/vww_notebook.py new file mode 100644 index 0000000..7481a89 --- /dev/null +++ b/akida2/model_zoo/vww/vww_notebook.py @@ -0,0 +1,221 @@ +# --- +# jupyter: +# jupytext: +# formats: ipynb,py:percent +# text_representation: +# extension: .py +# format_name: percent +# format_version: '1.3' +# jupytext_version: 1.19.5 +# kernelspec: +# display_name: Python 3 +# language: python +# name: python3 +# --- + +# %% [markdown] +# BrainChip Dev Hub +# +# # Visual Wake Words (VWW) — Akida 2 Training +# +# This notebook walks through the full Akida 2 pipeline for the Visual Wake Words (person / non-person) task using an AkidaNet-0.25 model at 96×96 resolution: float training, quantization with **quantizeml** (an 8-bit variant and a 4-bit QAT variant), conversion to Akida with **cnn2snn**, and evaluation on the Akida software backend. + +# %% +# Colab-only setup. Local users: ignore this cell — it does nothing for you. +import sys, os + +if 'google.colab' in sys.modules: + if not os.path.exists('colab_setup.py'): + !wget -q https://raw.githubusercontent.com/Brainchip-Inc/brainchip_devhub/main/akida2/model_zoo/vww/colab_setup.py + import colab_setup; colab_setup.setup() + +# %% [markdown] +# ## Setup + +# %% +import os +import numpy as np +import tensorflow as tf +from tqdm import tqdm + +DATA_PATH = './data/vw_coco2014_96' +MODELS_DIR = './models' +os.makedirs(MODELS_DIR, exist_ok=True) + +RUN_FLOAT_TRAINING = True + +SEED = 42 + +# Must be called before any TF ops to make GPU ops deterministic. +tf.config.experimental.enable_op_determinism() + +# %% [markdown] +# ## Dataset +# +# VWW is a directory dataset (`train/` + `val/`). `get_data` returns two batched datasets; images are uint8 in `[0, 255]` (the model rescales on-graph). + +# %% +from vww_data import get_data + +BATCH_SIZE = 32 +INPUT_SHAPE = (96, 96, 3) + +train_ds, val_ds = get_data(DATA_PATH, INPUT_SHAPE, BATCH_SIZE, seed=SEED) + +# %% [markdown] +# ## Model +# +# The model definition is lifted from the source example (`vww_model.py`). For Akida 2 it is built under `set_akida_version(AkidaVersion.v2)`; the factory is version-aware, so no per-layer changes are needed. + +# %% +from vww_model import build_vww_model +model = build_vww_model(seed=SEED) +model.summary() + +# %% [markdown] +# ## Float Training +# +# The Akida 2 VWW model is trained from scratch (no ImageNet transfer learning). Epoch count here is a provisional default — adjust for a real run. + +# %% +from vww_train import train_vww + +if RUN_FLOAT_TRAINING: + LEARNING_RATE = 1e-3 + EPOCHS = 50 + # Freshly set the dataset seed for reproducibility + train_ds, val_ds = get_data(DATA_PATH, INPUT_SHAPE, BATCH_SIZE, seed=SEED) + + train_vww(model, train_ds, val_ds, EPOCHS, LEARNING_RATE, seed=SEED) + + float_model_path = os.path.join(MODELS_DIR, 'akidanet_vww.h5') + model.save(float_model_path, include_optimizer=False) + print(f'Float model saved to {float_model_path}') +else: + from tf_keras.models import load_model + print('Training skipped. Loading an existing float model...') + model = load_model(os.path.join(MODELS_DIR, 'akidanet_vww.h5')) + +# %% [markdown] +# ### Evaluate float model + +# %% +model.compile(metrics=['accuracy']) +_, float_accuracy = model.evaluate(val_ds, verbose=0) +print(f'Float validation accuracy: {float_accuracy:.4f}') + +# %% [markdown] +# ## Quantization (quantizeml) +# +# Akida 2 quantizes with **quantizeml** (not `cnn2snn.quantize`). We produce two variants: +# +# * **8-bit** (i8/w8/a8) — post-training quantization only; 8-bit PTQ is accurate enough that QAT is not needed. +# * **4-bit** (i8/w4/a4) — quantization-aware training (QAT); 4-bit PTQ accuracy is poor, so we fine-tune. The input layer weights stay 8-bit in both variants. + +# %% +from quantizeml.models import quantize +from quantizeml.layers import QuantizationParams + +# --- 8-bit variant (i8 / w8 / a8), PTQ only --- +qparams_8bit = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8) +model_8bit = quantize(model, qparams=qparams_8bit) + +q8_path = os.path.join(MODELS_DIR, 'akidanet_vww_i8_w8_a8.h5') +model_8bit.save(q8_path, include_optimizer=False) + +model_8bit.compile(metrics=['accuracy']) +_, acc_8bit = model_8bit.evaluate(val_ds, verbose=0) +print(f'8-bit quantized validation accuracy: {acc_8bit:.4f}') + +# %% [markdown] +# ### 4-bit variant with QAT +# +# Quantize to 4-bit weights and activations, then fine-tune (QAT) to recover the accuracy lost at 4 bits. The intermediate PTQ model is not kept. + +# %% +# --- 4-bit variant (i8 / w4 / a4), QAT --- +qparams_4bit = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4) +model_4bit = quantize(model, qparams=qparams_4bit) + +# QAT fine-tune the quantized 4-bit model. quantizeml-quantized models are standard +# Keras models, so the same training loop applies. +QAT_EPOCHS = 5 +QAT_LR = 1e-4 +train_ds, val_ds = get_data(DATA_PATH, INPUT_SHAPE, BATCH_SIZE, seed=SEED) +train_vww(model_4bit, train_ds, val_ds, QAT_EPOCHS, QAT_LR, seed=SEED) + +q4_path = os.path.join(MODELS_DIR, 'akidanet_vww_i8_w4_a4_qat.h5') +model_4bit.save(q4_path, include_optimizer=False) + +model_4bit.compile(metrics=['accuracy']) +_, acc_4bit = model_4bit.evaluate(val_ds, verbose=0) +print(f'4-bit QAT validation accuracy: {acc_4bit:.4f}') + +# %% [markdown] +# ## Conversion to Akida Format +# +# `cnn2snn.convert` accepts quantizeml-quantized models and produces the Akida `.fbz`. We convert both variants. + +# %% +from cnn2snn import convert + +akida_8bit = convert(model_8bit) +akida_8bit.save(os.path.join(MODELS_DIR, 'akidanet_vww_i8_w8_a8.fbz')) + +akida_4bit = convert(model_4bit) +akida_4bit.save(os.path.join(MODELS_DIR, 'akidanet_vww_i8_w4_a4_qat.fbz')) + +akida_8bit.summary() + + +# %% [markdown] +# ## Evaluation on Akida (software backend) +# +# Run both Akida models over the validation set. No hardware is required — Akida runs on the software backend when no device is present. + +# %% +def evaluate_akida(akida_model, ds): + ds.reset() + labels_all, logits_all = [], [] + for _ in tqdm(range(len(ds)), desc='Evaluating on Akida'): + batch, label_batch = next(ds) + if not isinstance(batch, np.ndarray): + batch = batch.numpy() + logits = akida_model.predict(batch.astype(np.uint8)).squeeze(axis=(1, 2)) + labels_all.append(label_batch) + logits_all.append(logits) + labels_all = np.concatenate(labels_all) + preds = np.argmax(np.concatenate(logits_all), axis=1) + return float(np.mean(preds == labels_all)) + +akida_acc_8bit = evaluate_akida(akida_8bit, val_ds) +akida_acc_4bit = evaluate_akida(akida_4bit, val_ds) +print(f'Akida 8-bit accuracy: {akida_acc_8bit:.4f}') +print(f'Akida 4-bit QAT accuracy: {akida_acc_4bit:.4f}') + +# %% [markdown] +# ## Activation Sparsity +# +# Activation sparsity drives efficiency on Akida (zero activations are skipped). + +# %% +from akida_models.sparsity import compute_sparsity +from brainchip_utils.plot_utils import pretty_print_sparsity +from vww_data import get_samples + +NUM_SAMPLES = 1024 +samples = get_samples(DATA_PATH, INPUT_SHAPE, num_samples=NUM_SAMPLES) + +print('8-bit sparsity:') +pretty_print_sparsity(compute_sparsity(akida_8bit, samples=samples)) +print('\n4-bit QAT sparsity:') +pretty_print_sparsity(compute_sparsity(akida_4bit, samples=samples)) + +# %% [markdown] +# ## Summary + +# %% +print(f'{"Variant":<16}{"Keras acc":<12}{"Akida acc":<12}') +print(f'{"float":<16}{float_accuracy:<12.4f}{"-":<12}') +print(f'{"8-bit (w8a8)":<16}{acc_8bit:<12.4f}{akida_acc_8bit:<12.4f}') +print(f'{"4-bit QAT":<16}{acc_4bit:<12.4f}{akida_acc_4bit:<12.4f}') diff --git a/akida2/model_zoo/vww/vww_notebook_benchmark.ipynb b/akida2/model_zoo/vww/vww_notebook_benchmark.ipynb new file mode 100644 index 0000000..3030589 --- /dev/null +++ b/akida2/model_zoo/vww/vww_notebook_benchmark.ipynb @@ -0,0 +1,136 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"BrainChip\n\n# Visual Wake Words (VWW) \u2014 Akida 2 Benchmark\n\nThis notebook evaluates a converted Akida VWW model and, **if an Akida 2 device is connected**, benchmarks its latency. The Akida 2 reference platform is an FPGA running at 25 MHz; a projected latency at a higher clock is also reported (cycle count is clock-independent, so the projection is exact). Power measurement is not yet available, so benchmarking is latency-only.\n\n> **Note:** the hardware benchmark section requires a physical Akida 2 FPGA board. Without a device it will report \"no hardware found\" and skip \u2014 it cannot run on Colab." + ], + "id": "cell-00" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup" + ], + "id": "cell-01" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "import os\nimport numpy as np\nimport akida\n\nDATA_PATH = './data/vw_coco2014_96'\nMODELS_DIR = './models'\nINPUT_SHAPE = (96, 96, 3)\n\n# Which converted model to benchmark. Options produced by the training pipeline:\n# akidanet_vww_i8_w8_a8.fbz (8-bit)\n# akidanet_vww_i8_w4_a4_qat.fbz (4-bit QAT)\nMODEL_FBZ = os.path.join(MODELS_DIR, 'akidanet_vww_i8_w8_a8.fbz')\n\n# Clocks (see vww_benchmark.py).\nMEASURED_CLOCK = 25e6 # 25 MHz FPGA\nPROJECTED_CLOCK = 100e6 # provisional \u2014 confirm target clock" + ], + "id": "cell-02" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load the Akida model" + ], + "id": "cell-03" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "ak_model = akida.Model(MODEL_FBZ)\nak_model.summary()" + ], + "id": "cell-04" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Device\n\n`get_akida_device` returns `None` when no compatible hardware is present, in which case the benchmark below is skipped." + ], + "id": "cell-05" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from brainchip_utils.hardware_utils import get_akida_device\n\ndevice = get_akida_device(target_version=ak_model.ip_version)\nif device is None:\n print('No compatible Akida hardware device found \u2014 benchmark will be skipped.')\nelse:\n print('Akida device found:', device)" + ], + "id": "cell-06" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Samples\n\nAkida latency is activity-dependent (it exploits sparsity), so we benchmark on real inputs rather than random data." + ], + "id": "cell-07" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from vww_data import get_samples\n\nNUM_SAMPLES = 1000\nsamples = get_samples(DATA_PATH, INPUT_SHAPE, num_samples=NUM_SAMPLES)" + ], + "id": "cell-08" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Latency benchmark\n\nFull-model benchmark in both mapping modes, with measured (25 MHz) and projected latency. Cycle count is fixed for a given model + mapping, so `projected_ms = mean_inf_clk / PROJECTED_CLOCK * 1000`." + ], + "id": "cell-09" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from brainchip_utils.hardware_utils import full_model_benchmark, get_mapping_stats\n\nif device is not None:\n for mm in ['Minimal', 'AllNps']:\n map_mode = getattr(akida.MapMode, mm)\n res = full_model_benchmark(ak_model, device, samples,\n map_mode=map_mode, clock_freq=MEASURED_CLOCK)\n projected_ms = res['mean_inf_clk'] / PROJECTED_CLOCK * 1000\n ak_model.map(device, mode=map_mode)\n num_nps, num_passes, num_sequences = get_mapping_stats(ak_model)\n print(f'[{mm}] NPs={num_nps} passes={num_passes} '\n f'latency@25MHz={res[\"mean_clk_ms\"]:.3f} ms '\n f'projected@100MHz={projected_ms:.3f} ms')\nelse:\n print('Hardware not available \u2014 skipping latency benchmark.')" + ], + "id": "cell-10" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Per-layer benchmark & sparsity\n\nPer-layer latency (Minimal mapping) plus activation sparsity, which drives Akida efficiency." + ], + "id": "cell-11" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from brainchip_utils.hardware_utils import per_layer_benchmark\nfrom akida_models.sparsity import compute_sparsity\nfrom brainchip_utils.plot_utils import pretty_print_sparsity\n\nif device is not None:\n ak_model.map(device, mode=akida.MapMode.Minimal, hw_only=True)\n sparsity_dict = compute_sparsity(ak_model, samples=samples)\n pretty_print_sparsity(sparsity_dict)\n per_layer_results = per_layer_benchmark(ak_model, device, samples,\n repeats=NUM_SAMPLES, clock_freq=MEASURED_CLOCK)\n print('Per-layer benchmark complete.')\nelse:\n # Sparsity can still be computed on the software backend without hardware.\n sparsity_dict = compute_sparsity(ak_model, samples=samples)\n pretty_print_sparsity(sparsity_dict)\n print('Hardware not available \u2014 skipped latency; sparsity computed on software backend.')" + ], + "id": "cell-12" + } + ], + "metadata": { + "jupytext": { + "formats": "ipynb,py:percent" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/akida2/model_zoo/vww/vww_notebook_training.ipynb b/akida2/model_zoo/vww/vww_notebook_training.ipynb new file mode 100644 index 0000000..111260e --- /dev/null +++ b/akida2/model_zoo/vww/vww_notebook_training.ipynb @@ -0,0 +1,378 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cell-00", + "metadata": {}, + "source": [ + "\"BrainChip\n", + "\n", + "# Visual Wake Words (VWW) — Akida 2 Training\n", + "\n", + "This notebook walks through the full Akida 2 pipeline for the Visual Wake Words (person / non-person) task using an AkidaNet-0.25 model at 96×96 resolution: float training, quantization with **quantizeml** (an 8-bit variant and a 4-bit QAT variant), conversion to Akida with **cnn2snn**, and evaluation on the Akida software backend." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-01", + "metadata": {}, + "outputs": [], + "source": [ + "# Colab-only setup. Local users: ignore this cell — it does nothing for you.\n", + "import sys, os\n", + "\n", + "if 'google.colab' in sys.modules:\n", + " if not os.path.exists('colab_setup.py'):\n", + " !wget -q https://raw.githubusercontent.com/Brainchip-Inc/brainchip_devhub/main/akida2/model_zoo/vww/colab_setup.py\n", + " import colab_setup; colab_setup.setup()" + ] + }, + { + "cell_type": "markdown", + "id": "cell-02", + "metadata": {}, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-03", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import numpy as np\n", + "import tensorflow as tf\n", + "from tqdm import tqdm\n", + "\n", + "DATA_PATH = './data/vw_coco2014_96'\n", + "MODELS_DIR = './models'\n", + "os.makedirs(MODELS_DIR, exist_ok=True)\n", + "\n", + "RUN_FLOAT_TRAINING = True\n", + "\n", + "SEED = 42\n", + "\n", + "# Must be called before any TF ops to make GPU ops deterministic.\n", + "tf.config.experimental.enable_op_determinism()" + ] + }, + { + "cell_type": "markdown", + "id": "cell-04", + "metadata": {}, + "source": [ + "## Dataset\n", + "\n", + "VWW is a directory dataset (`train/` + `val/`). `get_data` returns two batched datasets; images are uint8 in `[0, 255]` (the model rescales on-graph)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-05", + "metadata": {}, + "outputs": [], + "source": [ + "from vww_data import get_data\n", + "\n", + "BATCH_SIZE = 32\n", + "INPUT_SHAPE = (96, 96, 3)\n", + "\n", + "train_ds, val_ds = get_data(DATA_PATH, INPUT_SHAPE, BATCH_SIZE, seed=SEED)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-06", + "metadata": {}, + "source": [ + "## Model\n", + "\n", + "The model definition is lifted from the source example (`vww_model.py`). For Akida 2 it is built under `set_akida_version(AkidaVersion.v2)`; the factory is version-aware, so no per-layer changes are needed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-07", + "metadata": {}, + "outputs": [], + "source": [ + "from vww_model import build_vww_model\n", + "model = build_vww_model(seed=SEED)\n", + "model.summary()" + ] + }, + { + "cell_type": "markdown", + "id": "cell-08", + "metadata": {}, + "source": [ + "## Float Training\n", + "\n", + "The Akida 2 VWW model is trained from scratch (no ImageNet transfer learning)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-09", + "metadata": {}, + "outputs": [], + "source": [ + "from vww_train import train_vww\n", + "\n", + "if RUN_FLOAT_TRAINING:\n", + " LEARNING_RATE = 1e-3\n", + " EPOCHS = 20\n", + " # Freshly set the dataset seed for reproducibility\n", + " train_ds, val_ds = get_data(DATA_PATH, INPUT_SHAPE, BATCH_SIZE, seed=SEED)\n", + "\n", + " train_vww(model, train_ds, val_ds, EPOCHS, LEARNING_RATE, seed=SEED)\n", + "\n", + " float_model_path = os.path.join(MODELS_DIR, 'akidanet_vww.h5')\n", + " model.save(float_model_path, include_optimizer=False)\n", + " print(f'Float model saved to {float_model_path}')\n", + "else:\n", + " from tf_keras.models import load_model\n", + " print('Training skipped. Loading an existing float model...')\n", + " model = load_model(os.path.join(MODELS_DIR, 'akidanet_vww.h5'))" + ] + }, + { + "cell_type": "markdown", + "id": "cell-10", + "metadata": {}, + "source": [ + "### Evaluate float model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-11", + "metadata": {}, + "outputs": [], + "source": [ + "model.compile(metrics=['accuracy'])\n", + "_, float_accuracy = model.evaluate(val_ds, verbose=0)\n", + "print(f'Float validation accuracy: {float_accuracy:.4f}')" + ] + }, + { + "cell_type": "markdown", + "id": "cell-12", + "metadata": {}, + "source": [ + "## Quantization (quantizeml)\n", + "\n", + "Akida 2 quantizes with **quantizeml** (not `cnn2snn.quantize`). We produce two variants:\n", + "\n", + "* **8-bit** (i8/w8/a8) — post-training quantization only; 8-bit PTQ is accurate enough that QAT is not needed.\n", + "* **4-bit** (i8/w4/a4) — quantization-aware training (QAT); 4-bit PTQ accuracy is poor, so we fine-tune. The input layer weights stay 8-bit in both variants." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-13", + "metadata": {}, + "outputs": [], + "source": [ + "from quantizeml.models import quantize\n", + "from quantizeml.layers import QuantizationParams\n", + "\n", + "# --- 8-bit variant (i8 / w8 / a8), PTQ only ---\n", + "qparams_8bit = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)\n", + "model_8bit = quantize(model, qparams=qparams_8bit)\n", + "\n", + "q8_path = os.path.join(MODELS_DIR, 'akidanet_vww_i8_w8_a8.h5')\n", + "model_8bit.save(q8_path, include_optimizer=False)\n", + "\n", + "model_8bit.compile(metrics=['accuracy'])\n", + "_, acc_8bit = model_8bit.evaluate(val_ds, verbose=0)\n", + "print(f'8-bit quantized validation accuracy: {acc_8bit:.4f}')" + ] + }, + { + "cell_type": "markdown", + "id": "cell-14", + "metadata": {}, + "source": [ + "### 4-bit variant with QAT\n", + "\n", + "Quantize to 4-bit weights and activations, then fine-tune (QAT) to recover the accuracy lost at 4 bits. The intermediate PTQ model is not kept." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-15", + "metadata": {}, + "outputs": [], + "source": [ + "# --- 4-bit variant (i8 / w4 / a4), QAT ---\n", + "qparams_4bit = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)\n", + "model_4bit = quantize(model, qparams=qparams_4bit)\n", + "\n", + "# QAT fine-tune the quantized 4-bit model. quantizeml-quantized models are standard\n", + "# Keras models, so the same training loop applies.\n", + "QAT_EPOCHS = 5\n", + "QAT_LR = 1e-4\n", + "train_ds, val_ds = get_data(DATA_PATH, INPUT_SHAPE, BATCH_SIZE, seed=SEED)\n", + "train_vww(model_4bit, train_ds, val_ds, QAT_EPOCHS, QAT_LR, seed=SEED)\n", + "\n", + "q4_path = os.path.join(MODELS_DIR, 'akidanet_vww_i8_w4_a4_qat.h5')\n", + "model_4bit.save(q4_path, include_optimizer=False)\n", + "\n", + "model_4bit.compile(metrics=['accuracy'])\n", + "_, acc_4bit = model_4bit.evaluate(val_ds, verbose=0)\n", + "print(f'4-bit QAT validation accuracy: {acc_4bit:.4f}')" + ] + }, + { + "cell_type": "markdown", + "id": "cell-16", + "metadata": {}, + "source": [ + "## Conversion to Akida Format\n", + "\n", + "`cnn2snn.convert` accepts quantizeml-quantized models and produces the Akida `.fbz`. We convert both variants." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-17", + "metadata": {}, + "outputs": [], + "source": [ + "from cnn2snn import convert\n", + "\n", + "akida_8bit = convert(model_8bit)\n", + "akida_8bit.save(os.path.join(MODELS_DIR, 'akidanet_vww_i8_w8_a8.fbz'))\n", + "\n", + "akida_4bit = convert(model_4bit)\n", + "akida_4bit.save(os.path.join(MODELS_DIR, 'akidanet_vww_i8_w4_a4_qat.fbz'))\n", + "\n", + "akida_8bit.summary()" + ] + }, + { + "cell_type": "markdown", + "id": "cell-18", + "metadata": {}, + "source": [ + "## Evaluation on Akida (software backend)\n", + "\n", + "Run both Akida models over the validation set. No hardware is required — Akida runs on the software backend when no device is present." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-19", + "metadata": {}, + "outputs": [], + "source": [ + "def evaluate_akida(akida_model, ds):\n", + " ds.reset()\n", + " labels_all, logits_all = [], []\n", + " for _ in tqdm(range(len(ds)), desc='Evaluating on Akida'):\n", + " batch, label_batch = next(ds)\n", + " if not isinstance(batch, np.ndarray):\n", + " batch = batch.numpy()\n", + " logits = akida_model.predict(batch.astype(np.uint8)).squeeze(axis=(1, 2))\n", + " labels_all.append(label_batch)\n", + " logits_all.append(logits)\n", + " labels_all = np.concatenate(labels_all)\n", + " preds = np.argmax(np.concatenate(logits_all), axis=1)\n", + " return float(np.mean(preds == labels_all))\n", + "\n", + "akida_acc_8bit = evaluate_akida(akida_8bit, val_ds)\n", + "akida_acc_4bit = evaluate_akida(akida_4bit, val_ds)\n", + "print(f'Akida 8-bit accuracy: {akida_acc_8bit:.4f}')\n", + "print(f'Akida 4-bit QAT accuracy: {akida_acc_4bit:.4f}')" + ] + }, + { + "cell_type": "markdown", + "id": "cell-20", + "metadata": {}, + "source": [ + "## Activation Sparsity\n", + "\n", + "Activation sparsity drives efficiency on Akida (zero activations are skipped)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-21", + "metadata": {}, + "outputs": [], + "source": [ + "from akida_models.sparsity import compute_sparsity\n", + "from brainchip_utils.plot_utils import pretty_print_sparsity\n", + "from vww_data import get_samples\n", + "\n", + "NUM_SAMPLES = 1024\n", + "samples = get_samples(DATA_PATH, INPUT_SHAPE, num_samples=NUM_SAMPLES)\n", + "\n", + "print('8-bit sparsity:')\n", + "pretty_print_sparsity(compute_sparsity(akida_8bit, samples=samples))\n", + "print('\\n4-bit QAT sparsity:')\n", + "pretty_print_sparsity(compute_sparsity(akida_4bit, samples=samples))" + ] + }, + { + "cell_type": "markdown", + "id": "cell-22", + "metadata": {}, + "source": [ + "## Summary" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell-23", + "metadata": {}, + "outputs": [], + "source": [ + "print(f'{\"Variant\":<16}{\"Keras acc\":<12}{\"Akida acc\":<12}')\n", + "print(f'{\"float\":<16}{float_accuracy:<12.4f}{\"-\":<12}')\n", + "print(f'{\"8-bit (w8a8)\":<16}{acc_8bit:<12.4f}{akida_acc_8bit:<12.4f}')\n", + "print(f'{\"4-bit QAT\":<16}{acc_4bit:<12.4f}{akida_acc_4bit:<12.4f}')" + ] + } + ], + "metadata": { + "jupytext": { + "formats": "ipynb,py:percent" + }, + "kernelspec": { + "display_name": "ak2191", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/akida2/model_zoo/vww/vww_train.py b/akida2/model_zoo/vww/vww_train.py new file mode 100644 index 0000000..7b52f0a --- /dev/null +++ b/akida2/model_zoo/vww/vww_train.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License +""" +VWW training + +Example +------- + python vww_train.py -d /data/vww_coco2014_96/ -e 50 \\ + -l akidanet_vww_untrained.h5 -s akidanet_vww.h5 +""" +import argparse +import tensorflow as tf + +from tf_keras.losses import SparseCategoricalCrossentropy +from tf_keras.optimizers.legacy import Adam +from tf_keras.optimizers.schedules import CosineDecay +from tf_keras import regularizers +from tf_keras.layers import ReLU +from tf_keras.utils import set_random_seed + +from cnn2snn import load_quantized_model + +from vww_data import get_data + +# Must be called before any TF ops to make GPU ops (conv backward passes, +# bilinear resize, etc.) deterministic. Has a small throughput cost. +tf.config.experimental.enable_op_determinism() + +def train_vww(model, train_ds, val_ds, epochs, learning_rate, regularization=None, seed=42): + set_random_seed(seed) + + steps_per_epoch = len(train_ds) + total_steps = steps_per_epoch * epochs + warmup_steps = int(0.1 * total_steps) # 10% of total steps for warmup + + lr_scheduler = CosineDecay( + initial_learning_rate=0.0, + decay_steps=total_steps - warmup_steps, + warmup_target=learning_rate, + warmup_steps=warmup_steps, + ) + # --------------------------------------------------------------------------- + # Model + # --------------------------------------------------------------------------- + if regularization is not None: + print('Adding Activity Regularization to ReLU layers') + regularizer = regularizers.L1L2(regularization, regularization) + for layer in model.layers: + if isinstance(layer, ReLU): + layer.activity_regularizer = regularizer + + model.compile(optimizer=Adam(learning_rate=lr_scheduler), + loss=SparseCategoricalCrossentropy(from_logits=True), + metrics=['accuracy']) + + + # --------------------------------------------------------------------------- + # Training + # --------------------------------------------------------------------------- + history = model.fit( + train_ds, + epochs=epochs, + validation_data=val_ds, + ) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-l', '--loadmodel', required=True, + help='Model to load (.h5 tf_keras or .fbz akida model)') + parser.add_argument('-s', '--savemodel', required=True, + help='Model save path') + + parser.add_argument('-d', '--data', default='./data/vw_coco2014_96', + help='VWW dataset root (contains train/ and val/ subdirs)') + + parser.add_argument('-b', '--batch_size', type=int, default=32) + parser.add_argument('-e', '--epochs', type=int, default=50) + parser.add_argument('-lr', '--learning_rate', type=float, default=1e-3, + help='Initial learning rate') + parser.add_argument('-reg', '--regularization', type=float, default=None, + help='Activity Regularization to increase sparsity') + parser.add_argument('--seed', type=int, default=42, + help='Random seed for reproducibility') + args = parser.parse_args() + + # --------------------------------------------------------------------------- + # Model + # --------------------------------------------------------------------------- + model = load_quantized_model(args.loadmodel) + + # --------------------------------------------------------------------------- + # Data loading + # --------------------------------------------------------------------------- + train_ds, val_ds = get_data(args.data, model.input_shape[1:], args.batch_size, seed=args.seed) + + train_vww(model=model, + train_ds=train_ds, + val_ds=val_ds, + epochs=args.epochs, + learning_rate=args.learning_rate, + regularization=args.regularization, + seed=args.seed) + + model.save(args.savemodel, include_optimizer=False) + print(f'Model saved as {args.savemodel}.') diff --git a/akida2/model_zoo/vww/vww_train.sh b/akida2/model_zoo/vww/vww_train.sh new file mode 100644 index 0000000..32215a3 --- /dev/null +++ b/akida2/model_zoo/vww/vww_train.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License +# +# End-to-end VWW pipeline for Akida 2. +# +# Quantization uses `quantizeml`. Two quantized variants are produced: +# * 8-bit (i8/w8/a8) via PTQ -- no QAT needed (8-bit PTQ is accurate enough). +# * 4-bit (i8/w4/a4) via QAT only -- 4-bit PTQ accuracy is poor, so the PTQ +# model is a throwaway on the way to QAT (not stored/evaluated). +# The input layer weights are always 8-bit (-i 8). +# Conversion to .fbz is done with `cnn2snn convert` (accepts quantizeml models). +# +# Usage: +# bash vww_train.sh [DATADIR] +# where the optional DATADIR overrides the default dataset location +# (./data/vw_coco2014_96). + +set -e + +DATADIR="${1:-}" +DATA_ARG=${DATADIR:+-d "$DATADIR"} + +# 1. Build untrained float model +python vww_model.py -s models/akidanet_vww_untrained.h5 + +# 2. Float training (from scratch). Epoch/LR provisional -- confirm on first run. +python vww_train.py -l models/akidanet_vww_untrained.h5 -s models/akidanet_vww.h5 -e 50 -lr 1e-3 $DATA_ARG +python vww_eval.py -l models/akidanet_vww.h5 $DATA_ARG + +# 8-BIT VARIANT (i8/w8/a8) -- PTQ only, no QAT +quantizeml quantize -m models/akidanet_vww.h5 -i 8 -w 8 -a 8 \ + -s models/akidanet_vww_i8_w8_a8.h5 +python vww_eval.py -l models/akidanet_vww_i8_w8_a8.h5 $DATA_ARG + +cnn2snn convert -m models/akidanet_vww_i8_w8_a8.h5 +python vww_eval.py -l models/akidanet_vww_i8_w8_a8.fbz $DATA_ARG + +python vww_benchmark.py -l models/akidanet_vww_i8_w8_a8.fbz $DATA_ARG || true + +# 4-BIT VARIANT (i8/w4/a4) -- QAT only. 4-bit PTQ is a throwaway (_pretmp). +quantizeml quantize -m models/akidanet_vww.h5 -i 8 -w 4 -a 4 \ + -s models/akidanet_vww_i8_w4_a4_pretmp.h5 + +python vww_train.py -l models/akidanet_vww_i8_w4_a4_pretmp.h5 -s models/akidanet_vww_i8_w4_a4_qat.h5 -e 5 -lr 1e-4 $DATA_ARG +python vww_eval.py -l models/akidanet_vww_i8_w4_a4_qat.h5 $DATA_ARG + +cnn2snn convert -m models/akidanet_vww_i8_w4_a4_qat.h5 +python vww_eval.py -l models/akidanet_vww_i8_w4_a4_qat.fbz $DATA_ARG + +python vww_benchmark.py -l models/akidanet_vww_i8_w4_a4_qat.fbz $DATA_ARG || true + +rm -f models/akidanet_vww_i8_w4_a4_pretmp.h5