Usage Guide

This guide shows how to run katamaran in both direct CLI mode and Kubernetes Job mode.

Command Overview

katamaran has two modes:

dest: destination-side listener and packet buffering setup
source: source-side migration orchestrator

Build the tool:

make

General form:

katamaran --mode <source|dest> [flags]

Flags

Common flags

Flag	Required	Default	Description
`--mode`	yes	`""`	Migration role: `source` or `dest`
`--qmp`	no	`/run/vc/vm/extra-monitor.sock`	QEMU QMP socket path
`--drive-id`	no	`drive-virtio-disk0`	QEMU block device ID(s), comma-separated for multi-disk migrations
`--shared-storage`	no	`false`	Skip NBD storage mirroring
`--multifd-channels`	no	`4`	Parallel TCP channels for RAM migration (0 to disable)
`--log-format`	no	`text`	Log output format: `text` or `json`
`--log-level`	no	`info`	Log level: `debug`, `info`, `warn`, or `error`
`--version`, `-v`	no	n/a	Show version and exit

Source mode flags

Flag	Required	Default	Description
`--dest-ip`	yes	`""`	Destination node IP
`--vm-ip`	when not in pod mode	`""`	VM pod IP used for route/tunnel cutover
`--pod-name`	alt to --vm-ip+--qmp	`""`	Source pod name; resolver finds sandbox + VM IP at runtime
`--pod-namespace`	with --pod-name	`""`	Source pod namespace
`--emit-cmdline-to`	no	`""`	Capture source QEMU `/proc/<pid>/cmdline` to this path before migration; used by replay-cmdline orchestration. The file is removed when the source run ends so the shared hostPath dir does not accumulate one file per migration
`--tunnel-mode`	no	`ipip`	`ipip`, `gre`, or `none`
`--downtime`	no	`25`	Maximum allowed downtime during VM pause, 1-60000 (ms)
`--auto-downtime`	no	`false`	Auto-calculate downtime based on RTT (overrides `--downtime`)
`--auto-downtime-floor-ms`	no	`0`	Lower bound + overhead for auto downtime; 0 uses the built-in 25 ms floor
`--cni-convergence-delay`	no	`0s`	Keep the source-to-dest tunnel alive after cutover; 0 uses the built-in 5s delay

Destination mode flags

Flag	Required	Default	Description
`--tap`	recommended	`""`	Destination tap interface for `tc sch_plug` buffering
`--tap-netns`	no	`""`	Network namespace path for tap interface (e.g. `/proc/PID/ns/net`)
`--dest-pod-name`	alt to --qmp	`""`	Destination pod name; resolver finds sandbox QMP socket at runtime
`--dest-pod-namespace`	with --dest-pod-name	`""`	Destination pod namespace
`--replay-cmdline`	no	`""`	Path to a captured source QEMU cmdline file. When set, dest spawns its own QEMU with the replayed cmdline + `-incoming defer` (no kata sandbox needed on dest).
`--replay-cmdline-from-pod`	no	`""`	Source pod reference (`<namespace>/<name>`) whose logs contain the captured cmdline marker for in-cluster replay

Direct CLI Usage

1) Destination node (run first)

sudo /usr/local/bin/katamaran \
  --mode dest \
  --qmp /run/vc/vm/<sandbox-id>/extra-monitor.sock \
  --tap tap0_kata

2) Source node

sudo /usr/local/bin/katamaran \
  --mode source \
  --qmp /run/vc/vm/<sandbox-id>/extra-monitor.sock \
  --dest-ip <destination-node-ip> \
  --vm-ip <vm-pod-ip> \
  --tunnel-mode ipip

Shared storage mode (Ceph/NFS)

# destination
sudo /usr/local/bin/katamaran --mode dest --qmp /run/vc/vm/<id>/extra-monitor.sock --tap tap0_kata --shared-storage

# source
sudo /usr/local/bin/katamaran --mode source --qmp /run/vc/vm/<id>/extra-monitor.sock \
  --dest-ip <destination-node-ip> --vm-ip <vm-pod-ip> --shared-storage

GRE mode (cloud VPC networks)

sudo /usr/local/bin/katamaran --mode source --qmp /run/vc/vm/<id>/extra-monitor.sock \
  --dest-ip <destination-node-ip> --vm-ip <vm-pod-ip> --tunnel-mode gre

Tap in a different network namespace

sudo /usr/local/bin/katamaran --mode dest --qmp /run/vc/vm/<id>/extra-monitor.sock \
  --tap tap0_kata --tap-netns /proc/12345/ns/net

Auto-downtime calculation

sudo /usr/local/bin/katamaran --mode source --qmp /run/vc/vm/<id>/extra-monitor.sock \
  --dest-ip <destination-node-ip> --vm-ip <vm-pod-ip> --auto-downtime

When --auto-downtime is set, the source binary measures network RTT to the destination node by sending three ICMP echo requests (the source pod runs privileged, so it can open a raw socket). The downtime limit programmed into QEMU is then rtt × 2 + floor, where the floor defaults to 25 ms so an idle kata-noble VM still converges. Override that floor with --auto-downtime-floor-ms when you need a larger minimum budget. Both the chosen limit and the measured RTT are surfaced:

in the source pod log via the structured marker KATAMARAN_DOWNTIME_LIMIT applied_ms=N rtt_ms=R auto=true,
on the dashboard log line as >>> transferring: downtime limit Nms (auto from Rms RTT) plus a recap on the final succeeded line, and
on the Migration CR as .status.appliedDowntimeMS, .status.rttMS, and .status.autoDowntime.

Kubernetes Job-Based Usage

The repository includes:

internal/orchestrator/templates/job-dest.yaml (canonical destination Job template, embedded into the binaries)
internal/orchestrator/templates/job-source.yaml (canonical source Job template, embedded into the binaries)
deploy/migrate.sh (legacy shell wrapper that renders the templates above)

deploy/migrate.sh renders the canonical templates with envsubst, starts destination job first, waits for readiness, starts source job, then collects logs.

Required inputs

All modes require:

source node name
destination node name
destination node IP
image reference

Legacy explicit-fields mode also requires:

destination tap interface name
source and destination QMP socket paths
VM pod IP

Pod-picker mode requires source pod name + namespace instead of source QMP, VM IP, and tap values; the resolver derives those at runtime. Use --replay-cmdline unless you also provide an existing destination pod or QMP socket.

Example (legacy explicit-fields mode)

deploy/migrate.sh \
  --source-node <source-node-name> \
  --dest-node <dest-node-name> \
  --tap <dest-tap-iface> \
  --qmp-source /run/vc/vm/<src-id>/extra-monitor.sock \
  --qmp-dest /run/vc/vm/<dst-id>/extra-monitor.sock \
  --dest-ip <destination-node-ip> \
  --vm-ip <vm-pod-ip> \
  --image localhost/katamaran:dev \
  --shared-storage \
  --tunnel-mode ipip \
  --downtime 25 \
  --context <kube-context>

Pod-picker mode (recommended)

Skip the manual sandbox/UUID/PID lookup. Pass a source pod name + namespace; the source job's resolver finds the QEMU PID, sandbox UUID, pod IP, and tap netns at runtime. Add --replay-cmdline so the destination job spawns its own QEMU (no kata pod needed on dest).

deploy/migrate.sh \
  --source-node <source-node-name> \
  --dest-node <dest-node-name> \
  --pod-name kata-demo \
  --pod-namespace default \
  --dest-ip <destination-node-ip> \
  --image localhost/katamaran:dev \
  --shared-storage \
  --replay-cmdline

Same flow from the Dashboard:

curl -sS -X POST http://127.0.0.1:8080/api/migrate \
  -d source_pod_namespace=default \
  -d source_pod_name=kata-demo \
  -d dest_node=<dest-node-name> \
  -d image=localhost/katamaran:dev \
  -d downtime=25 \
  -d shared_storage=true \
  -d replay_cmdline=true

See cmd/katamaran-dashboard/README.md for the full UI flow + screenshots.

Show orchestrator help:

deploy/migrate.sh --help

Structured CLI: `katamaran-orchestrator`

bin/katamaran-orchestrator is a thin wrapper around the same Go orchestrator package the dashboard uses. It reads a single orchestrator.Request JSON object on stdin, submits Jobs through client-go, and emits newline-delimited JSON StatusUpdate events on stdout. Exit codes: 0 on success, 1 on migration failure or runtime error, 2 on input error, 130 on signal-induced shutdown.

Useful for CI pipelines and local automation that need structured status instead of parsing migrate.sh output.

echo '{
  "SourceNode":"worker-a",
  "DestNode":"worker-b",
  "DestIP":"10.0.0.20",
  "Image":"localhost/katamaran:dev",
  "SourcePod":{"Namespace":"default","Name":"kata-demo"},
  "DestPod":{"Namespace":"default","Name":"kata-dest-shell"},
  "SharedStorage":true,
  "ReplayCmdline":true
}' | bin/katamaran-orchestrator

Sample stdout (one line per StatusUpdate; phases run submitted → dest-starting → src-starting → transferring → cutover → succeeded or failed, with periodic transferring updates in between):

{"id":"a1b2c3...","phase":"submitted","time":"2026-04-27T05:25:32.243Z"}
{"id":"a1b2c3...","phase":"dest-starting","time":"2026-04-27T05:25:33.012Z"}
{"id":"a1b2c3...","phase":"src-starting","time":"2026-04-27T05:25:40.881Z"}
{"id":"a1b2c3...","phase":"transferring","msg":"downtime limit applied: 25ms","applied_downtime_ms":25,"time":"2026-04-27T05:25:47.104Z"}
{"id":"a1b2c3...","phase":"transferring","msg":"status=active","ram_transferred":536870912,"ram_total":2147483648,"time":"2026-04-27T05:25:52.377Z"}
{"id":"a1b2c3...","phase":"cutover","msg":"source reported cutover","time":"2026-04-27T05:25:57.901Z"}
{"id":"a1b2c3...","phase":"succeeded","downtime_ms":14,"ram_transferred":2147483648,"ram_total":2147483648,"applied_downtime_ms":25,"time":"2026-04-27T05:25:58.314Z"}

The same Go package (internal/orchestrator) backs the dashboard's POST /api/migrate handler: anything callable from the dashboard is callable from the CLI and vice versa.

Environment Variables

Variable	Description
`KATAMARAN_MIGRATION_ID`	Correlation ID added to all log entries (set by the dashboard, controller, or job wrapper)

Validation Rules

Source mode requires --dest-ip plus either --vm-ip or --pod-name + --pod-namespace
CLI pod mode cannot be combined with explicit --qmp or --vm-ip; the resolver derives both at runtime
When --vm-ip is supplied explicitly, --dest-ip and --vm-ip must be the same address family
--tunnel-mode must be ipip, gre, or none
--downtime must be between 1 and 60000
Source-only flags in dest mode (and vice versa) produce warnings
Job orchestration requires --tap for zero-drop buffering path

Operational Notes

NBD storage mirror port: 10809
RAM migration port: 4444
--tap is critical for sch_plug buffering during STOP→RESUME cutover
On failure, deploy/migrate.sh keeps jobs for forensic debugging output

Troubleshooting

invalid --tunnel-mode
- use ipip, gre, or none
migration did not complete
- check logs from source and destination jobs/services

For the full error reference (dialing QMP socket, failed to add plug qdisc, storage/RAM timeouts, etc.), see the Troubleshooting table in the Testing Guide.

FilesExpand file tree

USAGE.md

Latest commit

History