Need a troubleshooting guide for actor developers

[git286]

There is no troubleshooting documentation for actor developers.

docs/observability.md is good, but it answers a different question. It is capability-oriented: "here is the logging model, here are the metrics we emit, here is how to reach Jaeger." A developer whose actor is stuck in STATUS_SUSPENDING at 2am does not need a model; they need"start here, run this, and here is what the output means."

This issue asks for a symptom-driven guide: start from what the developer observes, route them to the right signal and tool, and say what to do with the answer.

The second deliverable matters as much as the first. Writing this guide honestly means walking every path end-to-end, and several paths dead-end today. Those dead ends become a gap list derived from real user tasks rather than from architecture opinion, which is a much better basis for prioritizing observability work than a wish list.

---

Who this is for

Worth stating explicitly in the doc, because the answers differ:

	Actor developer	Substrate operator
Owns	their application code and its ActorTemplate	the fleet, worker pools, nodes
Cares about	"why is my actor misbehaving?"	"why is the platform degraded?"
Cluster access	limited or none	full
Placement (pod/node)	can't act on it	primary lever

This guide is for the first column. An operator runbook is a separate document, and conflating them is how observability docs become unusable for both.

---

What actor developers actually have today

Grounding the guide in what exists, not what we intend to build:

Tool / signal	Command or location	Reality check
Actor logs (live)	kubectl ate logs actors <name> -a <atespace> [-f]	Streams the worker pod's log, client-side filtered to the actor by ate.dev/actor_name + ate.dev/actor_atespace (logs_actors.go:309-346)
Actor logs (history)	logging backend, query on labels."ate.dev/actor_name"	Four documented query dimensions in docs/observability.md
Log metadata	ate.dev/actor_{name,atespace,uid}, ate.dev/actor_template_{name,namespace}, ate.dev/container_name	Injected by Substrate into container logs
Actor status	kubectl ate get actors
Resource usage	kubectl ate top workers	Workers only. There is no top actors — top.go has exactly one subcommand
System metrics	rpc.server.call.duration, atenet.router.route.duration, atelet.snapshot.size	Three total, all platform-scoped; none carries actor identity
Traces	--trace on CLI commands → Jaeger (kind) / Cloud Trace (GKE)	Control-plane operations
Local UIs	kubectl port-forward -n otel-system svc/prometheus 9090 / svc/jaeger 16686	kind only
Dashboards	tools/setup-gcp/dashboards/	Three, all system-facing: e2e-latency, grpc, snapshot. None actor-facing. GCP only

Proposed structure — symptom first

One section per symptom, each with the same shape: what you'd see → which signal answers it → the command → how to read the output → what to do next.

Symptom	Primary signal today	Status
Actor won't start / stuck in a status	get actors, actor logs	works
kubectl ate logs says "not currently running"	—	needs explaining, not fixing — see below
Can't find logs from before a resume	backend query on ate.dev/actor_uid	works, but only if a backend is wired up
Actor is slow to first response	—	gap: cold-start (resume) vs. warm latency is not separable by an actor developer
Actor using too much memory / OOM	—	gap (#550)
Custom metrics/traces not appearing	—	gap (#563, #569)
Custom telemetry has gaps around suspend/resume	—	known bug (#503)
Actor can't reach an external service	—	gap: no actor-visible egress/DNS signal
Requests to the actor fail or hang	atenet.router.route.duration	operator-only, and not collected at all in kind