Add recovery mechanics

[tom-ehh]

Adds recovery mechanics

When configured, Application launches the Recovery worker, which scans the provided stores for root agents (no parent_agent_id) which were interrupted mid run (status: :running). It then a Task which calls Legion.recover/2 and drives them to completion before killing them.

The recovery process is non-blocking when it comes to the application startup.

[Copilot]

Pull request overview

Adds an opt-in startup recovery workflow to automatically drive persisted, interrupted root runs (status: :running, parent_agent_id: nil) to completion without blocking application startup. This extends existing step-persistence and resume capabilities by introducing a dedicated recovery path (Legion.recover/2) and a temporary startup worker (Legion.Recovery).

Changes:

• Introduce Legion.Recovery startup worker and wire it into Legion.Application via :recovery config.
• Add Legion.recover/2, AgentServer.start_monitor/2, and resume/recover execution continuation via persisted conversation_state.execution.
• Update store/state encoding, docs, and tests to treat conversation_state.execution as always present (nil when not checkpointing).

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/legion/store/postgres_test.exs	Updates persisted conversation_state fixtures to include execution: nil.
test/legion/store/postgres_db_test.exs	Same as above for DB-backed store tests.
test/legion/recovery_test.exs	Adds coverage for startup recovery scanning, filtering, and concurrency limiting.
test/legion/application_test.exs	Adds coverage for passing recovery config into Legion.Application.children/0.
test/legion/agent_server_test.exs	Adds coverage for start_monitor/2, resume checkpoints, and recover/2 behavior.
test/integration/step_persistence_test.exs	Aligns assertions with execution being present (as nil) in snapshots.
README.md	Documents :recovery configuration and Legion.recover/2 usage.
mix.exs	Updates source URL and includes Legion.Recovery in docs grouping.
lib/legion/store/postgres.ex	Ensures decoded conversation state always includes execution (default nil).
lib/legion/store/payload.ex	Clarifies payload semantics and makes execution explicit in state type/docs.
lib/legion/store/migration/postgres.ex	Fixes migration docs wording (Legion vs Oban).
lib/legion/store.ex	Updates store contract docs to describe execution as nil or checkpoint map and references recovery.
lib/legion/recovery.ex	New startup worker that lists stores and calls Legion.recover/2 for eligible runs.
lib/legion/executor.ex	Adds optional execution param to run/5 to resume from checkpoints.
lib/legion/application.ex	Exposes children/0 and wires in the Recovery child with Application.fetch_env/2.
lib/legion/agent_server.ex	Adds start_monitor/2, stores/restores execution checkpoints, and runs resume/recover via handle_continue.
lib/legion.ex	Adds recover/2, refactors store lookup into store!/2, and updates resume/2 to use start modes.
CHANGELOG.md	Notes addition of recover/2 and :recovery startup configuration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dimamik]

Looks good!

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

lib/legion/agent_server.ex:52

• The :name option is no longer handled as a GenServer option; it will be treated as a config override and only produce an "Unknown Legion config keys" warning while being ignored. Since :name is documented as removed, it’s better to fail fast with a clear error so callers don’t think they successfully registered a name.

  defp start_args(agent_module, opts) do
    {store, opts} = Keyword.pop(opts, :store)
    {agent_id, opts} = Keyword.pop(opts, :agent_id)

    store = store || Vault.get(:store) || Application.get_env(:legion, :store)

lib/legion/agent_server.ex:153

• resume mode currently only continues execution when the last persisted message has role: "user". Step checkpoints persisted after an eval_result will have the last message as an assistant/user eval-result message, so the resumed agent will never continue and will remain stuck in :running state. Resume should continue when an executor_state checkpoint is present, regardless of the last message role.

  def handle_continue(%{start_mode: :resume, executor_state: executor_state}, state) do
    if match?(%{role: "user"}, List.last(state.messages)) do
      {_reply, state} = do_run(state, executor_state)
      {:noreply, state}
    else

lib/legion/recovery.ex:65

• Task.async_stream/3 results are fully discarded via Stream.run/1, so failed recoveries (e.g. {:error, :already_running} or unexpected exit reasons) are silent. This makes startup recovery failures hard to diagnose and can leave runs stuck as :running without any visibility. Consider forcing the stream and logging non-:ok outcomes (and task exits).

    |> Task.async_stream(
      fn {store, %Payload{agent_id: agent_id}} -> Legion.recover(agent_id, store: store) end,
      max_concurrency: concurrent_request_limit,
      ordered: false,
      timeout: :infinity