Add usage tracking - #13
Merged
Merged
Conversation
…truct; add default save/2 implementations
…at can only be updated together, it now supports updating any combination of columns at once through the Payload wrapper. To achieve this Ecto was added as a dependency, so Postgres could use it for DB operations instead of raw sql queries (main benefit was in the unconstrained save/1 function where we could skip tedious string ops)
Besides looking in the registry (lookup/1) resume/2 also checks whether the process is actually alive using running?/1
Legion now supports emitting live updates from within a single LLM turn. It's configurable through implementing the optional persistence_frequency/0 callback and providing either :turn or :step, default value is :turn and it persists the state at the end of a turn (that does inclde the intra state history, but the refresh rate is slower). When using :step writes also occur after each intermediate eval result or recovorable error to ensure full recoverability is achievable in the future
Change folder structure and make migrations use Ecto
Complete the resume/2 loop by creating a mechanism where not only is an agent's history recovered, but also it actually RESUMES executing, if it was interrupted before completion (switching back to :idle).
Users can now opt in for Legion to automatically drive interrupted (:running without PID) agents to completion.
Stores full ReqLLM.Response.Usage struct (as jsonb[] of each consequent message) instead of just total token count
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands Legion’s persistence and runtime lifecycle features by adding per-request LLM usage tracking to stored conversations, and introducing startup recovery for interrupted root runs. It updates the core executor/agent-server flow to collect and persist usage, and adds a Legion.Recovery worker plus supporting APIs (recover/2, enhanced resume/2) to continue interrupted executions.
Changes:
- Persist ordered
ReqLLM.Response.usageentries per successful LLM request (optionally disabled viaconfig :legion, :track_usage, false), including Postgres storage asjsonb[]. - Add startup recovery (
Legion.Recovery) and a synchronous recovery API (Legion.recover/2) for interrupted root runs. - Extend
Legion.Executor.run/…to return per-turn usage and to resume step checkpoints via anexecutionparameter.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/legion/store/postgres_test.exs | Updates store contract tests for new usage field and execution: nil in conversation snapshots. |
| test/legion/store/postgres_db_test.exs | Adds DB-level assertions for usage stored as jsonb[] and adjusts expectations for string-key reload. |
| test/legion/recovery_test.exs | New tests covering recovery worker scanning stores and enforcing concurrency/eligibility. |
| test/legion/parallel_and_pipeline_test.exs | Adjusts mocked ReqLLM.Response to include usage. |
| test/legion/executor_test.exs | Adds coverage for usage collection behavior and new Executor.run return shape. |
| test/legion/application_test.exs | New tests ensuring recovery config is passed into the supervision tree. |
| test/legion/agent_server_test.exs | Adds usage accumulation tests and resume/recover behavior validation; updates execution snapshot expectations. |
| test/integration/step_persistence_test.exs | Updates integration assertions to expect execution: nil and mocked responses with usage. |
| README.md | Documents usage tracking and startup recovery configuration and semantics. |
| mix.exs | Updates project source URL and includes Legion.Recovery in module grouping metadata. |
| lib/legion/store/postgres.ex | Adds usage column mapping and decodes it into payloads; documents JSONB behavior. |
| lib/legion/store/payload.ex | Extends payload schema/types with usage and makes execution explicit in conversation state. |
| lib/legion/store/migration/postgres/v01.ex | Adds usage column ({:array, :map}) with non-null default. |
| lib/legion/store/migration/postgres.ex | Fixes migration documentation wording to reference Legion (not Oban). |
| lib/legion/store.ex | Documents usage tracking and startup recovery behavior in the store contract docs. |
| lib/legion/recovery.ex | New startup worker that lists stores and concurrently invokes Legion.recover/2. |
| lib/legion/executor.ex | Adds checkpoint-resume support and returns turn_usage; improves structured object extraction error handling. |
| lib/legion/application.ex | Exposes children/0 and conditionally starts Legion.Recovery based on config. |
| lib/legion/agent_server.ex | Restores execution checkpoint + usage into server state; persists usage; adds resume/recover start modes and start_monitor/2. |
| lib/legion.ex | Refactors store lookup and adds recover/2; changes resume semantics to start in :resume mode. |
| CHANGELOG.md | Updates changelog entries to include recovery and usage tracking changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dimamik
approved these changes
Aug 5, 2026
| @@ -0,0 +1,71 @@ | |||
| defmodule Legion.Recovery do | |||
Member
There was a problem hiding this comment.
This PR probably should have been branched off of recovery?
Transfer Legion startup to Oban style supervisor to ensure Recovery
worker and rest of Legion starts AFTER the repo.
```elixir
defmodule MyApp.Application do
use Application
def start(_type, _args) do
children = [
MyApp.Repo,
{Legion, []}
]
Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end
end
```
Instead of Legion autostarting through `application.ex`
tom-ehh
force-pushed
the
feat/add-revive
branch
from
August 10, 2026 14:12
109472b to
2eec501
Compare
Thix fixes an issue where usage which was persisted, and then recovered would be string-keyed while new records would be atom keyed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expands persistence with usage tracking
Usage is collected for each successful LLM interaction and then appended at the end of turn. This means that if there is a crash mid turn, usage will NOT be logged.
Usage is tracked as a list of the ReqLLM.Response.Usage structs and not collapsed into single entry, in postgres it is stored as
jsonb[].There is a place for future expansion for saving more frequently than just at end of turn.