Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion apps/cli/src/commands/results/remote.ts
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,6 @@ export async function loadNormalizedResultsConfig(
}),
...(project.results.repoPath !== undefined && { repo_path: project.results.repoPath }),
...(project.results.branch !== undefined && { branch: project.results.branch }),
...(project.results.remote !== undefined && { remote: project.results.remote }),
...(project.results.path !== undefined && { path: project.results.path }),
...((project.results.sync?.autoPush !== undefined ||
project.results.sync?.pushConflictPolicy !== undefined) && {
Expand Down
7 changes: 2 additions & 5 deletions apps/web/src/content/docs/docs/tools/dashboard.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -291,10 +291,9 @@ projects:
branch: agentv/results/v1
sync:
auto_push: false
push_conflict_policy: block
```

`results.repo.remote` is the Git remote URL used when AgentV creates a fresh results checkout, and the intended remote URL for portable project config. `results.repo.path: .` stores completed run artifacts on a dedicated branch of the source repository without checking out that branch in the source worktree. AgentV does not add or rewrite remotes inside an existing checkout; the checkout's existing `origin` must already point at the repository you want to fetch and push. When `results.repo.remote` is omitted, `results.repo.path` means an existing local Git checkout whose object database and refs AgentV should write to, and the branch defaults to `agentv/results/v1`. AgentV creates the branch automatically on first publish and commits only AgentV result paths into it. `sync.auto_push: false` keeps the result commit local; set it to `true` to push the branch best-effort after each completed run. For CI workflows where a push failure should fail the command after local artifacts are written, invoke the run with `agentv eval run --results-require-push`. `sync.push_conflict_policy` defaults to `block`; the removed `backup_and_force_push` value is rejected with migration guidance because AgentV never force-pushes result branches. Non-fast-forward result branch pushes are auto-merged with artifact-aware Git merge drivers and pushed as a fast-forward, so the canonical results branch is never force-pushed or rewritten. Genuine overlay conflicts route to a timestamped temp branch plus a GitHub compare link for a human merge instead.
`results.repo.remote` is the Git remote URL used when AgentV creates a fresh results checkout, and the intended remote URL for portable project config. `results.repo.path: .` stores completed run artifacts on a dedicated branch of the source repository without checking out that branch in the source worktree. AgentV does not add or rewrite remotes inside an existing checkout; the checkout's existing `origin` must already point at the repository you want to fetch and push. When `results.repo.remote` is omitted, `results.repo.path` means an existing local Git checkout whose object database and refs AgentV should write to, and the branch defaults to `agentv/results/v1`. AgentV creates the branch automatically on first publish and commits only AgentV result paths into it. `sync.auto_push: false` keeps the result commit local; set it to `true` to push the branch best-effort after each completed run. For CI workflows where a push failure should fail the command after local artifacts are written, invoke the run with `agentv eval run --results-require-push`. The default conflict behavior is block-and-ask; the removed `backup_and_force_push` value is rejected with migration guidance because AgentV never force-pushes result branches. Non-fast-forward result branch pushes are auto-merged with artifact-aware Git merge drivers and pushed as a fast-forward, so the canonical results branch is never force-pushed or rewritten. Genuine overlay conflicts route to a timestamped temp branch plus a GitHub compare link for a human merge instead.

For a separate results repository, use `results.repo.remote` and an optional managed clone `results.repo.path`:

Expand All @@ -311,7 +310,6 @@ projects:
path: /home/entity/projects/EntityProcess/agentv-examples-eval-results
sync:
auto_push: true
push_conflict_policy: block
```

`results.repo.remote` is the Git remote URL used for clone and push operations, so use HTTPS when credentials are HTTP-token based and SSH when the runtime has SSH keys configured. When `results.repo.remote` is set and `results.repo.path` is missing or empty, AgentV creates that filesystem location with `git clone`. If `results.repo.path` already points at a Git checkout, AgentV treats that checkout's remotes as user-owned state: it fetches and pushes using the existing configured remote name (`origin` by default), but it does not run `git remote add` or `git remote set-url`. Omit `results.repo.remote` only when `results.repo.path` points at an already-existing local checkout such as `.`.
Expand All @@ -326,7 +324,6 @@ results:
branch: agentv/results/v1
sync:
auto_push: false
push_conflict_policy: block
```

Project-local `.agentv/config.yaml` is for portable eval defaults such as `execution`, `eval_patterns`, and `dashboard`. Do not put `projects` in project-local config; AgentV warns and ignores it there. `results_by_project` is deprecated; use `projects[].results` in `$AGENTV_HOME/config.yaml`.
Expand Down Expand Up @@ -374,7 +371,7 @@ projects:
auto_push: true
```

Current flat fields (`path`, `repo_url`, `ref`, `results.repo_url`, `results.repo_path`, `results.branch`, `results.remote`, and `results.path`) still load with migration warnings and are written back in nested form the next time AgentV saves the project registry. Older removed fields (`source`, `repository`, `results.mode`, `results.repo` as a string, `results.repository`, `results.local_path`, and `results.auto_push`) fail validation with migration guidance.
Current flat fields (`path`, `repo_url`, `ref`, `results.repo_url`, `results.repo_path`, `results.branch`, and `results.path`) still load with migration warnings and are written back in nested form the next time AgentV saves the project registry. Removed fields (`source`, `repository`, `results.mode`, `results.repo` as a string, `results.remote`, `results.repository`, `results.local_path`, and `results.auto_push`) fail validation with migration guidance.

Use project-level **Sync Project** as the results exchange workflow. It handles pulled remote runs, locally edited metadata, dirty state, and blocked conflict feedback in one project-scoped action.

Expand Down
2 changes: 1 addition & 1 deletion apps/web/src/content/docs/docs/tools/results.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ The CLI contract is deliberately narrow: `agentv results` manages local result a

Use these supported remote workflows instead:

- **Automatic publishing:** configure `projects[].results` or top-level `results`; new `agentv eval` and `agentv pipeline bench` runs publish completed artifacts after the run completes. Use `repo.remote` with `repo.path: .` and `repo.branch: agentv/results/v1` to store primary result records on a dedicated branch of the source repo. AgentV never adds or rewrites remotes in an existing checkout; that checkout's `origin` must already point at the repository you want to fetch and push. AgentV reserves `agentv/results/v1` for primary results and `agentv/artifacts/v1` for heavy artifact payloads. When `index.jsonl` rows point trace or transcript payloads at `agentv/artifacts/v1`, automatic publishing stores those bytes on that artifact branch in the same remote and publishes pointer keys such as `runs/<run-path>/<pointer.path>`. The configured results branch remains the metadata/control plane (`index.jsonl`, `summary.json`, tags, and pointers) instead of duplicating canonical trace/transcript payload bodies. Local pre-publish run workspaces can still contain those files beside the manifest so local tools keep working. Mutable run tags are stored as `tags.json` with a `tag_revision`; there is no tag event log in the normal results layout. `results.repo.path` without `results.repo.remote` means an existing local Git checkout, distinct from `workspace.repos[].repo`, which is a portable repository identity. Set `sync.auto_push: true` to push after publish. In CI, use `agentv eval run --results-require-push` when push failures should fail that invocation after local artifacts are written. Non-fast-forward result branch pushes never force-push: AgentV auto-merges concurrent remote writes with artifact-aware Git merge drivers (a union driver for the append-only `index.jsonl`, a JSON-union driver for tag and feedback overlays) and pushes the merge as a fast-forward, and routes a genuine overlay conflict to a timestamped `agentv/results-sync/...` branch plus a GitHub compare/PR link for a human merge. The removed `sync.push_conflict_policy: backup_and_force_push` value is rejected with migration guidance; remove the field or set it to `block`. While an eval is still running, [WIP checkpoints](/docs/tools/wip-checkpoints/) can keep partial run output durable on `agentv/wip/...` branches when auto-push is enabled.
- **Automatic publishing:** configure `projects[].results` or top-level `results`; new `agentv eval` and `agentv pipeline bench` runs publish completed artifacts after the run completes. Use `repo.remote` with `repo.path: .` and `repo.branch: agentv/results/v1` to store primary result records on a dedicated branch of the source repo. AgentV never adds or rewrites remotes in an existing checkout; that checkout's `origin` must already point at the repository you want to fetch and push. AgentV reserves `agentv/results/v1` for primary results and `agentv/artifacts/v1` for heavy artifact payloads. When `index.jsonl` rows point trace or transcript payloads at `agentv/artifacts/v1`, automatic publishing stores those bytes on that artifact branch in the same remote and publishes pointer keys such as `runs/<run-path>/<pointer.path>`. The configured results branch remains the metadata/control plane (`index.jsonl`, `summary.json`, tags, and pointers) instead of duplicating canonical trace/transcript payload bodies. Local pre-publish run workspaces can still contain those files beside the manifest so local tools keep working. Mutable run tags are stored as `tags.json` with a `tag_revision`; there is no tag event log in the normal results layout. `results.repo.path` without `results.repo.remote` means an existing local Git checkout, distinct from `workspace.repos[].repo`, which is a portable repository identity. Set `sync.auto_push: true` to push after publish. In CI, use `agentv eval run --results-require-push` when push failures should fail that invocation after local artifacts are written. Non-fast-forward result branch pushes never force-push: AgentV auto-merges concurrent remote writes with artifact-aware Git merge drivers (a union driver for the append-only `index.jsonl`, a JSON-union driver for tag and feedback overlays) and pushes the merge as a fast-forward, and routes a genuine overlay conflict to a timestamped `agentv/results-sync/...` branch plus a GitHub compare/PR link for a human merge. The removed `sync.push_conflict_policy: backup_and_force_push` value is rejected with migration guidance; remove the field and rely on the default block-and-ask behavior. While an eval is still running, [WIP checkpoints](/docs/tools/wip-checkpoints/) can keep partial run output durable on `agentv/wip/...` branches when auto-push is enabled.
- **Manual Dashboard sync:** run `agentv dashboard`, open the project, and use **Sync Project**.
- **Manual API sync:** while Dashboard is running, call `GET /api/projects/:projectId/remote/status` or `POST /api/projects/:projectId/remote/sync` for project-scoped automation. Single-project sessions also expose `GET /api/remote/status` and `POST /api/remote/sync`.
- **Git escape hatch:** for advanced recovery, inspect or repair the configured `projects[].results.repo.path` clone with `git` directly, then sync again.
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ force push.
`backup_and_force_push` is **hard-deprecated/removed** from supported config:
the value shipped only on the `next` npm tag before stable release, so AgentV
now rejects it with migration guidance instead of preserving a compatibility
alias. Remove the field or set `sync.push_conflict_policy: block`; AgentV never
force-pushes result branches.
alias. Remove the field and rely on the default block-and-ask behavior; AgentV
never force-pushes result branches.

## Consequences

Expand Down
24 changes: 13 additions & 11 deletions docs/plans/git-native-results.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,30 +20,32 @@ The configured results branch tree IS the index. `git ls-tree -r <storage-ref> -

### Storage

- `results.repo_path` points at an existing local Git checkout whose object database and refs AgentV may write to. Use `repo_path: .` to store results in a dedicated branch of the source repo without checking that branch out in the source worktree.
- `results.repo_url` points at a remote results repository. AgentV manages a local clone at `results.path`; omit `path` to use the default AgentV data dir.
- `results.branch` is the storage branch. `repo_path` configs default to `agentv/results/v1`.
- `results.repo.path` points at an existing local Git checkout whose object database and refs AgentV may write to. Use `path: .` to store results in a dedicated branch of the source repo without checking that branch out in the source worktree.
- `results.repo.remote` points at the portable Git endpoint URL. AgentV manages a local clone at `results.repo.path` when the path is separate or omitted.
- `results.repo.branch` is the storage branch. Existing-checkout configs default to `agentv/results/v1`.
- Local `.agentv/results/runs/` remains the active run workspace for local Dashboard, resume, and rerun flows. Publishing copies completed run artifacts into the branch-backed store under `runs/**` (the branch name already namespaces results, so no redundant `.agentv/results/` prefix on the branch). Editable tag overlays live alongside under `metadata/runs/**`.

```yaml
# Existing checkout, usually the eval source repo.
results:
repo_path: .
branch: agentv/results/v1
remote: origin
repo:
remote: https://github.com/OWNER/REPO.git
path: .
branch: agentv/results/v1
sync:
auto_push: true
auto_push: false

# Separate results repository.
results:
repo_url: git@github.com:myorg/eval-results.git
branch: agentv/results/v1
path: ~/data/agentv-results
repo:
remote: git@github.com:myorg/eval-results.git
branch: agentv/results/v1
path: ~/data/agentv-results
sync:
auto_push: true
```

The field is intentionally `repo_path`, not `repo`: `workspace.repos[].repo` is a portable repository identity, while `results.repo_path` is a filesystem path to an already-existing local checkout.
The nested `results.repo.path` field is a filesystem path to an existing local checkout or managed clone location; `workspace.repos[].repo` remains a separate portable repository identity.

### Writes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,15 +74,7 @@ results:

**Avoid making a checked-in config depend on a local alias:**

```yaml
results:
repo:
remote: origin
path: .agentv/remotes/agentv-private
branch: agentv/results/v1
```

If a local alias override must exist for backward compatibility, keep it on an explicitly advanced or legacy option, not on the ordinary project config contract.
Do not set `results.repo.remote` to a local alias such as `origin`. If AgentV is operating on an existing checkout, it may still use that checkout's local alias internally, but the persisted config should either use the endpoint URL or omit `results.repo.remote`.

## Related

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,8 @@ release channels:
especially when preserving the surface would encode a dangerous or misleading
contract.

For removed config values, make the correction explicit:

```yaml
results:
sync:
# Remove unsupported aliases and use the stable default.
push_conflict_policy: block
```
For removed config values, make the correction explicit: delete the unsupported
field and rely on the stable default behavior when no replacement is needed.

If existing local registries or generated config may contain the removed value,
either reject it with migration guidance or drop it during a registry migration
Expand Down
15 changes: 6 additions & 9 deletions packages/core/src/evaluation/loaders/config-loader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ export type ResultsConfig = {
readonly repo_path?: string;
/** Optional remote branch used as the canonical git-backed results store. */
readonly branch?: string;
/** Runtime-only local Git remote-name override. Persistent config should not set this. */
readonly remote?: string;
/** Local filesystem path for the results clone. Optional; defaults to ~/.agentv/results/<slug>/. */
readonly path?: string;
Expand Down Expand Up @@ -685,7 +686,6 @@ type NestedResultsRepoConfig = {
readonly repo_url?: string;
readonly repo_path?: string;
readonly branch?: string;
readonly remote?: string;
readonly path?: string;
};

Expand Down Expand Up @@ -811,15 +811,13 @@ export function parseResultsConfig(raw: unknown, configPath: string): ResultsCon
branch = obj.branch.trim();
}

let remote: string | undefined;
if (nestedRepo?.remote !== undefined) {
remote = nestedRepo.remote;
} else if (obj.remote !== undefined) {
if (typeof obj.remote !== 'string' || obj.remote.trim().length === 0) {
logWarning(`Invalid results.remote in ${configPath}, expected non-empty string`);
if (obj.remote !== undefined) {
if (!hasNestedRepo) {
logWarning(
`results.remote in ${configPath} is no longer supported in persistent config. Use results.repo.remote for a portable Git endpoint URL, or omit it and let AgentV use the local checkout remote alias internally.`,
);
return undefined;
}
remote = obj.remote.trim();
}

let resultsPath: string | undefined;
Expand Down Expand Up @@ -895,7 +893,6 @@ export function parseResultsConfig(raw: unknown, configPath: string): ResultsCon
...(repoUrl && { repo_url: repoUrl }),
...(repoPath && { repo_path: repoPath }),
...(branch !== undefined && { branch }),
...(remote !== undefined && { remote }),
...(resultsPath !== undefined && { path: resultsPath }),
...(typeof obj.auto_push === 'boolean' && { auto_push: obj.auto_push }),
...(sync && { sync }),
Expand Down
16 changes: 2 additions & 14 deletions packages/core/src/evaluation/validation/config-validator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -530,15 +530,12 @@ function validateFlatResultsRepoConfig(
);
}

if (
resultsRecord.remote !== undefined &&
(typeof resultsRecord.remote !== 'string' || resultsRecord.remote.trim().length === 0)
) {
if (resultsRecord.remote !== undefined) {
addError(
errors,
filePath,
`${location}.remote`,
`Field '${location}.remote' must be a non-empty string`,
`Field '${location}.remote' was removed from persistent config because it was a local Git remote-name alias. Use '${location}.repo.remote' for the portable Git endpoint URL, or omit it and let AgentV use the checkout remote alias internally.`,
);
}

Expand Down Expand Up @@ -590,15 +587,6 @@ function warnFlatResultsMigration(
);
}
}

if (resultsRecord.remote !== undefined) {
addWarning(
errors,
filePath,
`${location}.remote`,
`Field '${location}.remote' is a legacy local Git remote-name override. Prefer omitting it; nested '${location}.repo.remote' is the portable Git remote URL and AgentV manages local aliases automatically.`,
);
}
}

function validateResultsConfigBody(
Expand Down
Loading
Loading