Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ on-disk formats are versioned and refuse to open when newer than the build under

### Fixed

- Automatic task completion is fenced by the observed execution generation. A delayed result from an older run cannot complete or cancel a newer attempt of the same task.

- Automatic task cancellation now compares the observed attempt policy in the server transaction. Changing a limit during execution cannot be overwritten by a stale cancellation decision.

- Integration rechecks the proposal head and open state together with the stream inside its commit transaction. A proposal updated or closed after candidate computation aborts the old operation, including the already-integrated path.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ This is an early public beta: [status and limits](#status-and-limits) are below,

**Development (`main`):** operators can disable or re-enable new runs for an individual agent from the browser or `ag agent disable NAME --remote origin` (`enable` to restore admission). Existing sessions, keys and history are retained. This is not included in the beta.5 archives; see [ADR-0026](docs/adr/0026-agent-run-admission.md).

**Development core:** Tasks can carry a server-enforced attempt limit. `ag task attempts TASK --max 5 --expected 3` lets an operator extend an exhausted task without losing its history; restart recovery uses durable run records. Zero keeps the existing repository retry policy. Concurrent operator policy changes are protected from stale automatic cancellation. [Execution limits](docs/adr/0033-task-attempt-limits.md).
**Development core:** Tasks can carry a server-enforced attempt limit. `ag task attempts TASK --max 5 --expected 3` lets an operator extend an exhausted task without losing its history; restart recovery uses durable run records. Zero keeps the existing repository retry policy. Concurrent operator policy changes are protected from stale automatic cancellation, and late results cannot finish a newer task attempt. [Execution limits](docs/adr/0033-task-attempt-limits.md).

**Development upgrade:** persistent follow-up task links and durable task execution limits require metadata schema 8. Back up before opening existing repositories with this development build; rollback uses the pre-upgrade backup. [Source-link semantics](docs/adr/0027-task-source-links.md).

Expand Down
2 changes: 1 addition & 1 deletion README.ru.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ AgentGit — система контроля версий для репозит

**В разработке (`main`):** оператор может отключить или снова разрешить новые запуски отдельного агента из браузера или через `ag agent disable NAME --remote origin` (`enable` для включения). Ключ, история и работающие сессии сохраняются. В архивы beta.5 эта функция не включена; [ADR-0026](docs/adr/0026-agent-run-admission.md).

**Ядро в разработке:** Сервер ограничивает число запусков каждой задачи, сохраняя счётчик через историю запусков при перезапуске. Оператор может продолжить исчерпанную задачу командой `ag task attempts TASK --max 5 --expected 3`, не теряя её историю. Нулевой лимит сохраняет прежнюю политику повторов репозитория. Изменение лимита во время запуска защищено от отмены по устаревшей политике. [Лимиты исполнения](docs/adr/0033-task-attempt-limits.md).
**Ядро в разработке:** Сервер ограничивает число запусков каждой задачи, сохраняя счётчик через историю запусков при перезапуске. Оператор может продолжить исчерпанную задачу командой `ag task attempts TASK --max 5 --expected 3`, не теряя её историю. Нулевой лимит сохраняет прежнюю политику повторов репозитория. Изменение лимита во время запуска защищено от отмены по устаревшей политике; запоздавший результат не завершит более новую попытку. [Лимиты исполнения](docs/adr/0033-task-attempt-limits.md).

**Обновление разработки:** связи задач-продолжений и постоянные лимиты исполнения используют схему метаданных 8. Перед запуском этой сборки на существующих данных сделайте резервную копию; для отката восстановите её. [Семантика связей](docs/adr/0027-task-source-links.md).

Expand Down
2 changes: 2 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,3 +157,5 @@ hashing, merge semantics or the publication protocol need an ADR ([CONTRIBUTING]
Merged in [PR #116](https://github.com/Paffin/AgentGit/pull/116): server-transactional attempt admission, operator extension with optimistic concurrency, CLI/HTTP/gRPC/SDK contracts, and orchestrator restart recovery. Metadata schema 8 preserves existing task/run history. Evidence: agent coordination, real-server executor recovery, HTTP permissions/idempotency, CLI and SQLite/PostgreSQL upgrade scenarios. Web UI displays exhaustion; editing the limit in the browser remains follow-up work.

Automatic cancellation also carries an expected attempt policy. Concurrent operator changes survive stale session decisions, and reconciliation reloads durable history (ADR0034).

Task completion is also bound to its observed execution generation. A late executor or stale scheduler cannot finish a newer attempt; capability negotiation protects older-server deployments (ADR0035).
6 changes: 6 additions & 0 deletions apps/web/src/api/schema.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2541,6 +2541,12 @@ export interface components {
* log as an operator's, with the owner it overrode.
*/
FinishTaskRequest: {
/**
* Format: int64
* @description Reject a stale completion when another run was admitted after this task snapshot.
* @default null
*/
expected_attempts_started: number | null
/**
* Format: int32
* @description Optional execution-policy precondition, checked atomically with the state change.
Expand Down
52 changes: 45 additions & 7 deletions crates/ag-agent/src/tasks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -257,25 +257,65 @@ pub(crate) fn claim_in_tx(tx: &Tx<'_>, task: &str, owner: &str) -> Result<(), Me
/// (`agent:bot#run_…`) may only finish the task *that run* claimed - another run of the same agent has no say in
/// it. Use [`finish_task_as_operator`] for an operator that overrides the owner.
pub fn finish_task(meta: &Meta, task: &str, state: &str, actor: &str) -> Result<(), AgentError> {
finish_task_with_limit(meta, task, state, actor, false, None)
finish_task_checked(meta, task, state, actor, false, TaskFinishPreconditions::default())
}

/// Like [`finish_task`] for an operator: the task is finished whoever owns it. The audit entry says that it was
/// an operator's doing and whose task it overrode, so the log never shows an owner finishing what an operator
/// finished for it.
pub fn finish_task_as_operator(meta: &Meta, task: &str, state: &str, actor: &str) -> Result<(), AgentError> {
finish_task_with_limit(meta, task, state, actor, true, None)
finish_task_checked(meta, task, state, actor, true, TaskFinishPreconditions::default())
}

/// Finishes only if the optional execution-policy precondition still matches.
/// The check shares the task transition transaction; zero is a meaningful expected limit.
/// Finishes with an optional retry-policy comparison. Prefer the full preconditions
/// when completing work from a particular execution attempt.
pub fn finish_task_with_limit(
meta: &Meta,
task: &str,
state: &str,
actor: &str,
operator: bool,
expected_max_attempts: Option<u32>,
) -> Result<(), AgentError> {
finish_task_checked(
meta,
task,
state,
actor,
operator,
TaskFinishPreconditions { max_attempts: expected_max_attempts, attempts_started: None },
)
}

/// Observed execution state for a conditional task transition.
#[derive(Clone, Copy, Debug, Default)]
pub struct TaskFinishPreconditions {
/// Expected configured limit, including an explicitly unset limit of zero.
pub max_attempts: Option<u32>,
/// Expected durable count of admitted runs; a new run invalidates this snapshot.
pub attempts_started: Option<u64>,
}

impl TaskFinishPreconditions {
fn check(self, task: &TaskRow) -> Result<(), MetaError> {
if self.max_attempts.is_some_and(|expected| task.max_attempts != expected)
|| self.attempts_started.is_some_and(|expected| task.attempts_started != expected)
{
return Err(MetaError::Invalid("task execution changed; reload before finishing".into()));
}
Ok(())
}
}

/// Finishes only if the observed execution generation and policy still match.
/// Preconditions share the task transition transaction; zero remains meaningful.
pub fn finish_task_checked(
meta: &Meta,
task: &str,
state: &str,
actor: &str,
operator: bool,
expected: TaskFinishPreconditions,
) -> Result<(), AgentError> {
if !["done", "cancelled"].contains(&state) {
return Err(AgentError::Invalid("state must be 'done' or 'cancelled'".into()));
Expand All @@ -286,9 +326,7 @@ pub fn finish_task_with_limit(
};
meta.write(|tx| {
let t = tx.task(task)?.ok_or_else(|| MetaError::NotFound(format!("task '{task}'")))?;
if expected_max_attempts.is_some_and(|expected| t.max_attempts != expected) {
return Err(MetaError::Invalid("task attempt limit changed; reload before finishing".into()));
}
expected.check(&t)?;
let mut details = json!({"state": state});
match t.owner.as_ref() {
Some(o) if operator => {
Expand Down
19 changes: 17 additions & 2 deletions crates/ag-agent/tests/agents.rs
Original file line number Diff line number Diff line change
Expand Up @@ -827,10 +827,25 @@ fn stale_retry_policy_cannot_cancel_a_task_or_append_a_finish_audit() {
let task = tasks::create_task(&e.meta, "Preserve operator policy", "", &[], "user:creator").unwrap();
e.meta.write(|tx| tasks::set_attempt_limit_in(tx, &task.id, 2, 0, "user:operator", true)).unwrap();
let before = e.meta.read(|tx| tx.audit_list(None, 100)).unwrap().len();
let stale = tasks::finish_task_with_limit(&e.meta, &task.id, "cancelled", "user:operator", true, Some(0));
let stale = tasks::finish_task_checked(
&e.meta,
&task.id,
"cancelled",
"user:operator",
true,
tasks::TaskFinishPreconditions { max_attempts: Some(0), ..Default::default() },
);
assert!(matches!(stale, Err(AgentError::Conflict(_))));
assert_eq!(e.meta.read(|tx| tx.task(&task.id)).unwrap().unwrap().state, "open");
assert_eq!(e.meta.read(|tx| tx.audit_list(None, 100)).unwrap().len(), before);
tasks::finish_task_with_limit(&e.meta, &task.id, "cancelled", "user:operator", true, Some(2)).unwrap();
tasks::finish_task_checked(
&e.meta,
&task.id,
"cancelled",
"user:operator",
true,
tasks::TaskFinishPreconditions { max_attempts: Some(2), ..Default::default() },
)
.unwrap();
assert_eq!(e.meta.read(|tx| tx.task(&task.id)).unwrap().unwrap().state, "cancelled");
}
2 changes: 2 additions & 0 deletions crates/ag-protocol/proto/agwp/v1/agwp.proto
Original file line number Diff line number Diff line change
Expand Up @@ -1227,6 +1227,8 @@ message FinishTaskRequest {
// Optional execution-policy precondition, checked atomically with the state change.
// Explicit zero matches an unset limit; omission preserves ordinary manual completion.
optional uint32 expected_max_attempts = 5;
// Reject a stale completion when another run was admitted after this task snapshot.
optional uint64 expected_attempts_started = 6;
}

message Lease {
Expand Down
4 changes: 4 additions & 0 deletions crates/ag-protocol/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,9 @@ pub mod features {
/// Durable task-local attempt limits and optimistic policy updates.
pub const TASK_ATTEMPT_LIMITS: &str = "tasks.attempt_limits";

/// Task completion compares both the execution generation and retry policy.
pub const TASK_FINISH_PRECONDITIONS: &str = "tasks.finish_preconditions";

/// Every feature this implementation knows.
pub const ALL: &[&str] = &[
EVENTS_REPLAY,
Expand All @@ -113,6 +116,7 @@ pub mod features {
TASK_LINEAGE,
TASK_EDIT,
TASK_ATTEMPT_LIMITS,
TASK_FINISH_PRECONDITIONS,
RUN_OUTPUT,
TASK_ASSIGNMENT,
EXECUTOR_PRESENCE,
Expand Down
11 changes: 8 additions & 3 deletions crates/ag-server/src/api/agents.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ use ag_agent::leases::{acquire_with, extend, release};
use ag_agent::predict::predict;
use ag_agent::provenance::{ProvenanceRecord, SCHEMA, attach_signed, verify};
use ag_agent::registry::{finish_run_as, heartbeat, reap_runs, register_agent_with_key, start_run_with};
use ag_agent::tasks::{add_dependency, create_task_with, finish_task_with_limit, unfinished_dependencies};
use ag_agent::tasks::{
TaskFinishPreconditions, add_dependency, create_task_with, finish_task_checked, unfinished_dependencies,
};
use ag_crypto::{PublicKey, Signature};
use ag_protocol::v1;
use ag_storage::meta::{MetaError, TaskRow, Tx};
Expand Down Expand Up @@ -761,13 +763,16 @@ impl App {
let repo_name = require(&req.repo, "repo")?.to_string();
let operator = self.authorize(call, &repo_name, Cap::AgentOperate, None).await.is_ok();
self.with_repo_mut(call, &repo_name, Cap::TaskWrite, None, move |repo, actor| {
finish_task_with_limit(
finish_task_checked(
repo.meta(),
&req.id,
&req.state,
actor.as_str(),
operator,
req.expected_max_attempts,
TaskFinishPreconditions {
max_attempts: req.expected_max_attempts,
attempts_started: req.expected_attempts_started,
},
)?;
let t = repo.meta().read(|tx| tx.task(&req.id))?.ok_or_else(|| ApiError::not_found("task not found"))?;
task_view(repo, &t)
Expand Down
18 changes: 18 additions & 0 deletions crates/ag-server/tests/task_attempts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -63,5 +63,23 @@ async fn attempts_are_counted_once_and_an_operator_can_extend_the_same_task() {
assert_eq!(second.status, 200, "{}", second.json);
assert_ne!(second.json["run_id"], started.json["run_id"]);
assert_eq!(creator.get(&format!("{base}/tasks/{task}")).await.json["attempts_started"], 2);
let stale_run_finish = admin
.post(
&format!("{base}/tasks/{task}/finish"),
json!({"state":"cancelled", "expected_max_attempts":2, "expected_attempts_started":1}),
)
.await;
assert_eq!(stale_run_finish.status, 412, "{}", stale_run_finish.json);
assert_eq!(creator.get(&format!("{base}/tasks/{task}")).await.json["state"], "in_progress");
let second_run = second.json["run_id"].as_str().unwrap();
assert_eq!(
admin.post(&format!("{base}/runs/{second_run}/complete"), json!({"status":"completed"})).await.status,
200
);
let current_finish = admin
.post(&format!("{base}/tasks/{task}/finish"), json!({"state":"done", "expected_attempts_started":2}))
.await;
assert_eq!(current_finish.status, 200, "{}", current_finish.json);
assert_eq!(current_finish.json["state"], "done");
server.stop().await;
}
2 changes: 2 additions & 0 deletions dev/goal/backlog.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,5 @@ CLI explanations use the same server rule. The active priority remains agent-nat
Core integration: PR112 (atomic expected-head proposal updates) and PR114 (revalidate proposal and stream at final integration commit) are merged. PR115 (policy-required thread resolution) is also merged. PR113 adds review suggestions on the same head-precondition contract. These remain distinct from a published release.

Retry policy changes during an active run now reject stale automatic cancellation atomically; the scheduler rereads durable history before resuming. See ADR0034 and the evidence journal.

Execution completion now carries a durable generation precondition across core, API and orchestrator (ADR0035). A late result cannot finish a newer attempt; matching manual completion remains available.
18 changes: 18 additions & 0 deletions dev/goal/evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -615,3 +615,21 @@ history (0.01s); HTTP stale cancellation returned 412 (0.14s). SDK tours passed
`/workspace/agentgit-quality/task-policy-finish-red.log`; follow-up logs share that prefix.
No schema migration; ADR0034 describes the additive protocol field. Static Crapper/Dryer/
Mutator scans were run; these are not coverage or mutation-kill measurements.

## Completion is bound to the observed task execution

ADR0035 adds `expected_attempts_started` to task completion. It is checked together with the
optional expected attempt limit in the same state/audit transaction. Automatic session,
recovered-budget and landed-proposal completion carry the observed generation. The
orchestrator negotiates `tasks.finish_preconditions` and leaves a task unchanged if an old
server would ignore the condition. Explicit manual completion remains compatible.

Red: a real HTTP request carrying generation one cancelled a task already on its second
run (200 instead of 412). Green: stale generation is rejected and matching completion succeeds
(0.14s); previous policy/state/audit regression remains green. Five real-server/shell scheduler
scenarios passed in 1.50s, including a delayed first executor after a second run was admitted,
mid-run policy changes, restart and legacy cancellation. Protocol checks cover unsupported
servers and preservation of explicit zero (0.008s). TypeScript/Python SDK tours passed
(12/13 tests) using both conditions; Web types remain valid. Crapper/Dryer/Mutator scans are
static inspection, not a measured mutation score. Logs: task-completion-fence-* in the
implementation workspace.
24 changes: 24 additions & 0 deletions docs/adr/0035-task-completion-generation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# ADR 0035: fence automatic task completion by execution generation

Status: accepted.

## Context

An orchestrator finishes tasks with operator credentials after its run credential stops
working. A delayed result from an earlier attempt must not finish work owned by a newer
attempt, even if both attempts use the same agent and retry limit.

## Decision

Add optional `expected_attempts_started` to `FinishTaskRequest` (protobuf field 6). The core
compares it to the durable count of admitted task runs in the same write transaction as
state and audit. A newer admitted run changes that count and stale completion is rejected
with HTTP 412 / failed_precondition, without changing the task or adding a finish audit.
The count includes failed and abandoned runs and is never reset by retry-policy edits.

Automatic session completion, cancellation, history recovery and proposal-result completion
supply the observed count. Cancellation also supplies the expected retry limit (ADR0034).
On a rejected transition, the scheduler discards local assumptions and reloads the task and
its history. Explicit manual completion without preconditions keeps its existing behavior.
Servers advertise `tasks.finish_preconditions`; the orchestrator verifies it before automatic completion and leaves the task unchanged on an older server. No storage migration or canonical-format change is required. Upgrade all replicas and
orchestrators together; this extends the unreleased task execution contract.
2 changes: 2 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,5 @@ which test: [ADR traceability](../architecture/adr-traceability.md).
- [0033 — Task execution-attempt limits](0033-task-attempt-limits.md)

- [0034 — Task completion policy precondition](0034-task-finish-policy-precondition.md)

- [0035 — Task completion execution generation](0035-task-completion-generation.md)
Loading
Loading