diff --git a/Cargo.lock b/Cargo.lock index c3fd2c7..b583b5e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -205,7 +205,7 @@ dependencies = [ [[package]] name = "dotmend" -version = "0.2.0" +version = "0.3.0" dependencies = [ "axum", "base64 0.22.1", diff --git a/Cargo.toml b/Cargo.toml index 0c1537e..1bcdd52 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "dotmend" -version = "0.2.0" +version = "0.3.0" edition = "2024" publish = false license = "MIT" diff --git a/README.md b/README.md index d9dcc71..d58ca67 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,8 @@ When a detail needs your touch, the screen has four actions: Marks are saved separately from the artwork so your agent can inspect the exact locations. +When your agent offers alternatives, click **Choose** under any candidates you want to explore, then **Save**. You can choose several, or click a chosen option again to remove it. Your agent can refine those choices and bring back another set. Earlier candidates remain available. + Ask the agent to show a collection, filter the view, bring back an earlier candidate, or work on a particular area. You do not need to learn an editor to do that. ## What it supports diff --git a/resources/agent-usage.md b/resources/agent-usage.md index 9f2c75a..66b8df9 100644 --- a/resources/agent-usage.md +++ b/resources/agent-usage.md @@ -10,18 +10,22 @@ Dotmend uses the project directory supplied by the client at launch, or an expli ## Presenting work to a human -Humans paint with palette colors, mark suspected issues, undo the last stroke once and save. Handle collections, filters, past candidates, regions and reviews through conversation and tools. +Humans paint with palette colors, mark suspected issues, undo the last stroke once and save. They can also choose several alternatives when you offer candidate choices. Handle collections, filters, past candidates, regions and reviews through conversation and tools. When first sharing the screen, briefly explain its available controls in the user's language: choose a palette color, then click or drag to paint; **Mark issues** uses left clicks/drags to add marks and right clicks/drags to remove them; **Undo** (Ctrl/Cmd+Z) reverses the last stroke once; **Save** (Ctrl/Cmd+S) records the current result. A click or drag is one stroke, and painting and marking share the same undo. Tell the user to ask you for older results or other operations. Adapt the explanation to read-only and playback views; repeat it only if asked or needed. Do not send the user to this guide or require IDs and technical settings. For more than 20 allowed colors, the screen starts in **Mark issues** mode and shows up to 20 commonly used palette entries, including Erase when allowed. Explain this briefly in the user's language and invite them to mark areas and describe the desired change. Read saved concerns, focus the marked regions, resolve unclear intent, apply edits within the request's permissions and present the result. Keep color selection and palette details with the agent. The displayed subset stays fixed during editing; all original indices remain available to agent tools. -1. Choose a fresh random `control_id` per independent task (16..128 ASCII letters, digits, underscores or hyphens); retain it across calls and connections. Call `open_workbench({control_id})` for the instance ID and URL. Do not bypass another owner or the shared limit. Read `inspect_presentation({})`; no existing view returns `presentation:null`. -2. Call `present_art({control_id,workbench_id,view})`. Set `view.title`, `note` and `items` to describe the user's task. An art item is `{kind:"art",art_id,label,region,scale,editable,request_id?}`. Attach `request_id` to an editable item carrying out a protected request. -3. Copy the freshly read IDs into `view.expected_presentation_id` and `view.expected_state_id`. Both keys must be explicit `null` for the first view. On conflict, read the human's latest edits first. +1. Choose a fresh random `control_id` per independent task (16..128 ASCII letters, digits, underscores or hyphens); retain it across calls and connections. Call `open_workbench({control_id,work_state:"working"})` for the instance ID and URL. Do not bypass another owner or the shared limit. Read `inspect_presentation({})`; no existing view returns `presentation:null`. +2. Prepare `view.title`, `note` and `items` to describe the user's task. An art item is `{kind:"art",art_id,label,region,scale,editable,request_id?}`. Attach `request_id` to an editable item carrying out a protected request. Copy the freshly read IDs into `view.expected_presentation_id` and `view.expected_state_id`; both keys must be explicit `null` for the first view. +3. Call `present_art({control_id,workbench_id,view})`. On conflict, read the human's latest edits first. When ready for their input, call `open_workbench({control_id,work_state:"waiting"})` and share its returned URL. 4. Read the explicit save from `inspect_presentation` at `saved.art_ids` and `saved.state_id`. Current drafts are at `state.art_ids`; `dirty` distinguishes them from the last explicit save. Saving is not acceptance. -Do not close the screen while the human is working. Finish with `close_workbench({control_id,workbench_id})` and check its result. `inspect_workbench({control_id})` distinguishes `owned`, `closing`, `busy` and `closed` without renewing activity. Reopening with the same control_id reuses its ID and URL and renews activity; it does not change its timeout. Successful presentation and human actions renew activity; polling does not. The default idle timeout is 1800 seconds, configurable from 1 to 1800. HTTP ends when the MCP process hosting it exits or the idle deadline expires; art and drafts remain. +Do not close the screen while the human is working. Use `open_workbench({control_id,work_state:"working"})` at each follow-up before doing art work or calling an external generator. A declared working task stays alive through long calls and gaps between calls, including external work. Collaborators must use the same explicit task ID to change its work state; ordinary art calls never imply task ownership or activity. + +Before handing the screen to the human or awaiting their reply, call `open_workbench({control_id,work_state:"waiting"})`. Share only the URL returned by that call, never a URL from memory or an earlier answer. On failure, cancellation or interruption, switch to waiting or close the finished task. The server cannot infer that an external agent stopped: leaving working set will hold the screen while the host lives. Use `inspect_workbench({control_id})` to read the current instance and work_state without renewing activity. Finish with `close_workbench({control_id,workbench_id})` and check its result. + +Reopening preserves an active instance's ID, URL and timeout, and resets idle time. Omitting work_state preserves an active state; a new instance defaults to waiting. In waiting, presentation, successful human actions and real visible input (scrolling, color selection and unfinished dragging) renew activity. Art calls, failed calls, HTTP/browser polling, playback and inspect_workbench do not. The waiting idle timeout defaults to 1800 seconds, configurable from 1 to 1800. The hosting MCP process exiting always ends HTTP, including working tasks; art and drafts remain. After shutdown, open a new instance and inspect the preserved draft before continuing. One explicit controller owns each workspace's screen, with a shared maximum of four. The same control_id works across MCP connections; different tasks remain separate even on one connection. Share the workbench URL with the human; keep control IDs private. Independent candidate creation does not require screen control. Never bypass control using shell servers, alternate runtime directories or lock-file changes. The local HTTP interface is not an MCP transport or a public agent mutation API. Closing a forwarding MCP connection does not close the host's screen. If close returns closing, inspect until closed. @@ -29,6 +33,20 @@ For collections and filters, use `list_art` and present the selected candidates Reference items use `{kind:"reference",art_id,source_hash,label,region,scale}` and source-image coordinates; they are read-only. For playback, supply existing frames in a read-only art item's `playback`, show the full first `art_id`, and specify every frame's `duration_ms`. If timing is unknown, show frames side by side. All playback frames count toward the total display area. Prepare narrower crops or native-size views yourself; users should not have to learn IDs, coordinates or validation settings. +## Offering candidate choices + +Create alternatives as separate candidates, keeping the supplied intent and constraints. Label the differences clearly and show comparable regions at appropriate scales. The agent prepares the alternatives; choosing does not generate images automatically. + +Use the existing `present_art` view with `candidate_choices: {"item_indices":[1,2,3]}` to make those zero-based items selectable. For example, item 0 can be the original and items 1..3 the alternatives. Offer 2..16 distinct static art candidates with `editable:false`. References, playback and editable items cannot be choices. Other items can still provide context. Omit `candidate_choices` for an ordinary editing view. + +Tell the human in their language to click **Choose** under any candidates they want to explore, then **Save** (Ctrl/Cmd+S). They can choose several and click a chosen option again to remove it. Nothing is preselected. Choice changes are preserved immediately as drafts; Save communicates the current choice. Mark issues remains available for details. Undo affects only the last painting or marking stroke and leaves candidate choices unchanged. + +Read `saved.chosen_candidates`, a list of `{item_index,art_id}`, together with `saved.state_id` and `presentation_id`. `state.chosen_candidates` may contain a newer draft; absent or empty fields mean no choice. Do not infer choices from `saved.art_ids`, which includes every shown item. An empty choice is not rejection of the alternatives. Read the saved marks and any conversational feedback as well. + +Use the chosen IDs as the exact bases for separate refinements, preserving their target constraints and applicable permissions. Recreate any pixel selections and protection against each new base before protected edits. Show the next alternatives through a new presentation, which starts with no chosen candidates. Previous candidates, saved choices and draft states remain available through `inspect_presentation`. Do not discard unchosen alternatives or treat them as rejected. + +Choices express directions for further work. They do not submit results, record final acceptance, grant additional edit permission, validate constraints or approve game insertion. The agent continues the conversation using the saved choice; Dotmend does not invoke the agent automatically. On conflicts or lost responses, inspect the current presentation before retrying and never apply an old click to a different view. + ## Reading human issue marks The human toggles **Mark issues** to add pixels with a left click or drag and remove them with a right click or drag. Choosing a palette color returns to painting. Marks are a separate overlay, not palette edits. Static art can be marked even when read-only or protected. Reference images and playback cannot be marked; present a specific still-art candidate when needed. diff --git a/src/presentation.rs b/src/presentation.rs index 06d2a2c..d6dee87 100644 --- a/src/presentation.rs +++ b/src/presentation.rs @@ -49,6 +49,19 @@ impl PresentationItem { } #[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] +pub struct CandidateChoices { + /// Indices of distinct, static, read-only art items offered as alternatives. + #[schemars(length(min = 2, max = 16))] + pub item_indices: Vec, +} +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, JsonSchema)] +#[serde(deny_unknown_fields)] +pub struct ChosenCandidate { + pub item_index: usize, + pub art_id: String, +} +#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] +#[serde(deny_unknown_fields)] #[schemars(extend("required" = ["title", "items", "expected_presentation_id", "expected_state_id"]))] pub struct PresentArt { pub title: String, @@ -59,6 +72,9 @@ pub struct PresentArt { pub expected_presentation_id: Option, #[serde(deserialize_with = "required_nullable")] pub expected_state_id: Option, + /// Offer candidates for the human to choose and save as the basis for further work. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub candidate_choices: Option, } #[derive(Debug, Clone, Default, Serialize, Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] @@ -73,6 +89,7 @@ pub enum PresentationActionKind { Paint, Mark, Undo, + Choose, } #[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] @@ -95,6 +112,8 @@ pub struct PresentationState { pub concerns: Vec, #[serde(default, skip_serializing_if = "Option::is_none")] pub undo_concerns: Option>, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub chosen_candidates: Vec, } #[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] @@ -105,6 +124,8 @@ pub struct SavedPresentation { pub art_ids: Vec, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub concerns: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub chosen_candidates: Vec, } #[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] @@ -128,6 +149,12 @@ pub struct PresentedArt { #[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] #[serde(tag = "action", rename_all = "snake_case", deny_unknown_fields)] pub enum HumanAction { + Choose { + presentation_id: String, + expected_state_id: String, + /// The complete choice, or an empty list to clear it. + item_indices: Vec, + }, Mark { presentation_id: String, expected_state_id: String, @@ -153,7 +180,12 @@ pub enum HumanAction { impl HumanAction { pub fn identity(&self) -> (&str, &str) { match self { - Self::Mark { + Self::Choose { + presentation_id, + expected_state_id, + .. + } + | Self::Mark { presentation_id, expected_state_id, .. diff --git a/src/requests.rs b/src/requests.rs index 3fac2d1..1659c74 100644 --- a/src/requests.rs +++ b/src/requests.rs @@ -174,7 +174,7 @@ pub fn tool_schemas() -> Vec<(&'static str, &'static str, bool, Value)> { vec![ ( "open_workbench", - "Open a managed human workbench with an explicit control_id (fresh random ID per task, 16..128 ASCII letters/digits/_/-). Reuse the same ID across calls or MCP connections. Other controllers and the shared limit prevent launch. Optional idle_timeout_seconds: 1..1800; default 1800. Reopening renews activity without changing the existing timeout.", + "Open or resume a managed workbench with an explicit control_id (fresh random ID per task, 16..128 ASCII letters/digits/_/-). Set work_state: working before agent work, including lengthy external calls; it prevents idle shutdown until you explicitly switch to waiting or close. Set waiting at handoff, when awaiting user input, or after interruption; use that response's URL, never a cached address. Omitted state preserves an active state, or starts a new instance waiting. Same control_id works across connections. Other tasks' art calls do not renew activity. Reopening resets idle time and keeps the active ID, URL and timeout. Optional idle_timeout_seconds: 1..1800, default 1800. Only the owner can change state; host exit and the shared limit still apply.", false, input_schema::(), ), @@ -192,13 +192,13 @@ pub fn tool_schemas() -> Vec<(&'static str, &'static str, bool, Value)> { ), ( "present_art", - "Present art through a managed workbench. Pass its control_id and the exact workbench_id from open_workbench and a view containing title, items and the expected presentation/state IDs from inspect_presentation. Both expected IDs must be explicit null for the first view. Humans paint, mark issues, undo once and save.", + "Present art through a managed workbench. Pass its control_id and the exact workbench_id from open_workbench and a view containing title, items and the expected presentation/state IDs from inspect_presentation. Both expected IDs must be explicit null for the first view. Optionally set view.candidate_choices={item_indices:[...]} to offer 2..16 distinct static, read-only art items for the human to choose from. Multiple choices are allowed. Humans can also paint editable items, mark issues, undo one stroke and save.", false, input_schema::(), ), ( "inspect_presentation", - "Read the current or archived presentation, human drafts and explicitly saved candidates. state.concerns and saved.concerns contain item_index, art_id, bounds and exact pixels marked by the human; absent means empty. Marks indicate observation, not edit permission or rejection. Use state_id for past states; prepare history and filters yourself with present_art.", + "Read the current or archived presentation, human drafts and explicitly saved candidates. state.chosen_candidates and saved.chosen_candidates contain {item_index,art_id} for the human's draft or saved choices; absent means empty. Continue from saved choices, preserving the unchosen candidates. Choosing is a preference for further work, not final acceptance or rejection. state.concerns and saved.concerns contain item_index, art_id, bounds and exact marked pixels; absent means empty. Marks request observation, not edit permission. Use state_id for past states; prepare history and filters yourself with present_art.", true, input_schema::(), ), diff --git a/src/server.rs b/src/server.rs index 51acd8d..3df7485 100644 --- a/src/server.rs +++ b/src/server.rs @@ -252,7 +252,12 @@ async fn tool_call(State(state): State, Json(request): Json) Err(error) => failure(error), } } -async fn presentation(State(state): State) -> Response { +async fn presentation(State(state): State, headers: HeaderMap) -> Response { + if let Some(id) = headers.get("x-retro-art-workbench") + && let Err(error) = state.access.check_instance(id.to_str().unwrap_or("")) + { + return failure(error); + } match state .workspace .lock() @@ -273,14 +278,14 @@ async fn human_action( .and_then(|h| h.to_str().ok()) .unwrap_or("") .to_owned(); + let action_id = headers + .get("x-dotmend-action") + .and_then(|h| h.to_str().ok()) + .map(str::to_owned); let result = tokio::task::spawn_blocking(move || { - state.access.apply(&id, || { - state - .workspace - .lock() - .map_err(|_| ArtError::new("storage_error", "Failed to acquire lock")) - .and_then(|mut w| w.human_action(input)) - }) + state + .access + .human_action(&id, action_id.as_deref(), input, false, state.workspace) }) .await; match result { @@ -289,6 +294,42 @@ async fn human_action( Err(error) => failure(ArtError::new("storage_error", error.to_string())), } } +async fn recover_action( + State(state): State, + headers: HeaderMap, + Json(input): Json, +) -> Response { + let id = headers + .get("x-retro-art-workbench") + .and_then(|h| h.to_str().ok()) + .unwrap_or("") + .to_owned(); + let action_id = headers + .get("x-dotmend-action") + .and_then(|h| h.to_str().ok()) + .map(str::to_owned); + match tokio::task::spawn_blocking(move || { + state + .access + .human_action(&id, action_id.as_deref(), input, true, state.workspace) + }) + .await + { + Ok(Ok(output)) => Json(output.data).into_response(), + Ok(Err(error)) => failure(error), + Err(error) => failure(ArtError::new("storage_error", error.to_string())), + } +} +async fn human_activity(State(state): State, headers: HeaderMap) -> Response { + let id = headers + .get("x-retro-art-workbench") + .and_then(|h| h.to_str().ok()) + .unwrap_or(""); + match state.access.apply(id, || Ok(())) { + Ok(()) => Json(json!({"ok": true})).into_response(), + Err(error) => failure(error), + } +} async fn art_data(State(state): State, Path(id): Path) -> Response { match state.workspace.lock().map_err(|_| ArtError::new("storage_error","Failed to acquire lock")).and_then(|w| w.load(&id)) { Ok(art) => Json(json!({"art_id":id,"target":art.target,"indices":art.indices,"context":art.context,"references":art.references,"parents":art.parents})).into_response(), @@ -372,7 +413,7 @@ async fn control_call( if !(1..=DEFAULT_IDLE_SECONDS).contains(&idle) { return Err(invalid("idle_timeout_seconds must be 1..1800")); } - state.access.reopen(&input.control_id) + state.access.reopen(&input.control_id, input.work_state) } "inspect_workbench" => { let input: InspectWorkbench = decode(request.arguments)?; @@ -438,6 +479,7 @@ pub(crate) fn web_router( "/api/workbench", get(|State(state): State| async move { Json(state.access.instance.clone()) }), ) + .route("/api/workbench/activity", post(human_activity)) .route( "/", get(|| async { Html(include_str!("../web/index.html")) }), @@ -462,6 +504,7 @@ pub(crate) fn web_router( ) .route("/api/presentation", get(presentation)) .route("/api/presentation/action", post(human_action)) + .route("/api/presentation/recover", post(recover_action)) .route("/api/call", post(tool_call)) .route("/api/art/{id}", get(art_data)) .route("/api/art/{id}/image", get(art_image)) diff --git a/src/server_instructions.rs b/src/server_instructions.rs index b2f6a0a..fc85f09 100644 --- a/src/server_instructions.rs +++ b/src/server_instructions.rs @@ -7,12 +7,13 @@ INPUT For a PNG: (1) inspect an existing target art or create_art with supplied target constraints and initial={kind:"fill",index:}; target colors use #RRGGBB and transparent_index. (2) Call prepare_image with that art_id as target_art_id, a source_path relative to the server workspace, and explicit crop, resize, alpha, color_mapping and dither settings. Path errors provide the workspace and recovery steps. (3) Use the returned art_id for inspection, validation, editing and presentation. Preserve supplied constraints; ask for missing required target conditions. Use create_art with bundle_path for exported Dotmend art, or target and initial.kind=indices for caller-supplied index data. WORKBENCH -Choose a fresh random control_id per independent task; retain it across calls and connections. Pass it to open_workbench, inspect_workbench, close_workbench and present_art. Reopening with the same control_id reuses the instance. Also pass its workbench_id to present_art {control_id,workbench_id,view} and close_workbench. Share the URL with the human; keep control IDs private. -One controller per workspace; four workbenches per shared runtime directory. Never bypass control or limits with shell servers, PIDs or alternate paths. HTTP ends when its hosting MCP process exits or its returned idle timeout expires; polling does not renew activity. Close when finished, never while the human is working. Art and drafts persist. +Choose a fresh random control_id per independent task; retain it across calls and connections. At every follow-up and before agent or external work, call open_workbench with that ID and work_state:"working". This holds the screen even between calls and throughout lengthy external generation. At handoff or while awaiting user input, call open_workbench with work_state:"waiting" and share only the URL returned by that call, never a cached URL. On failure or interruption, switch to waiting or close. Also pass its workbench_id to present_art {control_id,workbench_id,view} and close_workbench. Keep control IDs private. +One controller per workspace; four workbenches per shared runtime directory. Never bypass control or limits with shell servers, PIDs or alternate paths. A working task stays active until its owner changes state or the host exits; you must clear working when work stops because external cancellation is not automatically observable. In waiting, successful presentation, human actions and real browser input renew activity; art calls, failures, inspect_workbench, HTTP/browser polling and playback do not. The default waiting idle timeout is 30 minutes. Close when finished, never while the human is working. Art and drafts persist; after shutdown reopen, inspect preserved work and share the newly returned URL. HUMAN SCREEN At first handoff, briefly explain available controls in the user's language: palette + click/drag paints; Mark issues: left adds, right removes; Undo once for the last stroke (Ctrl/Cmd+Z); Save (Ctrl/Cmd+S). For more than 20 allowed colors, explain that the screen shows common colors and opens in Mark issues mode. Invite the user to mark changes; read the saved concerns, focus those areas and make edits within the request's permissions. Clarify ambiguous intent. Offer older results through conversation. Repeat only when needed. Prepare collections, filters, past art, reference crops and known-time playback yourself. Read inspect_presentation; copy current IDs into view.expected_presentation_id/expected_state_id (both explicit null only for the first view). saved.art_ids holds the explicit save; state.art_ids may contain a newer draft. state.concerns and saved.concerns identify marked candidates and pixels; marks request observation, not edit permission or rejection. +For alternatives, create distinct candidates and offer their static, read-only art items through view.candidate_choices={item_indices:[...]}. The human can choose several, change the choice, and Save. Briefly explain this in their language. Read saved.chosen_candidates ({item_index,art_id}) with saved.state_id before continuing; state.chosen_candidates is a draft. Missing or empty means no choice. Refine the saved choices as separate branches and present the next alternatives. Preserve unchosen candidates; a choice is a direction for further work, not final acceptance or rejection. EDIT AND RECOVER Read the request and target. Preserve palette order, duplicate indices, transparency, dimensions and constraints. validate_art locations and focus_art grant no write permission. Pass request_id for protection; preview sets before applying the exact plan and bindings. diff --git a/src/workbench.rs b/src/workbench.rs index 7d1e6f5..2cceb62 100644 --- a/src/workbench.rs +++ b/src/workbench.rs @@ -1,6 +1,7 @@ use crate::server::{SharedWorkspace, web_router}; use dotmend::{art::*, workbench_protocol::*, workspace::ToolOutput}; use std::{ + collections::HashSet, fs::{self, File, OpenOptions, TryLockError}, future::IntoFuture, path::{Path, PathBuf}, @@ -12,7 +13,7 @@ use tokio::{sync::watch, task::JoinHandle}; fn storage(error: impl std::fmt::Display) -> ArtError { ArtError::new("storage_error", error.to_string()) } -fn locked_file(path: &Path) -> ArtResult> { +fn open_lock_file(path: &Path) -> ArtResult { let mut options = OpenOptions::new(); options.read(true).write(true).create(true).truncate(false); #[cfg(unix)] @@ -20,7 +21,10 @@ fn locked_file(path: &Path) -> ArtResult> { use std::os::unix::fs::OpenOptionsExt; options.mode(0o600); } - let file = options.open(path).map_err(storage)?; + options.open(path).map_err(storage) +} +fn locked_file(path: &Path) -> ArtResult> { + let file = open_lock_file(path)?; match file.try_lock() { Ok(()) => Ok(Some(file)), Err(TryLockError::WouldBlock) => Ok(None), @@ -50,11 +54,13 @@ fn runtime_directory() -> ArtResult { struct Activity { active: bool, last: Instant, + work_state: WorkState, } pub struct WorkbenchAccess { pub instance: WorkbenchInstance, control_id: String, activity: Mutex, + cancelled_actions: Mutex>, } impl WorkbenchAccess { pub fn apply(&self, id: &str, operation: impl FnOnce() -> ArtResult) -> ArtResult { @@ -71,6 +77,16 @@ impl WorkbenchAccess { activity.last = Instant::now(); Ok(result) } + pub fn check_instance(&self, id: &str) -> ArtResult<()> { + let activity = self.activity.lock().map_err(storage)?; + if id != self.instance.workbench_id || !activity.active { + return Err(ArtError::new( + "workbench_conflict", + "This workbench has ended. Reopen it through open_workbench", + )); + } + Ok(()) + } fn active(&self) -> bool { self.activity.lock().is_ok_and(|a| a.active) } @@ -83,6 +99,12 @@ impl WorkbenchAccess { let Ok(mut activity) = self.activity.lock() else { return true; }; + if !activity.active { + return true; + } + if activity.work_state == WorkState::Working { + return false; + } if activity.last.elapsed() >= Duration::from_secs(self.instance.idle_timeout_seconds) { activity.active = false; } @@ -147,6 +169,7 @@ impl Workbench { state, instance, max_workbenches: MAX_WORKBENCHES, + work_state: None, }) .map_err(storage)?, )) @@ -241,7 +264,7 @@ impl Workbench { } if let Some(running) = &mut self.running { if running.access.active() && !running.task.is_finished() { - return running.access.reopen(&input.control_id); + return running.access.reopen(&input.control_id, input.work_state); } if !running.task.is_finished() { running.finish().await; @@ -302,13 +325,16 @@ impl Workbench { file.as_file().sync_all().map_err(storage)?; file.persist(self.root.join(".dotmend/workbench.json")) .map_err(storage)?; + let work_state = input.work_state.unwrap_or(WorkState::Waiting); let access = Arc::new(WorkbenchAccess { control_id: input.control_id, instance: instance.clone(), activity: Mutex::new(Activity { active: true, last: Instant::now(), + work_state, }), + cancelled_actions: Mutex::new(HashSet::new()), }); let app = web_router(workspace, port, access.clone()); let (shutdown, mut requested) = watch::channel(false); @@ -348,12 +374,13 @@ impl Workbench { } lifetime.end(); }); + let output = access.inspect(&access.control_id)?; self.running = Some(RunningWorkbench { access, shutdown, task, }); - self.result(WorkbenchState::Owned, Some(instance)) + Ok(output) } pub async fn close(&mut self, input: CloseWorkbench) -> ArtResult { validate_control(&input.control_id)?; @@ -444,11 +471,16 @@ impl WorkbenchAccess { state, instance: owned.then(|| self.instance.clone()), max_workbenches: MAX_WORKBENCHES, + work_state: if owned { + Some(self.activity.lock().map_err(storage)?.work_state) + } else { + None + }, }) .map_err(storage)?, )) } - pub fn reopen(&self, id: &str) -> ArtResult { + pub fn reopen(&self, id: &str, work_state: Option) -> ArtResult { validate_control(id)?; if id != self.control_id { return Err(ArtError::new( @@ -456,9 +488,59 @@ impl WorkbenchAccess { "Another control_id owns this workspace's workbench. Its controller must close it first", )); } - self.apply(&self.instance.workbench_id, || Ok(()))?; + { + let mut activity = self.activity.lock().map_err(storage)?; + if !activity.active { + return Err(ArtError::new( + "workbench_conflict", + "This workbench has ended. Inspect and reopen it", + )); + } + if let Some(work_state) = work_state { + activity.work_state = work_state; + } + activity.last = Instant::now(); + } self.inspect(id) } + pub fn human_action( + &self, + id: &str, + action_id: Option<&str>, + input: dotmend::presentation::HumanAction, + recover: bool, + workspace: SharedWorkspace, + ) -> ArtResult { + if let Some(action_id) = action_id { + if !(16..=128).contains(&action_id.len()) + || !action_id + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_') + { + return Err(invalid( + "x-dotmend-action must contain 16..128 ASCII letters, digits, underscores or hyphens", + )); + } + } else if recover { + return Err(invalid( + "Recovery requires the original x-dotmend-action identifier", + )); + } + self.apply(id, || { + let mut cancelled = self.cancelled_actions.lock().map_err(storage)?; + let mut workspace = workspace.lock().map_err(storage)?; + if recover { + let output = workspace.recover_human_action(&input)?; + cancelled.insert(action_id.unwrap().to_owned()); + Ok(output) + } else { + if action_id.is_some_and(|id| cancelled.contains(id)) { + return Err(ArtError::new("action_cancelled", "This action was settled during recovery and cannot execute again. Read the current presentation before making a new edit")); + } + workspace.human_action(input) + } + }) + } pub async fn present( self: &Arc, input: ManagedPresentArt, diff --git a/src/workbench_protocol.rs b/src/workbench_protocol.rs index 4c47a6d..bb94ea8 100644 --- a/src/workbench_protocol.rs +++ b/src/workbench_protocol.rs @@ -10,6 +10,7 @@ pub const DEFAULT_IDLE_SECONDS: u64 = 1800; pub struct OpenWorkbench { pub control_id: String, pub idle_timeout_seconds: Option, + pub work_state: Option, } #[derive(Debug, Serialize, Deserialize, JsonSchema)] #[serde(deny_unknown_fields)] @@ -36,6 +37,12 @@ pub struct WorkbenchInstance { pub url: String, pub idle_timeout_seconds: u64, } +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)] +#[serde(rename_all = "snake_case")] +pub enum WorkState { + Working, + Waiting, +} #[derive(Debug, Serialize, Deserialize, JsonSchema)] #[serde(rename_all = "snake_case")] pub enum WorkbenchState { @@ -50,4 +57,6 @@ pub struct WorkbenchStatus { pub state: WorkbenchState, pub instance: Option, pub max_workbenches: usize, + #[serde(skip_serializing_if = "Option::is_none")] + pub work_state: Option, } diff --git a/src/workspace_presentation.rs b/src/workspace_presentation.rs index 165da9e..91f8876 100644 --- a/src/workspace_presentation.rs +++ b/src/workspace_presentation.rs @@ -96,6 +96,7 @@ impl Workspace { state_id: saved_id, art_ids: state.art_ids, concerns: state.concerns, + chosen_candidates: state.chosen_candidates, }) } else { None @@ -134,6 +135,28 @@ impl Workspace { { return Err(invalid("Provide a presentation title and 1..16 items")); } + if let Some(choices) = &input.candidate_choices { + if !(2..=MAX_SET_SIZE).contains(&choices.item_indices.len()) { + return Err(invalid("Offer 2..16 distinct candidate choices")); + } + let mut candidates = BTreeSet::new(); + for index in &choices.item_indices { + let Some(PresentationItem::Art { + art_id, + editable: false, + playback: None, + .. + }) = input.items.get(*index) + else { + return Err(invalid( + "Candidate choices must refer to static, read-only art items", + )); + }; + if !candidates.insert(art_id) { + return Err(invalid("Offer each candidate only once")); + } + } + } let mut pixels = 0u64; let mut editable = BTreeSet::new(); for item in &input.items { @@ -221,6 +244,7 @@ impl Workspace { action: PresentationActionKind::Open, concerns: vec![], undo_concerns: None, + chosen_candidates: vec![], }; let transaction = self .database @@ -254,6 +278,25 @@ impl Workspace { state_id: None, }) } + pub fn recover_human_action(&self, input: &HumanAction) -> ArtResult { + let command_id = identity("view_action", input)?; + let completed: Option = self + .database + .query_row( + "SELECT state_id FROM presentation_actions WHERE id=?1", + [&command_id], + |r| r.get(0), + ) + .optional() + .map_err(storage_error)?; + let mut output = if let Some(state_id) = &completed { + self.action_receipt(state_id.clone(), matches!(input, HumanAction::Save { .. }))? + } else { + self.inspect_presentation(InspectPresentation::default())? + }; + output.data["completed"] = json!(completed.is_some()); + Ok(output) + } pub fn human_action(&mut self, input: HumanAction) -> ArtResult { let (presentation_id, expected_state_id) = input.identity(); let command_id = identity("view_action", &input)?; @@ -274,6 +317,38 @@ impl Workspace { let mut arts = vec![]; let save = matches!(input, HumanAction::Save { .. }); match &input { + HumanAction::Choose { item_indices, .. } => { + let choices = snapshot + .presentation + .candidate_choices + .as_ref() + .ok_or_else(|| invalid("This presentation does not offer candidate choices"))?; + if item_indices.len() > choices.item_indices.len() { + return Err(invalid( + "The choice exceeds the number of candidates allowed by this presentation", + )); + } + let mut indices = BTreeSet::new(); + for index in item_indices { + if !choices.item_indices.contains(index) || !indices.insert(*index) { + return Err(invalid( + "Choose only distinct candidates offered by this presentation", + )); + } + } + let chosen: Vec<_> = indices + .into_iter() + .map(|item_index| ChosenCandidate { + item_index, + art_id: state.art_ids[item_index].clone(), + }) + .collect(); + if chosen != state.chosen_candidates { + state.chosen_candidates = chosen; + state.previous_state_id = Some(expected_state_id.into()); + state.action = PresentationActionKind::Choose; + } + } HumanAction::Mark { item_index, pixels, @@ -440,6 +515,7 @@ impl Workspace { state_id: state_id.clone(), art_ids: state.art_ids, concerns: state.concerns, + chosen_candidates: state.chosen_candidates, }; transaction .execute( diff --git a/tests/browser/workbench.spec.mjs b/tests/browser/workbench.spec.mjs index 6536700..9863deb 100644 --- a/tests/browser/workbench.spec.mjs +++ b/tests/browser/workbench.spec.mjs @@ -7,7 +7,7 @@ import { createInterface } from "node:readline"; import { randomUUID } from "node:crypto"; import { createConnection } from "node:net"; -const binary = resolve(`target/debug/dotmend${process.platform === "win32" ? ".exe" : ""}`); +const binary = resolve(process.env.DOTMEND_TEST_BINARY || `target/debug/dotmend${process.platform === "win32" ? ".exe" : ""}`); function mcpClient(workspace, runtimeDirectory=join(workspace,"runtime"), perRequestMetadata=true) { const child = spawn(binary, ["--workspace", workspace], { stdio: ["pipe", "pipe", "pipe"], env:{...process.env,DOTMEND_RUNTIME_DIR:runtimeDirectory} }); const controlId = randomUUID(); @@ -99,7 +99,7 @@ test("a stale browser action cannot write onto an agent's new presentation",asyn test("a lost save response is recovered without inventing a new result",async({page,workbench})=>{ const a=await create(workbench.client,"save");await show(workbench.client,[a]);await settle(page);await color(page);await point(page,1,1);await expect.poll(()=>result(workbench.client)).not.toBe(a);const painted=await result(workbench.client); await page.route("**/api/presentation/action",async route=>{await route.fetch();await route.abort();},{times:1});await page.locator("#save").click();await expect.poll(async()=>(await inspect(workbench.client)).saved?.art_ids[0]).toBe(painted); - await expect(page.locator("#status")).toContainText("Please check your saved work"); + await expect(page.locator("#status")).toHaveText("Saved."); await page.reload();await settle(page);await expect(page.locator("#status")).toHaveText("Saved.");await expect(page.locator("#save")).toBeDisabled();expect(await result(workbench.client)).toBe(painted); }); test("the agent records a user's explicit feedback while save alone does not approve art",async({page,workbench})=>{ @@ -209,11 +209,42 @@ test("browser polling does not prevent idle shutdown and reopening keeps the dra const id=await create(workbench.client,"idle");await show(workbench.client,[id]);const draft=await inspect(workbench.client); await workbench.client.tool("close_workbench",{workbench_id:workbench.client.workbenchId}); const short=await instance(workbench.client,{idle_timeout_seconds:2});await page.goto(short.url);await settle(page); + await page.evaluate(()=>setInterval(()=>document.dispatchEvent(new PointerEvent("pointermove",{bubbles:true})),100)); await expect.poll(async()=>(await workbench.client.tool("inspect_workbench",{})).structuredContent.state,{timeout:7000}).toBe("closed"); expect(await isServing(page,short.url)).toBe(false); const renewed=await instance(workbench.client);expect(renewed.workbench_id).not.toBe(short.workbench_id);expect((await inspect(workbench.client)).state_id).toBe(draft.state_id); expect((await workbench.client.tool("close_workbench",{workbench_id:short.workbench_id})).structuredContent.error.code).toBe("workbench_conflict"); expect(await isServing(page,renewed.url)).toBe(true); + expect((await (await page.request.get(renewed.url+"/api/workbench")).json()).workbench_id).toBe(renewed.workbench_id); +}); + +for(const activity of ["scrolling","choosing a color","dragging"])test(`${activity} renews idle time without changing the workbench address`,async({page,workbench})=>{ + await page.setViewportSize({width:600,height:1000}); + const id=await create(workbench.client,"active-view"); + await show(workbench.client,[id],{items:[{kind:"art",art_id:id,label:"Image to edit",region:{x:0,y:0,width:8,height:8},scale:64,editable:true}]}); + const original=await inspect(workbench.client); + await workbench.client.tool("close_workbench",{workbench_id:workbench.client.workbenchId}); + const opened=await instance(workbench.client,{idle_timeout_seconds:2});await page.goto(opened.url);await settle(page); + const canvas=page.locator("canvas.markable"),box=await canvas.boundingBox(); + if(activity==="scrolling")await page.locator(".drawing").hover(); + if(activity==="dragging"){ + await page.locator("#mark").click();await page.mouse.move(box.x+32,box.y+32);await page.mouse.down(); + } + // Keep interacting past the original deadline without completing a stored action. + for(let i=0;i<8;i++){ + await page.waitForTimeout(400); + if(activity==="scrolling")await page.mouse.wheel(i%2?-24:24,0); + else if(activity==="choosing a color")await color(page); + else await page.mouse.move(box.x+32+(i%2)*64,box.y+32); + } + const current=(await workbench.client.tool("inspect_workbench",{})).structuredContent; + expect(current.state).toBe("owned");expect(current.instance).toEqual(opened); + expect((await inspect(workbench.client)).state_id).toBe(original.state_id); + if(activity==="dragging"){ + await page.mouse.up();await expect.poll(async()=>(await marks(workbench.client))[0]?.pixels).toEqual([{x:0,y:0},{x:1,y:0}]); + } + // Stopping real input must still release an unattended screen despite browser polling. + await expect.poll(async()=>(await workbench.client.tool("inspect_workbench",{})).structuredContent.state,{timeout:7000}).toBe("closed"); }); test("HTTP cannot bypass managed MCP ownership for tool mutations",async({page,workbench})=>{ @@ -224,6 +255,201 @@ test("HTTP cannot bypass managed MCP ownership for tool mutations",async({page,w expect((await noIdentity.json()).error.code).toBe("workbench_conflict"); }); +test("an owning agent renews the same address across connections while continuing work",async({page,workbench})=>{ + await workbench.client.tool("close_workbench",{workbench_id:workbench.client.workbenchId}); + const opened=await instance(workbench.client,{idle_timeout_seconds:2}); + const other=mcpClient(workbench.workspace);await other.discover(); + try{ + for(let i=0;i<4;i++){ + await page.waitForTimeout(650); + const renewed=(await other.tool("open_workbench",{control_id:workbench.client.controlId,idle_timeout_seconds:1})).structuredContent; + expect(renewed.state).toBe("owned");expect(renewed.instance).toEqual(opened); + } + await stopClient(other); + expect(await isServing(page,opened.url)).toBe(true); + await expect.poll(async()=>(await workbench.client.tool("inspect_workbench",{})).structuredContent.state,{timeout:7000}).toBe("closed"); + }finally{await stopClient(other);} +}); + +test("unrelated art calls cannot keep an unattended workbench alive",async({page,workbench})=>{ + const id=await create(workbench.client,"agent-activity"); + await workbench.client.tool("close_workbench",{workbench_id:workbench.client.workbenchId}); + const opened=await instance(workbench.client,{idle_timeout_seconds:2}); + const other=mcpClient(workbench.workspace);await other.discover(); + try{ + for(let i=0;i<6;i++){ + await page.waitForTimeout(500); + const result=await other.tool(i%2?"render_art":"inspect_art",{art_id:id});expect(result.structuredContent.ok).toBe(true); + } + const current=(await workbench.client.tool("inspect_workbench",{})).structuredContent; + expect(current.state).toBe("closed"); + expect(await isServing(page,opened.url)).toBe(false); + }finally{await stopClient(other);} +}); + +test("declared agent work survives external waiting and returns to idle at handoff",async({page,workbench})=>{ + await workbench.client.tool("close_workbench",{workbench_id:workbench.client.workbenchId}); + const opened=await instance(workbench.client,{idle_timeout_seconds:1,work_state:"working"}); + const other=mcpClient(workbench.workspace);await other.discover(); + try{ + await page.waitForTimeout(2200); + const current=(await other.tool("inspect_workbench",{control_id:workbench.client.controlId})).structuredContent; + expect(current.state).toBe("owned");expect(current.instance).toEqual(opened);expect(current.work_state).toBe("working"); + expect((await other.tool("open_workbench",{control_id:workbench.client.controlId})).structuredContent.work_state).toBe("working"); + expect((await other.tool("open_workbench",{work_state:"waiting"})).structuredContent.error.code).toBe("workbench_busy"); + const handoff=(await other.tool("open_workbench",{control_id:workbench.client.controlId,work_state:"waiting"})).structuredContent; + expect(handoff.instance).toEqual(opened);expect(handoff.work_state).toBe("waiting"); + for(let i=0;i<7;i++){ + await page.waitForTimeout(250); + await other.tool("inspect_presentation",{}); + expect((await other.tool("inspect_art",{art_id:"missing"})).isError).toBe(true); + } + expect((await workbench.client.tool("inspect_workbench",{})).structuredContent.state).toBe("closed"); + }finally{await stopClient(other);} +}); + +test("a stalled committed save recovers without reloading or replaying it",async({page,workbench})=>{ + const id=await create(workbench.client,"stalled-save");await show(workbench.client,[id]);await settle(page);await color(page);await point(page,1,1); + await expect.poll(()=>result(workbench.client)).not.toBe(id); + let requests=0; + page.on("request",request=>{if(request.url().endsWith("/api/presentation/action"))requests++;}); + await page.evaluate(()=>{ + const fetch=window.fetch; + let release;const blocked=new Promise(resolve=>release=resolve); + window.delayedSave={ready:false,settled:false,release}; + window.fetch=async(...args)=>{ + const response=await fetch(...args); + if(String(args[0]).endsWith("/api/presentation/action")&&JSON.parse(args[1].body).action==="save"){ + window.fetch=fetch; + const data=await response.json(); + // Delay parsed data so abort cannot prevent its eventual completion. + response.json=async()=>{window.delayedSave.ready=true;await blocked;window.delayedSave.settled=true;return data;}; + } + return response; + }; + }); + try{ + await page.locator("#save").click();await expect.poll(()=>page.evaluate(()=>window.delayedSave.ready)).toBe(true); + const saved=(await inspect(workbench.client)).saved; + await expect(page.locator("#status")).toHaveText("Saved.",{timeout:12000}); + await expect(page.locator("#mark")).toBeEnabled();expect(requests).toBe(1); + await point(page,2,1);await expect.poll(()=>result(workbench.client)).not.toBe(saved.art_ids[0]); + await expect(page.locator("#status")).toHaveText("Click Save when you are ready."); + const edited=await result(workbench.client); + await page.evaluate(async()=>{window.delayedSave.release();await new Promise(resolve=>requestAnimationFrame(()=>requestAnimationFrame(resolve)));}); + expect(await page.evaluate(()=>window.delayedSave.settled)).toBe(true); + await expect(page.locator("#status")).toHaveText("Click Save when you are ready.");await expect(page.locator("#save")).toBeEnabled(); + expect(await page.locator("canvas").evaluate(c=>Array.from(c.getContext("2d").getImageData(80,48,1,1).data))).toEqual([224,116,64,255]); + expect(await result(workbench.client)).toBe(edited);expect((await inspect(workbench.client)).saved).toEqual(saved);expect(requests).toBe(2); + }finally{await page.evaluate(()=>window.delayedSave.release());} +}); + +for(const lostResponse of [false,true])test(`a ${lostResponse?"recovered":"confirmed"} edit follows new artwork after its image fails to load`,async({page,workbench})=>{ + const id=await create(workbench.client,"failed-edit-image"),replacement=await create(workbench.client,"replacement-image"); + await show(workbench.client,[id]);await settle(page);await color(page); + let failedArt; + await page.route("**/api/art/*",route=>{ + const artId=route.request().url().split("/").pop(); + if(artId!==id&&artId!==replacement){failedArt=artId;return route.fulfill({status:503,json:{ok:false,error:{code:"source_unavailable",message:"Temporary asset failure"}}});} + return route.continue(); + }); + if(lostResponse)await page.route("**/api/presentation/action",async route=>{await route.fetch();await route.abort();},{times:1}); + await point(page,1,1);await expect.poll(()=>result(workbench.client)).not.toBe(id);const edited=await result(workbench.client); + await expect(page.locator("#status")).toContainText("Connection lost");await expect(page.locator("#mark")).toBeDisabled();expect(failedArt).toBe(edited); + await show(workbench.client,[replacement],{title:"Replacement available"}); + await settle(page,"Replacement available");await expect(page.locator("#mark")).toBeEnabled(); + expect((await rows(workbench.client,edited))[1][1]).toBe(2); + await page.unroute("**/api/art/*");await color(page);await point(page,2,1); + await expect(page.locator("#undo")).toBeEnabled(); + expect((await rows(workbench.client,await result(workbench.client)))[1]).toEqual([0,0,2,0,0,0,0,0]); +}); + +test("recovery fences an undelivered stroke before allowing another edit",async({page,workbench})=>{ + const id=await create(workbench.client,"late-stroke");await show(workbench.client,[id]);await settle(page);await color(page); + let release;const blocked=new Promise(resolve=>release=resolve);let reached;const entered=new Promise(resolve=>reached=resolve);let request; + await page.route("**/api/presentation/action",async route=>{request=route.request();reached();await blocked;await route.abort().catch(()=>{});},{times:1}); + try{ + await point(page,1,1);await entered; + await expect(page.locator("#status")).toContainText("was not applied",{timeout:12000}); + await expect(page.locator("#mark")).toBeEnabled();expect(await result(workbench.client)).toBe(id); + const late=await page.request.post(request.url(),{headers:request.headers(),data:request.postDataJSON()}); + expect((await late.json()).error.code).toBe("action_cancelled");expect(await result(workbench.client)).toBe(id); + await point(page,2,1);await expect.poll(()=>result(workbench.client)).not.toBe(id); + expect((await rows(workbench.client,await result(workbench.client)))[1]).toEqual([0,0,2,0,0,0,0,0]); + }finally{release();} +}); + +test("a disconnected screen disables editing and recovers after reconnection",async({page,workbench})=>{ + const id=await create(workbench.client,"connection");await show(workbench.client,[id]);await settle(page);await color(page); + await page.context().setOffline(true); + await expect(page.locator("#status")).toContainText("Connection lost"); + await expect(page.locator("#mark")).toBeDisabled();await expect(page.locator(".swatch").first()).toBeDisabled(); + await point(page,1,1);expect(await result(workbench.client)).toBe(id); + await page.context().setOffline(false);await expect(page.locator("#mark")).toBeEnabled(); + await point(page,1,1);await expect.poll(()=>result(workbench.client)).not.toBe(id); + await workbench.client.tool("close_workbench",{workbench_id:workbench.client.workbenchId}); + // Shutdown can reject an in-flight request or close the connection first. + await expect(page.locator("#status")).toHaveText(/^(Connection lost\.|This view has ended\.)/); + await expect(page.locator("#mark")).toBeDisabled();await expect(page.locator(".swatch").first()).toBeDisabled(); + const closedArt=await result(workbench.client);await point(page,2,1);expect(await result(workbench.client)).toBe(closedArt); +}); + +test("an ended workbench is distinguished from newly presented artwork",async({page,workbench})=>{ + const id=await create(workbench.client,"execution-conflict");await show(workbench.client,[id]);await settle(page);await color(page); + const before=await inspect(workbench.client); + await page.route("**/api/presentation/action",route=>route.continue({headers:{...route.request().headers(),"x-retro-art-workbench":"old-instance"}}),{times:1}); + await point(page,1,1); + await expect(page.locator("#status")).toContainText("Ask your agent to reopen");await expect(page.locator("#mark")).toBeDisabled(); + expect((await inspect(workbench.client)).state_id).toBe(before.state_id); +}); + +test("failed recovery keeps edits disabled until the saved result is confirmed",async({page,workbench})=>{ + const id=await create(workbench.client,"retry-recovery");await show(workbench.client,[id]);await settle(page);await color(page);await point(page,1,1); + await expect.poll(()=>result(workbench.client)).not.toBe(id);await expect(page.locator("#undo")).toBeEnabled(); + let requests=0,recover=true; + page.on("request",request=>{if(request.url().endsWith("/api/presentation/action"))requests++;}); + await page.route("**/api/presentation/action",async route=>{await route.fetch();await route.abort();},{times:1}); + await page.route("**/api/presentation/recover",route=>recover?route.abort():route.continue()); + await page.locator("#save").click();await expect.poll(async()=>(await inspect(workbench.client)).saved).not.toBeNull(); + await expect(page.locator("#status")).toHaveClass(/error/); + await expect(page.locator("#mark")).toBeDisabled();await expect(page.locator("#save")).toBeDisabled(); + await expect(page.locator("#status")).toContainText("Connection lost"); + const saved=(await inspect(workbench.client)).saved; + expect(await page.locator("canvas").evaluate(c=>Array.from(c.getContext("2d").getImageData(48,48,1,1).data))).toEqual([224,116,64,255]); + recover=false;await expect(page.locator("#status")).toHaveText("Saved.");await expect(page.locator("#mark")).toBeEnabled(); + expect(requests).toBe(1);expect((await inspect(workbench.client)).saved).toEqual(saved); +}); + +test("art tool work in another project does not keep an unattended screen alive",async({page,workbench})=>{ + const other=mcpClient(join(workbench.workspace,"other-project"));await other.discover(); + try{ + const id=await create(other,"unrelated-activity"); + await workbench.client.tool("close_workbench",{workbench_id:workbench.client.workbenchId}); + await instance(workbench.client,{idle_timeout_seconds:2}); + for(let i=0;i<6;i++){ + await page.waitForTimeout(500); + expect((await other.tool("inspect_art",{art_id:id})).structuredContent.ok).toBe(true); + } + expect((await workbench.client.tool("inspect_workbench",{})).structuredContent.state).toBe("closed"); + }finally{await stopClient(other);} +}); + +test("missing or stale instance IDs and foreign origins cannot renew idle time",async({page,workbench})=>{ + const stale=workbench.client.workbenchId; + await workbench.client.tool("close_workbench",{workbench_id:stale}); + const opened=await instance(workbench.client,{idle_timeout_seconds:2}); + const url=opened.url+"/api/workbench/activity"; + const missing=await page.request.post(url);expect((await missing.json()).error.code).toBe("workbench_conflict"); + const foreign=await page.request.post(url,{headers:{"x-retro-art-workbench":opened.workbench_id,Origin:"https://example.com"}});expect(foreign.status()).toBe(403); + for(let i=0;i<4;i++){ + const response=await page.request.post(url,{headers:{"x-retro-art-workbench":stale}}); + expect((await response.json()).error.code).toBe("workbench_conflict"); + await page.waitForTimeout(400); + } + // Valid renewals during that interval would move shutdown beyond this deadline. + await expect.poll(async()=>(await workbench.client.tool("inspect_workbench",{})).structuredContent.state,{timeout:1200,intervals:[100]}).toBe("closed"); +}); + test("an unfinished HTTP body cannot keep a closed workbench alive",async({page,workbench})=>{ const address=new URL(workbench.url); @@ -324,6 +550,72 @@ async function stroke(page,canvas,from,to,button="left"){ } async function marks(client){return (await inspect(client)).state.concerns||[];} +test("the human saves several candidates and the agent continues from those exact choices",async({page,workbench},testInfo)=>{ + const base=await create(workbench.client,"candidate-options"); + const ids=[]; + for(const x of [1,3,5])ids.push((await workbench.client.tool("edit_art",{art_id:base,write_region:{x:0,y:0,width:8,height:8},operations:[{kind:"fill_rect",rect:{x,y:1,width:2,height:6},index:2}]})).structuredContent.art_id); + const items=[base,...ids].map((art_id,index)=>({kind:"art",art_id,label:index?`Option ${index}`:"Original",region:{x:0,y:0,width:8,height:8},scale:20,editable:false})); + const initial=await show(workbench.client,ids,{title:"Choose directions to explore",items,candidate_choices:{item_indices:[1,2,3]}}); + await expect(page.locator("#title")).toHaveText("Choose directions to explore"); + await expect(page.getByRole("button",{name:"Choose Original",exact:true})).toHaveCount(0); + const first=page.getByRole("button",{name:"Choose Option 1",exact:true}); + const second=page.getByRole("button",{name:"Choose Option 2",exact:true}); + const third=page.getByRole("button",{name:"Choose Option 3",exact:true}); + await first.click();await expect(first).toHaveAttribute("aria-pressed","true"); + await third.focus();await page.keyboard.press("Space");await expect(third).toHaveAttribute("aria-pressed","true");await expect(third).toBeFocused(); + await expect(second).toHaveAttribute("aria-pressed","false"); + const chosen=[{item_index:1,art_id:ids[0]},{item_index:3,art_id:ids[2]}]; + await expect.poll(async()=>(await inspect(workbench.client)).state.chosen_candidates).toEqual(chosen); + await page.keyboard.press(`${modifier}+s`);await expect(page.locator("#status")).toHaveText("Saved."); + const saved=(await inspect(workbench.client)).saved;expect(saved.chosen_candidates).toEqual(chosen);expect(saved.art_ids).toEqual([base,...ids]); + await page.reload();await expect(first).toHaveAttribute("aria-pressed","true");await expect(third).toHaveAttribute("aria-pressed","true"); + await first.click();await expect(first).toHaveAttribute("aria-pressed","false");expect((await inspect(workbench.client)).saved).toEqual(saved); + await first.click();await expect(first).toHaveAttribute("aria-pressed","true"); + await page.screenshot({path:testInfo.outputPath("candidate-choices.png"),fullPage:true}); + const next=[]; + for(const candidate of saved.chosen_candidates){ + const output=(await workbench.client.tool("edit_art",{art_id:candidate.art_id,write_region:{x:0,y:0,width:8,height:8},operations:[{kind:"set_pixels",pixels:[{x:0,y:0,index:2}]}]})).structuredContent; + expect(output.ok).toBe(true);expect(output.base_art_id).toBe(candidate.art_id);next.push(output.art_id); + } + await show(workbench.client,next,{title:"Refined choices",items:next.map((art_id,i)=>({kind:"art",art_id,label:`Refinement ${i+1}`,region:{x:0,y:0,width:8,height:8},scale:20,editable:false})),candidate_choices:{item_indices:[0,1]}}); + await expect(page.locator("#title")).toHaveText("Refined choices");await expect(page.locator('.candidate-choice[aria-pressed="true"]')).toHaveCount(0); + const history=(await workbench.client.tool("inspect_presentation",{presentation_id:initial.presentation_id,state_id:saved.state_id})).structuredContent.presentation; + expect(history.state.chosen_candidates).toEqual(chosen); +}); + +test("lost choice responses and a stale click preserve the right candidates",async({page,workbench})=>{ + const ids=[await create(workbench.client,"choice-a"),await create(workbench.client,"choice-b")]; + const options={items:ids.map((art_id,i)=>({kind:"art",art_id,label:`Option ${i+1}`,region:{x:0,y:0,width:8,height:8},scale:16,editable:false})),candidate_choices:{item_indices:[0,1]}}; + await show(workbench.client,ids,options); + const button=page.getByRole("button",{name:"Choose Option 1",exact:true});await expect(button).toBeVisible(); + await page.route("**/api/presentation/action",async route=>{await route.fetch();await route.abort();},{times:1}); + await button.click();await expect(button).toHaveAttribute("aria-pressed","true"); + expect((await inspect(workbench.client)).state.chosen_candidates).toEqual([{item_index:0,art_id:ids[0]}]); + let release,reached;const blocked=new Promise(resolve=>release=resolve),entered=new Promise(resolve=>reached=resolve); + await page.route("**/api/presentation/action",async route=>{reached();await blocked;await route.continue();},{times:1}); + await page.getByRole("button",{name:"Choose Option 2",exact:true}).click();await entered; + const next=await show(workbench.client,ids,{...options,title:"A new comparison"}); + release();await expect(page.locator("#title")).toHaveText("A new comparison");await expect(page.locator('.candidate-choice[aria-pressed="true"]')).toHaveCount(0); + const current=await inspect(workbench.client);expect(current.state_id).toBe(next.state_id);expect(current.state.chosen_candidates||[]).toEqual([]); +}); + +test("marking an overflowing picture preserves its scroll position through undo and save",async({page,workbench})=>{ + await page.setViewportSize({width:700,height:700}); + const id=(await workbench.client.tool("create_art",{target:{resource_id:"scrolling-art",width:64,height:64,palette:["#000000","#E07440"],transparent_index:0,allowed_indices:[0,1],constraints_ref:null,requirements:[]},initial:{kind:"fill",index:1}})).structuredContent.art_id; + await show(workbench.client,[id],{items:[{kind:"art",art_id:id,label:"Large picture",region:{x:0,y:0,width:64,height:64},scale:16,editable:true}]});await settle(page); + await page.locator("#mark").click(); + await page.locator(".drawing").evaluate(well=>{well.scrollLeft=160;window.scrollTo(0,300);}); + const scroll=()=>page.locator(".drawing").evaluate(well=>({left:well.scrollLeft,top:well.scrollTop,pageY:window.scrollY})); + const before=await scroll();expect(before.left).toBeGreaterThan(0);expect(before.pageY).toBeGreaterThan(0); + const box=await page.locator(".drawing").boundingBox(); + await page.mouse.move(box.x+50,300);await page.mouse.down();await page.mouse.move(box.x+220,350,{steps:12});await page.mouse.up(); + await expect.poll(async()=>(await marks(workbench.client))[0]?.pixels.length||0).toBeGreaterThan(1); + await expect(page.locator("#undo")).toBeEnabled(); + expect(await scroll()).toEqual(before); + await page.keyboard.press(`${modifier}+z`);await expect.poll(()=>marks(workbench.client)).toEqual([]);await expect(page.locator("#save")).toBeEnabled();await expect(page.locator("#undo")).toBeDisabled();expect(await scroll()).toEqual(before); + await page.keyboard.press(`${modifier}+s`);await expect(page.locator("#status")).toHaveText("Saved.");expect(await scroll()).toEqual(before); +}); + test("issue mode adds and removes exact pixels without painting and the agent reads the saved marks",async({page,workbench},testInfo)=>{ const id=await create(workbench.client,"marked-crop"); await show(workbench.client,[id],{items:[{kind:"art",art_id:id,label:"Hair detail",region:{x:2,y:3,width:4,height:3},scale:32,editable:true}]});await settle(page); @@ -382,7 +674,7 @@ test("lost marking responses and late strokes preserve the right presentation",a const a=await create(workbench.client,"mark-a"),b=await create(workbench.client,"mark-b");await show(workbench.client,[a]);await settle(page);await page.locator("#mark").click(); await page.route("**/api/presentation/action",async route=>{await route.fetch();await route.abort();},{times:1}); await stroke(page,page.locator("canvas.markable"),[1,1],[1,1]); - await expect(page.locator("#status")).toContainText("Please check your saved work"); + await expect(page.locator("#status")).toHaveText("Click Save when you are ready."); await expect.poll(async()=>(await marks(workbench.client))[0]?.pixels).toEqual([{x:1,y:1}]); const original=await inspect(workbench.client); let release,reached;const blocked=new Promise(r=>release=r),entered=new Promise(r=>reached=r); diff --git a/tests/presentation_workflows.rs b/tests/presentation_workflows.rs index 2d1c001..6bef54c 100644 --- a/tests/presentation_workflows.rs +++ b/tests/presentation_workflows.rs @@ -523,3 +523,173 @@ fn legacy_presentation_payloads_keep_their_ids_and_undo_after_marking_is_added() let marked = act(&mut w, mark(&undone, 0, json!([{"x":1,"y":1}]), true)); assert_eq!(marked["state"]["concerns"][0]["art_id"], base); } + +fn offer(w: &mut Workspace, ids: &[String]) -> Value { + let mut args = arguments(w, ids); + for item in args["items"].as_array_mut().unwrap() { + item["editable"] = json!(false); + } + args["candidate_choices"] = json!({"item_indices":(0..ids.len()).collect::>()}); + call(w, "present_art", args)["presentation"].clone() +} + +fn choose(view: &Value, indices: &[usize]) -> Value { + let mut input = command(view, "choose"); + input["item_indices"] = json!(indices); + input +} + +#[test] +fn multiple_candidate_choices_are_saved_without_editing_or_approving_art() { + let dir = tempfile::tempdir().unwrap(); + let mut w = Workspace::open(dir.path()).unwrap(); + let ids = [ + art(&mut w, "first"), + art(&mut w, "second"), + art(&mut w, "third"), + ]; + let request = call( + &mut w, + "request_edit", + json!({"base_art_id":ids[0],"write_region":{"x":0,"y":0,"width":8,"height":8},"instruction":"Explore alternative shapes"}), + ); + let view = offer(&mut w, &ids); + assert!(view["state"].get("chosen_candidates").is_none()); + let selected = act(&mut w, choose(&view, &[2, 0])); + let choices = json!([{"item_index":0,"art_id":ids[0]},{"item_index":2,"art_id":ids[2]}]); + assert_eq!(selected["state"]["chosen_candidates"], choices); + assert_eq!(selected["state"]["art_ids"], json!(ids)); + assert_eq!(selected["undo_available"], false); + let marked = act(&mut w, mark(&selected, 0, json!([{"x":1,"y":1}]), true)); + let changed = act(&mut w, choose(&marked, &[1])); + let undone = act(&mut w, command(&changed, "undo")); + assert_eq!( + undone["state"]["chosen_candidates"], + changed["state"]["chosen_candidates"] + ); + assert!(undone["state"].get("concerns").is_none()); + let selected = act(&mut w, choose(&undone, &[0, 2])); + let saved = act(&mut w, command(&selected, "save")); + assert_eq!(saved["saved"]["chosen_candidates"], choices); + assert_eq!(saved["dirty"], false); + drop(w); + let mut w = Workspace::open(dir.path()).unwrap(); + assert_eq!(current(&mut w)["saved"], saved["saved"]); + let cleared = act(&mut w, choose(&saved, &[])); + assert!(cleared["state"].get("chosen_candidates").is_none()); + assert_eq!(cleared["saved"]["chosen_candidates"], choices); + let cleared = act(&mut w, command(&cleared, "save")); + assert!(cleared["saved"].get("chosen_candidates").is_none()); + let old = call( + &mut w, + "inspect_presentation", + json!({"presentation_id":view["presentation_id"],"state_id":selected["state_id"]}), + ); + assert_eq!(old["presentation"]["state"]["chosen_candidates"], choices); + let next = offer(&mut w, &ids); + assert!(next["state"].get("chosen_candidates").is_none()); + assert_eq!(count(&mut w), ids.len()); + let request = call( + &mut w, + "inspect_edit_request", + json!({"request_id":request["request_id"]}), + ); + assert_eq!(request["human_review"], "pending"); + assert!(request["result_art_id"].is_null()); +} + +#[test] +fn choices_reject_invalid_candidates_and_leave_the_presentation_unchanged() { + let dir = tempfile::tempdir().unwrap(); + let mut w = Workspace::open(dir.path()).unwrap(); + let ids = [ + art(&mut w, "first"), + art(&mut w, "second"), + art(&mut w, "comparison"), + ]; + let view = show(&mut w, &ids); + assert!(view["presentation"].get("candidate_choices").is_none()); + assert!( + w.human_action(serde_json::from_value(choose(&view, &[0])).unwrap()) + .is_err() + ); + let mut args = arguments(&mut w, &ids); + for item in args["items"].as_array_mut().unwrap() { + item["editable"] = json!(false); + } + for indices in [json!([]), json!([0]), json!([0, 0]), json!([0, 3])] { + args["candidate_choices"] = json!({"item_indices":indices}); + assert!(w.call("present_art", args.clone()).is_err()); + } + args["candidate_choices"] = json!({"item_indices":[0,1]}); + args["items"][0]["editable"] = json!(true); + assert!(w.call("present_art", args.clone()).is_err()); + args["items"][0]["editable"] = json!(false); + let mut duplicate = args.clone(); + duplicate["items"][1]["art_id"] = json!(ids[0]); + assert!(w.call("present_art", duplicate).is_err()); + let mut playback = args.clone(); + playback["items"][0]["playback"] = json!([]); + assert!(w.call("present_art", playback).is_err()); + assert_eq!(current(&mut w)["state_id"], view["state_id"]); + let offered = call(&mut w, "present_art", args)["presentation"].clone(); + for indices in [&[0, 0][..], &[2], &[0, 1, 2], &[usize::MAX]] { + assert!( + w.human_action(serde_json::from_value(choose(&offered, indices)).unwrap()) + .is_err() + ); + assert_eq!(current(&mut w)["state_id"], offered["state_id"]); + } + assert!( + w.human_action(serde_json::from_value(paint(&offered, 0, 0)).unwrap()) + .is_err() + ); + assert_eq!(count(&mut w), ids.len()); +} + +#[test] +fn candidate_choice_conflicts_retries_and_storage_failures_preserve_the_latest_choice() { + let dir = tempfile::tempdir().unwrap(); + let mut w = Workspace::open(dir.path()).unwrap(); + let ids = [art(&mut w, "first"), art(&mut w, "second")]; + let view = offer(&mut w, &ids); + let input = choose(&view, &[0]); + let db = rusqlite::Connection::open(dir.path().join(".dotmend/art.sqlite")).unwrap(); + db.execute_batch("CREATE TRIGGER reject_choice BEFORE INSERT ON presentation_states WHEN json_extract(NEW.payload,'$.action')='choose' BEGIN SELECT RAISE(FAIL,'test write failure'); END;").unwrap(); + assert_eq!( + w.human_action(serde_json::from_value(input.clone()).unwrap()) + .err() + .unwrap() + .code, + "storage_error" + ); + assert_eq!(current(&mut w)["state_id"], view["state_id"]); + db.execute_batch("DROP TRIGGER reject_choice").unwrap(); + let first = act(&mut w, input.clone()); + let mut other = Workspace::open(dir.path()).unwrap(); + assert_eq!( + other + .human_action(serde_json::from_value(choose(&view, &[1])).unwrap()) + .err() + .unwrap() + .code, + "presentation_conflict" + ); + let second = act(&mut other, choose(&first, &[1])); + assert_eq!(act(&mut w, input)["state_id"], second["state_id"]); + let save = command(&second, "save"); + let saved = act(&mut w, save.clone()); + let changed = act(&mut w, choose(&saved, &[0, 1])); + let latest = act(&mut w, command(&changed, "save")); + assert_eq!(act(&mut w, save)["saved"], latest["saved"]); + let next = offer(&mut w, &ids); + assert_eq!( + w.human_action(serde_json::from_value(choose(&latest, &[])).unwrap()) + .err() + .unwrap() + .code, + "presentation_conflict" + ); + assert_eq!(current(&mut w)["state_id"], next["state_id"]); + assert_eq!(count(&mut w), ids.len()); +} diff --git a/tests/project_workspaces.rs b/tests/project_workspaces.rs index e4c29b5..0222eb5 100644 --- a/tests/project_workspaces.rs +++ b/tests/project_workspaces.rs @@ -16,6 +16,7 @@ impl Client { fn open(project: &Path) -> Self { let mut child = Command::new(env!("CARGO_BIN_EXE_dotmend")) .current_dir(project) + .env("DOTMEND_RUNTIME_DIR", project.join("runtime")) .stdin(Stdio::piped()) .stdout(Stdio::piped()) .spawn() @@ -28,6 +29,10 @@ impl Client { } } fn call(&mut self, name: &str, arguments: Value) -> Value { + self.send(name, arguments); + self.receive() + } + fn send(&mut self, name: &str, arguments: Value) { self.sequence += 1; let request = json!({"jsonrpc":"2.0","id":self.sequence,"method":"tools/call","params":{ "name":name,"arguments":arguments,"_meta":{ @@ -36,6 +41,8 @@ impl Client { } }}); writeln!(self.child.stdin.as_mut().unwrap(), "{request}").unwrap(); + } + fn receive(&mut self) -> Value { let mut line = String::new(); self.output.read_line(&mut line).unwrap(); let reply: Value = serde_json::from_str(&line).unwrap(); @@ -86,6 +93,71 @@ fn launch_directory_isolates_projects_and_shares_records_on_reconnect() { assert_eq!(resumed.call("list_art", json!({}))["arts"][0]["art_id"], id); } +#[test] +fn declared_work_preserves_long_calls_until_handoff_or_host_death() { + for killed in [false, true] { + let project = tempfile::tempdir().unwrap(); + let mut host = Client::open(project.path()); + let mut worker = Client::open(project.path()); + assert_eq!(worker.call("list_art", json!({}))["ok"], true); + let control = "active-tool-workbench-owner"; + let opened = host.call( + "open_workbench", + json!({"control_id":control,"idle_timeout_seconds":1,"work_state":"working"}), + ); + assert_eq!(opened["ok"], true, "{opened}"); + let database = + rusqlite::Connection::open(project.path().join(".dotmend/art.sqlite")).unwrap(); + database.execute_batch("BEGIN IMMEDIATE").unwrap(); + worker.send( + "create_art", + json!({"target":{ + "resource_id":"busy-art","width":1,"height":1,"palette":["#000000"], + "transparent_index":null,"allowed_indices":[0],"constraints_ref":null,"requirements":[] + },"initial":{"kind":"fill","index":0}}), + ); + // A real tool waits on the SQLite write lock longer than the idle deadline. + std::thread::sleep(std::time::Duration::from_millis(1600)); + let current = host.call("inspect_workbench", json!({"control_id":control})); + if killed { + worker.child.kill().unwrap(); + worker.child.wait().unwrap(); + } + database.execute_batch("ROLLBACK").unwrap(); + if !killed { + let result = worker.receive(); + assert_eq!(result["ok"], true, "{result}"); + } + assert_eq!(current["state"], "owned", "{current}"); + assert_eq!(current["instance"], opened["instance"]); + if killed { + host.child.kill().unwrap(); + host.child.wait().unwrap(); + let mut replacement = Client::open(project.path()); + let reopened = replacement.call("open_workbench", json!({"control_id":control})); + assert_eq!(reopened["state"], "owned"); + assert_ne!( + reopened["instance"]["workbench_id"], + opened["instance"]["workbench_id"] + ); + assert_eq!(reopened["work_state"], "waiting"); + } else { + assert_eq!( + host.call( + "open_workbench", + json!({"control_id":control,"work_state":"waiting"}) + )["work_state"], + "waiting" + ); + std::thread::sleep(std::time::Duration::from_millis(1600)); + assert_eq!( + host.call("inspect_workbench", json!({"control_id":control}))["state"], + "closed" + ); + } + } +} + #[test] fn project_storage_is_ignored_without_changing_existing_ignore_rules() { let temp = tempfile::tempdir().unwrap(); diff --git a/web/app.js b/web/app.js index a6e06fb..73eb2e7 100644 --- a/web/app.js +++ b/web/app.js @@ -1,7 +1,8 @@ "use strict"; const $=id=>document.getElementById(id); -const state={workbenchId:null,view:null,arts:new Map(),sources:new Map(),colors:new Map(),paletteChoices:new Map(),marking:false,stroke:null,busy:false,refresh:0,polling:false,timers:new Set()}; +const state={workbenchId:null,connection:"checking",pending:null,activityInterval:30000,lastActivity:-Infinity,renewing:false,view:null,arts:new Map(),sources:new Map(),colors:new Map(),paletteChoices:new Map(),marking:false,stroke:null,busy:false,refresh:0,polling:null,timers:new Set()}; const MAX_VISIBLE_COLORS=20; +const REQUEST_TIMEOUT=5000; function visibleColors(art,region){ const allowed=art.target.allowed_indices; if(allowed.length<=MAX_VISIBLE_COLORS)return [...allowed]; @@ -15,23 +16,58 @@ function visibleColors(art,region){ return [...erase,...ranked.slice(0,MAX_VISIBLE_COLORS-erase.length)]; } function status(text,error=false){$("status").textContent=text;$("status").classList.toggle("error",error);} -async function json(url,options){const response=await fetch(url,{cache:"no-store",...options,headers:{...options?.headers,...(state.workbenchId?{"x-retro-art-workbench":state.workbenchId}:{})}});const data=await response.json();if(!response.ok||data.ok===false){const error=new Error(data.error?.message||"Please check your connection.");error.code=data.error?.code;throw error;}return data;} +async function deadline(operation,cancel=()=>{}){ + let timer; + try{return await Promise.race([operation,new Promise((_,reject)=>{timer=setTimeout(()=>{const error=new Error("The response timed out.");error.code="request_timeout";reject(error);cancel();},REQUEST_TIMEOUT);})]);} + finally{clearTimeout(timer);} +} +async function json(url,options){ + const controller=new AbortController(); + return deadline((async()=>{ + const response=await fetch(url,{cache:"no-store",...options,signal:controller.signal,headers:{...options?.headers,...(state.workbenchId?{"x-retro-art-workbench":state.workbenchId}:{})}}); + const data=await response.json(); + if(!response.ok||data.ok===false){const error=new Error(data.error?.message||"Please check your connection.");error.code=data.error?.code;throw error;} + return data; + })(),()=>controller.abort()); +} +function canAct(){return state.connection==="ready"&&!state.pending&&!state.busy&&!state.stroke;} +function connectionFailed(error){ + state.connection=error.code==="workbench_conflict"?"closed":"offline";controls(); + status(state.connection==="closed"?"This view has ended. Ask your agent to reopen it.":state.pending?"Connection lost. Checking your last change before you continue.":"Connection lost. Checking your saved work.",true); +} + +async function renewActivity(event){ + if(state.connection!=="ready"||!event.isTrusted||document.visibilityState!=="visible"||!state.workbenchId||state.renewing||performance.now()-state.lastActivity{state.arts.delete(id);throw error;}));return state.arts.get(id);} -function loadSource(hash){if(!state.sources.has(hash))state.sources.set(hash,new Promise((resolve,reject)=>{const img=new Image();img.onload=()=>resolve(img);img.onerror=()=>{state.sources.delete(hash);reject(new Error("Could not load the reference image."));};img.src=`/api/source/${hash}`;}));return state.sources.get(hash);} -async function loadAnimation(item){const result=await json("/api/call",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({tool:"render_art_set",arguments:{frames:item.playback,scale:item.scale,view:"frames"}})});result.loaded=await Promise.all(result.images.map(src=>new Promise((resolve,reject)=>{const image=new Image();image.onload=()=>resolve(image);image.onerror=()=>reject(new Error("Could not load the frames."));image.src=src;})));return result;} +function loadImage(src){const img=new Image();return deadline(new Promise((resolve,reject)=>{img.onload=()=>resolve(img);img.onerror=()=>reject(new Error("Could not load the image."));img.src=src;}),()=>{img.src="";});} +function loadSource(hash){if(!state.sources.has(hash))state.sources.set(hash,loadImage(`/api/source/${hash}`).catch(error=>{state.sources.delete(hash);throw error;}));return state.sources.get(hash);} +async function loadAnimation(item){const result=await json("/api/call",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({tool:"render_art_set",arguments:{frames:item.playback,scale:item.scale,view:"frames"}})});result.loaded=await Promise.all(result.images.map(loadImage));return result;} + function stopAnimations(){for(const timer of state.timers)clearTimeout(timer);state.timers.clear();} function later(fn,delay){const timer=setTimeout(()=>{state.timers.delete(timer);fn();},Math.min(delay,60000));state.timers.add(timer);} function animate(canvas,result){let index=0;const token=state.refresh;canvas.width=result.display_width;canvas.height=result.display_height;const ctx=canvas.getContext("2d");ctx.imageSmoothingEnabled=false;function frame(){if(token!==state.refresh)return;ctx.clearRect(0,0,canvas.width,canvas.height);checker(ctx,Math.ceil(canvas.width/8),Math.ceil(canvas.height/8),8);ctx.drawImage(result.loaded[result.frames[index].image_index],0,0);const end=performance.now()+result.frames[index].duration_ms;function wait(){if(token!==state.refresh)return;const remaining=end-performance.now();if(remaining>0)later(wait,remaining);else{index=(index+1)%result.frames.length;frame();}}later(wait,result.frames[index].duration_ms);}frame();} function editable(){return state.view?.presentation.items.some(i=>i.kind==="art"&&i.editable);} +function choiceIndices(){return state.view?.presentation.candidate_choices?.item_indices||[];} +function chosenIndices(){return (state.view?.state.chosen_candidates||[]).map(candidate=>candidate.item_index);} +function choiceHint(){return "Choose any candidates you want to explore, then Save to share your choice. Click a chosen candidate again to remove it.";} function markable(item){return item.kind==="art"&&!item.playback;} function canMark(){return state.view?.presentation.items.some(markable);} function controls(){ - const working=state.busy||!!state.stroke; + const working=!canAct(); $("undo").disabled=working||!state.view?.undo_available; - $("save").disabled=working||!(editable()||canMark())||!state.view?.dirty; + $("save").disabled=working||!(editable()||canMark()||choiceIndices().length)||!state.view?.dirty; $("mark").disabled=working||!canMark();$("mark").setAttribute("aria-pressed",String(state.marking)); - $("hint").hidden=!(editable()||canMark()); - $("hint").textContent=state.marking?"Left-click or drag to mark pixels. Right-click or drag to unmark. Choose a palette color to paint.":editable()?"Choose a color, then click the pixels you want to change. Use Mark issues to point out an area.":"Use Mark issues to point out pixels for your agent."; + $("hint").hidden=!(editable()||canMark()||choiceIndices().length); + $("hint").textContent=state.marking?(editable()?"Left-click or drag to mark pixels. Right-click or drag to unmark. Choose a palette color to paint.":"Left-click or drag to mark pixels. Right-click or drag to unmark."):choiceIndices().length?choiceHint():editable()?"Choose a color, then click the pixels you want to change. Use Mark issues to point out an area.":"Use Mark issues to point out pixels for your agent."; + const chosen=chosenIndices(); + for(const button of document.querySelectorAll(".candidate-choice")){ + const selected=chosen.includes(Number(button.dataset.item)); + button.disabled=working;button.setAttribute("aria-pressed",String(selected));button.textContent=selected?"Chosen":"Choose"; + button.closest(".picture").classList.toggle("chosen",selected); + } for(const canvas of document.querySelectorAll("canvas.markable"))canvas.classList.toggle("marking",state.marking); for(const swatch of document.querySelectorAll(".swatch")){ swatch.disabled=working; @@ -70,7 +106,7 @@ function traceStroke(event,canvas,item){ function attachBrush(canvas,item,index,art){ canvas.oncontextmenu=event=>{if(state.marking)event.preventDefault();}; canvas.onpointerdown=event=>{ - if(state.busy||state.stroke||!(state.marking?[0,2].includes(event.button):item.editable&&event.button===0))return; + if(!canAct()||!(state.marking?[0,2].includes(event.button):item.editable&&event.button===0))return; const color=state.colors.get(index); if(!state.marking&&color===undefined){status("Choose a color below first.");return;} event.preventDefault();canvas.setPointerCapture(event.pointerId); @@ -97,7 +133,11 @@ async function display(view,force=false){ if(!view){stopAnimations();state.view=null;$("pictures").replaceChildren();$("title").textContent="Waiting for artwork.";$("note").textContent="Tell your agent what you want to see.";$("note").hidden=false;$("hint").hidden=true;controls();return;} const assets=await Promise.all(view.presentation.items.map((item,i)=>item.playback?loadAnimation(item):item.kind==="art"?loadArt(view.state.art_ids[i]):loadSource(item.source_hash))); if(token!==state.refresh||state.busy||state.stroke)return; - stopAnimations();if(state.view?.presentation_id!==view.presentation_id){state.colors.clear();state.paletteChoices.clear();state.marking=view.presentation.items.some((item,index)=>markable(item)&&assets[index].target.allowed_indices.length>MAX_VISIBLE_COLORS);}state.view=view; + // A completed stroke replaces the drawing nodes; keep the viewport within this same presentation. + const samePresentation=state.view?.presentation_id===view.presentation_id; + const scrollPositions=samePresentation?[...document.querySelectorAll(".drawing")].map(well=>({left:well.scrollLeft,top:well.scrollTop})):[]; + const focusedChoice=samePresentation&&document.activeElement?.classList.contains("candidate-choice")?document.activeElement.dataset.item:null; + stopAnimations();if(state.view?.presentation_id!==view.presentation_id){state.colors.clear();state.paletteChoices.clear();state.marking=!view.presentation.candidate_choices&&view.presentation.items.some((item,index)=>markable(item)&&assets[index].target.allowed_indices.length>MAX_VISIBLE_COLORS);}state.view=view; $("title").textContent=view.presentation.title;$("note").textContent=view.presentation.note;$("note").hidden=!view.presentation.note;$("hint").hidden=!editable();$("pictures").replaceChildren(); view.presentation.items.forEach((item,index)=>{ const figure=document.createElement("figure");figure.className="picture";figure.dataset.item=String(index);const caption=document.createElement("figcaption");caption.textContent=item.label;const well=document.createElement("div");well.className="drawing";const canvas=document.createElement("canvas");canvas.width=item.region.width*item.scale;canvas.height=item.region.height*item.scale;canvas.setAttribute("aria-label",item.label);well.append(canvas);figure.append(caption,well); @@ -107,21 +147,69 @@ async function display(view,force=false){ if(!state.paletteChoices.has(index))state.paletteChoices.set(index,visibleColors(art,item.region)); for(const color of state.paletteChoices.get(index)){const swatch=document.createElement("button");swatch.className="swatch";swatch.dataset.index=String(color);swatch.dataset.item=String(index);const transparent=color===art.target.transparent_index;if(transparent)swatch.classList.add("transparent");else swatch.style.backgroundColor=art.target.palette[color];swatch.title=transparent?"Erase":art.target.palette[color];swatch.setAttribute("aria-label",transparent?"Erase":`Color ${art.target.palette[color]} · ${color+1}`);swatch.setAttribute("aria-pressed",String(state.colors.get(index)===color));swatch.onclick=()=>{state.colors.set(index,color);state.marking=false;controls();status("Click the pixels you want to change.");};palette.append(swatch);}figure.append(palette); if(art.target.allowed_indices.length>MAX_VISIBLE_COLORS){const note=document.createElement("p");note.textContent="Showing up to 20 common colors. Mark areas to change and ask your agent to edit them.";figure.append(note);} - }}$("pictures").append(figure); - });controls();status((editable()||canMark())?(view.dirty?"Click Save when you are ready.":"Saved."):""); + }} + if(choiceIndices().includes(index)){ + const button=document.createElement("button");button.className="candidate-choice";button.dataset.item=String(index);button.setAttribute("aria-label",`Choose ${item.label}`); + button.onclick=()=>{const selected=chosenIndices();void act("choose",{item_indices:selected.includes(index)?selected.filter(i=>i!==index):[...selected,index]});};figure.append(button); + } + $("pictures").append(figure); + }); + document.querySelectorAll(".drawing").forEach((well,index)=>{const position=scrollPositions[index];if(position){well.scrollLeft=position.left;well.scrollTop=position.top;}}); + if(focusedChoice!==null)document.querySelector(`.candidate-choice[data-item="${focusedChoice}"]`)?.focus({preventScroll:true}); + controls();status((editable()||canMark()||choiceIndices().length)?(view.dirty?"Click Save when you are ready.":"Saved."):""); +} +function actionMessage(pending){ + const action=pending.input.action,error=pending.error; + if(pending.completed){ + if(state.view?.presentation_id!==pending.input.presentation_id)status("Your earlier change was recovered. Please check the current view."); + return; + } + status(error?.code==="presentation_conflict"?"New artwork has arrived. Please check the current view.":error?.code==="protected_pixel"||error?.code==="out_of_bounds"?"This area is protected. Tell your agent where you want to edit.":action==="mark"&&error?.code==="limit_exceeded"?"Too many marked pixels. Unmark some pixels or use a smaller area.":action==="save"?"Save was not applied. Your previous saved work is preserved.":"Change was not applied. Please try again.",true); +} +async function synchronize(force=false){ + if(state.connection==="closed"||state.polling||state.busy||state.stroke)return; + const flight={};state.polling=flight;let token=state.refresh; + try{ + if(state.connection!=="ready"){ + const instance=await json("/api/workbench"); + if(token!==state.refresh)return; + if(state.workbenchId&&state.workbenchId!==instance.workbench_id){const error=new Error("The workbench has changed.");error.code="workbench_conflict";throw error;} + state.workbenchId=instance.workbench_id;state.activityInterval=Math.min(30000,instance.idle_timeout_seconds*250); + } + const pending=state.pending; + if(pending&&pending.completed===undefined){ + const receipt=await json("/api/presentation/recover",{method:"POST",headers:{"Content-Type":"application/json","x-dotmend-action":pending.id},body:JSON.stringify(pending.input)}); + if(token!==state.refresh)return; + pending.completed=receipt.completed; + } + // Keep action completion across retries, but always read the current view. + const data=await json("/api/presentation"); + if(token!==state.refresh)return; + const displaying=display(data.presentation,force||!!pending||state.connection!=="ready");token=state.refresh;await displaying; + if(token!==state.refresh)return; + state.pending=null;state.connection="ready";controls(); + if(pending){actionMessage(pending);if(pending.focusedChoice!==null&&state.view?.presentation_id===pending.input.presentation_id)document.querySelector(`.candidate-choice[data-item="${pending.focusedChoice}"]`)?.focus({preventScroll:true});} + }catch(error){if(token===state.refresh&&!state.busy&&!state.stroke)connectionFailed(error);} + finally{if(state.polling===flight)state.polling=null;} } -async function synchronize(force=false){if(!state.workbenchId||state.polling||state.busy||state.stroke)return;state.polling=true;const token=state.refresh;try{const data=await json("/api/presentation");if(token===state.refresh)await display(data.presentation,force);}catch{if(token===state.refresh&&!state.busy&&!state.stroke)status("Connection lost. Checking your saved work.",true);}finally{state.polling=false;}} async function act(action,extra={},view=state.view){ - if(!view||state.busy||state.stroke)return;state.busy=true;state.refresh++;controls();status(action==="save"?"Saving…":""); - let error=null;let receipt=null; - try{receipt=await json("/api/presentation/action",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({action,presentation_id:view.presentation_id,expected_state_id:view.state_id,...extra})});} - catch(cause){error=cause;} + if(!view||!canAct())return; + const pending={id:crypto.randomUUID(),input:{action,presentation_id:view.presentation_id,expected_state_id:view.state_id,...extra},focusedChoice:document.activeElement?.classList.contains("candidate-choice")?document.activeElement.dataset.item:null}; + state.pending=pending;state.busy=true;state.polling=null;state.refresh++;controls();status(action==="save"?"Saving…":"Applying your change…"); + try{await json("/api/presentation/action",{method:"POST",headers:{"Content-Type":"application/json","x-dotmend-action":pending.id},body:JSON.stringify(pending.input)});pending.completed=true;} + catch(error){pending.error=error;} finally{state.busy=false;} - if(receipt)await display(receipt.presentation,true);else {await display(state.view,true);await synchronize(true);} - controls(); - if(error)status(action==="mark"&&error.code==="limit_exceeded"?"Too many marked pixels. Unmark some pixels or use a smaller area.":error.code==="protected_pixel"||error.code==="out_of_bounds"?"This area is protected. Tell your agent where you want to edit.":(error.code==="presentation_conflict"||error.code==="workbench_conflict")?"New artwork has arrived. Please check the current view.":(action==="save"?"Save was not confirmed. Please check your saved work.":"Change was not confirmed. Please check your saved work."),true); + state.connection="checking"; + // Restore the confirmed view while an uncertain action is settled by the server. + try{await display(state.view,true);}catch(error){connectionFailed(error);} + if(pending.error?.code==="workbench_conflict"){connectionFailed(pending.error);return;} + status(pending.completed===undefined?"Checking whether your change was applied…":"Checking the current view…"); + await synchronize(true); } -$("mark").onclick=()=>{state.marking=!state.marking;controls();status(state.marking?"Marked pixels are notes for your agent. The artwork stays unchanged.":"Choose a palette color to paint.");}; + +$("mark").onclick=()=>{state.marking=!state.marking;controls();status(state.marking?"Marked pixels are notes for your agent. The artwork stays unchanged.":choiceIndices().length?choiceHint():editable()?"Choose a palette color to paint.":"Use Mark issues to point out pixels for your agent.");}; $("undo").onclick=()=>void act("undo");$("save").onclick=()=>void act("save"); document.addEventListener("keydown",event=>{if(!(event.metaKey||event.ctrlKey)||event.altKey)return;const key=event.key.toLowerCase();if(key!=="z"&&key!=="s")return;event.preventDefault();if(event.shiftKey)return;const button=$(key==="z"?"undo":"save");if(!button.disabled)button.click();}); -window.addEventListener("focus",()=>void synchronize());setInterval(()=>{if(document.visibilityState==="visible")void synchronize();},1000);void json("/api/workbench").then(instance=>{state.workbenchId=instance.workbench_id;return synchronize();}).catch(()=>status("This view has closed. Ask your agent to reopen it.")); +// Only real input renews idle time; polling, playback and unattended tabs do not. +for(const type of ["pointerdown","pointermove","pointerup","wheel","keydown"])document.addEventListener(type,renewActivity,{capture:true,passive:true}); +window.addEventListener("focus",()=>void synchronize());window.addEventListener("online",()=>void synchronize());setInterval(()=>{if(document.visibilityState==="visible")void synchronize();},1000);controls();void synchronize(); diff --git a/web/style.css b/web/style.css index eb502e9..b7f177d 100644 --- a/web/style.css +++ b/web/style.css @@ -1 +1,2 @@ :root{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;color:#24382f;background:#f6f7f3;font-synthesis:none;--line:#dfe4db;--accent:#246e59}*{box-sizing:border-box}body{margin:0}button{font:inherit;cursor:pointer;touch-action:manipulation;border:1px solid var(--line);border-radius:8px;background:white;color:inherit;min-height:42px;padding:9px 16px}button:hover:enabled{background:#edf4ee}button:focus-visible{outline:3px solid #92bdaa;outline-offset:3px}button:disabled{opacity:.38;cursor:default}.primary{background:var(--accent);border-color:var(--accent);color:#fff}.primary:hover:enabled{background:#18533f}header{position:sticky;top:0;z-index:2;display:flex;align-items:center;justify-content:space-between;padding:16px 32px;background:#f6f7f3f5;border-bottom:1px solid var(--line);gap:18px}.brand{font-size:16px;font-weight:650;white-space:nowrap}.actions{display:flex;gap:8px;flex-wrap:wrap;justify-content:flex-end}#mark[aria-pressed=true]{background:#fff0c5;border-color:#995400;color:#704000}.drawing canvas.marking{outline:2px dashed #b17816;outline-offset:3px;cursor:cell}main{max-width:1440px;margin:auto;padding:40px 32px 84px}.heading{margin-bottom:28px}h1{font-size:24px;font-weight:600;margin:0 0 10px;line-height:1.4}p{font-size:14px;line-height:1.65;margin:0;color:#637267;white-space:pre-wrap}#hint{margin-top:10px;color:var(--accent)}.pictures{display:flex;flex-wrap:wrap;align-items:start;justify-content:center;gap:24px}.picture{flex:1 1 340px;min-width:0;max-width:100%;margin:0;padding:22px;border:1px solid var(--line);border-radius:14px;background:white}.picture figcaption{font-size:14px;font-weight:550;margin-bottom:18px;white-space:pre-wrap}.drawing{overflow:auto;display:flex;align-items:center;min-height:120px;padding:12px;border-radius:8px;background:#f1f3ee}.drawing canvas{display:block;margin:auto;image-rendering:pixelated;flex-shrink:0}.drawing canvas.editable{cursor:crosshair;touch-action:none}.drawing canvas.markable.marking{cursor:cell;touch-action:none}.palette{display:flex;justify-content:center;flex-wrap:wrap;gap:10px;max-width:620px;margin:22px auto 0;padding:4px}.swatch{width:36px;height:36px;min-height:36px;border:2px solid #fff;padding:0;border-radius:7px;box-shadow:0 0 0 1px #d7ded6;position:relative}.swatch[aria-pressed=true]{outline:3px solid var(--accent);outline-offset:3px}.swatch:hover:enabled{transform:translateY(-1px)}.swatch.transparent{background:repeating-conic-gradient(#c6cdc6 0% 25%,#fff 0% 50%) 0 0/10px 10px}.swatch[aria-pressed=true]::after{content:"";position:absolute;inset:10px;border:2px solid white;border-radius:50%;box-shadow:0 0 0 1px #273e36}footer{position:fixed;bottom:0;width:100%;min-height:48px;padding:14px 24px;text-align:center;background:#f6f7f3f5;color:#647368;font-size:13px;z-index:2}#status.error{color:#a5423f}[hidden]{display:none!important}@media(max-width:600px){header{padding:12px 16px}.brand{font-size:14px}button{padding:9px 12px}main{padding:28px 16px 74px}h1{font-size:21px}.picture{padding:16px}.pictures{gap:18px}} +.picture.chosen{border-color:var(--accent);box-shadow:0 0 0 2px var(--accent)}.candidate-choice{display:block;min-width:120px;margin:18px auto 0}.candidate-choice[aria-pressed=true]{background:var(--accent);border-color:var(--accent);color:white}.candidate-choice[aria-pressed=true]:hover:enabled{background:#18533f}