Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "dotmend"
version = "0.2.0"
version = "0.3.0"
edition = "2024"
publish = false
license = "MIT"
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ When a detail needs your touch, the screen has four actions:

Marks are saved separately from the artwork so your agent can inspect the exact locations.

When your agent offers alternatives, click **Choose** under any candidates you want to explore, then **Save**. You can choose several, or click a chosen option again to remove it. Your agent can refine those choices and bring back another set. Earlier candidates remain available.

Ask the agent to show a collection, filter the view, bring back an earlier candidate, or work on a particular area. You do not need to learn an editor to do that.

## What it supports
Expand Down
28 changes: 23 additions & 5 deletions resources/agent-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,43 @@ Dotmend uses the project directory supplied by the client at launch, or an expli

## Presenting work to a human

Humans paint with palette colors, mark suspected issues, undo the last stroke once and save. Handle collections, filters, past candidates, regions and reviews through conversation and tools.
Humans paint with palette colors, mark suspected issues, undo the last stroke once and save. They can also choose several alternatives when you offer candidate choices. Handle collections, filters, past candidates, regions and reviews through conversation and tools.

When first sharing the screen, briefly explain its available controls in the user's language: choose a palette color, then click or drag to paint; **Mark issues** uses left clicks/drags to add marks and right clicks/drags to remove them; **Undo** (Ctrl/Cmd+Z) reverses the last stroke once; **Save** (Ctrl/Cmd+S) records the current result. A click or drag is one stroke, and painting and marking share the same undo. Tell the user to ask you for older results or other operations. Adapt the explanation to read-only and playback views; repeat it only if asked or needed. Do not send the user to this guide or require IDs and technical settings.

For more than 20 allowed colors, the screen starts in **Mark issues** mode and shows up to 20 commonly used palette entries, including Erase when allowed. Explain this briefly in the user's language and invite them to mark areas and describe the desired change. Read saved concerns, focus the marked regions, resolve unclear intent, apply edits within the request's permissions and present the result. Keep color selection and palette details with the agent. The displayed subset stays fixed during editing; all original indices remain available to agent tools.

1. Choose a fresh random `control_id` per independent task (16..128 ASCII letters, digits, underscores or hyphens); retain it across calls and connections. Call `open_workbench({control_id})` for the instance ID and URL. Do not bypass another owner or the shared limit. Read `inspect_presentation({})`; no existing view returns `presentation:null`.
2. Call `present_art({control_id,workbench_id,view})`. Set `view.title`, `note` and `items` to describe the user's task. An art item is `{kind:"art",art_id,label,region,scale,editable,request_id?}`. Attach `request_id` to an editable item carrying out a protected request.
3. Copy the freshly read IDs into `view.expected_presentation_id` and `view.expected_state_id`. Both keys must be explicit `null` for the first view. On conflict, read the human's latest edits first.
1. Choose a fresh random `control_id` per independent task (16..128 ASCII letters, digits, underscores or hyphens); retain it across calls and connections. Call `open_workbench({control_id,work_state:"working"})` for the instance ID and URL. Do not bypass another owner or the shared limit. Read `inspect_presentation({})`; no existing view returns `presentation:null`.
2. Prepare `view.title`, `note` and `items` to describe the user's task. An art item is `{kind:"art",art_id,label,region,scale,editable,request_id?}`. Attach `request_id` to an editable item carrying out a protected request. Copy the freshly read IDs into `view.expected_presentation_id` and `view.expected_state_id`; both keys must be explicit `null` for the first view.
3. Call `present_art({control_id,workbench_id,view})`. On conflict, read the human's latest edits first. When ready for their input, call `open_workbench({control_id,work_state:"waiting"})` and share its returned URL.
4. Read the explicit save from `inspect_presentation` at `saved.art_ids` and `saved.state_id`. Current drafts are at `state.art_ids`; `dirty` distinguishes them from the last explicit save. Saving is not acceptance.

Do not close the screen while the human is working. Finish with `close_workbench({control_id,workbench_id})` and check its result. `inspect_workbench({control_id})` distinguishes `owned`, `closing`, `busy` and `closed` without renewing activity. Reopening with the same control_id reuses its ID and URL and renews activity; it does not change its timeout. Successful presentation and human actions renew activity; polling does not. The default idle timeout is 1800 seconds, configurable from 1 to 1800. HTTP ends when the MCP process hosting it exits or the idle deadline expires; art and drafts remain.
Do not close the screen while the human is working. Use `open_workbench({control_id,work_state:"working"})` at each follow-up before doing art work or calling an external generator. A declared working task stays alive through long calls and gaps between calls, including external work. Collaborators must use the same explicit task ID to change its work state; ordinary art calls never imply task ownership or activity.

Before handing the screen to the human or awaiting their reply, call `open_workbench({control_id,work_state:"waiting"})`. Share only the URL returned by that call, never a URL from memory or an earlier answer. On failure, cancellation or interruption, switch to waiting or close the finished task. The server cannot infer that an external agent stopped: leaving working set will hold the screen while the host lives. Use `inspect_workbench({control_id})` to read the current instance and work_state without renewing activity. Finish with `close_workbench({control_id,workbench_id})` and check its result.

Reopening preserves an active instance's ID, URL and timeout, and resets idle time. Omitting work_state preserves an active state; a new instance defaults to waiting. In waiting, presentation, successful human actions and real visible input (scrolling, color selection and unfinished dragging) renew activity. Art calls, failed calls, HTTP/browser polling, playback and inspect_workbench do not. The waiting idle timeout defaults to 1800 seconds, configurable from 1 to 1800. The hosting MCP process exiting always ends HTTP, including working tasks; art and drafts remain. After shutdown, open a new instance and inspect the preserved draft before continuing.

One explicit controller owns each workspace's screen, with a shared maximum of four. The same control_id works across MCP connections; different tasks remain separate even on one connection. Share the workbench URL with the human; keep control IDs private. Independent candidate creation does not require screen control. Never bypass control using shell servers, alternate runtime directories or lock-file changes. The local HTTP interface is not an MCP transport or a public agent mutation API. Closing a forwarding MCP connection does not close the host's screen. If close returns closing, inspect until closed.

For collections and filters, use `list_art` and present the selected candidates in order. For history, inspect parent IDs or archived presentations and states. `inspect_presentation({presentation_id,state_id})` is read-only; use `present_art` to change the visible screen. A response with `is_current:false` is not the current screen.

Reference items use `{kind:"reference",art_id,source_hash,label,region,scale}` and source-image coordinates; they are read-only. For playback, supply existing frames in a read-only art item's `playback`, show the full first `art_id`, and specify every frame's `duration_ms`. If timing is unknown, show frames side by side. All playback frames count toward the total display area. Prepare narrower crops or native-size views yourself; users should not have to learn IDs, coordinates or validation settings.

## Offering candidate choices

Create alternatives as separate candidates, keeping the supplied intent and constraints. Label the differences clearly and show comparable regions at appropriate scales. The agent prepares the alternatives; choosing does not generate images automatically.

Use the existing `present_art` view with `candidate_choices: {"item_indices":[1,2,3]}` to make those zero-based items selectable. For example, item 0 can be the original and items 1..3 the alternatives. Offer 2..16 distinct static art candidates with `editable:false`. References, playback and editable items cannot be choices. Other items can still provide context. Omit `candidate_choices` for an ordinary editing view.

Tell the human in their language to click **Choose** under any candidates they want to explore, then **Save** (Ctrl/Cmd+S). They can choose several and click a chosen option again to remove it. Nothing is preselected. Choice changes are preserved immediately as drafts; Save communicates the current choice. Mark issues remains available for details. Undo affects only the last painting or marking stroke and leaves candidate choices unchanged.

Read `saved.chosen_candidates`, a list of `{item_index,art_id}`, together with `saved.state_id` and `presentation_id`. `state.chosen_candidates` may contain a newer draft; absent or empty fields mean no choice. Do not infer choices from `saved.art_ids`, which includes every shown item. An empty choice is not rejection of the alternatives. Read the saved marks and any conversational feedback as well.

Use the chosen IDs as the exact bases for separate refinements, preserving their target constraints and applicable permissions. Recreate any pixel selections and protection against each new base before protected edits. Show the next alternatives through a new presentation, which starts with no chosen candidates. Previous candidates, saved choices and draft states remain available through `inspect_presentation`. Do not discard unchosen alternatives or treat them as rejected.

Choices express directions for further work. They do not submit results, record final acceptance, grant additional edit permission, validate constraints or approve game insertion. The agent continues the conversation using the saved choice; Dotmend does not invoke the agent automatically. On conflicts or lost responses, inspect the current presentation before retrying and never apply an old click to a different view.

## Reading human issue marks

The human toggles **Mark issues** to add pixels with a left click or drag and remove them with a right click or drag. Choosing a palette color returns to painting. Marks are a separate overlay, not palette edits. Static art can be marked even when read-only or protected. Reference images and playback cannot be marked; present a specific still-art candidate when needed.
Expand Down
34 changes: 33 additions & 1 deletion src/presentation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,19 @@ impl PresentationItem {
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
pub struct CandidateChoices {
/// Indices of distinct, static, read-only art items offered as alternatives.
#[schemars(length(min = 2, max = 16))]
pub item_indices: Vec<usize>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
pub struct ChosenCandidate {
pub item_index: usize,
pub art_id: String,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
#[schemars(extend("required" = ["title", "items", "expected_presentation_id", "expected_state_id"]))]
pub struct PresentArt {
pub title: String,
Expand All @@ -59,6 +72,9 @@ pub struct PresentArt {
pub expected_presentation_id: Option<String>,
#[serde(deserialize_with = "required_nullable")]
pub expected_state_id: Option<String>,
/// Offer candidates for the human to choose and save as the basis for further work.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub candidate_choices: Option<CandidateChoices>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
Expand All @@ -73,6 +89,7 @@ pub enum PresentationActionKind {
Paint,
Mark,
Undo,
Choose,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
Expand All @@ -95,6 +112,8 @@ pub struct PresentationState {
pub concerns: Vec<Concern>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub undo_concerns: Option<Vec<Concern>>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub chosen_candidates: Vec<ChosenCandidate>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
Expand All @@ -105,6 +124,8 @@ pub struct SavedPresentation {
pub art_ids: Vec<String>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub concerns: Vec<Concern>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub chosen_candidates: Vec<ChosenCandidate>,
}
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
Expand All @@ -128,6 +149,12 @@ pub struct PresentedArt {
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
#[serde(tag = "action", rename_all = "snake_case", deny_unknown_fields)]
pub enum HumanAction {
Choose {
presentation_id: String,
expected_state_id: String,
/// The complete choice, or an empty list to clear it.
item_indices: Vec<usize>,
},
Mark {
presentation_id: String,
expected_state_id: String,
Expand All @@ -153,7 +180,12 @@ pub enum HumanAction {
impl HumanAction {
pub fn identity(&self) -> (&str, &str) {
match self {
Self::Mark {
Self::Choose {
presentation_id,
expected_state_id,
..
}
| Self::Mark {
presentation_id,
expected_state_id,
..
Expand Down
6 changes: 3 additions & 3 deletions src/requests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ pub fn tool_schemas() -> Vec<(&'static str, &'static str, bool, Value)> {
vec![
(
"open_workbench",
"Open a managed human workbench with an explicit control_id (fresh random ID per task, 16..128 ASCII letters/digits/_/-). Reuse the same ID across calls or MCP connections. Other controllers and the shared limit prevent launch. Optional idle_timeout_seconds: 1..1800; default 1800. Reopening renews activity without changing the existing timeout.",
"Open or resume a managed workbench with an explicit control_id (fresh random ID per task, 16..128 ASCII letters/digits/_/-). Set work_state: working before agent work, including lengthy external calls; it prevents idle shutdown until you explicitly switch to waiting or close. Set waiting at handoff, when awaiting user input, or after interruption; use that response's URL, never a cached address. Omitted state preserves an active state, or starts a new instance waiting. Same control_id works across connections. Other tasks' art calls do not renew activity. Reopening resets idle time and keeps the active ID, URL and timeout. Optional idle_timeout_seconds: 1..1800, default 1800. Only the owner can change state; host exit and the shared limit still apply.",
false,
input_schema::<OpenWorkbench>(),
),
Expand All @@ -192,13 +192,13 @@ pub fn tool_schemas() -> Vec<(&'static str, &'static str, bool, Value)> {
),
(
"present_art",
"Present art through a managed workbench. Pass its control_id and the exact workbench_id from open_workbench and a view containing title, items and the expected presentation/state IDs from inspect_presentation. Both expected IDs must be explicit null for the first view. Humans paint, mark issues, undo once and save.",
"Present art through a managed workbench. Pass its control_id and the exact workbench_id from open_workbench and a view containing title, items and the expected presentation/state IDs from inspect_presentation. Both expected IDs must be explicit null for the first view. Optionally set view.candidate_choices={item_indices:[...]} to offer 2..16 distinct static, read-only art items for the human to choose from. Multiple choices are allowed. Humans can also paint editable items, mark issues, undo one stroke and save.",
false,
input_schema::<ManagedPresentArt>(),
),
(
"inspect_presentation",
"Read the current or archived presentation, human drafts and explicitly saved candidates. state.concerns and saved.concerns contain item_index, art_id, bounds and exact pixels marked by the human; absent means empty. Marks indicate observation, not edit permission or rejection. Use state_id for past states; prepare history and filters yourself with present_art.",
"Read the current or archived presentation, human drafts and explicitly saved candidates. state.chosen_candidates and saved.chosen_candidates contain {item_index,art_id} for the human's draft or saved choices; absent means empty. Continue from saved choices, preserving the unchosen candidates. Choosing is a preference for further work, not final acceptance or rejection. state.concerns and saved.concerns contain item_index, art_id, bounds and exact marked pixels; absent means empty. Marks request observation, not edit permission. Use state_id for past states; prepare history and filters yourself with present_art.",
true,
input_schema::<InspectPresentation>(),
),
Expand Down
Loading