From b490f7691f7c85cb0d5f8f85ce6e1fc94b32eb21 Mon Sep 17 00:00:00 2001 From: Tomas McMonigal <74835766+sudoTomas@users.noreply.github.com> Date: Thu, 5 Nov 2026 21:41:52 +0000 Subject: [PATCH 1/4] feat(slop): the Slop-O-Meter's scoring as a settings section The meter kept all of its policy in code: the file weights and which directories are vendored or tests, the pace and cap constants, the small-repository rule, the level bounds and names and the judge's prompt. An organization could not say what counts as code, how fast its team works or where its levels fall, and changing a constant silently rescored every repository. Its reading pace was three variables, and an owner's own bots sat on its organization row. Add a `slop_meter` section to the settings document. Every constant is a key whose default is the value the code held, so nothing written scores bit for bit as before: a fixture history reproduces the pinned pre-change numbers, and 86,000 paths weigh what the old rules gave them. An organization's layer may set every key but `enabled` and `interval_hours`; a repository's what counts as code in its layout. `weights` maps a glob to a weight, merged glob by glob through every layer, the longest glob winning. Each score row records `config_version` (migration 144), a digest of the keys its repository's score is computed from. The loop now looks at the section every five minutes: a change that gives a repository it read another version rescores at once, `enabled` and `interval_hours` apply within the look, and turning the meter on reads at the next one. The forge budget is read per organization; organizations sharing one credential get the fewest calls and the largest reserve. SLOP_METER_INTERVAL_SECONDS (0 is `enabled: false`), SLOP_METER_API_CALLS_PER_TICK and SLOP_METER_RATE_RESERVE are imported once and then ignored. The organization setting `slop_meter_bots` moves into its layer as `bots`, copied at the first start and still taken by PATCH /orgs until 0.2.0. The process limits stay variables. The unread LEAN_LEVEL constant is deleted: triage's lean belongs to the routing section a later pull request adds. Multi-line texts in the settings YAML are now written as literal blocks, so the judge's prompt reads as written. Refs #169 --- .env.example | 24 +- docs/api.md | 6 +- docs/architecture.md | 1 + docs/configuration.md | 45 +- docs/examples/settings.yaml | 288 +++++++++ docs/glossary.md | 2 +- docs/slop-meter.md | 96 ++- embry0/api/app.py | 25 +- embry0/api/v1/slop_meter.py | 7 +- embry0/config.py | 24 +- embry0/mcp_server/server.py | 6 +- embry0/services/organizations.py | 29 +- embry0/services/slop_meter/context.py | 3 - embry0/services/slop_meter/globs.py | 132 +++++ embry0/services/slop_meter/history.py | 25 +- embry0/services/slop_meter/judge.py | 35 +- embry0/services/slop_meter/loop.py | 353 ++++++++--- embry0/services/slop_meter/score.py | 123 ++-- embry0/services/slop_meter/weights.py | 151 +++-- embry0/settings/document.py | 18 +- embry0/settings/sections/__init__.py | 2 + embry0/settings/sections/slop_meter.py | 556 ++++++++++++++++++ embry0/storage/migrations/runner.py | 13 + embry0/storage/repositories/slop_meter.py | 16 +- frontend/src/api/slopMeter.ts | 11 +- tests/unit/api/test_background_scope_live.py | 17 +- .../unit/api/test_slop_meter_settings_live.py | 270 +++++++++ .../v1/test_organization_register_settings.py | 10 +- tests/unit/api/v1/test_slop_meter_api.py | 8 +- tests/unit/services/slop_meter/test_loop.py | 74 ++- .../services/slop_meter/test_loop_settings.py | 277 +++++++++ .../slop_meter/test_mirror_subgroups.py | 4 +- tests/unit/services/slop_meter/test_prune.py | 28 +- .../slop_meter/test_scoring_settings.py | 317 ++++++++++ .../test_background_scope_transients.py | 6 +- tests/unit/settings/test_budget_section.py | 1 + tests/unit/settings/test_registry.py | 6 +- .../unit/settings/test_slop_meter_section.py | 310 ++++++++++ tests/unit/storage/test_migration_143.py | 6 +- tests/unit/storage/test_migration_144.py | 71 +++ 40 files changed, 3040 insertions(+), 356 deletions(-) create mode 100644 embry0/services/slop_meter/globs.py create mode 100644 embry0/settings/sections/slop_meter.py create mode 100644 tests/unit/api/test_slop_meter_settings_live.py create mode 100644 tests/unit/services/slop_meter/test_loop_settings.py create mode 100644 tests/unit/services/slop_meter/test_scoring_settings.py create mode 100644 tests/unit/settings/test_slop_meter_section.py create mode 100644 tests/unit/storage/test_migration_144.py diff --git a/.env.example b/.env.example index f9181943..0afb37b9 100644 --- a/.env.example +++ b/.env.example @@ -182,17 +182,19 @@ PROXY_ADMIN_TOKEN= # settings document, per organization and repository (docs/configuration.md); # none until one is enrolled. A repository that leaves loses its mirror and # its rows background.prune_after_days later (docs/ops/background-data.md). -# 0 turns the loop off. Each tick's -# work per repository is bounded; a large history is read over ticks. Forge -# API calls have one budget per credential per tick, dealt round-robin among -# its repositories, and stop when that credential's rate limit drops below -# the reserve (jobs use the same credential); git fetches are not API calls. -# SLOP_METER_REPOS and SLOP_METER_EXCLUDE are read once, at the first start of -# the release that added the background section, into the repositories' -# layers, and ignored after that. Do not add them here. -# SLOP_METER_API_CALLS_PER_TICK=200 -# SLOP_METER_RATE_RESERVE=2000 -# SLOP_METER_INTERVAL_SECONDS=86400 +# How it scores, how often it reads (and whether it reads at all) and its +# forge budget per credential are the `slop_meter` section of the settings +# document, not variables: edit it with PUT /api/v1/config/settings. +# SLOP_METER_INTERVAL_SECONDS, SLOP_METER_API_CALLS_PER_TICK and +# SLOP_METER_RATE_RESERVE are read once, at the first start of the release +# that added the section, and ignored after that; the next release removes +# them. SLOP_METER_REPOS and SLOP_METER_EXCLUDE are read once, at the first +# start of the release that added the background section, into the +# repositories' layers, and ignored after that. Do not add them here. +# The variables below are the process's own limits: each tick's work per +# repository is bounded, so a large history is read over ticks, coming back +# after the catch-up interval while one is behind; git fetches are not API +# calls, and every git call has a timeout. # SLOP_METER_CATCHUP_SECONDS=300 # SLOP_METER_DATA_PATH=/data/embry0/slop-meter # SLOP_METER_GIT_TIMEOUT_SECONDS=900 diff --git a/docs/api.md b/docs/api.md index 14163453..16f66f6a 100644 --- a/docs/api.md +++ b/docs/api.md @@ -90,7 +90,7 @@ curl http://localhost:8200/api/v1/jobs/{job_id}/agents/developer/session A job's `area` (on the list and on `GET /jobs/{job_id}`) is the part of a monorepo the pull request embry0 opened for it changes, from the repository's `.embry0/areas.yml` ([areas](areas.md)); `null` when the repository names none or the change falls in none. `GET /repos/{repo}/-/areas` reads that file, and `POST /repos/{repo}/-/areas/map` starts the agent that writes it as a pull request ([map areas](areas.md#map-areas)). -`GET /repos/{repo}/-/slop-meter?area=` is the repository's [Slop-O-Meter](glossary.md#slop-o-meter) (after [Slop-O-Meter](https://github.com/slop-o-meter/slop-o-meter)'s published slop score algorithm): `meter` (the repository's `{score, level, name, provisional, computed_at}`, `null` before its first score), `areas` (each scored area's `{area, meter}`), `area` (the series' area, `null` for the repository), `weeks` (`{week, cost, capacity, slop, lines, score}` oldest first, in weighted lines; `score` cumulative) and `reading` (`{commits_read, commits_total, computed_at, error}`, how far the history is read). 404 for an unregistered repository or an area with no reading. A repository the meter does not read (`background.slop_meter`, [which repositories](configuration.md#which-repositories-background-features-read)) gets no new reading: what the meter kept of one that left is removed `background.prune_after_days` later, and `meter` is then `null`. An inactive one keeps its reading. +`GET /repos/{repo}/-/slop-meter?area=` is the repository's [Slop-O-Meter](glossary.md#slop-o-meter) (after [Slop-O-Meter](https://github.com/slop-o-meter/slop-o-meter)'s published slop score algorithm): `meter` (the repository's `{score, level, name, provisional, computed_at}`, `null` before its first score), `areas` (each scored area's `{area, meter}`), `area` (the series' area, `null` for the repository), `weeks` (`{week, cost, capacity, slop, lines, score}` oldest first, in weighted lines; `score` cumulative) and `reading` (`{commits_read, commits_total, computed_at, error, config_version}`, how far the history is read, and the version of the `slop_meter` settings the repository's score was computed with, `null` for a score from before #169: [how a change rescores](slop-meter.md#how-a-change-rescores)). 404 for an unregistered repository or an area with no reading. A repository the meter does not read (`background.slop_meter`, [which repositories](configuration.md#which-repositories-background-features-read)) gets no new reading: what the meter kept of one that left is removed `background.prune_after_days` later, and `meter` is then `null`. An inactive one keeps its reading. `POST /jobs/{job_id}/resume` takes `{"choice": "continue_retrying"|"merge_as_is"|"abandon", "guidance", "extra_budget_usd"}`. On a job over its budget, `continue_retrying` lets it spend one more cap past what it has spent (`budget.continue_default_extra_usd` of the [settings document](configuration.md#settings-document) when set), or `extra_budget_usd` when given: at most `budget.continue_extra_max_usd`, 1,000 by default, read when the request comes in, and a 422 naming that key above it. A continue that names nothing is held to the same most. The pause's `interrupt.summary.budget` says `{spent, cap, continue_adds}` ([per-job cap](architecture.md#budget-controls)). A continue goes on top of the fleet ceiling, `budget.max_budget_per_job_usd`, unless `budget.continue_may_exceed_ceiling` is false: then it adds at most what is left under the ceiling, and a job at it is not offered `continue_retrying` (one answered anyway is abandoned). Each continue that raises a job's cap writes a `job.budget_raised` audit row naming who continued it, with `job_id`, `spent_usd`, `cap_before_usd`, `cap_after_usd` and the `extra_usd` asked for. @@ -741,9 +741,9 @@ A repository whose `context_audit.days` is set (the [settings document](configur | `context_audit_budget_usd` | number more than 0 and at most 50, or `null` | `null` | Deprecated: the organization's layer of the settings document's `budget.context_audit_job_usd` now. Taken until 0.2.0, see below | | `context_audit_max_jobs_per_tick` | integer from 0 to 50, or `null` | `null` | Deprecated: the organization's layer of `budget.context_audit_max_jobs_per_tick` now. Taken until 0.2.0, see below | | `context_audit_open_prs` | boolean | off | Deprecated: the organization's layer of `context_audit.open_prs` now. Taken until 0.2.0, see below | -| `slop_meter_bots` | list of up to 100 strings, each at most 200 characters | empty | The owner's own bots: commit names, addresses and forge logins the Slop-O-Meter counts as a bot's, not a person's attention. embry0's own commit identities are always counted so; the list applies to every reading, including history read before it was set | +| `slop_meter_bots` | list of up to 100 strings, each at most 200 characters | empty | Deprecated: the organization's layer of the settings document's `slop_meter.bots` now. Taken until 0.2.0, see below | -**Moved settings.** `context_audit_budget_usd` and `context_audit_max_jobs_per_tick` are the organization's layer of the settings document's `budget` section now, as `context_audit_job_usd` and `context_audit_max_jobs_per_tick` (`PATCH /api/v1/config/settings/budget?owner=acme`), where an organization may also set `area_map_job_usd` and `fix_job_usd`. `context_audit_days` and `context_audit_open_prs` are its layer of the `context_audit` section, as `days` and `open_prs` (`PATCH /api/v1/config/settings/context_audit?owner=acme`), where a repository's own layer may set them too and `days: false` turns the schedule off whatever the layer below says (#170). The first start of each release that moved them copied each organization's value into its layer once, when the old reader would have used it (a value it read as the deployment's is nothing to copy); the value left on the organization is read by nothing, and `GET /api/v1/orgs` no longer shows it. Until 0.2.0, `PATCH` and `POST /api/v1/orgs` still take them, checked as above, and write them into the organization's layer instead, with every rule of that route (a revision with source `person`, a `settings.updated` audit row, a key with no repository allowlist); `null` takes the key out of the layer, so the organization reads the deployment's value. The answer then carries `deprecated`: one `{"setting", "moved_to", "removed_in", "detail"}` per moved setting it took. 0.2.0 refuses them. +**Moved settings.** `context_audit_budget_usd` and `context_audit_max_jobs_per_tick` are the organization's layer of the settings document's `budget` section now, as `context_audit_job_usd` and `context_audit_max_jobs_per_tick` (`PATCH /api/v1/config/settings/budget?owner=acme`), where an organization may also set `area_map_job_usd` and `fix_job_usd`. `context_audit_days` and `context_audit_open_prs` are its layer of the `context_audit` section, as `days` and `open_prs` (`PATCH /api/v1/config/settings/context_audit?owner=acme`), where a repository's own layer may set them too and `days: false` turns the schedule off whatever the layer below says (#170). `slop_meter_bots` is its layer of the `slop_meter` section, as `bots` (`PATCH /api/v1/config/settings/slop_meter?owner=acme`): the owner's own bots, which the Slop-O-Meter counts as a bot's work, not a person's attention, beside embry0's own commit identities, applied to every reading, history read before they were listed included (#169). Each organization's list was copied with its blanks left out and each name once, without regard to case. The first start of each release that moved them copied each organization's value into its layer once, when the old reader would have used it (a value it read as the deployment's is nothing to copy); the value left on the organization is read by nothing, and `GET /api/v1/orgs` no longer shows it. Until 0.2.0, `PATCH` and `POST /api/v1/orgs` still take them, checked as above, and write them into the organization's layer instead, with every rule of that route (a revision with source `person`, a `settings.updated` audit row, a key with no repository allowlist); `null` takes the key out of the layer, so the organization reads the deployment's value. The answer then carries `deprecated`: one `{"setting", "moved_to", "removed_in", "detail"}` per moved setting it took. 0.2.0 refuses them. `GET` and `POST /api/v1/orgs/{owner}/webhook` inspect and install the organization's one GitHub webhook, and the receiver's gate handles deliveries for repositories nobody registered ([webhooks.md](webhooks.md#organization-webhooks-github), [ADR 0012](adr/0012-organization-webhook.md)). diff --git a/docs/architecture.md b/docs/architecture.md index 4c751cce..08461ae0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1390,6 +1390,7 @@ The whole pass is serialised across processes by a session-level Postgres adviso | 141 | the repository of a job posted with a git context alone: fills `jobs.repo` from `context->>'repo'` where `repo` is NULL and the context is git, so `GET /jobs?repo=`, the world view and the stats find it (`POST /jobs` stores it now); a row naming a repository and any other context are left alone, `updated_at` is kept, and a second run changes nothing | `jobs.repo` (data) | | 142 | the skills triage may add to an agent: `agent_definitions.triage_may_add_skills`, a JSON array of `source:skill` names (NOT NULL, default `[]`); triage adds one only from a source the agent's skills already load. The built-in developer row gets `superpowers:brainstorming` and `superpowers:writing-plans`, the two the triage prompt always offered it, so an upgraded deployment keeps its behaviour; every other row, and a deployment without that row, gets none | `agent_definitions.triage_may_add_skills` | | 143 | behaviour from agent and template rows, not built-in names (#167): `pipeline_templates.fixed_route` (NOT NULL, default false), `offered_to_triage` (NOT NULL, default true), `job_kind` (an override; NULL reads the route), `delivers` (NOT NULL, default `''`) and `leader_agent`; `agent_definitions.station` (NOT NULL, default `desk`) and `quest_template` (NULL none, `''` a task naming no template); `jobs.coding_agent` and `jobs.leader_agent`, which plan_route records and progression credits from. Every row gets what the code said of its name before: the ten specialist templates fixed and not offered, the leaders and deliverables of the old maps, the old stations and guild tasks; a slop-cleanup job coded by the Slopmaster and every job led by its template's old leader. A template named `qa`, `onboard` or `pr-review` is renamed `-template` | `pipeline_templates`, `agent_definitions`, `jobs` | +| 144 | each Slop-O-Meter score's configuration version: `slop_meter_scores.config_version` (NOT NULL, default `''`), a digest of the `slop_meter` settings the score was computed from (#169). A score stored before it names none, which no digest equals, so the meter's next run rescores it and records one | `slop_meter_scores.config_version` | Note: LangGraph's checkpoint tables (`checkpoints`, `checkpoint_blobs`, `checkpoint_writes`) are created and mutated by `langgraph-checkpoint-postgres` / `orchestration/checkpoint.py`, outside `embry0_migrations` entirely — they're excluded from this ledger by design, not an oversight. diff --git a/docs/configuration.md b/docs/configuration.md index 99b0ac62..4f012328 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -162,12 +162,14 @@ Both loops are one brief writer (`embry0/services/brief_writer.py`), each on its ### Slop-O-Meter +How the [Slop-O-Meter](slop-meter.md) scores, how often it reads and how hard it reads the forge are the settings document's `slop_meter` section ([the reference below](#slop_meter)); the variables left are the process's own limits. + | Variable | Default | Description | |----------|---------|-------------| -| `SLOP_METER_INTERVAL_SECONDS` | `86400` | How often the [Slop-O-Meter](slop-meter.md) reads and scores its repositories; `0` turns it off. The loop is on by default, and reads the repositories enrolled in `background.slop_meter`: none on a new deployment ([which repositories](#which-repositories-background-features-read)) | +| `SLOP_METER_INTERVAL_SECONDS` | `86400` | Governed by the settings document (`slop_meter.interval_hours`, and `slop_meter.enabled: false` for `0`). Read once, at the first start of the release that added the section, and ignored after that | | `SLOP_METER_REPOS` / `SLOP_METER_EXCLUDE` | (empty) | Governed by the settings document (`background.slop_meter`, in each repository's layer). Read once, at the first start of the release that added the `background` section, as `SCOUT_REPOS` and `SCOUT_EXCLUDE` are, and ignored after that | +| `SLOP_METER_API_CALLS_PER_TICK` / `SLOP_METER_RATE_RESERVE` | `200` / `2000` | Governed by the settings document (`slop_meter.api_calls_per_tick` and `slop_meter.rate_reserve`, which an organization's layer may set). Read once, at the first start of the release that added the section, and ignored after that | | `SLOP_METER_CATCHUP_SECONDS` | `300` | How soon the meter comes back while a repository's history is still being read | -| `SLOP_METER_API_CALLS_PER_TICK` / `SLOP_METER_RATE_RESERVE` | `200` / `2000` | The forge API calls one tick may make with each credential, shared round-robin by the repositories it reads, and that credential's remaining rate limit below which it stops calling (jobs use the same credential). Git fetches are not API calls | | `SLOP_METER_DATA_PATH` | `/data/embry0/slop-meter` | Where the meter keeps a bare mirror of each repository, inside the orchestrator's data volume | | `SLOP_METER_GIT_TIMEOUT_SECONDS` | `900` | The longest one git clone or fetch of a mirror may take | | `SLOP_METER_COMMITS_PER_TICK` / `SLOP_METER_EVENT_PAGES` / `SLOP_METER_REVIEWS_PER_TICK` / `SLOP_METER_LOGINS_PER_TICK` | `5000` / `10` / `50` / `20` | What one tick reads at most per repository: commits of its history, pages of forge events, pull requests' reviews, and logins mapped to commit emails. A larger history is read over several ticks | @@ -305,7 +307,7 @@ asks: - **In the dashboard**, Settings → Policy holds the same three things, for one document at a time. The **Layer** picker above them chooses the deployment's document, an organization's layer or a repository's: it offers every organization and repository the lists show and every stored layer, each marked with how many keys its layer sets, and keeps the choice in the URL (`/settings?tab=policy&owner=acme`, `&repo=acme/widgets`), so a layer can be linked to; an organization's Settings tab and a repository's page link to their own. Picking one with no layer opens an empty one that its first save creates, and picking another while an edit is unsaved asks first. The document is a YAML text box: **Preview** runs the dry run and lists each key the text would change, before and after, with the text diff; **Save** then stores exactly the previewed text, with an optional reason. Both name the generation the edit started from (0 for a layer nobody wrote), so if someone else saved in between they are refused, the new version is read in and your text stays in the box to preview again. A layer lists above its text the keys it may set, per section. A refused document lists every problem next to the text, each with its key, and marks a key the layer may not set. When a save, its preview or an undo makes values of the layers under the document read as inherited (`layers_refused`), the tab lists them, each linked to its layer. Below it, **Values in effect** shows every key of every section with its value, its default and where it came from, and warns when a stored value is refused, an imported variable is still set, or the document holds a section or key this release does not know; for a layer, as that organization or repository reads it: the layer each value comes from, what the layer's own value overrides ("Overrides "09-18" from the deployment"), what it inherits and from whom, and, dimmed, the keys it may not set. **History** lists the document's revisions, opens each one to show what it changed, and offers Undo on the newest. Anyone signed in can read the tab, with the people behind changes shown as the audit log shows them (someone else's name only to an admin); only an admin can preview, save or undo, and an API key with a repository allowlist only on a repository's layer the list covers. A document [kept in a file](#from-a-file-the-command-line-and-mcp) is shown read-only, to an admin too. - **From the environment.** When a release moves a feature's policy into a section, the first start of that release imports each of its variables you set, when the value differs from the default, once per key (source `bootstrap`); a key a later release adds to the section is imported at that release's first start. A variable left at its default is not imported, so a default the section changes reaches every deployment that never set it; where a section turns a default the other way on purpose, it declares the value a deployment that relied on the old meaning keeps, and how to tell one from its records (for `leads.enabled`: a registered repository, a job or a lead a finder stored). Only records older than the moment the release first asked about the key count: the import records that moment once in the document's ledger (`leads.enabled@