diff --git a/CLAUDE.md b/CLAUDE.md index 6514f32..d1d6d04 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,8 @@ supervisor) came from **audiosilo-sidecars** with its history (`git log --follow and `git blame` reach the reliability fixes). sidecars stays usable until Phase 4, then is archived with the manager. Phase 0 imported and tested the engine; Phase 1a runs it (`internal/app/engine.go`) behind the `sidecars.generate` job -(`internal/generate`), which writes the results into the book's `.audiosilo/`. +(`internal/generate`), which writes the results into the book's `.audiosilo/`; +Phase 1c adds the description step and the `sidecars.describe` job beside it. ## Build / test / gate @@ -98,7 +99,8 @@ internal/ 0004 workers) auth/ the page's single password + hashed sessions + login limiter -- Phase 1a -- - generate/ sidecars.generate: server job -> engine book -> .audiosilo/ + rescan + generate/ sidecars.generate: server job -> engine book -> .audiosilo/ + rescan; + sidecars.describe (Phase 1c): a description from a finished run's notes agentproto/ the staging protocol shared with the agent container (a leaf) agentbox/ the Bridge's side: an agent.Runner over the staging dir agentrun/ the agent container's side: `agent-runner` (claim, run, harvestable out/) @@ -143,8 +145,9 @@ in PACKAGES.md. use, checked again before each write), temp file then rename, umask 022, a targeted rescan after, deletes to `.audiosilo-trash/`. A library that isn't mapped and writable refuses a new write job (a resumed one parks and waits). Today the - only writer is `generate`: `characters.json`, `recaps.json` and `bridge.json` in the - book's `.audiosilo/` (a single-file book's prefixed `.`), nothing else. + only writer is `generate` (its two jobs): `characters.json`, `recaps.json`, + `description.json` (Phase 1c) and `bridge.json` in the book's `.audiosilo/` (a + single-file book's prefixed `.`), nothing else. - **The agent container's files are untrusted** (and the Bridge's are to it): every read goes through `agentproto.OpenRegular`/`ReadRegular` inside an `os.Root`, the harvest is capped, and a credential never leaves the agent container (only its @@ -191,7 +194,28 @@ install`/`uninstall` (launchd agent, systemd user unit), worker tokens on the pa the protocol-version refusal, and the engine's ASR as `workers.Router`: remote when a worker is connected, else `asr.worker_fallback` (`local`, the default, which yields to a worker that connects mid-decode; or `wait`, which parks `asr_worker_waiting` -until one connects). See BRIDGE-PLAN.md's Status table for what is next. +until one connects). + +**Phase 1c (own-words descriptions):** the description step +(`pipeline.Executor.Describe`, not an engine stage: its own `describing` stage run +on the book, which the reconcile and the supervisor skip, run under the +scheduler's `Hold` on the book, or inside the contributing stage) writes a spoiler-free +description from the fact notes of the book's opening chapters ONLY (the staged dir +holds `opening.md` and nothing else: no transcript, no `knowledge-final.md`, no later +chapter), validated (shape, length, voice, meta's description schema, the n-gram +check against whatever source text is left: the retention removes an audio book's +transcripts, and an ebook's text goes as it reaches done, so the contributing stage +of a book a `sidecars.generate` job follows runs it first, best effort and once, +while that text is there) and composed with the run's work +and sources. `sidecars.generate` runs it at its finish, after the characters and +recaps are saved (reusing the contributing stage's for free, its check kept; once +more at done when it first ran at a contributing park), and +writes `description.json` with them, but never fails, parks or holds for it (the +result says `description: false` and why; a regenerate that gets none keeps the +earlier file vouched for in `bridge.json`); `sidecars.describe` (advertised beside +`sidecars.generate`) writes one for a book already generated here, parking for the +agent's parks. Local only: a contribution never carries it. See BRIDGE-PLAN.md's +Status table for what is next. Operative caveats carried from sidecars (PACKAGES.md has the detail): - **audiosilo-meta is pinned** and the schema drift guard diff --git a/PACKAGES.md b/PACKAGES.md index ea48842..4e126e9 100644 --- a/PACKAGES.md +++ b/PACKAGES.md @@ -57,8 +57,10 @@ internal/ is resolved once at startup). M5 made agent.* LIVE: backend (""|claude|codex), capacity, claude_path/codex_path, timeout_minutes, and the per-stage claude/openai model maps (keys are agent - stage names). Validate rejects an unknown backend, an unknown model-map key, - or timeout_minutes < 1; Default() seeds the claude map. M7 added + stage names, and Phase 1c's description step, "describing": + state.RoutesModel). Validate rejects an unknown backend, an unknown model-map key, + or timeout_minutes < 1; Default() seeds the claude map ("describing": + sonnet). M7 added contribution.{mode [issue|local], core_repo (add-work), community_repo (sidecars), auto_purge, poll_minutes, api_base_url} (restart-to-apply; api_base_url for tests/GHE). Retired settings load via @@ -226,6 +228,8 @@ internal/ Confined like Purge (the denied case is tested); the file names come from the owning packages' constants (spelling.RefsDir/PriorPrefix/ SeriesGlossaryFile/CorrectionsLogFile, qa.AcceptedFile), never literals. + Phase 1c: it keeps the description step's chapter_positions.json and + sidecars/description.json (neither quotes the transcript). qa/ M4: the mechanical transcript-QA degeneration sweep, a faithful Go port of the historical Python detectors (qa_sweep/cross_segment/ within_segment/multi_loop/tail_rate scans - thresholds and per-detector @@ -338,6 +342,11 @@ internal/ Availability.RetryAfter marks an unavailable runner that comes back by itself (the agent container restarting): the preflight park then gets a retry_at instead of waiting for a human. + Phase 1c: prompts/description.md is the description step's prompt (the + AUTHORING.md `description` contract restated: setup only, grounded only + in opening.md, own words, neutral voice, 200-1500 characters, hyphens, + out/description.json as exactly {"text": ...}); a prompts test renders + both source branches and holds the output shape. benchmark/ private, provider-neutral post-ASR evaluation: allow-listed corpus preparation with input digests; fresh per-run work trees and databases; stage-specific model/effort routes; the real fact/synthesis/audit/fix @@ -661,6 +670,82 @@ internal/ names the validated characters.json/recaps.json generate copies into the library (SidecarsView reads the same two). ManifestChangedMsg now says to cancel the job and generate again. + Phase 1c, the DESCRIPTION STEP (describe_stage.go): Executor.Describe(ctx, + bookID, noWait) writes a finished book's own-words description. + It is NOT an engine stage (the state machine, scheduler and supervisor + never run it): the Bridge's sidecars.generate (at its finish) and + sidecars.describe call it, under the scheduler's Hold on the book, and + the contributing stage calls it first (describeAtContributing, at the + stage's start, outside its rate sample, under its heartbeat, no hold: + the book is the scheduler's in-flight book), while the source text is + still there: an ebook's text goes as the book reaches done, before the + job's pass can ask, so a description first written then is checked + against nothing (a 13-word run copied from the epub was accepted). Best + effort: only for a book an open sidecars.generate job follows + (store.GenerateBookFollowed), only when the book has no describing run + yet (ok or failed: a contributing park runs the stage again and again, + and never pays again), noWait, any error logged and never the stage's. + The job's pass then reuses it (below), keeping its checked_against. The + SPOILER BOUND is the staging: the agent's dir holds ONLY opening.md (and + out/), built mechanically from the fact pass's chunks (chunk_plan.json, + named by factsChunkName) covering the chosen files: the `## Chapter N` sections (file numbering kept, + never renumbered, a "Chapter 0" too) of the files whose LOGICAL position + (classifyBookEdges -> PositionOf) is 0 (front matter) or among the + book's first K story positions, K = max(2, ceil(n/4)) capped at n-1 + (never the last chapter: a book of two gets its first, a book of one no + description, ErrOneChapter), n the count of distinct story positions the + files hold capped at the logical count (never LogicalCount alone: narrated numbering takes it from the + highest announcement believed, so a misheard "Forty." or a year, or a + sequel numbering on from 45, would widen or empty the window); + excluded edge files (an Audible intro, credits) and end matter never. No transcript, no knowledge-final.md (it + carries FINAL/ENDING), no later chapter, no publisher text (a test lists + the staged dir). The classification reads the transcripts, which the + retention deletes, so each one is kept as chapter_positions.json (keyed + by manifestFingerprint; an unchanged one isn't rewritten; + PurgeTranscripts leaves it) and a book whose + transcripts are gone uses it, else its FILE numbers with the logical count + from the recaps' last through (a file's number is never below its story + chapter, so that bound is safe, only less generous); an ebook classifies + from its manifest alone. The run's characters.json supplies the work and + sources (a run whose own work/sources can't make a valid description + fails before any spend); the per-book and monthly budget preflight + (budgetPreflight, which runAgent runs for every other stage and + runAgentWith leaves to its caller) runs after the preparation (local + reads, which keep chapter_positions.json even for a call the budget + parks) and before the stage run, so a budget-parked retry costs nothing + and opens no run. The model is config's route for "describing" + (state.Describing, which agent.claude/agent.openai accept; agent.claude + defaults it to sonnet). The validator (patch feedback on a retry): + strict {"text"} decode (decodeSidecarBytes) with a 64 KiB cap; trimmed text 200-1500 runes; no + control character (nor U+2028/U+2029) but newline (the server refuses + control characters); no em dash; at most two paragraphs; no second + person, no sentence ending in "?", no sales words, with word edges by + Unicode letter and the book's own title and series exempt (a book may + be "Me Before You" or end in "?") (descriptionProblems); then the composed model.Description + (license CC-BY-SA-4.0) against meta's description.schema.json + (sidecarSchemas now compiles it too), then extract.NGram against the + transcript text layers through ngramScan (ngramCheck's scan without its + "no layer" refusal): a hit is a retry naming the run; a local fault of + the check (the disk, a transcript that can't be read) is no agent + retry: the answer is taken and the run fails with a plain error; no layer at all + (the retention purged the transcripts; an ebook's text goes as the + book reaches done, so only a later sidecars.describe meets an ebook + without it) skips the check and records checked_against "none". Spend: the call opens its own stage_run (DescribeStage + "describing", numbered store.NextStageAttempt as the scheduler's runs + are, so each run has its own _runs/describing-aNN) on the book, so + invocation tracking, + SumStageRunBudgetCost, the monthly sum and the estimates see it, and + finishes it ok/failed with {chapters, logical_chapters, basis, + checked_against, usage}; runAgentWith is runAgent after the preflight with + the rate-limit backoff given (noWait passes an empty schedule: a rate + limit parks at once), stageMaxTurns gives it 10. Its parks come back + as *scheduler.ParkError (the caller decides). The composed file is kept + as sidecars/description.json (FinalDescriptionPath, canonical JSON); a + later call reuses it (re-composed with the current work/sources, nothing + spent) when a successful describing run recorded how it was made + (store.LatestOkStageRun), its chapters and checked_against read from + that run's metrics (the check it was made with, never one run again + against layers since gone). ErrNoNotes: no characters.json, no manifest, or no notes for the opening. contrib/ M7: everything GitHub-facing for contribution. TokenSource (secrets GitHubPAT first, else `gh auth token` - the token NEVER enters argv/logs/ errors, leak-canary tested), a stdlib REST client (injectable base URL; typed RateLimitError; APIError @@ -745,6 +830,13 @@ internal/ toward the month. budgetCostSQL is the one budget-basis expression (the provider's cost, else a complete estimate, else 0) every sum shares. GetBookBySourcePath finds an earlier run of the same book. + Phase 1c: StageRunBudgetCost(runID) is one run's budget-basis spend (the + description step reports its own); LatestGenerateBookAt(library, path) + is the newest generate job's surviving engine book at a path + (sidecars.describe finds a moved book's run with it); NextStageAttempt + (one past every recorded run: the scheduler's and the description step's + run numbers) and LatestOkStageRun (a stage's newest success, or + ErrNotFound). state/ per-book pipeline state machine: table-driven states/lanes/transitions, CanStart/NextState guards, the audit fix-loop cap. Pure, no I/O. M6 added ParkCode (typed park reasons - M7 added contrib_unavailable, core_needed, @@ -765,7 +857,11 @@ internal/ takes the status - a FAILED book (a cancel is a failure) holds nothing, since the job that wanted it ended and nothing will finish it (a later volume runs without its carryover, like a series' first book), while a - parked or paused one still holds it. + parked or paused one still holds it. Phase 1c: Describing ("describing") + names the Bridge's description step's stage runs, NOT a state (no table + row: IsStage/IsAgent false, Order 0), declared here for the one spelling + pipeline and config share; RoutesModel (an agent stage, or Describing) + is the model maps' key rule. eta/ the PURE ETA engine (no I/O, no clock): per-stage unit kinds (chapter/chunk/book) + seed rates from the historical extraction metrics, EWMA Observe (alpha 0.3), book ETA = rate x remaining units over the @@ -842,6 +938,14 @@ internal/ SetAgentCapacity(n) changes the agent lane's book capacity (atomic, clamped to 1, notifies on a change) while it runs: the Bridge follows the agent container's slots. + Phase 1c: reconcileBook skips a succeeded stage the state machine doesn't + know (the description step's "describing": no sentinel, state.Order 0, + so it would rewind a contributing-parked book to a state nothing runs). + Hold(id) (release, ok) is PurgeScratch's reservation for work outside the + stages (the Bridge's description step, its transcript purges): no stage + dispatched, Delete/PurgeScratch busy, the re-admits and startup GC skip + it, a supervisor terminate finds no worker; ok false when busy already; + release is idempotent. runStage numbers a run store.NextStageAttempt. supervisor/ the health-tick babysitter (config supervisor.*): classifies incidents over book/stage-run snapshots (deterministic classifiers + optional model-assisted lane), decides bounded recovery actions, and applies them @@ -910,6 +1014,12 @@ internal/ time limit would stop a long CPU transcription again and again), and any action on a book no open sidecars.generate job follows (store.GenerateBookFollowed: it would run, and spend, for nobody). + Phase 1c: engineRuns drops the runs of a stage outside the state machine + (the description step's "describing") at the run reads (engineRunsAll for + the tick, engineRunsOf for the confirmation and Ask; the input slice back + as it is when nothing is dropped): an open one has no scheduler worker + and a done one no sentinel, so they read as a missing process or a + broken artifact on a finished book. metaops/ meta.audiosilo.app client (coverage/lookup, capped 1h TTL caches, graceful degrade) + the PathAllowed/AllowedPath containment checks (Phase 1a passes the mapped mounts as the roots). glossary.go adds @@ -992,7 +1102,8 @@ internal/ monthly_budget_usd, month_spent_usd and can_contribute; omitted until the engine has started); RescanAttachments(library, path) is POST /bridge/rescan with mode "attachments" (the book's .audiosilo/ only, never - its audio), answering which of characters/recaps/bridge are indexed; and + its audio), answering which of characters/recaps/description (Phase 1c; + false from an older server)/bridge are indexed; and a job update's answer carries the job's current path, which the server moves with its book. mapping/ where the Bridge sees each library, CHECKED (decision 13): Check stats the @@ -1035,6 +1146,9 @@ internal/ report gets 409 and the row is lost, and the claim loop takes the job again as the next attempt (the kill test in the handoff log). release frees only its own run (a later claim of the job may run under its id). + Job.Resumed (Phase 1c) tells a handler the runner resumed the job under + its claim (a restart's recovery, resumeStranded) rather than claimed it + now, so an earlier run of it may have spent already. Recovery runs once the link is up: a kind no longer run is failed. Nothing is sent that the server refuses for good: a result that isn't a JSON object of at most 64 KiB fails the job (errUnreportable) and @@ -1159,7 +1273,98 @@ internal/ Retention (decision 5) is policy here: SweepTranscripts (daily) purges each finished (done, or failed/cancelled) book whose job ended at least retention_days ago and wasn't swept, through scratch.PurgeTranscripts, - then marks it swept. + then marks it swept. Phase 1c: each purge takes the engine's Hold on the + book (Engine.Hold, the scheduler's reservation); the sweep leaves a held + book for the next one, and a job end's purge (retention_days 0) waits a + few polls for it (a stage the end just stopped) before leaving it too. + Under the hold the book is looked at again (purgeHeld): one no longer + finished, or that an open job follows again (a later claim of its job + re-opened and re-admitted it while the purge waited), keeps its + transcripts and stays unswept. + Phase 1c, DESCRIPTIONS (describejob.go): Options.Describer is the + engine's description step (pipeline.Executor.Describe). A generate job's + write() saves the characters and recaps first, then runs it + (describeOnce, with noWait: the step can still wait for an agent slot + behind other books, so it never holds the files back) when the sidecars + are final (done, or parked at contributing; a contributing park is + reported before the write), and a description it got follows in a + second save. It asks once per run, and once more at done when it first + asked at a contributing park: the contributing stage may have resolved + the work and sources since, and the step's free re-compose makes the + description name the work the characters do; a try at the park that + failed without spending gets its second go there too. The engine's own + try at the start of contributing is what it usually finds (reused, + nothing spent, its checked_against kept); when the book's latest + describing run failed after spending (that try's, or an earlier run of + this job's before a restart), it asks no more (describeFailedPaid: no + second payment, and an ebook's second try at done would be checked + against nothing). The + description NEVER fails, parks or holds the job: any error (the engine + keeping its hold, see below, included) is logged and + the result says {"description": false, "description_error": ""} (descriptionErrorText), else "description": true and + description.json is written beside the others in the same finish (same + rules: known names only, once per content hash, at the book's current + path, mapping re-checked) with bridge.json's description object + {job_id, generated_at, from "opening_notes", chapters, + checked_against}. Only the description the run's step read back under + the hold is written (the work dir's file is never read again for it); + the cost is read again after the step (a failed try's spend included). + The contribution never carries it. A regenerate whose step got none + leaves an earlier run's description.json in place (nothing deletes into + a library yet), and its bridge.json carries over the existing one's + description object (keptDescription: only beside a description.json + that is a regular file, read through readProvenance), so the kept file + stays vouched for. + sidecars.describe (DescribeJob, DescribeKind, pipeline lane): params + decoded leniently (any JSON object). prepare as generate does, strict + when new (nothing spent yet) and not when resumed (jobs.Job.Resumed: an + earlier run may have paid; it parks library_not_writable like a + resumed generate job); notesBook finds the engine book (the latest at the + book's source path, else LatestGenerateBookAt for library+path, else the + generate run the book's own bridge.json names, for a book that moved + after its job ended (that file moved with it, the Bridge's records did + not; the job's mapping then follows the book); never a run whose source + lies outside this library's folder: library ids and paths repeat across + servers): none is + errDescribeNoNotes ("The Bridge has no notes for this book ..."), a + single-chapter book errDescribeOneChapter, one an + open generate job follows or the engine still has queued/running is + errDescribeGenerating. Both jobs describe through describeHeld: the + scheduler's Hold on the engine book (holdBook tries every poll for + holdTries polls, else errBookBusy, "The engine is busy with this book; + try again in a minute.") for the call and the read-back of the file, so + no stage, Retry's dispatch, Delete, purge, sweep or supervisor terminate + acts on the book meanwhile (a new generate job waits holdTries polls for + a held earlier run, deleteEarlier: the server asks a describe job to give + way to it, which it does at its next report; then "try again in a + minute"); a park waits with the hold let go (the job's in-process + rate-limit backoff, noWait false, does not). The job sets r.bookID once + it has its engine book, so its own parks are lifted by the save that + clears them, never between two tries. describeParked runs it: an agent park + (rate limit, budget, credential, unavailable) is the job's park (code, + hint, retry_at in progress) and it tries again at retry_at or on Retry + (the shared pokes, so the page's Retry and Handler.Retry reach it), + staying parked while it tries so a park that holds isn't announced + again (a Retry pressed during a try at a timed park is answered by that + try; the success lifts the park, one the runner restored after a restart + included); validation exhausted fails it ("Couldn't write a description + that passed the checks."). Handler.Retry only pokes a sidecars.describe + job (its id may map to another server's generate run in generate_books). + saveDescription writes description.json and + bridge.json (describedProvenance: the existing file's fields kept, its + description object added or replaced; a fresh one when there is none or + it isn't a regular file holding a JSON object; readProvenance reads it + inside the .audiosilo as an os.Root, the opened file the one looked at, + non-blocking and capped, as agentproto.ReadRegular does) through save, the write + loop generate's write() uses too (saveSpec: the files composed once the + book is located, relocate following a moved .epub, parking + book_not_writable until it can, adoptMove following a move: no mapping + to record), then RescanAttachments. Progress {step describing|writing| + rescanning|done, message, done, total, stage "describing", source, + cost_usd}; result {library_id, path, description: true, files, cost_usd, + checked_against} ("description" is always true; the server console's + shared result parser reads it). agentproto/ the staging protocol between the Bridge and the agent container (decision 15), shared by both sides so they can't drift; a LEAF (standard library only). Protocol 1 (both sides refuse another). A job dir /<16 hex>/: diff --git a/internal/agent/prompts/description.md b/internal/agent/prompts/description.md new file mode 100644 index 0000000..641a2c8 --- /dev/null +++ b/internal/agent/prompts/description.md @@ -0,0 +1,55 @@ +You write the own-words description of "{{.Title}}" by {{.Authors}}{{if .Series}} ({{.Series}}{{if .SeriesPos}} book {{.SeriesPos}}{{end}}){{end}}: +the one paragraph a stranger reads before deciding to listen. It appears where a +publisher's blurb would, and everyone who opens the book sees it, whatever they +have heard so far. + +## What you have + +You work in the current directory. It contains ONLY: + +- `opening.md` - private fact notes for the opening of the book: its first + {{.Chapters}} of {{.ChapterCount}} chapters, plus any front matter before chapter 1. + This is your whole source.{{if .IsEbook}} The `## Chapter N` headings are the + ebook's chapter numbers.{{else}} The `## Chapter N` headings number the audio + files, not the story's chapters (an opening announcement or preface can take a + file), so read them as labels only.{{end}} +- `out/` - the ONLY place you write. + +You deliberately have nothing else: no book text, no transcript, no notes on +later chapters, no publisher copy. Do not use any tool other than reading and +writing files in this directory. No web access, and do not draw on your own +memory of this book or of anything written about it. + +## The rules + +- **Setup only.** The premise, the situation, who the book is about and what it + is asking. Nothing past the opening act: no reveals, no turns, no deaths, no + roles a character only takes on later, and never the ending. A note in + `opening.md` that points ahead ("named later", "revealed to be") is not yours + to use. When a detail is arguably a reveal, leave it out. +- **Grounded in the notes.** Every claim must come from `opening.md`. Leave out + what the notes don't support: skip over invent, always. Don't name a place, + time or person the notes don't name. +- **Your own words.** Never a blurb, and never the book's own sentences: an + 8-word run copied from the book fails a mechanical check. Quote nothing. +- **Neutral reference-guide voice.** Say what the book is and let the reader + decide. No sales adjectives (gripping, unputdownable, page-turner, must-read, + masterpiece, breathtaking, captivating, riveting, thrilling, unforgettable, and + their like), no second person (no "you" or "your"), no rhetorical questions + (no sentence ends in a question mark), no star-rating by adjective. +- **Length:** 200 to 1500 characters. Aim for 80 to 150 words in one paragraph; + two paragraphs at most. +- **Punctuation:** use a hyphen, never an em dash. +- Present tense, third person. A sequel may say which series it continues, but + must not summarise the earlier books' endings. + +## Output + +Write `out/description.json` containing exactly one JSON object with exactly one +key, and nothing else: + +```json +{"text": "The description paragraph."} +``` + +No other keys, no Markdown, no comments, nothing before or after the object. diff --git a/internal/agent/prompts/prompts_test.go b/internal/agent/prompts/prompts_test.go index 19bf18b..38a59c8 100644 --- a/internal/agent/prompts/prompts_test.go +++ b/internal/agent/prompts/prompts_test.go @@ -213,3 +213,27 @@ func TestFactPassBranchesShareTheSpoilerContract(t *testing.T) { t.Error("ebook branch names the wrong text directory") } } + +// The description prompt renders both source branches, states the exact output +// shape its strict reader enforces, and keeps to hyphens. +func TestDescriptionPromptRenders(t *testing.T) { + type data struct { + Title, Authors, Series, SeriesPos string + Chapters, ChapterCount int + IsEbook bool + } + for _, ebook := range []bool{false, true} { + out, err := Render("description.md", data{Title: "T", Authors: "A", Series: "S", SeriesPos: "2", Chapters: 3, ChapterCount: 12, IsEbook: ebook}) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"opening.md", `{"text": "The description paragraph."}`, "exactly one", "3 of 12 chapters", "S book 2", "skip over invent"} { + if !strings.Contains(out, want) { + t.Errorf("ebook=%v: prompt lacks %q", ebook, want) + } + } + if strings.Contains(out, "\u2014") { + t.Error("the prompt has an em dash") + } + } +} diff --git a/internal/app/app.go b/internal/app/app.go index ebdaa5f..99a6685 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -150,7 +150,8 @@ func New(ctx context.Context, opts Options) (*Bridge, string, error) { } return nil }) - b.gen = generate.New(generate.Options{DB: db, Engine: b.engine.generateEngine, Libraries: b.libs, Mount: b.mount, + b.gen = generate.New(generate.Options{DB: db, Engine: b.engine.generateEngine, Describer: b.engine.describer, + Libraries: b.libs, Mount: b.mount, Server: func() generate.Server { if cl := b.conn.Client(); cl != nil { return cl @@ -160,7 +161,7 @@ func New(ctx context.Context, opts Options) (*Bridge, string, error) { WorkRoot: filepath.Join(opts.DataDir, "work"), Version: opts.Version, RetentionDays: cfg.Transcripts.RetentionDays, CanContribute: b.canContribute, Log: opts.Logger}) b.runner = jobs.New(db, b.server, b.tokenRefused, jobs.Options{Log: opts.Logger, - ServerID: func() string { return b.conn.Pairing().ServerID }}, verifier, b.gen) + ServerID: func() string { return b.conn.Pairing().ServerID }}, verifier, b.gen, b.gen.DescribeJob()) conn.onChange = b.linkChanged return b, oneTime, nil } diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 8c7aded..b615c35 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -138,7 +138,7 @@ func TestPairBeatMapAndRevoke(t *testing.T) { return ok && len(st.Libraries) == 1 && st.Libraries[0].Mapped && st.Libraries[0].Writable }) st, _ := fs.lastBeat() - if st.Version != "0.1.0" || strings.Join(st.Kinds, ",") != "bridge.verify,sidecars.generate" || st.UIURL != "http://bridge.lan:8091" { + if st.Version != "0.1.0" || strings.Join(st.Kinds, ",") != "bridge.verify,sidecars.describe,sidecars.generate" || st.UIURL != "http://bridge.lan:8091" { t.Fatalf("heartbeat = %+v", st) } // Once the engine runs, every heartbeat carries the estimates. @@ -277,7 +277,7 @@ func TestEngineViewAndJobRoutes(t *testing.T) { if _, err := b.Job(context.Background(), 5); !errors.Is(err, api.ErrNotFound) { t.Fatalf("an unknown job = %v", err) } - if k := strings.Join(b.runner.Kinds(), ","); k != "bridge.verify,sidecars.generate" { + if k := strings.Join(b.runner.Kinds(), ","); k != "bridge.verify,sidecars.describe,sidecars.generate" { t.Fatalf("kinds = %s", k) } } diff --git a/internal/app/backend.go b/internal/app/backend.go index a451898..68ee68f 100644 --- a/internal/app/backend.go +++ b/internal/app/backend.go @@ -226,8 +226,9 @@ func (b *Bridge) Job(ctx context.Context, id int64) (*store.ServerJob, error) { return job, err } -// RetryJob retries a parked sidecars.generate job: the engine re-admits its book, -// or a job waiting for its book's folder to be writable checks again now. +// RetryJob retries a parked sidecars job: the engine re-admits a generate job's +// book, or a job waiting in a park of its own (its book's folder not writable, a +// description waiting out an agent park) tries again now. func (b *Bridge) RetryJob(ctx context.Context, id int64) error { err := b.gen.Retry(ctx, id) switch { diff --git a/internal/app/engine.go b/internal/app/engine.go index 75f5d47..3c8e028 100644 --- a/internal/app/engine.go +++ b/internal/app/engine.go @@ -65,6 +65,15 @@ func (s *engineState) generateEngine() generate.Engine { return nil } +// describer is the engine's description step for the generate and describe +// jobs (nil until it started: never a typed nil inside the interface). +func (s *engineState) describer() generate.Describer { + if e := s.get(); e != nil { + return e.exec + } + return nil +} + // agentBackend is the CLI the agent container runs (claude unless codex is set). func agentBackend(cfg config.Config) string { if cfg.Agent.Backend == config.AgentBackendCodex { diff --git a/internal/config/config.go b/internal/config/config.go index c7b6ae2..6c41467 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -408,18 +408,21 @@ func Default() Config { // defaultClaudeModels seeds the claude model routing map: which model runs each // agent stage under the claude backend. Cheap stages get sonnet; the load-bearing -// synthesis/audit/fix stages get opus. Model aliases ("sonnet"/"opus"/"haiku") are -// what the claude CLI accepts - never a pinned vendor model id. The openai map is -// intentionally empty (an unset codex model = the codex CLI's default model). +// synthesis/audit/fix stages get opus. The description step (state.Describing, one +// small notes file in, one paragraph out) is a cheap one too. Model aliases +// ("sonnet"/"opus"/"haiku") are what the claude CLI accepts - never a pinned vendor +// model id. The openai map is intentionally empty (an unset codex model = the codex +// CLI's default model). func defaultClaudeModels() map[string]string { return map[string]string{ - "markers_normalizing": "sonnet", - "qa_adjudicating": "sonnet", - "spelling_research": "sonnet", - "fact_pass": "sonnet", - "synthesizing": "opus", - "auditing": "opus", - "fixing": "opus", + "markers_normalizing": "sonnet", + "qa_adjudicating": "sonnet", + "spelling_research": "sonnet", + "fact_pass": "sonnet", + "synthesizing": "opus", + "auditing": "opus", + "fixing": "opus", + string(state.Describing): "sonnet", } } @@ -837,14 +840,15 @@ func (c Config) Validate() error { return fmt.Errorf("pricing.rates[%q] values must be >= 0", key) } } - // Model-map keys must name agent-lane stages; a typo would silently never route. + // Model-map keys must name agent-lane stages (or the description step); a typo + // would silently never route. for key := range c.Agent.Claude { - if !state.IsAgent(state.State(key)) { + if !state.RoutesModel(state.State(key)) { return fmt.Errorf("agent.claude has key %q, which is not an agent stage", key) } } for key := range c.Agent.OpenAI { - if !state.IsAgent(state.State(key)) { + if !state.RoutesModel(state.State(key)) { return fmt.Errorf("agent.openai has key %q, which is not an agent stage", key) } } diff --git a/internal/config/config_test.go b/internal/config/config_test.go index c049cb3..45a2e07 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -239,8 +239,8 @@ func TestValidateAgentFields(t *testing.T) { good := Default() good.Agent.Backend = AgentBackendClaude good.Agent.TimeoutMinutes = 1 - good.Agent.Claude = map[string]string{"fact_pass": "sonnet", "auditing": "opus"} - good.Agent.OpenAI = map[string]string{"synthesizing": "gpt-x"} + good.Agent.Claude = map[string]string{"fact_pass": "sonnet", "auditing": "opus", "describing": "haiku"} + good.Agent.OpenAI = map[string]string{"synthesizing": "gpt-x", "describing": "gpt-y"} if err := good.Validate(); err != nil { t.Errorf("Validate() = %v, want nil for valid agent config", err) } @@ -259,6 +259,7 @@ func TestDefaultSeedsClaudeModels(t *testing.T) { "synthesizing": "opus", "auditing": "opus", "fixing": "opus", + "describing": "sonnet", } for k, v := range want { if cfg.Agent.Claude[k] != v { diff --git a/internal/generate/attach.go b/internal/generate/attach.go index 102b18f..8efeeca 100644 --- a/internal/generate/attach.go +++ b/internal/generate/attach.go @@ -10,6 +10,7 @@ import ( "os" "path/filepath" "strings" + "syscall" "time" "github.com/kodestar/audiosilo-meta/pkg/model" @@ -24,9 +25,10 @@ const AttachDir = ".audiosilo" // The attachment names (a single-file book's get "." in front). const ( - charactersName = "characters.json" - recapsName = "recaps.json" - provenanceName = "bridge.json" + charactersName = "characters.json" + recapsName = "recaps.json" + descriptionName = "description.json" + provenanceName = "bridge.json" ) // maxAttachment is the largest attachment the server reads (4 MiB); a bigger one @@ -34,15 +36,24 @@ const ( const maxAttachment = 4 << 20 // sidecars are a finished engine book's validated characters and recaps, as the -// pipeline left them (pipeline.FinalSidecarsPaths). +// pipeline left them (pipeline.FinalSidecarsPaths), and the description the run's +// description step read back (pipeline.FinalDescriptionPath), when it has one. type sidecars struct { - characters, recaps []byte - nChars, nRecaps int + characters, recaps, description []byte + nChars, nRecaps int // workID is the slug the files name (the resolved work, or the pipeline's // placeholder). workID string - // hash identifies the content, so the same files aren't written twice. - hash string +} + +// hash identifies the content, so the same files aren't written twice. +func (s sidecars) hash() string { + h := sha256.New() + for _, b := range [][]byte{s.characters, s.recaps, s.description} { + _, _ = h.Write(b) + _, _ = h.Write([]byte{0}) + } + return hex.EncodeToString(h.Sum(nil)) } // sidecarsSig is the size and modification time of the engine's sidecar files @@ -64,35 +75,53 @@ func sidecarsSig(workDir string) string { return "" } +// readAttachment reads one of the engine's files for the library: nil when it +// isn't there, an error over the server's size cap. +func readAttachment(path string) ([]byte, error) { + f, err := os.Open(path) //nolint:gosec // the engine's own work dir + if errors.Is(err, os.ErrNotExist) { + return nil, nil + } + if err != nil { + return nil, err + } + defer func() { _ = f.Close() }() + b, err := io.ReadAll(io.LimitReader(f, maxAttachment+1)) + if err != nil { + return nil, err + } + if len(b) > maxAttachment { + return nil, fmt.Errorf("%s is larger than %d bytes, which the server won't read", filepath.Base(path), maxAttachment) + } + return b, nil +} + +// readDescription reads the engine's description.json at path (nil when there is +// none): it must decode into the community model, which it returns too. +func readDescription(path string) ([]byte, model.Description, error) { + var d model.Description + b, err := readAttachment(path) + if err != nil || b == nil { + return nil, d, err + } + if err := json.Unmarshal(b, &d); err != nil { + return nil, d, fmt.Errorf("the engine's description.json doesn't decode: %w", err) + } + return b, d, nil +} + // readSidecars reads the engine's characters.json and recaps.json: each must -// decode into the community model and fit the server's size cap. A missing file -// is skipped; neither present is an error. +// decode into the community model and fit the server's size cap. A missing file is +// skipped; neither present is an error. The description is never read here: the +// run ships only the one its own step read back (write). func readSidecars(workDir string) (sidecars, error) { var s sidecars - read := func(path string) ([]byte, error) { - f, err := os.Open(path) //nolint:gosec // the engine's own work dir - if errors.Is(err, os.ErrNotExist) { - return nil, nil - } - if err != nil { - return nil, err - } - defer func() { _ = f.Close() }() - b, err := io.ReadAll(io.LimitReader(f, maxAttachment+1)) - if err != nil { - return nil, err - } - if len(b) > maxAttachment { - return nil, fmt.Errorf("%s is larger than %d bytes, which the server won't read", filepath.Base(path), maxAttachment) - } - return b, nil - } charsPath, recapsPath := pipeline.FinalSidecarsPaths(workDir) var err error - if s.characters, err = read(charsPath); err != nil { + if s.characters, err = readAttachment(charsPath); err != nil { return sidecars{}, err } - if s.recaps, err = read(recapsPath); err != nil { + if s.recaps, err = readAttachment(recapsPath); err != nil { return sidecars{}, err } if s.characters == nil && s.recaps == nil { @@ -115,11 +144,6 @@ func readSidecars(workDir string) (sidecars, error) { s.workID = r.Work } } - h := sha256.New() - _, _ = h.Write(s.characters) - _, _ = h.Write([]byte{0}) - _, _ = h.Write(s.recaps) - s.hash = hex.EncodeToString(h.Sum(nil)) return s, nil } @@ -133,6 +157,29 @@ type Provenance struct { Source string `json:"source"` WorkID string `json:"work_id"` Inputs []InputFile `json:"inputs"` + // Description is how description.json was made (Phase 1c; the server reads + // only the fields above). Absent when no description was written with these. + Description *DescriptionProvenance `json:"description,omitempty"` +} + +// DescriptionProvenance is bridge.json's description object: which job wrote +// description.json and when, from what (the opening chapters' fact notes), and +// what its no-verbatim check measured against. +type DescriptionProvenance struct { + JobID int64 `json:"job_id"` + GeneratedAt string `json:"generated_at"` + From string `json:"from"` + Chapters int `json:"chapters"` + CheckedAgainst string `json:"checked_against"` +} + +// descriptionFrom is the only source a description is written from. +const descriptionFrom = "opening_notes" + +// descriptionProvenance composes the description object for res, written by job. +func descriptionProvenance(jobID int64, res pipeline.DescribeResult) *DescriptionProvenance { + return &DescriptionProvenance{JobID: jobID, GeneratedAt: res.GeneratedAt.UTC().Format(time.RFC3339), From: descriptionFrom, + Chapters: res.Chapters, CheckedAgainst: res.CheckedAgainst} } // InputFile is one file the run read, its path relative to the library root. @@ -162,49 +209,157 @@ func inputsOf(mount string, files []link.BookFile, epub string) []InputFile { return out } -// writeAttachments writes the sidecars and the provenance into dir/.audiosilo/ -// (contract section 4); attachmentNames names what it writes. The folder is made -// 0755 if missing and refused when it is a link or not a folder; each file goes -// to a temp file inside it, 0644, then is renamed into place, so the server never -// reads half a file. Only these JSON names are ever written there. -func writeAttachments(dir, prefix string, s sidecars, prov Provenance) error { +// attachmentFiles are the sidecars and the provenance as the files written into +// the book's .audiosilo/ (contract section 4); attachmentNames names them. +func attachmentFiles(s sidecars, prov Provenance) ([]attachFile, error) { + provJSON, err := indentJSON(prov) + if err != nil { + return nil, err + } + return []attachFile{{charactersName, s.characters}, {recapsName, s.recaps}, {descriptionName, s.description}, + {provenanceName, provJSON}}, nil +} + +// indentJSON is v as bridge.json is written: indented, a newline at the end. +func indentJSON(v any) ([]byte, error) { + out, err := json.MarshalIndent(v, "", " ") + if err != nil { + return nil, err + } + return append(out, '\n'), nil +} + +// attachFile is one file for the book's .audiosilo/ (nil data: not written). +type attachFile struct { + name string + data []byte +} + +// writeAttachFiles writes files into dir/.audiosilo/. The folder is made 0755 if +// missing and refused when it is a link or not a folder; each file goes to a temp +// file inside it, 0644, then is renamed into place, so the server never reads half +// a file. Only the known JSON names are ever written there. +func writeAttachFiles(dir, prefix string, files []attachFile) error { + adir, err := attachFolder(dir) + if err != nil { + return err + } + for _, f := range files { + if f.data == nil { + continue + } + // 0644: the server (another user) reads it. + if err := fsutil.WriteFileAtomic(filepath.Join(adir, prefix+f.name), f.data, 0o644); err != nil { + return err + } + } + return nil +} + +// attachFolder is dir/.audiosilo/, made 0755 when missing; a link or a file +// there is refused. +func attachFolder(dir string) (string, error) { adir := filepath.Join(dir, AttachDir) info, err := os.Lstat(adir) switch { case errors.Is(err, os.ErrNotExist): if err := os.Mkdir(adir, 0o755); err != nil { //nolint:gosec // the server (another user) reads it - return err + return "", err } // The umask may have narrowed it; the server must be able to list it. if err := os.Chmod(adir, 0o755); err != nil { //nolint:gosec // as above - return err + return "", err } case err != nil: - return err + return "", err case info.Mode()&os.ModeSymlink != 0 || !info.IsDir(): - return fmt.Errorf("the book's %s exists and isn't a folder; the Bridge won't write through it", AttachDir) + return "", fmt.Errorf("the book's %s exists and isn't a folder; the Bridge won't write through it", AttachDir) + } + return adir, nil +} + +// maxProvenance bounds the read of an existing bridge.json (the Bridge's own, +// a few hundred bytes). +const maxProvenance = 256 << 10 + +// describedProvenance is bridge.json after a description-only job: the existing +// file with its description object added or replaced (every other field kept as +// it is), or, when there is none (or it isn't a JSON object, or isn't a regular +// file), fresh with the usual fields. +func describedProvenance(dir, prefix string, fresh Provenance) ([]byte, error) { + existing := readProvenance(filepath.Join(dir, AttachDir), prefix+provenanceName) + if existing == nil { + return indentJSON(fresh) } - provJSON, err := json.MarshalIndent(prov, "", " ") + desc, err := json.Marshal(fresh.Description) if err != nil { - return err + return nil, err } - for _, f := range []struct { - name string - data []byte - }{{charactersName, s.characters}, {recapsName, s.recaps}, {provenanceName, append(provJSON, '\n')}} { - if f.data == nil { - continue - } - // 0644: the server (another user) reads it. - if err := fsutil.WriteFileAtomic(filepath.Join(adir, prefix+f.name), f.data, 0o644); err != nil { - return err - } + existing["description"] = desc + return indentJSON(existing) +} + +// keptDescription is the description object of the book's bridge.json, when a +// description.json is in place beside it: a run that writes no description of its +// own leaves an earlier run's file there (nothing deletes into a library), and the +// new bridge.json carries over what vouches for it. nil when either is missing, +// isn't a regular file, or holds no description object. +func keptDescription(dir, prefix string) *DescriptionProvenance { + adir := filepath.Join(dir, AttachDir) + if info, err := os.Lstat(filepath.Join(adir, prefix+descriptionName)); err != nil || !info.Mode().IsRegular() { + return nil } - return nil + raw, ok := readProvenance(adir, prefix+provenanceName)["description"] + if !ok { + return nil + } + var d *DescriptionProvenance + if json.Unmarshal(raw, &d) != nil { + return nil + } + return d +} + +// readProvenance reads adir/name (a bridge.json) as a JSON object, nil when there +// is none fit to keep. The library is another program's folder, so the read is +// the agent container's kind (agentproto.ReadRegular, which generate may not +// import): inside adir as an os.Root, of the regular file that was looked at (a +// link, FIFO or device swapped in after the look is refused, the open never +// blocking), capped. A .audiosilo that is itself a link is not read through. +func readProvenance(adir, name string) map[string]json.RawMessage { + if info, err := os.Lstat(adir); err != nil || !info.IsDir() { + return nil + } + root, err := os.OpenRoot(adir) + if err != nil { + return nil + } + defer func() { _ = root.Close() }() + before, err := root.Lstat(name) + if err != nil || !before.Mode().IsRegular() || before.Size() > maxProvenance { + return nil + } + f, err := root.OpenFile(name, os.O_RDONLY|syscall.O_NONBLOCK, 0) + if err != nil { + return nil + } + defer func() { _ = f.Close() }() + if info, err := f.Stat(); err != nil || !os.SameFile(before, info) { + return nil + } + raw, err := io.ReadAll(io.LimitReader(f, maxProvenance+1)) + if err != nil || len(raw) > maxProvenance { + return nil + } + var m map[string]json.RawMessage + if json.Unmarshal(raw, &m) != nil || m == nil { + return nil + } + return m } -// provenance composes bridge.json. -func provenance(version string, jobID int64, now time.Time, source, workID string, inputs []InputFile) Provenance { - return Provenance{Generator: "audiosilo-bridge", Version: version, Kind: Kind, JobID: jobID, +// provenance composes bridge.json for a job of kind. +func provenance(version, kind string, jobID int64, now time.Time, source, workID string, inputs []InputFile) Provenance { + return Provenance{Generator: "audiosilo-bridge", Version: version, Kind: kind, JobID: jobID, GeneratedAt: now.UTC().Format(time.RFC3339), Source: source, WorkID: workID, Inputs: inputs} } diff --git a/internal/generate/describe.go b/internal/generate/describe.go index 072cd6f..16a9277 100644 --- a/internal/generate/describe.go +++ b/internal/generate/describe.go @@ -102,6 +102,16 @@ var parkHints = map[string]string{ // ParkHint is the advice for a park code ("" for one with none). func ParkHint(code string) string { return parkHints[code] } +// parkExtra is a park as a job's progress carries it: its code, its hint and, +// when it tries again by itself, its retry_at ("" for none). +func parkExtra(code, retryAt string) map[string]any { + p := map[string]any{"code": code, "hint": ParkHint(code)} + if retryAt != "" { + p["retry_at"] = retryAt + } + return p +} + // view is what a poll of the engine book found, for describe. type view struct { book store.Book @@ -134,11 +144,7 @@ func describe(v view) jobs.Progress { extra := map[string]any{"stage": b.State, "source": v.source, "cost_usd": Cents(v.costUSD)} switch state.Status(b.Status) { case state.StatusNeedsAttention: - park := map[string]any{"code": b.ParkCode, "hint": ParkHint(b.ParkCode)} - if b.RetryAt != "" { - park["retry_at"] = b.RetryAt - } - extra["park"] = park + extra["park"] = parkExtra(b.ParkCode, b.RetryAt) pr.Message = "Needs attention: " + pr.Message case state.StatusPaused: pr.Message = "Paused: " + pr.Message diff --git a/internal/generate/describefix_test.go b/internal/generate/describefix_test.go new file mode 100644 index 0000000..cf9dc14 --- /dev/null +++ b/internal/generate/describefix_test.go @@ -0,0 +1,626 @@ +package generate + +import ( + "context" + "encoding/json" + "errors" + "io" + "log/slog" + "os" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/kodestar/audiosilo-bridge/internal/link" + "github.com/kodestar/audiosilo-bridge/internal/pipeline" + "github.com/kodestar/audiosilo-bridge/internal/scheduler" + "github.com/kodestar/audiosilo-bridge/internal/state" + "github.com/kodestar/audiosilo-bridge/internal/store" +) + +// newParks counts a job's moves into needs_attention with park code: each one the +// server announces as a new park. +func (f *fakeServer) newParks(id int64, code string) int { + f.mu.Lock() + defer f.mu.Unlock() + n, prev := 0, "" + for _, u := range f.updates[id] { + if u.State == nil || *u.State == prev { + continue + } + var p struct { + Park *struct { + Code string `json:"code"` + } `json:"park"` + } + _ = json.Unmarshal(u.Progress, &p) + if *u.State == link.StateNeedsAttention && p.Park != nil && p.Park.Code == code { + n++ + } + prev = *u.State + } + return n +} + +// A new generate job finds the earlier run held by a description (the server asked +// it to give way, which it does at its next report): it waits for the engine to +// let go, then generates, instead of failing. +func TestGenerateWaitsForADescriptionToLetGo(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(73) + b := hr.engineBook(73) + release, ok := hr.sched.Hold(b.ID) + if !ok { + t.Fatal("the finished book couldn't be held") + } + h := hr.handler() + h.o.Poll = 50 * time.Millisecond // holdTries polls: a second to let go in + hr.srv.add(74, "Author/Book", `{}`) + stop := hr.runner(h) + defer stop() + hr.waitFor("the wait for the earlier run", func() bool { + return hr.srv.saw(74, func(_ link.Update, p map[string]any) bool { + return p["message"] == "Waiting for the engine to let go of the book's earlier run" + }) + }) + release() + hr.waitFor("done", func() bool { s, _ := hr.srv.final(74); return s == link.StateDone }) + if _, err := hr.db.GetBook(context.Background(), b.ID); !errors.Is(err, store.ErrNotFound) { + t.Fatalf("the earlier run wasn't cleared: %v", err) + } +} + +// A description written at a contributing park is composed again at done (the +// step's free re-compose): the contributing stage resolved the work since, and the +// library's description names the work its characters do. +func TestTheParkDescriptionFollowsTheResolvedWork(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &fakeDescriber{db: hr.db, fromWork: true} + h := hr.describeHandler(d) + hr.exec.parks["contributing"] = 1 + hr.srv.add(33, "Author/Book", `{}`) + stop := hr.runAll(h) + defer stop() + adir := filepath.Join(hr.root, "Author", "Book", AttachDir) + hr.waitFor("the park's description", func() bool { + raw, err := os.ReadFile(filepath.Join(adir, "description.json")) + return err == nil && strings.Contains(string(raw), `"work":"the-book"`) + }) + // The contributing stage resolves the work (its reconcile rewrites + // characters.json; the description is left as it was). + chars, _ := pipeline.FinalSidecarsPaths(hr.engineBook(33).WorkDir) + raw, err := os.ReadFile(chars) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(chars, []byte(strings.Replace(string(raw), `"work":"the-book"`, `"work":"the-real-book"`, 1)), 0o600); err != nil { + t.Fatal(err) + } + if err := h.Retry(context.Background(), 33); err != nil { + t.Fatal(err) + } + hr.waitFor("done", func() bool { s, _ := hr.srv.final(33); return s == link.StateDone }) + for _, name := range []string{"characters.json", "description.json"} { + if raw, err := os.ReadFile(filepath.Join(adir, name)); err != nil || !strings.Contains(string(raw), `"work":"the-real-book"`) { + t.Fatalf("%s = %s, %v", name, raw, err) + } + } + if calls := d.calls(); len(calls) != 2 { + t.Fatalf("describe calls = %v, want the park's and done's", calls) + } +} + +// At a contributing park the job reports the park before it writes, and every +// report while parked carries it (the page shows the park, never a bare error), +// the write's and the description's steps included. +func TestAContributingParkIsReportedThroughTheWrite(t *testing.T) { + t.Parallel() + hr := newHarness(t) + gate := make(chan struct{}) + d := &fakeDescriber{db: hr.db, fail: func(int) error { <-gate; return nil }} + h := hr.describeHandler(d) + hr.exec.parks["contributing"] = 1 + hr.srv.add(38, "Author/Book", `{}`) + stop := hr.runAll(h) + defer stop() + // The description step is held: the park is the job's meanwhile. + hr.waitFor("the park while describing", func() bool { + return hr.srv.saw(38, func(u link.Update, p map[string]any) bool { + park, _ := p["park"].(map[string]any) + return u.State != nil && *u.State == link.StateNeedsAttention && p["step"] == pipeline.DescribeStage && + park["code"] == string(state.ParkCoreNeeded) + }) + }) + close(gate) + if hr.srv.saw(38, func(u link.Update, p map[string]any) bool { + return u.State != nil && *u.State == link.StateNeedsAttention && p["park"] == nil + }) { + t.Fatal("a parked report without its park") + } + if err := h.Retry(context.Background(), 38); err != nil { + t.Fatal(err) + } + hr.waitFor("done", func() bool { s, _ := hr.srv.final(38); return s == link.StateDone }) +} + +// A try at the park that failed without spending (a rate limit) gets a second go +// at done; one that spent (its checks exhausted) doesn't. +func TestTheParkDescriptionFailureAtDone(t *testing.T) { + t.Parallel() + for name, tc := range map[string]struct { + park error + free bool + calls int + desc bool + }{ + "free": {scheduler.ParkWithCodeAfter(state.ParkAgentRateLimited, "rate limited", time.Now().Add(time.Hour)), true, 2, true}, + "spent": {scheduler.ParkWithCode(state.ParkAgentValidationExhausted, "kept failing"), false, 1, false}, + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &fakeDescriber{db: hr.db, freeFail: tc.free, fail: func(call int) error { + if call == 1 { + return tc.park + } + return nil + }} + h := hr.describeHandler(d) + hr.exec.parks["contributing"] = 1 + hr.srv.add(34, "Author/Book", `{}`) + stop := hr.runAll(h) + defer stop() + hr.waitFor("the park's try", func() bool { + return len(d.calls()) == 1 && hr.srv.saw(34, func(u link.Update, p map[string]any) bool { + park, _ := p["park"].(map[string]any) + return park["code"] == "core_needed" + }) + }) + if err := h.Retry(context.Background(), 34); err != nil { + t.Fatal(err) + } + hr.waitFor("done", func() bool { s, _ := hr.srv.final(34); return s == link.StateDone }) + var res Result + hr.result(34, &res) + if calls := d.calls(); len(calls) != tc.calls || res.Description != tc.desc { + t.Fatalf("calls %v, result %+v", calls, res) + } + }) + } +} + +// When the re-compose at done fails, the description the park wrote stands. +func TestTheParkDescriptionStandsWhenDoneCantRecompose(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &fakeDescriber{db: hr.db, fail: func(call int) error { + if call == 2 { + return errors.New("the disk is full") + } + return nil + }} + h := hr.describeHandler(d) + hr.exec.parks["contributing"] = 1 + hr.srv.add(37, "Author/Book", `{}`) + stop := hr.runAll(h) + defer stop() + hr.waitFor("the park's description", func() bool { return len(d.calls()) == 1 }) + hr.waitFor("the park", func() bool { return hr.srv.newParks(37, string(state.ParkCoreNeeded)) > 0 }) + if err := h.Retry(context.Background(), 37); err != nil { + t.Fatal(err) + } + hr.waitFor("done", func() bool { s, _ := hr.srv.final(37); return s == link.StateDone }) + var res Result + hr.result(37, &res) + raw, err := os.ReadFile(filepath.Join(hr.root, "Author", "Book", AttachDir, "bridge.json")) + if len(d.calls()) != 2 || !res.Description || err != nil || !strings.Contains(string(raw), `"description"`) { + t.Fatalf("calls %v, result %+v, bridge.json %s", d.calls(), res, raw) + } +} + +// A description that spent and then failed its checks still counts in the +// generate job's cost, as the budgets count it. +func TestGenerateCostCountsAFailedDescription(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &spendingDescriber{db: hr.db} + hr.srv.add(35, "Author/Book", `{}`) + stop := hr.runAll(hr.describeHandler(d)) + defer stop() + hr.waitFor("done", func() bool { s, _ := hr.srv.final(35); return s == link.StateDone }) + var res Result + hr.result(35, &res) + sum, _, err := hr.db.SumStageRunBudgetCost(context.Background(), hr.engineBook(35).ID) + if err != nil || res.Description || res.CostUSD != Cents(sum) || res.CostUSD != 3.65 { + t.Fatalf("result %+v, the book's spend %v, %v", res, sum, err) + } +} + +// spendingDescriber spends on a describing stage run as the real step does (three +// tries at 0.05), then fails its checks. +type spendingDescriber struct{ db *store.DB } + +func (s *spendingDescriber) Describe(ctx context.Context, bookID int64, _ bool) (pipeline.DescribeResult, error) { + attempt, err := s.db.NextStageAttempt(ctx, bookID, pipeline.DescribeStage) + if err != nil { + return pipeline.DescribeResult{}, err + } + runID, err := s.db.StartStageRun(ctx, bookID, pipeline.DescribeStage, attempt) + if err != nil { + return pipeline.DescribeResult{}, err + } + for range 3 { + id, err := s.db.StartAgentInvocation(ctx, bookID, pipeline.DescribeStage, "describing-a01", "claude", "sonnet") + if err != nil { + return pipeline.DescribeResult{}, err + } + if err := s.db.FinishAgentInvocation(ctx, id, "validation_failed", "sonnet", 900, 120, 0, 0.05, true, 0, false, "too short"); err != nil { + return pipeline.DescribeResult{}, err + } + } + _ = s.db.FinishStageRun(ctx, runID, false, nil) + cost, _ := s.db.StageRunBudgetCost(ctx, runID) + return pipeline.DescribeResult{CostUSD: cost}, scheduler.ParkWithCode(state.ParkAgentValidationExhausted, "kept failing") +} + +// The characters and recaps reach the library before the description step runs: +// the step may wait a long time for an agent slot, and never holds them back. +func TestGenerateSavesBeforeTheDescription(t *testing.T) { + t.Parallel() + hr := newHarness(t) + var savedFirst atomic.Bool + chars := filepath.Join(hr.root, "Author", "Book", AttachDir, "characters.json") + d := &fakeDescriber{db: hr.db, fail: func(int) error { + _, err := os.Stat(chars) + savedFirst.Store(err == nil) + return nil + }} + hr.srv.add(36, "Author/Book", `{}`) + stop := hr.runAll(hr.describeHandler(d)) + defer stop() + hr.waitFor("done", func() bool { s, _ := hr.srv.final(36); return s == link.StateDone }) + var res Result + hr.result(36, &res) + if !savedFirst.Load() || !res.Description || + strings.Join(res.Files, ",") != "characters.json,recaps.json,description.json,bridge.json" { + t.Fatalf("saved before describing: %v, result %+v", savedFirst.Load(), res) + } +} + +// A description-only job whose book's .audiosilo/ can't be written parks +// book_not_writable once and stays parked while it tries again (never running in +// between, which the server would hear as a new park each time). +func TestDescribeJobParkDoesNotFlap(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(41) + adir := filepath.Join(hr.root, "Author", "Book", AttachDir) + if err := os.RemoveAll(adir); err != nil { + t.Fatal(err) + } + mustWrite(t, adir, "a file in the folder's place") + h := hr.describeHandler(&fakeDescriber{db: hr.db}) + h.o.Recheck = 20 * time.Millisecond + hr.srv.addDescribe(42, "Author/Book") + stop := hr.runAll(h) + defer stop() + hr.waitFor("the park", func() bool { return hr.srv.newParks(42, parkBookNotWritable) > 0 }) + time.Sleep(400 * time.Millisecond) // many rechecks + if n := hr.srv.newParks(42, parkBookNotWritable); n != 1 { + t.Fatalf("parked anew %d times", n) + } + if err := os.Remove(adir); err != nil { + t.Fatal(err) + } + if err := h.Retry(context.Background(), 42); err != nil { + t.Fatal(err) + } + hr.waitFor("done", func() bool { s, _ := hr.srv.final(42); return s == link.StateDone }) +} + +// heldRescan holds the attachments rescan until gate closes. +type heldRescan struct { + Server + gate chan struct{} +} + +func (g heldRescan) RescanAttachments(ctx context.Context, libraryID int64, path string) (*link.Attachments, error) { + select { + case <-g.gate: + case <-ctx.Done(): + return nil, ctx.Err() + } + return g.Server.RescanAttachments(ctx, libraryID, path) +} + +// A description-only job parked when the Bridge stopped resumes with its park +// restored; the try that succeeds lifts it, so the run's later reports aren't +// the old park. +func TestDescribeJobLiftsARestoredPark(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(41) + d := &fakeDescriber{db: hr.db, fail: func(call int) error { + if call == 1 { + return scheduler.ParkWithCodeAfter(state.ParkAgentRateLimited, "rate limited", time.Now().Add(time.Hour)) + } + return nil + }} + stop := hr.runAll(hr.describeHandler(d)) + hr.srv.addDescribe(42, "Author/Book") + hr.waitFor("the park", func() bool { return hr.srv.newParks(42, string(state.ParkAgentRateLimited)) > 0 }) + stop() // the Bridge stops with the job parked + hr.srv.set(func() { hr.srv.updates[42] = nil }) + + gate := make(chan struct{}) + h := hr.describeHandler(d) + h.o.Server = func() Server { return heldRescan{Server: hr.cl, gate: gate} } + stop2 := hr.runAll(h) + defer stop2() + hr.waitFor("the rescan", func() bool { + return hr.srv.saw(42, func(_ link.Update, p map[string]any) bool { return p["step"] == stepRescanning }) + }) + time.Sleep(100 * time.Millisecond) // a keepalive or two + if hr.srv.saw(42, func(u link.Update, p map[string]any) bool { + return (p["step"] == stepWriting || p["step"] == stepRescanning) && u.State != nil && *u.State == link.StateNeedsAttention + }) { + t.Fatal("the restored park went on after the description succeeded") + } + close(gate) + hr.waitFor("done", func() bool { s, _ := hr.srv.final(42); return s == link.StateDone }) +} + +// A Retry pressed while a parked description-only job tries again at its +// retry_at is answered by that try: when it parks again, the job waits for its new +// retry_at rather than trying once more at once. +func TestDescribeJobRetryDuringATry(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(41) + trying := make(chan struct{}) + gate := make(chan struct{}) + d := &fakeDescriber{db: hr.db, freeFail: true} + d.fail = func(call int) error { + switch call { + case 1: + return scheduler.ParkWithCodeAfter(state.ParkAgentRateLimited, "rate limited", time.Now().Add(20*time.Millisecond)) + case 2: + close(trying) + <-gate + return scheduler.ParkWithCodeAfter(state.ParkAgentRateLimited, "rate limited", time.Now().Add(time.Hour)) + } + return nil + } + h := hr.describeHandler(d) + stop := hr.runAll(h) + defer stop() + hr.srv.addDescribe(42, "Author/Book") + <-trying + if err := h.Retry(context.Background(), 42); err != nil { + t.Fatal(err) + } + close(gate) + time.Sleep(200 * time.Millisecond) + if n := len(d.calls()); n != 2 { + t.Fatalf("describe calls = %d after the press during a try", n) + } + if err := h.Retry(context.Background(), 42); err != nil { + t.Fatal(err) + } + hr.waitFor("done", func() bool { s, _ := hr.srv.final(42); return s == link.StateDone }) +} + +// unmappedLibs is library 1, checked mapped and writable or not, as told. +type unmappedLibs struct { + root string + mapped atomic.Bool +} + +func (l *unmappedLibs) Library(id int64) (link.Library, bool) { + return link.Library{ID: 1, Name: "Books", Root: l.root}, id == 1 +} + +func (l *unmappedLibs) Status(id int64) (link.LibraryStatus, bool) { + m := l.mapped.Load() + return link.LibraryStatus{LibraryID: 1, MountPath: l.root, Mapped: m, Writable: m}, id == 1 +} + +// A description-only job resumed after a restart onto a library that isn't mapped +// (yet) parks library_not_writable and goes on once it is (denied: a new one is +// refused at once). +func TestDescribeJobResumedOntoAnUnmappedLibrary(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(71) + libs := &unmappedLibs{root: hr.root} + libs.mapped.Store(true) + handler := func(d Describer) *Handler { + return New(Options{DB: hr.db, Engine: func() Engine { return hr.sched }, Describer: func() Describer { return d }, + Libraries: libs, Mount: func(int64, string) (string, bool) { return hr.root, false }, + Server: func() Server { return hr.cl }, WorkRoot: hr.work, Version: "0.2.0", RetentionDays: 30, + Poll: 10 * time.Millisecond, IdlePoll: 10 * time.Millisecond, Recheck: time.Hour, Log: slog.New(slog.NewTextHandler(io.Discard, nil))}) + } + d := &fakeDescriber{db: hr.db, fail: func(call int) error { + if call == 1 { + return scheduler.ParkWithCode(state.ParkBudgetExceeded, "the budget is spent") + } + return nil + }} + stop := hr.runAll(handler(d)) + hr.srv.addDescribe(72, "Author/Book") + hr.waitFor("the budget park", func() bool { return hr.srv.newParks(72, string(state.ParkBudgetExceeded)) > 0 }) + stop() + libs.mapped.Store(false) + stop2 := hr.runAll(handler(d)) + defer stop2() + hr.waitFor("library_not_writable", func() bool { + return hr.srv.saw(72, func(_ link.Update, p map[string]any) bool { + park, _ := p["park"].(map[string]any) + return park["code"] == parkLibraryNotWritable + }) + }) + if s, _ := hr.srv.final(72); s != "" { + t.Fatalf("the resumed job ended: %s", s) + } + libs.mapped.Store(true) + hr.waitFor("done", func() bool { s, _ := hr.srv.final(72); return s == link.StateDone }) + + // A new one on the unmapped library is refused at once. + libs.mapped.Store(false) + hr.srv.addDescribe(73, "Author/Book") + hr.waitFor("the refusal", func() bool { s, _ := hr.srv.final(73); return s == link.StateFailed }) +} + +// A book that moved after its generate job ended: the Bridge's records stayed at +// the old place, but the book's bridge.json moved with it and names the run, whose +// notes the description-only job reads; the job's mapping follows the book. +func TestDescribeJobFindsARunThatMovedAfterItsJob(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(93) + if err := os.Rename(filepath.Join(hr.root, "Author", "Book"), filepath.Join(hr.root, "Author", "Book (2026)")); err != nil { + t.Fatal(err) + } + hr.srv.set(func() { + hr.srv.books["Author/Book (2026)"] = link.Book{LibraryID: 1, Path: "Author/Book (2026)", Files: []link.BookFile{ + {Path: "Author/Book (2026)/01.mp3", Size: 9}, {Path: "Author/Book (2026)/02.mp3", Size: 10}}} + delete(hr.srv.books, "Author/Book") + }) + d := &fakeDescriber{db: hr.db} + stop := hr.runAll(hr.describeHandler(d)) + defer stop() + hr.srv.addDescribe(94, "Author/Book (2026)") + hr.waitFor("done", func() bool { s, _ := hr.srv.final(94); return s == link.StateDone }) + if len(d.calls()) != 1 { + t.Fatalf("describe calls = %v", d.calls()) + } + if m, err := hr.db.GetGenerateBook(context.Background(), 93); err != nil || m.RelPath != "Author/Book (2026)" { + t.Fatalf("the run's mapping = %+v, %v", m, err) + } +} + +// A description-only job doesn't take another server's run of the same library id +// and path for its notes: that run's book lies outside this library's folder. +func TestDescribeJobIgnoresAnotherServersRun(t *testing.T) { + t.Parallel() + hr := newHarness(t) + ctx := context.Background() + other := filepath.Join(t.TempDir(), "elsewhere", "Author", "Book") + b, err := hr.db.CreateBook(ctx, store.NewBook{SourcePath: other, WorkDir: filepath.Join(hr.work, "other"), Title: "Other"}) + if err != nil { + t.Fatal(err) + } + _ = hr.db.SetBookState(ctx, b.ID, "done", "", "", "") + if err := hr.db.PutGenerateBook(ctx, store.GenerateBook{ServerJobID: 9, BookID: &b.ID, LibraryID: 1, RelPath: "Author/Book"}); err != nil { + t.Fatal(err) + } + _ = hr.db.EndGenerateBook(ctx, 9, time.Now()) + d := &fakeDescriber{db: hr.db} + stop := hr.runAll(hr.describeHandler(d)) + defer stop() + hr.srv.addDescribe(42, "Author/Book") + hr.waitFor("the refusal", func() bool { s, _ := hr.srv.final(42); return s == link.StateFailed }) + if _, u := hr.srv.final(42); u.Error == nil || *u.Error != string(errDescribeNoNotes) || len(d.calls()) != 0 { + t.Fatalf("failed with %v, calls %v", u.Error, d.calls()) + } +} + +// A regenerate whose own description fails leaves the earlier description.json in +// place (nothing deletes into a library), and its bridge.json carries over the +// earlier one's description object, so the file is still vouched for. With no +// description.json in place there is nothing to vouch for (see +// TestGenerateSurvivesADescriptionFailure). +func TestARegenerateKeepsTheKeptDescriptionVouchedFor(t *testing.T) { + t.Parallel() + hr := newHarness(t) + adir := filepath.Join(hr.root, "Author", "Book", AttachDir) + hr.srv.add(45, "Author/Book", `{}`) + stop := hr.runAll(hr.describeHandler(&fakeDescriber{db: hr.db})) + hr.waitFor("the first run", func() bool { s, _ := hr.srv.final(45); return s == link.StateDone }) + stop() + + failing := &fakeDescriber{db: hr.db, fail: func(int) error { + return scheduler.ParkWithCode(state.ParkAgentValidationExhausted, "kept failing") + }} + hr.srv.add(46, "Author/Book", `{}`) + stop = hr.runAll(hr.describeHandler(failing)) + defer stop() + hr.waitFor("the regenerate", func() bool { s, _ := hr.srv.final(46); return s == link.StateDone }) + var res Result + hr.result(46, &res) + if res.Description || len(failing.calls()) != 1 { + t.Fatalf("result %+v, calls %v", res, failing.calls()) + } + if raw, err := os.ReadFile(filepath.Join(adir, "description.json")); err != nil || string(raw) != testDescription { + t.Fatalf("description.json = %s, %v", raw, err) + } + var prov Provenance + raw, _ := os.ReadFile(filepath.Join(adir, "bridge.json")) + if err := json.Unmarshal(raw, &prov); err != nil || prov.JobID != 46 || prov.Description == nil || + prov.Description.JobID != 45 || prov.Description.CheckedAgainst != pipeline.CheckedTranscripts || + prov.Description.GeneratedAt != "2026-10-16T12:00:00Z" { + t.Fatalf("bridge.json = %s, %v", raw, err) + } +} + +// keptDescription reads the description object only beside a description.json +// that is a regular file, and only from a bridge.json that holds one. +func TestKeptDescription(t *testing.T) { + t.Parallel() + dir := t.TempDir() + adir := filepath.Join(dir, AttachDir) + mustWrite(t, filepath.Join(adir, "bridge.json"), + `{"job_id":9,"description":{"job_id":8,"generated_at":"2026-10-16T12:00:00Z","from":"opening_notes","chapters":2,"checked_against":"book text"}}`) + if d := keptDescription(dir, ""); d != nil { + t.Fatalf("no description.json, kept %+v", d) + } + if err := os.Symlink(filepath.Join(t.TempDir(), "elsewhere.json"), filepath.Join(adir, "description.json")); err != nil { + t.Fatal(err) + } + if d := keptDescription(dir, ""); d != nil { + t.Fatalf("a linked description.json, kept %+v", d) + } + _ = os.Remove(filepath.Join(adir, "description.json")) + mustWrite(t, filepath.Join(adir, "description.json"), testDescription) + if d := keptDescription(dir, ""); d == nil || d.JobID != 8 || d.CheckedAgainst != "book text" { + t.Fatalf("kept %+v", d) + } + for _, prov := range []string{`{"job_id":9}`, `{"job_id":9,"description":null}`, `{"description":"x"}`, `not json`} { + mustWrite(t, filepath.Join(adir, "bridge.json"), prov) + if d := keptDescription(dir, ""); d != nil { + t.Fatalf("bridge.json %s, kept %+v", prov, d) + } + } +} + +// The engine's try at the start of contributing spent and failed: the job's pass +// at done doesn't pay for a second (for an ebook it would be checked against +// nothing), and says why there is none. +func TestGenerateDoesNotPayTwiceForADescription(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &fakeDescriber{db: hr.db} + gate := hr.exec.gate("contributing") + hr.srv.add(47, "Author/Book", `{}`) + stop := hr.runAll(hr.describeHandler(d)) + defer stop() + hr.waitFor("contributing", func() bool { + return hr.srv.saw(47, func(_ link.Update, p map[string]any) bool { return p["stage"] == "contributing" }) + }) + // The engine's own try, as the real step records one that failed its checks. + var park *scheduler.ParkError + if _, err := (&spendingDescriber{db: hr.db}).Describe(context.Background(), hr.engineBook(47).ID, true); !errors.As(err, &park) { + t.Fatalf("the engine's try = %v", err) + } + close(gate) + hr.waitFor("done", func() bool { s, _ := hr.srv.final(47); return s == link.StateDone }) + var res Result + hr.result(47, &res) + if len(d.calls()) != 0 || res.Description || !strings.Contains(res.DescriptionError, "an earlier try") || res.CostUSD != 3.65 { + t.Fatalf("calls %v, result %+v", d.calls(), res) + } +} diff --git a/internal/generate/describejob.go b/internal/generate/describejob.go new file mode 100644 index 0000000..d2aa5fb --- /dev/null +++ b/internal/generate/describejob.go @@ -0,0 +1,480 @@ +package generate + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "path/filepath" + "slices" + + "github.com/kodestar/audiosilo-bridge/internal/agent" + "github.com/kodestar/audiosilo-bridge/internal/jobs" + "github.com/kodestar/audiosilo-bridge/internal/metaops" + "github.com/kodestar/audiosilo-bridge/internal/pipeline" + "github.com/kodestar/audiosilo-bridge/internal/scheduler" + "github.com/kodestar/audiosilo-bridge/internal/state" + "github.com/kodestar/audiosilo-bridge/internal/store" +) + +// DescribeKind is the description-only job (BRIDGE-PLAN.md Phase 1c): an +// own-words description for a book the Bridge already generated, from the notes +// its run kept (no new speech recognition), written as the book's +// .audiosilo/description.json. +const DescribeKind = "sidecars.describe" + +// describeFailure is a job failure worded for the console as it is. +type describeFailure string + +func (f describeFailure) Error() string { return string(f) } + +// The ways a description-only job fails that need an admin's words. +const ( + errDescribeNoNotes describeFailure = "The Bridge has no notes for this book (it was not generated here, or they were removed). " + + "Generate characters and recaps again to get a description." + errDescribeGenerating describeFailure = "A generate job is still running for this book; it writes the description when it finishes." + errDescribeChecks describeFailure = "Couldn't write a description that passed the checks." + errBookBusy describeFailure = "The engine is busy with this book; try again in a minute." + errDescribeOneChapter describeFailure = "This book has a single chapter, so no description could leave out its ending." +) + +// holdTries bounds the wait for the engine's hold on a book (a stage just +// finishing or re-admitted, a purge): that many polls, then the description (or a +// job end's transcript purge) gives up. +const holdTries = 20 + +// DescribeJob runs sidecars.describe. It shares the generate handler's parts: +// the checked mapping, the book's folder, its parks, its Retry. +type DescribeJob struct{ h *Handler } + +// DescribeJob is the sidecars.describe handler beside this one. +func (h *Handler) DescribeJob() *DescribeJob { return &DescribeJob{h: h} } + +// Kind is sidecars.describe. +func (d *DescribeJob) Kind() string { return DescribeKind } + +// Lane is the pipeline lane: a park (a rate limit, the budget) can hold the job +// for a long time. +func (d *DescribeJob) Lane() string { return jobs.LanePipeline } + +// DescribeJobResult is what a done description-only job reports. +type DescribeJobResult struct { + LibraryID int64 `json:"library_id"` + Path string `json:"path"` + Description bool `json:"description"` + Files []string `json:"files"` + CostUSD float64 `json:"cost_usd"` + CheckedAgainst string `json:"checked_against"` +} + +// ParseDescribeParams checks a description-only job's params: nothing is asked +// for, and a field a later server adds is ignored, so anything but a JSON object +// (or none) is refused. +func ParseDescribeParams(raw json.RawMessage) error { + if len(raw) == 0 || string(raw) == "null" { + return nil + } + var m map[string]json.RawMessage + if err := json.Unmarshal(raw, &m); err != nil { + return fmt.Errorf("bad params: %w", err) + } + return nil +} + +// Run checks the book as a generate job does, finds the engine book whose notes +// it reads, writes the description (parking for the agent's parks) and saves it +// with the book. +func (d *DescribeJob) Run(ctx context.Context, job jobs.Job, rep *jobs.Reporter) (any, error) { + h := d.h + if err := ParseDescribeParams(job.Params); err != nil { + return nil, err + } + r, stop, err := h.startRun(job, rep, DescribeKind) + if err != nil { + return nil, err + } + defer stop() + if _, err := h.waitEngine(ctx, r); err != nil { + return nil, err + } + desc := h.describer() + if desc == nil { + return nil, errors.New("the Bridge's engine can't write descriptions") + } + // Strict when new: nothing is spent yet, so a library that isn't mapped and + // writable fails the job at once. A resumed one (its description may be paid + // for already) parks library_not_writable and waits, as a resumed generate job + // does. + if err := h.prepare(ctx, r, !job.Resumed); err != nil { + return nil, err + } + b, err := h.notesBook(ctx, r) + if err != nil { + return nil, err + } + // The job has its engine book now: from here a park of its own is lifted by + // the step that clears it (the save that goes through), never between two tries + // (unparkOwn), which the server would hear as a new park each time. + r.bookID = b.ID + got, spent, err := h.describeParked(ctx, r, desc, b.ID) + if err != nil { + return nil, err + } + files, err := h.saveDescription(ctx, r, got, spent) + if err != nil { + return nil, err + } + out := DescribeJobResult{LibraryID: r.lib.ID, Path: r.rel, Description: true, Files: files, CostUSD: Cents(spent), + CheckedAgainst: got.res.CheckedAgainst} + r.rep.Progress(jobs.Progress{Step: string(state.Done), Message: "Finished", Done: 1, Total: 1, + Extra: map[string]any{"stage": string(state.Done), "source": r.source, "cost_usd": out.CostUSD}}) + return out, nil +} + +// notesBook is the engine book whose notes describe the book: the latest run at +// its source path, else the latest generate job's book at its library path (a book +// that moved while that job ran), else the run its own bridge.json names (a book +// that moved after). None, or one still running, fails the job. +func (h *Handler) notesBook(ctx context.Context, r *run) (store.Book, error) { + b, err := h.bookAtSource(ctx, r) + if errors.Is(err, store.ErrNotFound) { + b, err = h.latestGenerated(ctx, r) + } + if errors.Is(err, store.ErrNotFound) { + b, err = h.namedRun(ctx, r) + } + if errors.Is(err, store.ErrNotFound) { + return store.Book{}, errDescribeNoNotes + } + if err != nil { + return store.Book{}, err + } + open, err := h.o.DB.OpenGenerateJobsForBook(ctx, b.ID, 0) + if err != nil { + return store.Book{}, err + } + // A book the engine still has queued or running (no park, not done) is + // working too, whether or not a job follows it. + if len(open) > 0 || (b.Status == "" && state.State(b.State) != state.Done) { + return store.Book{}, errDescribeGenerating + } + r.sourceOf(b) + return b, nil +} + +// bookAtSource is the engine's book at the book's source path (store.ErrNotFound +// for none, or a path outside the library's folder). +func (h *Handler) bookAtSource(ctx context.Context, r *run) (store.Book, error) { + sp, ok, err := metaops.AllowedPath(r.place.full, []string{r.mount}) + if err != nil || !ok { + return store.Book{}, store.ErrNotFound + } + return h.o.DB.GetBookBySourcePath(ctx, sp) +} + +// latestGenerated is the latest generate job's surviving engine book at the book's +// library path (store.ErrNotFound for none). A library id and path are the +// server's, and repeat across servers: a run made for another server's library +// (the Bridge paired anew) lies outside this library's folder, so it is none. +func (h *Handler) latestGenerated(ctx context.Context, r *run) (store.Book, error) { + id, err := h.o.DB.LatestGenerateBookAt(ctx, r.lib.ID, r.rel) + if err != nil { + return store.Book{}, err + } + b, err := h.o.DB.GetBook(ctx, id) + if err != nil { + return store.Book{}, err + } + if _, ok, err := metaops.AllowedPath(b.SourcePath, []string{r.mount}); err != nil || !ok { + return store.Book{}, store.ErrNotFound + } + return b, nil +} + +// namedRun is the engine book of the generate job the book's own bridge.json +// names (store.ErrNotFound for none): a book that moved after its job ended took +// that file along, while the Bridge's records (the book's source path, the job's +// mapping) stayed at the old place. Only this library's run, at a source inside its +// folder, counts; the job's mapping follows the book from then on. +func (h *Handler) namedRun(ctx context.Context, r *run) (store.Book, error) { + prov := readProvenance(filepath.Join(r.place.dir, AttachDir), r.place.prefix+provenanceName) + var kind string + var jobID int64 + if prov == nil || json.Unmarshal(prov["kind"], &kind) != nil || json.Unmarshal(prov["job_id"], &jobID) != nil || + kind != Kind || jobID == 0 { + return store.Book{}, store.ErrNotFound + } + m, err := h.o.DB.GetGenerateBook(ctx, jobID) + if err != nil { + return store.Book{}, err + } + if m.BookID == nil || m.LibraryID != r.lib.ID { + return store.Book{}, store.ErrNotFound + } + b, err := h.o.DB.GetBook(ctx, *m.BookID) + if err != nil { + return store.Book{}, err + } + if _, ok, err := metaops.AllowedPath(b.SourcePath, []string{r.mount}); err != nil || !ok { + return store.Book{}, store.ErrNotFound + } + if err := h.o.DB.SetGenerateBookPath(ctx, jobID, r.rel); err != nil { + h.o.Log.Warn("generate: recording the book's new path failed", "job", jobID, "err", err) + } + return b, nil +} + +// described is a description the step wrote (or reused), read back from the +// engine's work dir while the book was held. +type described struct { + res pipeline.DescribeResult + raw []byte + work string +} + +// describeHeld runs the description step under the engine's hold on the book +// (waiting a few polls for it; errBookBusy when it never comes) and reads the +// description back before letting go, so no stage, delete, purge or sweep acts on +// the book meanwhile. The result carries the step's spend whatever happened. +func (h *Handler) describeHeld(ctx context.Context, d Describer, bookID int64, noWait bool) (described, error) { + release, err := h.holdBook(ctx, bookID) + if err != nil { + return described{}, err + } + defer release() + res, err := d.Describe(ctx, bookID, noWait) + if err != nil { + return described{res: res}, err + } + raw, desc, err := readDescription(res.Path) + if err == nil && raw == nil { + err = errors.New("the engine kept no description") + } + return described{res: res, raw: raw, work: desc.Work}, err +} + +// holdBook takes the engine's hold on an engine book, trying again every poll +// for holdTries polls before giving up (errBookBusy). +func (h *Handler) holdBook(ctx context.Context, bookID int64) (func(), error) { + for i := 1; ; i++ { + if release, ok := h.hold(bookID); ok { + return release, nil + } + if i == holdTries { + return nil, errBookBusy + } + if err := agent.SleepCtx(ctx, h.o.Poll); err != nil { + return nil, err + } + } +} + +// describeParked runs the description step until it writes one: an agent park +// (a rate limit, the budget, a refused credential, no agent) parks the job and +// tries again at the park's retry_at or on Retry; the job stays parked while it +// tries, so a park that holds isn't announced again. It returns the spend across +// the tries. +func (h *Handler) describeParked(ctx context.Context, r *run, d Describer, bookID int64) (described, float64, error) { + var spent float64 + parked := false + for { + if !parked { + r.rep.Progress(jobs.Progress{Step: pipeline.DescribeStage, Message: "Writing a description", Total: 1, + Extra: r.describeExtra(spent)}) + } + got, err := h.describeHeld(ctx, d, bookID, false) + spent += got.res.CostUSD + var park *scheduler.ParkError + switch { + case err == nil: + // Lifts this loop's park, and one the runner restored for a job that + // resumed after a restart (nothing when there is none). + r.rep.Unpark() + return got, spent, nil + case ctx.Err() != nil: + return described{}, spent, ctx.Err() + case errors.Is(err, errBookBusy): + return described{}, spent, err + case errors.Is(err, pipeline.ErrNoNotes): + return described{}, spent, errDescribeNoNotes + case errors.Is(err, pipeline.ErrOneChapter): + return described{}, spent, errDescribeOneChapter + case errors.As(err, &park) && park.Code == state.ParkAgentValidationExhausted: + h.o.Log.Warn("generate: the description kept failing its checks", "job", r.job.ID, "err", err) + return described{}, spent, errDescribeChecks + case errors.As(err, &park): + parked = true + if !park.RetryAfter.IsZero() { + // A Retry pressed while that try ran was answered by it: only one + // pressed once this park is reported tries again before its + // retry_at. (A park without one waits for a press, so a press made + // meanwhile, after raising the budget say, is kept.) + select { + case <-r.poke: + default: + } + } + h.parkDescribe(r, park, spent) + if err := h.recheckWait(ctx, r, park.RetryAfter); err != nil { + return described{}, spent, err + } + default: + return described{}, spent, fmt.Errorf("could not write a description: %w", err) + } + } +} + +// parkDescribe reports an agent park as the job's, with its hint and retry_at. +func (h *Handler) parkDescribe(r *run, park *scheduler.ParkError, spent float64) { + code, retryAt := string(park.Code), "" + if !park.RetryAfter.IsZero() { + retryAt = store.Timestamp(park.RetryAfter) + } + extra := r.describeExtra(spent) + extra["park"] = parkExtra(code, retryAt) + r.rep.Progress(jobs.Progress{Step: pipeline.DescribeStage, Message: "Needs attention: Writing a description", Total: 1, + Extra: extra}) + r.rep.Park(code, park.Reason) +} + +// describeExtra is a description-only job's progress Extra: the describing +// stage, the source and the spend so far. +func (r *run) describeExtra(spent float64) map[string]any { + return map[string]any{"stage": pipeline.DescribeStage, "source": r.source, "cost_usd": Cents(spent)} +} + +// saveDescription writes the description and bridge.json's description object +// into the book's .audiosilo/ at the path it is at now, through the mapping +// checked again, parking book_not_writable until it can; then asks the server to +// re-read the attachments. It returns the names written. +func (h *Handler) saveDescription(ctx context.Context, r *run, got described, spent float64) ([]string, error) { + err := h.save(ctx, r, saveSpec{ + what: "the description", + wait: true, + progress: jobs.Progress{Step: stepWriting, Message: "Saving the description with the book", Done: 1, Total: 1, + Extra: r.describeExtra(spent)}, + files: func(p place) ([]attachFile, error) { + fresh := provenance(h.o.Version, DescribeKind, r.job.ID, h.now(), r.source, got.work, inputsOf(r.mount, r.files, r.epub)) + fresh.Description = descriptionProvenance(r.job.ID, got.res) + prov, err := describedProvenance(p.dir, p.prefix, fresh) + if err != nil { + return nil, err + } + return []attachFile{{descriptionName, got.raw}, {provenanceName, prov}}, nil + }, + }) + if err != nil { + return nil, err + } + h.o.Log.Info("generate: wrote the book's description", "job", r.job.ID, "path", r.rel) + r.rep.Progress(jobs.Progress{Step: stepRescanning, Message: "Asking the server to read it", Done: 1, Total: 1, + Extra: r.describeExtra(spent)}) + h.rescan(ctx, r) + return attachmentNames(r.place.prefix, sidecars{description: got.raw}), nil +} + +// describer is the engine's description step (nil until it started). +func (h *Handler) describer() Describer { + if h.o.Describer == nil { + return nil + } + return h.o.Describer() +} + +// describeOnce runs the description step once per run of a generate job, when the +// engine's sidecars are final (done, or parked at contributing). It never fails, +// parks or holds the job: a rate limit is not waited out, an engine that keeps the +// book (a few polls) is given up on, and any error leaves the job to finish with +// its characters and recaps, its result saying why there is no description. A run +// that asked at a contributing park asks once more at done: the contributing +// stage may have resolved the work and sources since, and the step re-composes the +// kept text with them (nothing spent), so the description never ships beside +// characters and recaps that name another work; a try at the park that failed +// without spending (a rate limit, the budget, a busy engine) gets its second go. +// It reports whether it asked (the description may have changed). +func (h *Handler) describeOnce(ctx context.Context, r *run, b store.Book) bool { + done := state.State(b.State) == state.Done + if r.descTried && (!done || r.descDone || (r.desc == nil && r.descPaid)) { + return false + } + r.descTried, r.descDone = true, done + if r.desc == nil && h.describeFailedPaid(ctx, b.ID) { + // The book's last try (the engine's at the start of contributing, or an + // earlier run of this job's) spent and failed: the run doesn't pay twice, + // and an ebook's second try at done would be checked against nothing. + r.descErr, r.descPaid = "an earlier try at the description failed (see the Bridge's log)", true + return false + } + d := h.describer() + if d == nil { + r.descErr = "the Bridge's engine can't write descriptions" + return false + } + r.rep.Progress(jobs.Progress{Step: pipeline.DescribeStage, Message: "Writing a description", + Extra: r.bookExtra(b)}) + got, err := h.describeHeld(ctx, d, b.ID, true) + if err != nil && r.desc != nil { + // The re-compose at done failed: the park's description stands (as the + // park had its work). + h.o.Log.Warn("generate: the description wasn't composed again at done; the park's stands", "job", r.job.ID, "err", err) + return false + } + if err != nil { + r.descErr, r.descPaid = descriptionErrorText(err), got.res.CostUSD > 0 + h.o.Log.Warn("generate: no description; the job finishes without one", "job", r.job.ID, "err", err) + return true + } + r.desc, r.descRaw, r.descErr = &got.res, got.raw, "" + return true +} + +// describeFailedPaid reports whether the book's latest describing run failed +// after it spent. +func (h *Handler) describeFailedPaid(ctx context.Context, bookID int64) bool { + runs, err := h.o.DB.ListStageRuns(ctx, bookID) + if err != nil { + return false + } + for _, run := range slices.Backward(runs) { + if run.Stage != pipeline.DescribeStage { + continue + } + if run.Ok == nil || *run.Ok { + return false + } + cost, err := h.o.DB.StageRunBudgetCost(ctx, run.ID) + return err == nil && cost > 0 + } + return false +} + +// descriptionErrorText is a generate result's short plain reason for having no +// description. +func descriptionErrorText(err error) string { + var park *scheduler.ParkError + switch { + case errors.Is(err, pipeline.ErrNoNotes): + return "the Bridge has no notes for this book" + case errors.Is(err, pipeline.ErrOneChapter): + return "the book has a single chapter, so no description leaves out its ending" + case errors.Is(err, errBookBusy): + return "the engine was busy with this book" + case errors.As(err, &park): + switch park.Code { + case state.ParkAgentRateLimited: + return "the agent account hit its usage limit" + case state.ParkBudgetExceeded: + return "the agent budget is spent" + case state.ParkAgentCredentialRefused: + return "the agent's credential was refused" + case state.ParkAgentUnavailable: + return "the agent isn't available" + case state.ParkAgentValidationExhausted: + return "the description kept failing its checks" + } + return "the agent stopped (" + string(park.Code) + ")" + case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded): + return "the job stopped while it was being written" + } + return "the description step failed (see the Bridge's log)" +} diff --git a/internal/generate/describejob_test.go b/internal/generate/describejob_test.go new file mode 100644 index 0000000..73b8d42 --- /dev/null +++ b/internal/generate/describejob_test.go @@ -0,0 +1,536 @@ +package generate + +import ( + "context" + "encoding/json" + "errors" + "io" + "log/slog" + "os" + "path/filepath" + "slices" + "strings" + "sync" + "testing" + "time" + + "github.com/kodestar/audiosilo-bridge/internal/jobs" + "github.com/kodestar/audiosilo-bridge/internal/link" + "github.com/kodestar/audiosilo-bridge/internal/pipeline" + "github.com/kodestar/audiosilo-bridge/internal/scheduler" + "github.com/kodestar/audiosilo-bridge/internal/state" + "github.com/kodestar/audiosilo-bridge/internal/store" +) + +// testDescription is a composed description as the pipeline keeps it. +const testDescription = `{"license":"CC-BY-SA-4.0","sources":[{"type":"community"}],"text":"A paragraph.","work":"the-book"}` + +// fakeDescriber stands in for the pipeline's description step: it writes the +// composed file into the engine book's work dir, or returns fail's error for a +// call (1-based), having spent 0.01 unless freeFail. With fromWork the file names +// the work characters.json names now, as the real step's re-compose does. +type fakeDescriber struct { + db *store.DB + mu sync.Mutex + runs []bool // each call's noWait + fail func(call int) error + freeFail bool + fromWork bool +} + +func (f *fakeDescriber) Describe(ctx context.Context, bookID int64, noWait bool) (pipeline.DescribeResult, error) { + f.mu.Lock() + f.runs = append(f.runs, noWait) + call := len(f.runs) + fail, freeFail, fromWork := f.fail, f.freeFail, f.fromWork + f.mu.Unlock() + if fail != nil { + if err := fail(call); err != nil { + if freeFail { + return pipeline.DescribeResult{}, err + } + return pipeline.DescribeResult{CostUSD: 0.01}, err + } + } + b, err := f.db.GetBook(ctx, bookID) + if err != nil { + return pipeline.DescribeResult{}, err + } + path := pipeline.FinalDescriptionPath(b.WorkDir) + desc := testDescription + if fromWork { + chars, _ := pipeline.FinalSidecarsPaths(b.WorkDir) + var c struct { + Work string `json:"work"` + } + raw, _ := os.ReadFile(chars) //nolint:gosec // the test's work dir + _ = json.Unmarshal(raw, &c) + desc = strings.Replace(testDescription, `"work":"the-book"`, `"work":"`+c.Work+`"`, 1) + } + if err := os.WriteFile(path, []byte(desc), 0o600); err != nil { + return pipeline.DescribeResult{}, err + } + return pipeline.DescribeResult{Path: path, Chapters: 2, CheckedAgainst: pipeline.CheckedTranscripts, + CostUSD: 0.25, GeneratedAt: time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)}, nil +} + +// calls is each call's noWait. +func (f *fakeDescriber) calls() []bool { + f.mu.Lock() + defer f.mu.Unlock() + return slices.Clone(f.runs) +} + +// describeHandler is a handler whose engine writes descriptions with d. +func (hr *harness) describeHandler(d Describer) *Handler { + return New(Options{DB: hr.db, Engine: func() Engine { return hr.sched }, Describer: func() Describer { return d }, + Libraries: hr.libs, Mount: func(int64, string) (string, bool) { return hr.libs.root, false }, + Server: func() Server { return hr.cl }, WorkRoot: hr.work, Version: "0.2.0", RetentionDays: 30, + Poll: 10 * time.Millisecond, IdlePoll: 10 * time.Millisecond, Recheck: time.Hour, Log: slog.New(slog.NewTextHandler(io.Discard, nil))}) +} + +// runAll runs the generate and describe jobs of h until stopped. +func (hr *harness) runAll(h *Handler) (stop func()) { + r := jobs.New(hr.db, func() jobs.Server { return hr.cl }, nil, jobs.Options{Keepalive: 30 * time.Millisecond, + ProgressEvery: 5 * time.Millisecond, Wait: 10 * time.Millisecond, RetryEvery: 20 * time.Millisecond, + ServerID: func() string { return "srv" }, Log: slog.New(slog.NewTextHandler(io.Discard, nil))}, h, h.DescribeJob()) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { defer close(done); r.Run(ctx) }() + return func() { cancel(); <-done } +} + +// addDescribe queues a sidecars.describe job. +func (f *fakeServer) addDescribe(id int64, path string) { + f.mu.Lock() + defer f.mu.Unlock() + lib := int64(1) + f.queue = append(f.queue, link.Job{ID: id, Kind: DescribeKind, LibraryID: &lib, Path: &path, Params: json.RawMessage(`{}`), Attempts: 1, State: "claimed"}) +} + +// result decodes a done job's result. +func (hr *harness) result(id int64, v any) { + hr.t.Helper() + _, final := hr.srv.final(id) + if err := json.Unmarshal(final.Result, v); err != nil { + hr.t.Fatalf("result of %d = %s: %v", id, final.Result, err) + } +} + +// A generate job writes the description beside the characters and recaps in the +// same finish, asking for it once, without waiting out a rate limit. +func TestGenerateWritesTheDescription(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &fakeDescriber{db: hr.db} + h := hr.describeHandler(d) + hr.exec.parks["contributing"] = 1 // a contributing park writes first, then done + hr.srv.add(31, "Author/Book", `{}`) + stop := hr.runAll(h) + defer stop() + hr.waitFor("the contributing park", func() bool { + return hr.srv.saw(31, func(u link.Update, p map[string]any) bool { + park, _ := p["park"].(map[string]any) + return park["code"] == "core_needed" + }) + }) + if err := h.Retry(context.Background(), 31); err != nil { + t.Fatal(err) + } + hr.waitFor("done", func() bool { s, _ := hr.srv.final(31); return s == link.StateDone }) + var res Result + hr.result(31, &res) + if !res.Description || res.DescriptionError != "" || + strings.Join(res.Files, ",") != "characters.json,recaps.json,description.json,bridge.json" { + t.Fatalf("result = %+v", res) + } + // Asked at the park, and again at done (to re-compose it with the work the + // contributing stage resolved), never waiting out a rate limit. + if calls := d.calls(); len(calls) != 2 || !calls[0] || !calls[1] { + t.Fatalf("describe calls = %+v, want two with noWait", calls) + } + adir := filepath.Join(hr.root, "Author", "Book", AttachDir) + if raw, err := os.ReadFile(filepath.Join(adir, "description.json")); err != nil || string(raw) != testDescription { + t.Fatalf("description.json = %s, %v", raw, err) + } + if info, _ := os.Stat(filepath.Join(adir, "description.json")); info.Mode().Perm() != 0o644 { + t.Fatalf("description.json is %v", info.Mode().Perm()) + } + var prov Provenance + raw, _ := os.ReadFile(filepath.Join(adir, "bridge.json")) + if err := json.Unmarshal(raw, &prov); err != nil || prov.Description == nil || prov.Description.JobID != 31 || + prov.Description.From != "opening_notes" || prov.Description.Chapters != 2 || + prov.Description.CheckedAgainst != "transcripts" || prov.Description.GeneratedAt != "2026-10-16T12:00:00Z" { + t.Fatalf("bridge.json = %s, %v", raw, err) + } +} + +// A description that can't be written never fails, parks or holds the job: it +// finishes with the characters and recaps and says why. +func TestGenerateSurvivesADescriptionFailure(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &fakeDescriber{db: hr.db, fail: func(int) error { + return scheduler.ParkWithCodeAfter(state.ParkAgentRateLimited, "rate limited", time.Now().Add(time.Hour)) + }} + hr.srv.add(32, "Author/Book", `{}`) + stop := hr.runAll(hr.describeHandler(d)) + defer stop() + hr.waitFor("done", func() bool { s, _ := hr.srv.final(32); return s == link.StateDone }) + var res Result + hr.result(32, &res) + if res.Description || res.DescriptionError != "the agent account hit its usage limit" || res.Characters != 2 || + strings.Join(res.Files, ",") != "characters.json,recaps.json,bridge.json" { + t.Fatalf("result = %+v", res) + } + if hr.srv.saw(32, func(u link.Update, _ map[string]any) bool { + return u.State != nil && *u.State == link.StateNeedsAttention + }) { + t.Fatal("the description parked the generate job") + } + if _, err := os.Stat(filepath.Join(hr.root, "Author", "Book", AttachDir, "description.json")); !os.IsNotExist(err) { + t.Fatal("a description was written") + } + var prov map[string]any + raw, _ := os.ReadFile(filepath.Join(hr.root, "Author", "Book", AttachDir, "bridge.json")) + if err := json.Unmarshal(raw, &prov); err != nil || prov["description"] != nil { + t.Fatalf("bridge.json = %s", raw) + } +} + +// generated runs a generate job for Author/Book to done with no description. +func (hr *harness) generated(id int64) { + hr.t.Helper() + hr.srv.add(id, "Author/Book", `{}`) + stop := hr.runner(hr.h) + defer stop() + hr.waitFor("the generate job", func() bool { s, _ := hr.srv.final(id); return s == link.StateDone }) +} + +// The description-only job: the notes of the book's earlier run, the file and +// bridge.json's description object saved with the book (the run's own bridge.json +// fields kept), the server asked to re-read them. +func TestDescribeJob(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(41) + d := &fakeDescriber{db: hr.db} + hr.srv.addDescribe(42, "Author/Book") + stop := hr.runAll(hr.describeHandler(d)) + defer stop() + hr.waitFor("done", func() bool { s, _ := hr.srv.final(42); return s == link.StateDone }) + var res DescribeJobResult + hr.result(42, &res) + if res.LibraryID != 1 || res.Path != "Author/Book" || !res.Description || res.CostUSD != 0.25 || res.CheckedAgainst != "transcripts" || + strings.Join(res.Files, ",") != "description.json,bridge.json" { + t.Fatalf("result = %+v", res) + } + if calls := d.calls(); len(calls) != 1 || calls[0] { + t.Fatalf("describe calls = %+v", calls) + } + if !hr.srv.saw(42, func(_ link.Update, p map[string]any) bool { + return p["stage"] == "describing" && p["step"] == "describing" + }) { + t.Fatal("the job never reported the describing stage") + } + adir := filepath.Join(hr.root, "Author", "Book", AttachDir) + if raw, err := os.ReadFile(filepath.Join(adir, "description.json")); err != nil || string(raw) != testDescription { + t.Fatalf("description.json = %s, %v", raw, err) + } + var prov Provenance + raw, _ := os.ReadFile(filepath.Join(adir, "bridge.json")) + if err := json.Unmarshal(raw, &prov); err != nil || prov.Kind != Kind || prov.JobID != 41 || len(prov.Inputs) != 2 || + prov.Description == nil || prov.Description.JobID != 42 || prov.Description.Chapters != 2 { + t.Fatalf("bridge.json = %s, %v", raw, err) + } + hr.srv.mu.Lock() + defer hr.srv.mu.Unlock() + if n := len(hr.srv.rescans); n != 2 || hr.srv.rescans[1]["mode"] != "attachments" { + t.Fatalf("rescans = %+v", hr.srv.rescans) + } +} + +// A description-only job for an ebook-sourced book that moved since its run: the +// .epub went with it, and a fresh bridge.json names it where it is now (before, +// the description's save didn't follow it and named the old, missing path). +func TestDescribeJobFollowsAMovedEpub(t *testing.T) { + old := readEpub + readEpub = func(string) error { return nil } + t.Cleanup(func() { readEpub = old }) + hr := newHarness(t) + mustWrite(t, filepath.Join(hr.root, "Author", "Book", "The Book.epub"), "epub!") + hr.srv.add(91, "Author/Book", `{"source":"auto"}`) + stop := hr.runner(hr.h) + hr.waitFor("generated", func() bool { s, _ := hr.srv.final(91); return s == link.StateDone }) + stop() + if err := os.Rename(filepath.Join(hr.root, "Author", "Book"), filepath.Join(hr.root, "Author", "Book (2026)")); err != nil { + t.Fatal(err) + } + // No bridge.json to keep: the description writes a fresh one. + if err := os.Remove(filepath.Join(hr.root, "Author", "Book (2026)", AttachDir, "bridge.json")); err != nil { + t.Fatal(err) + } + hr.srv.set(func() { + hr.srv.books["Author/Book (2026)"] = link.Book{LibraryID: 1, Path: "Author/Book (2026)", Files: []link.BookFile{ + {Path: "Author/Book (2026)/01.mp3", Size: 9}, {Path: "Author/Book (2026)/02.mp3", Size: 10}}} + delete(hr.srv.books, "Author/Book") + }) + // The server followed the book; so did the generate job's mapping. + if err := hr.db.SetGenerateBookPath(context.Background(), 91, "Author/Book (2026)"); err != nil { + t.Fatal(err) + } + hr.srv.addDescribe(92, "Author/Book (2026)") + stopAll := hr.runAll(hr.describeHandler(&fakeDescriber{db: hr.db})) + defer stopAll() + hr.waitFor("done", func() bool { s, _ := hr.srv.final(92); return s == link.StateDone }) + var prov Provenance + raw, _ := os.ReadFile(filepath.Join(hr.root, "Author", "Book (2026)", AttachDir, "bridge.json")) + if err := json.Unmarshal(raw, &prov); err != nil || prov.Source != SourceEbook || len(prov.Inputs) != 1 || + prov.Inputs[0].Path != "Author/Book (2026)/The Book.epub" || prov.Inputs[0].Size != 5 { + t.Fatalf("bridge.json = %s, %v", raw, err) + } +} + +// No run to read (never generated here), and a run still going: each fails +// with the console's sentence, before any spend. +func TestDescribeJobRefusals(t *testing.T) { + t.Parallel() + hr := newHarness(t) + d := &fakeDescriber{db: hr.db} + h := hr.describeHandler(d) + gate := hr.exec.gate("asr") + hr.srv.add(51, "Author/Book", `{}`) + stop := hr.runAll(h) + defer stop() + hr.waitFor("asr", func() bool { + return hr.srv.saw(51, func(_ link.Update, p map[string]any) bool { return p["stage"] == "asr" }) + }) + hr.srv.addDescribe(52, "Author/Book") + hr.srv.addDescribe(53, "Loose.m4b") + for id, want := range map[int64]describeFailure{52: errDescribeGenerating, 53: errDescribeNoNotes} { + hr.waitFor("the refusal", func() bool { s, _ := hr.srv.final(id); return s == link.StateFailed }) + if _, u := hr.srv.final(id); u.Error == nil || *u.Error != string(want) { + t.Fatalf("job %d failed with %v, want %q", id, u.Error, want) + } + } + close(gate) + if len(d.calls()) != 0 { + t.Fatal("a refused job described") + } +} + +// An agent park parks the job with its code and hint and tries again on Retry (no +// retry_at) or at its retry_at; validation exhausted fails it. +func TestDescribeJobParks(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(61) + var mu sync.Mutex + mode := "budget" + d := &fakeDescriber{db: hr.db, fail: func(call int) error { + mu.Lock() + defer mu.Unlock() + switch { + case mode == "budget" && call == 1: + return scheduler.ParkWithCode(state.ParkBudgetExceeded, "the budget is spent") + case mode == "rate" && call == 3: + return scheduler.ParkWithCodeAfter(state.ParkAgentRateLimited, "rate limited", time.Now().Add(50*time.Millisecond)) + case mode == "checks": + return scheduler.ParkWithCode(state.ParkAgentValidationExhausted, "kept failing") + } + return nil + }} + h := hr.describeHandler(d) + stop := hr.runAll(h) + defer stop() + + hr.srv.addDescribe(62, "Author/Book") + hr.waitFor("the budget park", func() bool { + return hr.srv.saw(62, func(u link.Update, p map[string]any) bool { + park, _ := p["park"].(map[string]any) + return u.State != nil && *u.State == link.StateNeedsAttention && u.Error != nil && *u.Error == "the budget is spent" && + park["code"] == "budget_exceeded" && park["hint"] != "" && park["retry_at"] == nil + }) + }) + if err := h.Retry(context.Background(), 62); err != nil { + t.Fatal(err) + } + hr.waitFor("done after the retry", func() bool { s, _ := hr.srv.final(62); return s == link.StateDone }) + + mu.Lock() + mode = "rate" + mu.Unlock() + if err := os.Remove(filepath.Join(hr.root, "Author", "Book", AttachDir, "description.json")); err != nil { + t.Fatal(err) + } + hr.srv.addDescribe(63, "Author/Book") + hr.waitFor("done at retry_at", func() bool { s, _ := hr.srv.final(63); return s == link.StateDone }) + if !hr.srv.saw(63, func(_ link.Update, p map[string]any) bool { + park, _ := p["park"].(map[string]any) + return park["code"] == "agent_rate_limited" && park["retry_at"] != nil + }) { + t.Fatal("the rate limit was never reported") + } + + mu.Lock() + mode = "checks" + mu.Unlock() + hr.srv.addDescribe(64, "Author/Book") + hr.waitFor("failed", func() bool { s, _ := hr.srv.final(64); return s == link.StateFailed }) + if _, u := hr.srv.final(64); u.Error == nil || *u.Error != string(errDescribeChecks) { + t.Fatalf("failed with %v", u.Error) + } +} + +// A new generate job doesn't delete the earlier run a description holds: the +// engine's Delete says busy, and the job asks to try again. +func TestGenerateWaitsForADescription(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(71) + b := hr.engineBook(71) + release, ok := hr.sched.Hold(b.ID) + if !ok { + t.Fatal("the finished book couldn't be held") + } + defer release() + hr.srv.add(72, "Author/Book", `{}`) + stop := hr.runner(hr.h) + defer stop() + hr.waitFor("failed", func() bool { s, _ := hr.srv.final(72); return s == link.StateFailed }) + if _, u := hr.srv.final(72); u.Error == nil || !strings.Contains(*u.Error, "try again in a minute") { + t.Fatalf("failed with %v", u.Error) + } + if _, err := hr.db.GetBook(context.Background(), b.ID); err != nil { + t.Fatalf("the earlier run was deleted: %v", err) + } +} + +// A description that can't get the engine's hold on the book: the description-only +// job fails asking to try again, and a generate job finishes without one, saying +// why. Neither describes. +func TestDescriptionNeedsTheHold(t *testing.T) { + t.Parallel() + hr := newHarness(t) + hr.generated(81) + b := hr.engineBook(81) + release, ok := hr.sched.Hold(b.ID) + if !ok { + t.Fatal("the finished book couldn't be held") + } + d := &fakeDescriber{db: hr.db} + stop := hr.runAll(hr.describeHandler(d)) + hr.srv.addDescribe(82, "Author/Book") + hr.waitFor("failed", func() bool { s, _ := hr.srv.final(82); return s == link.StateFailed }) + if _, u := hr.srv.final(82); u.Error == nil || *u.Error != string(errBookBusy) { + t.Fatalf("failed with %v", u.Error) + } + stop() + release() + + // The generate finish, its engine refusing every hold. + h := hr.describeHandler(d) + h.o.Engine = func() Engine { return refusingEngine{hr.sched} } + stopGen := hr.runAll(h) + defer stopGen() + hr.srv.add(83, "Loose.m4b", `{}`) + hr.waitFor("the job done", func() bool { s, _ := hr.srv.final(83); return s == link.StateDone }) + var res Result + hr.result(83, &res) + if res.Description || res.DescriptionError != "the engine was busy with this book" { + t.Fatalf("result = %+v", res) + } + if len(d.calls()) != 0 { + t.Fatalf("described %d times without the hold", len(d.calls())) + } +} + +// refusingEngine is the scheduler with every hold refused (a stage runs on the book). +type refusingEngine struct{ Engine } + +func (refusingEngine) Hold(int64) (func(), bool) { return func() {}, false } + +// bridge.json after a description-only job keeps every field it had (unknown +// ones included) and only adds or replaces description; with none fit to keep +// it is written fresh. +func TestDescribedProvenance(t *testing.T) { + t.Parallel() + dir := t.TempDir() + adir := filepath.Join(dir, AttachDir) + fresh := provenance("0.3.0", DescribeKind, 9, time.Date(2026, 10, 16, 0, 0, 0, 0, time.UTC), SourceAudio, "w", nil) + fresh.Description = &DescriptionProvenance{JobID: 9, From: descriptionFrom, Chapters: 3, CheckedAgainst: "none"} + + got := func() map[string]any { + t.Helper() + raw, err := describedProvenance(dir, "", fresh) + if err != nil { + t.Fatal(err) + } + var m map[string]any + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + return m + } + if m := got(); m["kind"] != DescribeKind || m["job_id"] != float64(9) { + t.Fatalf("no bridge.json: %v", m) + } + mustWrite(t, filepath.Join(adir, "bridge.json"), `{"kind":"sidecars.generate","job_id":3,"generated_at":"2026-10-01T00:00:00Z",`+ + `"later_field":true,"description":{"job_id":1}}`) + m := got() + desc, _ := m["description"].(map[string]any) + if m["kind"] != Kind || m["job_id"] != float64(3) || m["generated_at"] != "2026-10-01T00:00:00Z" || m["later_field"] != true || + desc["job_id"] != float64(9) || desc["chapters"] != float64(3) { + t.Fatalf("kept = %v", m) + } + for _, bad := range []string{`[1,2]`, `not json`, `null`} { + mustWrite(t, filepath.Join(adir, "bridge.json"), bad) + if m := got(); m["kind"] != DescribeKind { + t.Fatalf("%s: %v", bad, m) + } + } + // A link in its place is not read through. + _ = os.Remove(filepath.Join(adir, "bridge.json")) + other := filepath.Join(dir, "elsewhere.json") + mustWrite(t, other, `{"kind":"leaked"}`) + if err := os.Symlink(other, filepath.Join(adir, "bridge.json")); err != nil { + t.Fatal(err) + } + if m := got(); m["kind"] != DescribeKind { + t.Fatalf("read through a link: %v", m) + } +} + +func TestParseDescribeParams(t *testing.T) { + t.Parallel() + for _, ok := range []string{``, `null`, `{}`, `{"later":1}`} { + if err := ParseDescribeParams(json.RawMessage(ok)); err != nil { + t.Errorf("%q refused: %v", ok, err) + } + } + for _, bad := range []string{`[]`, `"x"`, `{`} { + if err := ParseDescribeParams(json.RawMessage(bad)); err == nil { + t.Errorf("%q accepted", bad) + } + } +} + +func TestDescriptionErrorText(t *testing.T) { + t.Parallel() + cases := map[error]string{ + pipeline.ErrNoNotes: "the Bridge has no notes for this book", + pipeline.ErrOneChapter: "the book has a single chapter, so no description leaves out its ending", + scheduler.ParkWithCode(state.ParkBudgetExceeded, "x"): "the agent budget is spent", + scheduler.ParkWithCode(state.ParkAgentCredentialRefused, "x"): "the agent's credential was refused", + scheduler.ParkWithCode(state.ParkAgentUnavailable, "x"): "the agent isn't available", + scheduler.ParkWithCode(state.ParkAgentValidationExhausted, "x"): "the description kept failing its checks", + scheduler.ParkWithCode(state.ParkSupervisorEscalated, "x"): "the agent stopped (supervisor_escalated)", + context.Canceled: "the job stopped while it was being written", + errors.New("disk on fire"): "the description step failed (see the Bridge's log)", + } + for err, want := range cases { + if got := descriptionErrorText(err); got != want { + t.Errorf("%v: %q, want %q", err, got, want) + } + } +} diff --git a/internal/generate/generate.go b/internal/generate/generate.go index c0d7884..db4c9b5 100644 --- a/internal/generate/generate.go +++ b/internal/generate/generate.go @@ -29,6 +29,7 @@ import ( "github.com/kodestar/audiosilo-bridge/internal/link" "github.com/kodestar/audiosilo-bridge/internal/mapping" "github.com/kodestar/audiosilo-bridge/internal/metaops" + "github.com/kodestar/audiosilo-bridge/internal/pipeline" "github.com/kodestar/audiosilo-bridge/internal/scheduler" "github.com/kodestar/audiosilo-bridge/internal/state" "github.com/kodestar/audiosilo-bridge/internal/store" @@ -45,6 +46,15 @@ type Engine interface { Cancel(ctx context.Context, id int64) error Delete(ctx context.Context, id int64) error ETASeconds(id int64) (int64, bool) + // Hold reserves a book for work outside its stages (the description step, a + // transcript purge); ok is false while a stage runs on it or another holds it. + Hold(id int64) (release func(), ok bool) +} + +// Describer writes a finished engine book's own-words description +// (*pipeline.Executor's Describe, Phase 1c). +type Describer interface { + Describe(ctx context.Context, bookID int64, noWait bool) (pipeline.DescribeResult, error) } // Server is the part of the server link the job calls. @@ -63,7 +73,10 @@ type Libraries interface { type Options struct { DB *store.DB // Engine is the scheduler, nil until the engine has started (the job waits). - Engine func() Engine + Engine func() Engine + // Describer is the engine's description step, nil until the engine has + // started (a nil func: no descriptions). + Describer func() Describer Libraries Libraries Mount mapping.MountFunc // Server is the server link, nil while unpaired. @@ -134,17 +147,21 @@ func (h *Handler) Kind() string { return Kind } // Lane is the pipeline lane. func (h *Handler) Lane() string { return jobs.LanePipeline } -// Result is what a done job reports (contract section 1). +// Result is what a done job reports (contract section 1). Description says +// whether description.json was written with the others (Phase 1c); when it +// wasn't, DescriptionError says why in a few plain words. type Result struct { - LibraryID int64 `json:"library_id"` - Path string `json:"path"` - Source string `json:"source"` - Files []string `json:"files"` - Characters int `json:"characters"` - Recaps int `json:"recaps"` - CostUSD float64 `json:"cost_usd"` - WorkID string `json:"work_id"` - Contributed map[string]string `json:"contributed,omitempty"` + LibraryID int64 `json:"library_id"` + Path string `json:"path"` + Source string `json:"source"` + Files []string `json:"files"` + Characters int `json:"characters"` + Recaps int `json:"recaps"` + CostUSD float64 `json:"cost_usd"` + WorkID string `json:"work_id"` + Contributed map[string]string `json:"contributed,omitempty"` + Description bool `json:"description"` + DescriptionError string `json:"description_error,omitempty"` } // run is one job's state as it goes. @@ -167,9 +184,9 @@ type run struct { progress []store.Progress costUSD float64 // side is the sidecars last read and sideSig their files' signature, so - // unchanged files aren't read again; written is the hash of the sidecars - // last written ("" for none yet); writeErr the last failed write's error, - // logged once. + // unchanged files aren't read again; written is the hash of the sidecars last + // written ("" for none yet); writeErr the last failed write's error, logged + // once. side sidecars sideSig string written string @@ -186,6 +203,16 @@ type run struct { poke chan struct{} ownPark bool resuming bool + // descTried says this run asked for the description, descDone that it asked + // with the book done (a run asks at most twice: at a contributing park, then at + // done); desc is the description it got and descRaw its file as read back, + // descErr why it got none and descPaid that the try which failed spent. + descTried bool + descDone bool + desc *pipeline.DescribeResult + descRaw []byte + descErr string + descPaid bool } // bookKey is what changes on an engine book row when the scheduler moves it. @@ -207,21 +234,12 @@ func (h *Handler) Run(ctx context.Context, job jobs.Job, rep *jobs.Reporter) (an if err != nil { return nil, err } - if job.LibraryID == nil || job.Path == nil || strings.TrimSpace(*job.Path) == "" { - return nil, errors.New("sidecars.generate needs a library and a book path") - } - r := &run{job: job, rep: rep, params: params, rel: *job.Path, jobPath: *job.Path, poke: make(chan struct{}, 1)} - if p := rep.Path(); p != "" { - // The server moved the job with its book since the claim (the first report's - // answer says where it is now). - r.rel, r.jobPath = p, p + r, stop, err := h.startRun(job, rep, Kind) + if err != nil { + return nil, err } - // One wake-up channel for the whole run: a Retry reaches whichever park the - // job waits in. It is this run's own, so a later attempt of the same job - // keeps its channel when this one ends. - h.setPoke(job.ID, r.poke) - defer h.dropPoke(job.ID, r.poke) - rep.Progress(jobs.Progress{Step: stepWaiting, Message: "Starting"}) + defer stop() + r.params = params eng, err := h.waitEngine(ctx, r) if err != nil { return nil, err @@ -252,7 +270,9 @@ func (h *Handler) Run(ctx context.Context, job jobs.Job, rep *jobs.Reporter) (an return h.end(ctx, r, nil, err) } if resumed && ownBook() && mapped.RelPath != r.rel { - h.recordMove(ctx, r, mapped.RelPath) + h.o.Log.Info("generate: the book moved while the job was away; the job follows it", "job", job.ID, + "from", mapped.RelPath, "to", r.rel) + h.recordMove(ctx, r) } if resumed && !ownBook() { // That run's job is gone (its id is this job's now), so nothing follows it. @@ -273,6 +293,25 @@ func (h *Handler) Run(ctx context.Context, job jobs.Job, rep *jobs.Reporter) (an return h.end(ctx, r, res, err) } +// startRun begins a run of either job kind: the run at the job's path (the server +// may have moved the job with its book since the claim; the first report's answer +// says where it is now), its wake-up channel registered and "Starting" reported. +// One wake-up channel serves the whole run, so a Retry reaches whichever park the +// job waits in; it is this run's own, so a later attempt of the same job keeps its +// channel when this one ends. stop drops it. +func (h *Handler) startRun(job jobs.Job, rep *jobs.Reporter, kind string) (r *run, stop func(), err error) { + if job.LibraryID == nil || job.Path == nil || strings.TrimSpace(*job.Path) == "" { + return nil, nil, fmt.Errorf("%s needs a library and a book path", kind) + } + r = &run{job: job, rep: rep, rel: *job.Path, jobPath: *job.Path, poke: make(chan struct{}, 1)} + if p := rep.Path(); p != "" { + r.rel, r.jobPath = p, p + } + h.setPoke(job.ID, r.poke) + rep.Progress(jobs.Progress{Step: stepWaiting, Message: "Starting"}) + return r, func() { h.dropPoke(job.ID, r.poke) }, nil +} + // errCantContribute is a contribute=issue job on a Bridge with no GitHub token. var errCantContribute = errors.New("this job asks to contribute the characters and recaps (contribute: issue), but " + "the Bridge has no GitHub token: set AUDIOSILO_BRIDGE_GITHUB_TOKEN on the Bridge and restart it, or generate " + @@ -359,6 +398,20 @@ func (h *Handler) stopBook(ctx context.Context, jobID, bookID int64) { } } +// hold takes the engine's hold on an engine book for work outside its stages (the +// description step, a transcript purge): ok is false while a stage runs on it or +// something else holds it. With no engine nothing runs on it, so it holds. +func (h *Handler) hold(bookID int64) (release func(), ok bool) { + var eng Engine + if h.o.Engine != nil { + eng = h.o.Engine() + } + if eng == nil { + return func() {}, true + } + return eng.Hold(bookID) +} + // waitEngine waits for the engine to have started. func (h *Handler) waitEngine(ctx context.Context, r *run) (Engine, error) { for { @@ -458,7 +511,7 @@ func (h *Handler) waitMapped(ctx context.Context, r *run, strict bool) error { if st.Error != "" { why += " (" + st.Error + ")" } - return fmt.Errorf("library %q %s, so the Bridge can't write the book's characters and recaps", r.lib.Name, why) + return fmt.Errorf("library %q %s, so the Bridge can't write into the book's folder", r.lib.Name, why) default: h.park(r, parkLibraryNotWritable, "library "+r.lib.Name+" isn't mapped and writable on the Bridge any more") } @@ -479,22 +532,28 @@ func (h *Handler) waitWritable(ctx context.Context, r *run) error { return nil } h.park(r, parkBookNotWritable, "the Bridge can't write in the book's folder ("+writeProblem(err)+")") - if err := h.recheckWait(ctx, r); err != nil { + if err := h.recheckWait(ctx, r, time.Now().Add(h.o.Recheck)); err != nil { return err } } } -// recheckWait waits for the next look at a park of the Bridge's own: Recheck, or -// a Retry (which pokes). -func (h *Handler) recheckWait(ctx context.Context, r *run) error { - t := time.NewTimer(h.o.Recheck) - defer t.Stop() +// recheckWait waits for the next look at a park: until at (the zero time: no +// timer, only a Retry), a Retry (which pokes), or the job's end. A park of the +// Bridge's own looks again every Recheck; a description's agent park at its +// retry_at. +func (h *Handler) recheckWait(ctx context.Context, r *run, at time.Time) error { + var timer <-chan time.Time + if !at.IsZero() { + t := time.NewTimer(max(time.Until(at), 0)) + defer t.Stop() + timer = t.C + } select { case <-ctx.Done(): return ctx.Err() case <-r.poke: - case <-t.C: + case <-timer: } return nil } @@ -517,7 +576,7 @@ func writeProblem(err error) string { // park parks the job with one of the Bridge's own codes. func (h *Handler) park(r *run, code, msg string) { r.rep.Progress(jobs.Progress{Step: stepWaiting, Message: "Needs attention: " + msg, - Extra: map[string]any{"source": r.source, "park": map[string]any{"code": code, "hint": ParkHint(code)}}}) + Extra: map[string]any{"source": r.source, "park": parkExtra(code, "")}}) r.rep.Park(code, msg) r.ownPark = true } @@ -576,8 +635,8 @@ func (h *Handler) create(ctx context.Context, eng Engine, r *run) error { if len(open) > 0 { return fmt.Errorf("job %d is already generating this book; cancel it first", open[0]) } - if err := eng.Delete(ctx, old.ID); errors.Is(err, scheduler.ErrBusy) { - return errors.New("the engine is still running a stage of this book's earlier run; try again in a minute") + if err := h.deleteEarlier(ctx, eng, r, old.ID); errors.Is(err, scheduler.ErrBusy) { + return errors.New("the engine is still busy with this book's earlier run; try again in a minute") } else if err != nil && !errors.Is(err, store.ErrNotFound) { return fmt.Errorf("could not clear the book's earlier run: %w", err) } @@ -605,6 +664,24 @@ func (h *Handler) create(ctx context.Context, eng Engine, r *run) error { return nil } +// deleteEarlier deletes the engine book of the book's earlier run, waiting a few +// polls (holdTries) while the engine is busy with it: a description reading it, +// which the server asked to give way to this job and which lets go at its next +// report, or a stage the earlier job's end just stopped. scheduler.ErrBusy when it +// stays busy. +func (h *Handler) deleteEarlier(ctx context.Context, eng Engine, r *run, id int64) error { + for i := 1; ; i++ { + err := eng.Delete(ctx, id) + if !errors.Is(err, scheduler.ErrBusy) || i == holdTries { + return err + } + r.rep.Progress(jobs.Progress{Step: stepWaiting, Message: "Waiting for the engine to let go of the book's earlier run"}) + if err := agent.SleepCtx(ctx, h.o.Poll); err != nil { + return err + } + } +} + // newBook is the engine book for the job, as sidecars' intake built it from a // scanned candidate: the identity from the server's book block, the work id from // the params or else the server's metadata cache. @@ -659,11 +736,7 @@ func (h *Handler) attach(ctx context.Context, eng Engine, r *run, m store.Genera return err } } - if b.Kind == string(state.KindEbook) { - r.source, r.epub = SourceEbook, b.EbookPath - } else { - r.source, r.epub = SourceAudio, "" - } + r.sourceOf(b) if scheduler.IsCancelled(b) { if err := eng.Retry(ctx, b.ID); err != nil && !errors.Is(err, scheduler.ErrInvalidOp) { return err @@ -674,6 +747,27 @@ func (h *Handler) attach(ctx context.Context, eng Engine, r *run, m store.Genera return nil } +// bookExtra is a progress Extra for a step on the engine book b: its stage and +// the run's source, and its park while it is parked (a write at a contributing +// park keeps the park in what the job reports). +func (r *run) bookExtra(b store.Book) map[string]any { + extra := map[string]any{"stage": b.State, "source": r.source} + if state.Status(b.Status) == state.StatusNeedsAttention { + extra["park"] = parkExtra(b.ParkCode, b.RetryAt) + } + return extra +} + +// sourceOf takes the run's source (and its .epub) from the engine book whose notes +// it follows: an ebook-kind book read its .epub, any other the audio. +func (r *run) sourceOf(b store.Book) { + if b.Kind == string(state.KindEbook) { + r.source, r.epub = SourceEbook, b.EbookPath + } else { + r.source, r.epub = SourceAudio, "" + } +} + // follow reports the engine book's progress until it is done (then writes and // rescans), fails, or the job's context ends. func (h *Handler) follow(ctx context.Context, eng Engine, r *run) (any, error) { @@ -729,7 +823,11 @@ func (h *Handler) step(ctx context.Context, eng Engine, r *run, b store.Book) (a return nil, true, fmt.Errorf("the engine failed the book at %s: %s", b.State, b.Error) } r.rep.Progress(describe(v)) - if status == state.StatusNeedsAttention && st == state.Contributing { + parked := status == state.StatusNeedsAttention + // The park is the job's before the write below, which can take a while (the + // description step may wait for an agent slot): the server hears it at once. + r.engineParked(parked, b.ParkCode, b.Error) + if parked && st == state.Contributing { _, err := h.write(ctx, r, b, false) if msg := errText(err); msg != r.writeErr { r.writeErr = msg @@ -738,8 +836,10 @@ func (h *Handler) step(ctx context.Context, eng Engine, r *run, b store.Book) (a } } r.rep.Progress(describe(v)) // the park again, after the write's own steps + // A park of the Bridge's own the write raised and lifted leaves the + // engine's to report again. + r.engineParked(true, b.ParkCode, b.Error) } - r.engineParked(status == state.StatusNeedsAttention, b.ParkCode, b.Error) return nil, false, nil } @@ -750,8 +850,19 @@ func (h *Handler) finish(ctx context.Context, r *run, b store.Book, cost float64 if err != nil { return nil, err } + if r.descTried { + // The description step may have spent after the cost was read (a try that + // failed its checks is paid for too), and the budgets count it. + if c, _, err := h.o.DB.SumStageRunBudgetCost(ctx, b.ID); err == nil { + cost = c + } + } res := Result{LibraryID: r.lib.ID, Path: r.rel, Source: r.source, Files: attachmentNames(r.place.prefix, s), - Characters: s.nChars, Recaps: s.nRecaps, CostUSD: Cents(cost), WorkID: cmp.Or(b.WorkID, s.workID)} + Characters: s.nChars, Recaps: s.nRecaps, CostUSD: Cents(cost), WorkID: cmp.Or(b.WorkID, s.workID), + Description: s.description != nil} + if !res.Description { + res.DescriptionError = cmp.Or(r.descErr, "no description was written") + } if r.params.Contribute == ContributeIssue { rows, err := h.o.DB.ListContributionsByBook(ctx, b.ID) if err == nil { @@ -779,75 +890,153 @@ func (h *Handler) finish(ctx context.Context, r *run, b store.Book, cost float64 // every Recheck (or on Retry) until it goes through: the sidecars stay in the // engine's work dir meanwhile, never lost to a failed job. Without (a book parked // at contributing, looked at every poll) the failure is returned. +// +// The characters and recaps go first, then the description step runs (once per +// run, see describeOnce) and a description it got follows in a second save: the +// step can wait a long time for an agent slot behind other books, and the +// optional description never holds back the files the job exists for. func (h *Handler) write(ctx context.Context, r *run, b store.Book, wait bool) (sidecars, error) { + s, err := h.writeSidecars(ctx, r, b, wait) + if err != nil || !h.describeOnce(ctx, r, b) { + return s, err + } + return h.writeSidecars(ctx, r, b, wait) +} + +// writeSidecars is one write of the sidecars as they are now (the description the +// run has so far with them), skipped when that content was written already. +func (h *Handler) writeSidecars(ctx context.Context, r *run, b store.Book, wait bool) (sidecars, error) { s, err := r.loadSidecars(b.WorkDir) if err != nil { return sidecars{}, err } - if s.hash == r.written { + // Only the description this run's step read back, under the engine's hold (it + // can say how that was made); none when it got none. + s.description = r.descRaw + sum := s.hash() + if sum == r.written { return s, nil } workID := cmp.Or(b.WorkID, s.workID) + err = h.save(ctx, r, saveSpec{ + what: "the characters and recaps", + wait: wait, + record: true, + progress: jobs.Progress{Step: stepWriting, Message: "Saving the characters and recaps with the book", + Extra: r.bookExtra(b)}, + files: func(p place) ([]attachFile, error) { + prov := provenance(h.o.Version, Kind, r.job.ID, h.now(), r.source, workID, inputsOf(r.mount, r.files, r.epub)) + if s.description != nil { + prov.Description = descriptionProvenance(r.job.ID, *r.desc) + } else { + // An earlier run's description.json stays in place; what vouches + // for it stays with it. + prov.Description = keptDescription(p.dir, p.prefix) + } + return attachmentFiles(s, prov) + }, + }) + if err != nil { + return sidecars{}, err + } + r.written = sum + h.o.Log.Info("generate: wrote the book's attachments", "job", r.job.ID, "path", r.rel, "files", attachmentNames(r.place.prefix, s)) + r.rep.Progress(jobs.Progress{Step: stepRescanning, Message: "Asking the server to read them", + Extra: r.bookExtra(b)}) + h.rescan(ctx, r) + return s, nil +} + +// saveSpec is one save into the book's .audiosilo/ (the generate job's +// attachments, or a description-only job's two files). +type saveSpec struct { + // what is what the save writes, as a park names it ("the description"). + what string + // wait parks book_not_writable and tries again until the save goes through; + // without it the failure is returned. + wait bool + // record records a move in the job's mapping (a description-only job has + // none to record it in). + record bool + // progress is reported once the book is located, before the write. + progress jobs.Progress + // files composes what is written once the book is located (bridge.json names + // the inputs where the book is now). + files func(p place) ([]attachFile, error) +} + +// save writes spec's files into the book's .audiosilo/ at the path it is at now +// (the server moves the job with its book), through the mapping checked again +// (mapped and writable at the mount the job uses). With spec.wait, a write that +// fails parks the job book_not_writable and tries again every Recheck (or on +// Retry) until it goes through. +func (h *Handler) save(ctx context.Context, r *run, spec saveSpec) error { for { if err := h.waitMapped(ctx, r, false); err != nil { - return sidecars{}, err + return err + } + if h.adoptMove(ctx, r) && spec.record { + h.recordMove(ctx, r) } - h.followMove(ctx, r) - err := h.writeOnce(r, b, s, workID) + err := h.saveOnce(r, spec) if err == nil { break } - if !wait { - return sidecars{}, err + if !spec.wait { + return err } - h.park(r, parkBookNotWritable, "the Bridge can't save the characters and recaps with the book: "+err.Error()) - if err := h.recheckWait(ctx, r); err != nil { - return sidecars{}, err + h.park(r, parkBookNotWritable, "the Bridge can't save "+spec.what+" with the book: "+err.Error()) + if err := h.recheckWait(ctx, r, time.Now().Add(h.o.Recheck)); err != nil { + return err } } if r.ownPark { r.rep.Unpark() r.ownPark = false } - r.written = s.hash - h.o.Log.Info("generate: wrote the book's attachments", "job", r.job.ID, "path", r.rel, "files", attachmentNames(r.place.prefix, s)) - r.rep.Progress(jobs.Progress{Step: stepRescanning, Message: "Asking the server to read them", - Extra: map[string]any{"stage": b.State, "source": r.source}}) - h.rescan(ctx, r) - return s, nil + return nil +} + +// saveOnce locates the book through the mount and writes spec's files, its error +// worded without the Bridge's own paths. +func (h *Handler) saveOnce(r *run, spec saveSpec) error { + if err := r.relocate(); err != nil { + return err + } + r.rep.Progress(spec.progress) + files, err := spec.files(r.place) + if err == nil { + err = writeAttachFiles(r.place.dir, r.place.prefix, files) + } + if err != nil { + return fmt.Errorf("its %s folder can't be written (%s)", AttachDir, writeProblem(err)) + } + return nil } -// writeOnce locates the book through the mount and writes its attachments, its -// error worded without the Bridge's own paths. -func (h *Handler) writeOnce(r *run, b store.Book, s sidecars, workID string) error { +// relocate locates the book where the job has it now (r.place), and its .epub +// with it: a book that moved took its .epub along. +func (r *run) relocate() error { p, err := locate(r.mount, r.rel) if err != nil { return err } r.place = p if r.epub != "" && !fsutil.IsFile(r.epub) { - // The book moved: its .epub went with it. if moved := filepath.Join(p.dir, filepath.Base(r.epub)); fsutil.IsFile(moved) { r.epub = moved } } - r.rep.Progress(jobs.Progress{Step: stepWriting, Message: "Saving the characters and recaps with the book", - Extra: map[string]any{"stage": b.State, "source": r.source}}) - prov := provenance(h.o.Version, r.job.ID, h.now(), r.source, workID, inputsOf(r.mount, r.files, r.epub)) - if err := writeAttachments(r.place.dir, r.place.prefix, s, prov); err != nil { - return fmt.Errorf("its %s folder can't be written (%s)", AttachDir, writeProblem(err)) - } return nil } -// followMove adopts the job's path when the server moved the job with its book -// (a rename its scan noticed; the answer to a report says so): the book's files -// are asked for again there, and the mapping follows, so a restart finds the -// same book. -func (h *Handler) followMove(ctx context.Context, r *run) { +// adoptMove adopts the job's path when the server moved the job with its book (a +// rename its scan noticed; the answer to a report says so): the book's files are +// asked for again there. It reports whether the book moved. +func (h *Handler) adoptMove(ctx context.Context, r *run) bool { cur := r.rep.Path() if cur == "" || cur == r.jobPath { - return + return false } from := r.rel r.jobPath, r.rel = cur, cur @@ -859,15 +1048,16 @@ func (h *Handler) followMove(ctx context.Context, r *run) { } else { h.o.Log.Warn("generate: asking the server for the moved book failed", "job", r.job.ID, "err", err) } - h.recordMove(ctx, r, from) + h.o.Log.Info("generate: the book moved; the job follows it", "job", r.job.ID, "from", from, "to", r.rel) + return true } -// recordMove records the job's book at r.rel (it was at from). -func (h *Handler) recordMove(ctx context.Context, r *run, from string) { +// recordMove records the job's book at r.rel in its mapping, so a restart finds +// the same book. +func (h *Handler) recordMove(ctx context.Context, r *run) { if err := h.o.DB.SetGenerateBookPath(ctx, r.job.ID, r.rel); err != nil { h.o.Log.Warn("generate: recording the book's new path failed", "job", r.job.ID, "err", err) } - h.o.Log.Info("generate: the book moved; the job follows it", "job", r.job.ID, "from", from, "to", r.rel) } // loadSidecars reads the engine's sidecars, unless their files are unchanged @@ -894,6 +1084,9 @@ func attachmentNames(prefix string, s sidecars) []string { if s.recaps != nil { out = append(out, prefix+recapsName) } + if s.description != nil { + out = append(out, prefix+descriptionName) + } return append(out, prefix+provenanceName) } @@ -906,7 +1099,8 @@ func (h *Handler) rescan(ctx context.Context, r *run) { for i := 1; ; i++ { got, err := r.srv.RescanAttachments(ctx, r.lib.ID, r.rel) if err == nil { - h.o.Log.Info("generate: the server read the attachments", "job", r.job.ID, "characters", got.Characters, "recaps", got.Recaps) + h.o.Log.Info("generate: the server read the attachments", "job", r.job.ID, "characters", got.Characters, "recaps", got.Recaps, + "description", got.Description) return } if i == rescanTries || errors.Is(err, link.ErrNotIndexable) || ctx.Err() != nil { @@ -940,6 +1134,15 @@ func (h *Handler) Retry(ctx context.Context, jobID int64) error { } return waiting } + // A description-only job's parks are all its own (they wait for the poke); its + // id may still map to an engine book in generate_books, an earlier server's + // generate job under the same id, which is not this job's to re-admit. + if sj, err := h.o.DB.GetServerJob(ctx, jobID); err == nil && sj.Kind == DescribeKind { + if poke() { + return nil + } + return store.ErrNotFound + } m, err := h.o.DB.GetGenerateBook(ctx, jobID) if errors.Is(err, store.ErrNotFound) || (err == nil && m.BookID == nil) { if poke() { diff --git a/internal/generate/generate_test.go b/internal/generate/generate_test.go index 076b86c..f3c22bf 100644 --- a/internal/generate/generate_test.go +++ b/internal/generate/generate_test.go @@ -88,7 +88,7 @@ func (f *fakeServer) ServeHTTP(w http.ResponseWriter, r *http.Request) { var body map[string]any _ = json.NewDecoder(r.Body).Decode(&body) f.rescans = append(f.rescans, body) - _ = json.NewEncoder(w).Encode(map[string]any{"path": body["path"], "attachments": map[string]bool{"characters": true, "recaps": true, "bridge": true}}) + _ = json.NewEncoder(w).Encode(map[string]any{"path": body["path"], "attachments": map[string]bool{"characters": true, "recaps": true, "description": true, "bridge": true}}) default: http.NotFound(w, r) } diff --git a/internal/generate/provenance_unix_test.go b/internal/generate/provenance_unix_test.go new file mode 100644 index 0000000..2758853 --- /dev/null +++ b/internal/generate/provenance_unix_test.go @@ -0,0 +1,52 @@ +//go:build unix + +package generate + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "syscall" + "testing" + "time" +) + +// bridge.json read back for a description-only job is never read through a FIFO +// (the open never blocks) or through a .audiosilo that is a link out of the book's +// folder: either is written fresh. +func TestDescribedProvenanceRefusesSpecialFiles(t *testing.T) { + t.Parallel() + fresh := provenance("0.3.0", DescribeKind, 9, time.Date(2026, 10, 16, 0, 0, 0, 0, time.UTC), SourceAudio, "w", nil) + fresh.Description = &DescriptionProvenance{JobID: 9, From: descriptionFrom, Chapters: 3, CheckedAgainst: "none"} + + dir := t.TempDir() + adir := filepath.Join(dir, AttachDir) + if err := os.MkdirAll(adir, 0o755); err != nil { + t.Fatal(err) + } + if err := syscall.Mkfifo(filepath.Join(adir, "bridge.json"), 0o644); err != nil { + t.Fatal(err) + } + done := make(chan map[string]json.RawMessage, 1) + go func() { done <- readProvenance(adir, "bridge.json") }() + select { + case m := <-done: + if m != nil { + t.Fatalf("a FIFO read as %v", m) + } + case <-time.After(5 * time.Second): + t.Fatal("reading a FIFO blocked") + } + + other := t.TempDir() + mustWrite(t, filepath.Join(other, "bridge.json"), `{"kind":"leaked"}`) + linked := t.TempDir() + if err := os.Symlink(other, filepath.Join(linked, AttachDir)); err != nil { + t.Fatal(err) + } + raw, err := describedProvenance(linked, "", fresh) + if err != nil || strings.Contains(string(raw), "leaked") { + t.Fatalf("read through a linked .audiosilo: %s, %v", raw, err) + } +} diff --git a/internal/generate/retention.go b/internal/generate/retention.go index 262866a..e2cf5bb 100644 --- a/internal/generate/retention.go +++ b/internal/generate/retention.go @@ -19,32 +19,69 @@ func finished(st, status string) bool { return state.IsTerminal(state.State(st)) || state.Status(status) == state.StatusFailed } -// purge deletes one finished book's transcripts, updates its disk gauge and -// records the sweep, so the daily sweep never looks at it again. -func (h *Handler) purge(ctx context.Context, b store.SweepBook) error { +// purge deletes one finished book's transcripts for the sweep, holding the engine +// book while it does: one the engine holds (a description reading it) is left for +// a later sweep, and swept reports false. +func (h *Handler) purge(ctx context.Context, b store.SweepBook) (swept bool, err error) { + release, ok := h.hold(b.ID) + if !ok { + return false, nil + } + defer release() + return h.purgeHeld(ctx, b.ID) +} + +// purgeHeld deletes one held book's transcripts, updates its disk gauge and records +// the sweep, so the daily sweep never looks at it again. The book is looked at +// again under the hold: one a job follows again since it was found finished (a +// later claim of its job re-opening and re-admitting it while the purge waited +// for the hold) runs on its transcripts, so it is left alone, and swept reports +// false. +func (h *Handler) purgeHeld(ctx context.Context, bookID int64) (swept bool, err error) { + b, err := h.o.DB.GetBook(ctx, bookID) + if errors.Is(err, store.ErrNotFound) { + return false, nil + } + if err != nil || !finished(b.State, b.Status) { + return false, err + } + if followed, err := h.o.DB.GenerateBookFollowed(ctx, bookID); err != nil || followed { + return false, err + } if err := scratch.PurgeTranscripts(h.o.WorkRoot, b.WorkDir, state.ParseKind(b.Kind)); err != nil { - return err + return false, err } // The disk gauge follows what is left (the durables). if size, err := scratch.DirSize(b.WorkDir); err == nil { _ = h.o.DB.UpdateScratchBytes(ctx, b.ID, size) } - return h.o.DB.MarkGenerateBookSwept(ctx, b.ID, h.now()) + return true, h.o.DB.MarkGenerateBookSwept(ctx, b.ID, h.now()) } // purgeEnded deletes the transcripts of the job's own engine book as its job -// ends (transcripts.retention_days 0). +// ends (transcripts.retention_days 0). It waits a few polls for the engine to let +// go of the book (a stage the job's end just stopped is still finishing); one it +// keeps is left for the sweep. func (h *Handler) purgeEnded(ctx context.Context, bookID int64) error { b, err := h.o.DB.GetBook(ctx, bookID) if err != nil || !finished(b.State, b.Status) { return err } - return h.purge(ctx, store.SweepBook{ID: b.ID, WorkDir: b.WorkDir, Kind: b.Kind, State: b.State, Status: b.Status}) + release, err := h.holdBook(ctx, b.ID) + if errors.Is(err, errBookBusy) { + return nil + } + if err != nil { + return err + } + defer release() + _, err = h.purgeHeld(ctx, b.ID) + return err } // SweepTranscripts deletes the transcripts of every finished engine book whose -// job ended at least retention ago and that wasn't swept yet, and returns how -// many it swept. +// job ended at least retention ago and that wasn't swept yet (one the engine +// holds is left for the next sweep), and returns how many it swept. func (h *Handler) SweepTranscripts(ctx context.Context, retention time.Duration) (int, error) { books, err := h.o.DB.UnsweptGenerateBooks(ctx, h.now().Add(-retention)) if err != nil { @@ -59,11 +96,14 @@ func (h *Handler) SweepTranscripts(ctx context.Context, retention time.Duration) if !finished(b.State, b.Status) { continue } - if err := h.purge(ctx, b); err != nil { + swept, err := h.purge(ctx, b) + if err != nil { errs = append(errs, err) continue } - n++ + if swept { + n++ + } } return n, errors.Join(errs...) } diff --git a/internal/generate/unit_test.go b/internal/generate/unit_test.go index 8576507..be04cb0 100644 --- a/internal/generate/unit_test.go +++ b/internal/generate/unit_test.go @@ -185,7 +185,7 @@ func TestWriteAttachments(t *testing.T) { t.Parallel() dir := t.TempDir() s := sidecars{characters: []byte(`{"work":"w","characters":[]}`), recaps: []byte(`{"work":"w","recaps":[]}`)} - prov := provenance("0.2.0", 12, time.Date(2026, 10, 15, 9, 0, 0, 0, time.UTC), SourceAudio, "w", + prov := provenance("0.2.0", Kind, 12, time.Date(2026, 10, 15, 9, 0, 0, 0, time.UTC), SourceAudio, "w", []InputFile{{Path: "A/B/01.mp3", Size: 3, ModTime: 1700000000}}) if err := writeAttachments(dir, "", s, prov); err != nil { t.Fatal(err) @@ -284,14 +284,14 @@ func TestLoadSidecarsSkipsUnchangedFiles(t *testing.T) { if err := os.Chtimes(chars, info.ModTime(), info.ModTime()); err != nil { t.Fatal(err) } - if s2, _ := r.loadSidecars(wd); s2.hash != s.hash { + if s2, _ := r.loadSidecars(wd); s2.hash() != s.hash() { t.Fatal("unchanged files were read again") } later := info.ModTime().Add(time.Second) if err := os.Chtimes(chars, later, later); err != nil { t.Fatal(err) } - if s3, _ := r.loadSidecars(wd); s3.hash == s.hash { + if s3, _ := r.loadSidecars(wd); s3.hash() == s.hash() { t.Fatal("a changed file wasn't read again") } } @@ -312,6 +312,15 @@ func TestInputsOfEpub(t *testing.T) { } } +// writeAttachments writes the attachments as a generate job's save does. +func writeAttachments(dir, prefix string, s sidecars, prov Provenance) error { + files, err := attachmentFiles(s, prov) + if err != nil { + return err + } + return writeAttachFiles(dir, prefix, files) +} + // The sweep deletes the transcripts of finished books whose job ended at least // the retention ago, and leaves a recent one and an unfinished one alone. func TestSweepTranscripts(t *testing.T) { @@ -410,6 +419,114 @@ func TestPurgeEndedIsTheJobsBookOnly(t *testing.T) { } } +// holdEngine is an engine whose holds are refused for the books in held. +type holdEngine struct { + Engine + held map[int64]bool +} + +func (e holdEngine) Hold(id int64) (func(), bool) { return func() {}, !e.held[id] } + +// A book the engine holds (a description reading it) isn't swept: the sweep leaves +// it, unmarked, for the next one. +func TestSweepSkipsAHeldBook(t *testing.T) { + t.Parallel() + db, err := store.Open(context.Background(), ":memory:") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + ctx := context.Background() + root := t.TempDir() + eng := holdEngine{held: map[int64]bool{}} + h := New(Options{DB: db, WorkRoot: root, Engine: func() Engine { return eng }, Poll: time.Millisecond}) + wd := filepath.Join(root, "held") + mustWrite(t, filepath.Join(wd, "transcripts-raw", "ch001.json"), "x") + b, err := db.CreateBook(ctx, store.NewBook{SourcePath: "/lib/held", WorkDir: wd, Title: "held"}) + if err != nil { + t.Fatal(err) + } + _ = db.SetBookState(ctx, b.ID, "done", "", "", "") + _ = db.PutGenerateBook(ctx, store.GenerateBook{ServerJobID: 1, BookID: &b.ID, LibraryID: 1, RelPath: "held"}) + _ = db.EndGenerateBook(ctx, 1, time.Now().Add(-time.Hour)) + + eng.held[b.ID] = true + if n, err := h.SweepTranscripts(ctx, 0); err != nil || n != 0 || !present(wd) { + t.Fatalf("a held book: swept %d, %v, present %v", n, err, present(wd)) + } + if err := h.purgeEnded(ctx, b.ID); err != nil || !present(wd) { + t.Fatalf("a held book purged at its job's end: %v", err) + } + delete(eng.held, b.ID) + if n, err := h.SweepTranscripts(ctx, 0); err != nil || n != 1 || present(wd) { + t.Fatalf("once released: swept %d, %v", n, err) + } +} + +// readmitEngine is an engine that re-admits a book as it hands out the hold: the +// Retry of a resumed job that landed while a purge waited for the hold. +type readmitEngine struct { + Engine + readmit func(id int64) +} + +func (e readmitEngine) Hold(id int64) (func(), bool) { + e.readmit(id) + return func() {}, true +} + +// A purge reads the book again under the hold: one re-admitted while it waited +// runs on its transcripts, so neither the job's end nor the sweep deletes them +// (denied); a book still finished under the hold is purged (allowed). +func TestPurgeRechecksTheBookUnderTheHold(t *testing.T) { + t.Parallel() + db, err := store.Open(context.Background(), ":memory:") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + ctx := context.Background() + root := t.TempDir() + readmitted := false + eng := readmitEngine{readmit: func(id int64) { + if readmitted { + _ = db.SetBookState(ctx, id, "fact_pass", "", "", "") + } + }} + h := New(Options{DB: db, WorkRoot: root, Engine: func() Engine { return eng }, Poll: time.Millisecond}) + wd := filepath.Join(root, "readmitted") + mustWrite(t, filepath.Join(wd, "transcripts-raw", "ch001.json"), "x") + b, err := db.CreateBook(ctx, store.NewBook{SourcePath: "/lib/readmitted", WorkDir: wd, Title: "readmitted"}) + if err != nil { + t.Fatal(err) + } + // Cancelled by its job's end: failed, so finished when the purge looks. + _ = db.SetBookState(ctx, b.ID, "fact_pass", "failed", "cancelled", "") + _ = db.PutGenerateBook(ctx, store.GenerateBook{ServerJobID: 1, BookID: &b.ID, LibraryID: 1, RelPath: "readmitted"}) + _ = db.EndGenerateBook(ctx, 1, time.Now().Add(-time.Hour)) + + readmitted = true + if err := h.purgeEnded(ctx, b.ID); err != nil || !present(wd) { + t.Fatalf("a book re-admitted under the job end's purge lost its transcripts: %v", err) + } + _ = db.SetBookState(ctx, b.ID, "fact_pass", "failed", "cancelled", "") + if n, err := h.SweepTranscripts(ctx, 0); err != nil || n != 0 || !present(wd) { + t.Fatalf("a book re-admitted under the sweep: swept %d, %v, present %v", n, err, present(wd)) + } + readmitted = false + _ = db.SetBookState(ctx, b.ID, "fact_pass", "failed", "cancelled", "") + // Its job's mapping open again (the later claim re-opened it, its Retry not + // landed yet): still followed, so still not purged. + _ = db.ReopenGenerateBook(ctx, 1) + if err := h.purgeEnded(ctx, b.ID); err != nil || !present(wd) { + t.Fatalf("a book its job follows again lost its transcripts: %v", err) + } + _ = db.EndGenerateBook(ctx, 1, time.Now().Add(-time.Hour)) + if n, err := h.SweepTranscripts(ctx, 0); err != nil || n != 1 || present(wd) { + t.Fatalf("a finished book: swept %d, %v, present %v", n, err, present(wd)) + } +} + // present reports whether a book's raw transcript layer is still on disk. func present(workDir string) bool { _, err := os.Stat(filepath.Join(workDir, "transcripts-raw")) diff --git a/internal/jobs/jobs.go b/internal/jobs/jobs.go index 08e8bae..dbb4c4b 100644 --- a/internal/jobs/jobs.go +++ b/internal/jobs/jobs.go @@ -34,6 +34,10 @@ type Job struct { Path *string Params json.RawMessage Attempt int + // Resumed: the runner resumed the job under its claim (after a restart, or + // once a refused token works again) instead of claiming it now, so an earlier + // run of it may have spent already. + Resumed bool } // Handler runs one job kind. @@ -352,6 +356,7 @@ func (r *Runner) recover(ctx context.Context, srv Server, wg *sync.WaitGroup) bo continue } r.log.Info("bridge: resuming a job after a restart", "job", job.ID, "kind", job.Kind, "attempt", job.Attempt) + job.Resumed = true r.launch(ctx, srv, job, wg, sj.Error) } return true @@ -402,6 +407,7 @@ func (r *Runner) resumeStranded(ctx context.Context, srv Server, wg *sync.WaitGr continue } r.log.Info("bridge: resuming a job its refused token stopped", "job", job.ID, "kind", job.Kind, "attempt", job.Attempt) + job.Resumed = true r.launch(ctx, srv, job, wg, sj.Error) } } diff --git a/internal/link/link.go b/internal/link/link.go index 4c2b54c..5dd9a2a 100644 --- a/internal/link/link.go +++ b/internal/link/link.go @@ -187,7 +187,9 @@ type BookInfo struct { type Attachments struct { Characters bool `json:"characters"` Recaps bool `json:"recaps"` - Bridge bool `json:"bridge"` + // Description: description.json (Phase 1c; an older server sends none). + Description bool `json:"description"` + Bridge bool `json:"bridge"` } // BookFile is one of a Book's files, its path relative to the library root. diff --git a/internal/link/link_test.go b/internal/link/link_test.go index 2b3b652..676999e 100644 --- a/internal/link/link_test.go +++ b/internal/link/link_test.go @@ -256,14 +256,19 @@ func TestHeartbeatEstimates(t *testing.T) { func TestRescanAttachments(t *testing.T) { t.Parallel() - f, c := newFake(t, http.StatusOK, `{"path":"A/B","attachments":{"characters":true,"recaps":true,"bridge":false}}`) + f, c := newFake(t, http.StatusOK, `{"path":"A/B","attachments":{"characters":true,"recaps":true,"description":true,"bridge":false}}`) got, err := c.RescanAttachments(context.Background(), 1, "A/B") - if err != nil || !got.Characters || !got.Recaps || got.Bridge { + if err != nil || !got.Characters || !got.Recaps || !got.Description || got.Bridge { t.Fatalf("RescanAttachments = %+v, %v", got, err) } if f.last.URL.Path != "/api/v1/bridge/rescan" || f.sent != `{"library_id":1,"mode":"attachments","path":"A/B"}` { t.Fatalf("request = %s %s", f.last.URL.Path, f.sent) } + // An older server sends no description: it reads as not indexed. + f.body = `{"path":"A/B","attachments":{"characters":true,"recaps":true,"bridge":true}}` + if got, err := c.RescanAttachments(context.Background(), 1, "A/B"); err != nil || got.Description { + t.Fatalf("older server = %+v, %v", got, err) + } f.status, f.body = http.StatusNotFound, `{"error":"no indexed book there","code":"not_indexable"}` if _, err := c.RescanAttachments(context.Background(), 1, "A/B"); !errors.Is(err, ErrNotIndexable) { t.Fatalf("404 not_indexable = %v", err) diff --git a/internal/pipeline/agent_stage.go b/internal/pipeline/agent_stage.go index 3385956..c312fd9 100644 --- a/internal/pipeline/agent_stage.go +++ b/internal/pipeline/agent_stage.go @@ -302,6 +302,9 @@ func stageMaxTurns(stage state.State) int { return 80 case state.FactPass, state.Synthesizing, state.Auditing, state.Fixing: return 32 + case state.State(DescribeStage): + // One small notes file in, one paragraph out. + return 10 default: return 32 } @@ -412,28 +415,42 @@ func (t *trackedAgentRunner) finish(res agent.Result, err error) { // (actionable, Retry-able); an exhausted output validator PARKS naming why; any other // error (render, transport, timeout) is returned as a plain error (StatusFailed). func (e *Executor) runAgent(ctx context.Context, book store.Book, stage state.State, r scheduler.StageReport, st *agent.Staging, promptName string, promptData any, web bool, validate func(agent.Result, *agent.Staging) error) (agentUsage, error) { - // Per-book budget preflight: park (with everything already recorded) BEFORE spending - // another invocation once the book's summed agent cost has reached the budget. It runs - // per runAgent call, so fact_pass (one runAgent per chunk) naturally checks between - // chunks; a single invocation is never aborted mid-flight. The budget sum prefers - // provider cost and falls back to complete configured API-equivalent estimates, and - // includes superseded rows, so a Retry never lowers the spend the guard sees. + if err := e.budgetPreflight(ctx, book, stage); err != nil { + return agentUsage{}, err + } + return e.runAgentWith(ctx, book, stage, r, st, promptName, promptData, web, validate, e.backoff) +} + +// budgetPreflight is the per-book budget preflight: park (with everything already +// recorded) BEFORE spending another invocation once the book's summed agent cost has +// reached the budget. It runs per runAgent call, so fact_pass (one runAgent per +// chunk) naturally checks between chunks; a single invocation is never aborted +// mid-flight. The budget sum prefers provider cost and falls back to complete +// configured API-equivalent estimates, and includes superseded rows, so a Retry never +// lowers the spend the guard sees. The monthly budget is checked after it. The +// description step calls it itself, before its preparation, then runAgentWith. +func (e *Executor) budgetPreflight(ctx context.Context, book store.Book, stage state.State) error { if e.db != nil && e.bookBudgetUSD > 0 { spent, complete, serr := e.db.SumStageRunBudgetCost(ctx, book.ID) if serr != nil { - return agentUsage{}, fmt.Errorf("%s: read book cost: %w", stage, serr) + return fmt.Errorf("%s: read book cost: %w", stage, serr) } if !complete { e.log.Warn("agent: book budget has unpriced usage; known subtotal is not treated as the complete cost", "book", book.ID, "known_usd", spent, "pricing_version", e.pricing.Version) } if spent >= e.bookBudgetUSD { - return agentUsage{}, scheduler.ParkWithCode(state.ParkBudgetExceeded, budgetExceededMsg(spent, e.bookBudgetUSD)) + return scheduler.ParkWithCode(state.ParkBudgetExceeded, budgetExceededMsg(spent, e.bookBudgetUSD)) } } - if err := e.checkMonthlyBudget(ctx, stage, time.Now()); err != nil { - return agentUsage{}, err - } + return e.checkMonthlyBudget(ctx, stage, time.Now()) +} + +// runAgentWith is runAgent after the budget preflight (its caller's), with the +// rate-limit backoff schedule given: nil is the default (agent.DefaultBackoff), an +// empty schedule waits out no rate limit at all (the limit parks at once, which +// the description step needs: it must never hold the job that asked for it). +func (e *Executor) runAgentWith(ctx context.Context, book store.Book, stage state.State, r scheduler.StageReport, st *agent.Staging, promptName string, promptData any, web bool, validate func(agent.Result, *agent.Staging) error, backoff []time.Duration) (agentUsage, error) { runner, av := e.ensureAgent(ctx) if runner == nil || !av.Available { // PREFLIGHT unavailability: no CLI is configured/resolvable at all (not a transient @@ -489,7 +506,6 @@ func (e *Executor) runAgent(ctx context.Context, book store.Book, stage state.St } }, } - backoff := e.backoff if backoff == nil { backoff = agent.DefaultBackoff() } diff --git a/internal/pipeline/contrib_stage.go b/internal/pipeline/contrib_stage.go index 821de31..0b9b718 100644 --- a/internal/pipeline/contrib_stage.go +++ b/internal/pipeline/contrib_stage.go @@ -114,6 +114,9 @@ func (e *Executor) contribute(ctx context.Context, book store.Book, r scheduler. if r.Progress != nil { r.Progress(0, 1) } + // The description first, while the source text is still there (best effort, + // outside the stage's rate sample). + e.describeAtContributing(ctx, book, r) start := time.Now() // 1. Load which sidecars exist. At least one must be present (synthesis normally diff --git a/internal/pipeline/describe_contrib_test.go b/internal/pipeline/describe_contrib_test.go new file mode 100644 index 0000000..ae8ecfa --- /dev/null +++ b/internal/pipeline/describe_contrib_test.go @@ -0,0 +1,186 @@ +package pipeline + +import ( + "context" + "encoding/json" + "errors" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/kodestar/audiosilo-meta/pkg/model" + + "github.com/kodestar/audiosilo-bridge/internal/agent" + "github.com/kodestar/audiosilo-bridge/internal/audio" + "github.com/kodestar/audiosilo-bridge/internal/ebook" + "github.com/kodestar/audiosilo-bridge/internal/scheduler" + "github.com/kodestar/audiosilo-bridge/internal/state" + "github.com/kodestar/audiosilo-bridge/internal/store" + "github.com/kodestar/audiosilo-bridge/internal/transcript" +) + +// copiedRun is a run of the ebook's text a description must not carry. +const copiedRun = "the lamplighter climbed the crooked stair and counted every one of the forty steps aloud" + +// contribEbook seeds an ebook book at contributing: six chapters, their text (the +// second carrying copiedRun), the fact notes and the run's sidecars. followed maps +// a sidecars.generate job to it. It returns the executor (fake agent, local +// contribution mode, real store) and the book row. +func contribEbook(t *testing.T, fake *fakeRunner, followed bool) (*Executor, store.Book, *store.DB) { + t.Helper() + dir := t.TempDir() + work := filepath.Join(dir, "work", "book") + if err := os.MkdirAll(work, 0o750); err != nil { + t.Fatal(err) + } + m := audio.Manifest{Source: filepath.Join(dir, "b.epub"), Style: audio.StyleEbook, ChapterCount: 6} + for i := 1; i <= 6; i++ { + m.Chapters = append(m.Chapters, audio.Chapter{Chapter: i}) + text := strings.Repeat("filler ", 200) + if i == 2 { + text += copiedRun + strings.Repeat(" filler", 200) + } + if err := transcript.WriteText(filepath.Join(work, ebook.TextDir), i, text); err != nil { + t.Fatal(err) + } + } + writeManifestStruct(t, work, m) + seedDescribeFacts(t, work) + chars := baseChars("the-carol") + chars.Sources = []model.Source{{Type: sourceTypeCommunity, Ref: "audiosilo-meta:work:the-carol"}} + seedWorkSidecars(t, work, chars, baseRecaps("the-carol")) + db, err := store.Open(context.Background(), filepath.Join(dir, store.FileName)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + book, err := db.CreateBook(context.Background(), store.NewBook{SourcePath: filepath.Join(dir, "carol"), WorkDir: work, + Title: "A Carol", Authors: []string{"An Author"}, WorkID: "reacher-01", Kind: string(state.KindEbook), + EbookPath: m.Source, ContribMode: contribModeLocal}) + if err != nil { + t.Fatal(err) + } + if followed { + if err := db.PutGenerateBook(context.Background(), store.GenerateBook{ServerJobID: 7, BookID: &book.ID, + LibraryID: 1, RelPath: "carol"}); err != nil { + t.Fatal(err) + } + } + cfg := withAgentConfig(dir, fake) + cfg.DB = db + cfg.ContribMode = contribModeLocal + cfg.ContribCommunityRepo = testCommunityRepo + cfg.ExportRoot = t.TempDir() + e := NewExecutor(cfg) + e.backoff = []time.Duration{time.Millisecond} + return e, book, db +} + +// contributeOnce runs the contributing stage, which must succeed. +func contributeOnce(t *testing.T, e *Executor, book store.Book) { + t.Helper() + if _, err := e.Execute(context.Background(), book, state.Contributing, scheduler.StageReport{}); err != nil { + t.Fatalf("contributing: %v", err) + } +} + +// describingRuns is the book's describing stage runs. +func describingRuns(t *testing.T, db *store.DB, bookID int64) []store.StageRun { + t.Helper() + runs, err := db.ListStageRuns(context.Background(), bookID) + if err != nil { + t.Fatal(err) + } + var out []store.StageRun + for _, r := range runs { + if r.Stage == DescribeStage { + out = append(out, r) + } + } + return out +} + +// An ebook's text is deleted as the book reaches done, before the Bridge's job can +// ask for a description, so the contributing stage writes it first, checked +// against the book text: a run copied from the epub is refused. The job's pass at +// done (the text gone by then) reuses it with no second agent call and keeps the +// check it was made with, never "none". +func TestContributingDescribesAnEbookAgainstItsText(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, attempt int) (agent.Result, error) { + text := cleanDescription + if attempt == 1 { + text = cleanDescription + " It opens: " + copiedRun + "." + } + writeDescription(t, req, text) + return agent.Result{}, nil + } + e, book, db := contribEbook(t, fake, true) + contributeOnce(t, e, book) + if p := fake.lastPrompt(DescribeStage); fake.count(DescribeStage) != 2 || !strings.Contains(p, "copies the book word for word") { + t.Fatalf("calls %d, last prompt:\n%s", fake.count(DescribeStage), p) + } + runs := describingRuns(t, db, book.ID) + var m map[string]any + if len(runs) != 1 || runs[0].Ok == nil || !*runs[0].Ok || json.Unmarshal(runs[0].Metrics, &m) != nil || + m["checked_against"] != CheckedBookText { + t.Fatalf("describing runs = %+v", runs) + } + if !scheduler.SentinelExists(book.WorkDir, string(state.Contributing)) { + t.Fatal("contributing didn't finish") + } + + // Done: the ebook's text goes, then the job's pass asks. + if err := os.RemoveAll(filepath.Join(book.WorkDir, ebook.TextDir)); err != nil { + t.Fatal(err) + } + res, err := e.Describe(context.Background(), book.ID, true) + if err != nil || !res.Reused || res.CostUSD != 0 || res.CheckedAgainst != CheckedBookText || res.Chapters != 2 { + t.Fatalf("done's pass = %+v, %v", res, err) + } + if fake.count(DescribeStage) != 2 || len(describingRuns(t, db, book.ID)) != 1 { + t.Fatalf("done's pass called the agent again (%d calls)", fake.count(DescribeStage)) + } + // Re-composed with the work contributing resolved. + var d model.Description + if err := decodeSidecarFile(FinalDescriptionPath(book.WorkDir), &d); err != nil || d.Work != "reacher-01" || + d.Text != cleanDescription { + t.Fatalf("description = %+v, %v", d, err) + } +} + +// A description that fails never fails the contributing stage, and the stage +// doesn't pay for it again when it runs again (a contributing park is run again +// and again); a book no sidecars.generate job follows gets none at all. +func TestContributingSurvivesAFailingDescription(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, _ int) (agent.Result, error) { + writeDescription(t, req, "Too short.") + return agent.Result{Usage: agent.Usage{CostUSD: 0.02, CostReported: true}}, nil + } + e, book, db := contribEbook(t, fake, true) + contributeOnce(t, e, book) + tries := fake.count(DescribeStage) + runs := describingRuns(t, db, book.ID) + if tries == 0 || len(runs) != 1 || runs[0].Ok == nil || *runs[0].Ok { + t.Fatalf("calls %d, describing runs %+v", tries, runs) + } + if _, err := os.Stat(FinalDescriptionPath(book.WorkDir)); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("a failed description was kept: %v", err) + } + contributeOnce(t, e, book) + if fake.count(DescribeStage) != tries || len(describingRuns(t, db, book.ID)) != 1 { + t.Fatalf("the stage paid again: calls %d (was %d)", fake.count(DescribeStage), tries) + } + + unfollowed := newFakeRunner() + e, book, db = contribEbook(t, unfollowed, false) + contributeOnce(t, e, book) + if unfollowed.count(DescribeStage) != 0 || len(describingRuns(t, db, book.ID)) != 0 { + t.Fatalf("a book no job follows was described (%d calls)", unfollowed.count(DescribeStage)) + } +} diff --git a/internal/pipeline/describe_stage.go b/internal/pipeline/describe_stage.go new file mode 100644 index 0000000..9412ad6 --- /dev/null +++ b/internal/pipeline/describe_stage.go @@ -0,0 +1,789 @@ +package pipeline + +import ( + "bytes" + "cmp" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "regexp" + "slices" + "strconv" + "strings" + "time" + "unicode" + "unicode/utf8" + + "github.com/kodestar/audiosilo-meta/pkg/canonical" + "github.com/kodestar/audiosilo-meta/pkg/model" + + "github.com/kodestar/audiosilo-bridge/internal/agent" + "github.com/kodestar/audiosilo-bridge/internal/audio" + "github.com/kodestar/audiosilo-bridge/internal/fsutil" + "github.com/kodestar/audiosilo-bridge/internal/scheduler" + "github.com/kodestar/audiosilo-bridge/internal/state" + "github.com/kodestar/audiosilo-bridge/internal/store" + "github.com/kodestar/audiosilo-bridge/internal/transcript" +) + +// DescribeStage names the description step's stage_runs (BRIDGE-PLAN.md Phase 1c). +// It is NOT a state of the engine's state machine: the step runs once a book's +// sidecars are final (at the start of contributing for a book a sidecars.generate +// job follows, then called by the Bridge's sidecars.generate and sidecars.describe +// jobs, never dispatched by the scheduler), so the readers of stage_runs that drive +// the state machine (reconcile, the supervisor) skip it. Its runs count wherever spend is summed (the budgets, the estimates). +// config's model maps route it under the same name (state.Describing). +const DescribeStage = string(state.Describing) + +// What the description's no-verbatim check measured against +// (DescribeResult.CheckedAgainst, bridge.json's description.checked_against). +const ( + CheckedTranscripts = "transcripts" + CheckedBookText = "book text" + // CheckedNone: every source layer was gone, so nothing was checked: an audio + // book's transcripts after the retention purged them, and an ebook's text, + // which the engine deletes as the book reaches done (a licensing rule; the + // contributing stage describes a generated book before that, so only a later + // sidecars.describe meets it). The agent never saw any of them either. + CheckedNone = "none" +) + +const ( + descriptionFileName = "description.json" + openingNotesName = "opening.md" + chapterPositionsName = "chapter_positions.json" + // The description schema's bounds on the text, in code points (a drift test + // holds them to meta's description.schema.json). + descriptionMinRunes = 200 + descriptionMaxRunes = 1500 + // descriptionMaxParagraphs: one paragraph, two at the outside (AUTHORING.md). + descriptionMaxParagraphs = 2 + // descriptionOutCap bounds the read of the agent's out/description.json. + descriptionOutCap = 64 << 10 +) + +// ErrNoNotes is a book whose work dir holds nothing to describe it from: no +// validated characters.json (whose work and sources the description carries), no +// manifest, or no fact notes for its opening chapters. +var ErrNoNotes = errors.New("the book has no notes to describe it from") + +// ErrOneChapter is a book with a single story chapter: its opening is the whole +// book, ending included, so the step writes no description for it (the window +// never stages the last chapter). +var ErrOneChapter = errors.New("the book has a single chapter, so no opening leaves out its ending") + +// DescribeResult is a written (or reused) description. +type DescribeResult struct { + // Path is the composed model.Description in the work dir + // (FinalDescriptionPath), canonical JSON. + Path string + // Chapters is K, how many opening chapters' notes the agent saw. + Chapters int + CheckedAgainst string + // CostUSD is this call's spend on the budget basis (0 for a reuse). + CostUSD float64 + GeneratedAt time.Time + // Reused: the work dir already held the description, so nothing was spent. + Reused bool +} + +// FinalDescriptionPath is where a book's description lives in its work dir once +// the description step wrote it: what the Bridge copies into the library. +func FinalDescriptionPath(workDir string) string { + return filepath.Join(workDir, sidecarsDir, descriptionFileName) +} + +// descriptionPromptData feeds description.md (missingkey=error). +type descriptionPromptData struct { + Title string + Authors string + Series string + SeriesPos string + Chapters int + ChapterCount int + IsEbook bool +} + +// Describe writes the book's spoiler-free, own-words description from the fact +// notes of its opening chapters ONLY: never a transcript, never knowledge-final.md +// (which carries the ending), never a later chapter's notes. That staging is the +// spoiler bound, by construction. The text is validated (shape, length, voice, the +// description schema, the no-verbatim check against whatever transcript layer is +// still there), composed into a model.Description carrying the run's work and +// sources, and kept in the work dir. A description already there is reused, so a +// resumed job doesn't pay twice. +// +// It is not an engine stage: it opens its own stage_run (DescribeStage) on the +// book so the spend counts, and runs through the shared agent driver, whose parks +// (rate limit, budget, credential, unavailable, validation exhausted) come back as +// *scheduler.ParkError for the caller to handle. noWait parks a rate limit at once +// instead of waiting it out in-process: sidecars.generate's description must never +// hold its job. +func (e *Executor) Describe(ctx context.Context, bookID int64, noWait bool) (DescribeResult, error) { + if e.db == nil { + return DescribeResult{}, errors.New("describe: no store") + } + book, err := e.db.GetBook(ctx, bookID) + if err != nil { + return DescribeResult{}, err + } + chars, err := describeBasis(book.WorkDir) + if err != nil { + return DescribeResult{}, err + } + if res, ok := e.reuseDescription(ctx, book, chars); ok { + return res, nil + } + // The run's own work and sources must make a valid description before any + // spend: no agent answer could repair them. + if err := checkComposable(chars); err != nil { + return DescribeResult{}, err + } + // The preparation (local reads) before the budgets, so a call the budget parks + // still keeps the chapter positions while the transcripts are there; the budgets + // before the stage run and any spend: a budget-parked retry costs nothing. + sel, err := e.openingSelection(book) + if err != nil { + return DescribeResult{}, err + } + if sel.Logical == 1 { + return DescribeResult{}, ErrOneChapter + } + notes, err := buildOpeningNotes(book.WorkDir, sel.Files) + if err != nil { + return DescribeResult{}, err + } + if notes == "" { + return DescribeResult{}, fmt.Errorf("%w (no fact notes for the opening chapters)", ErrNoNotes) + } + if err := e.budgetPreflight(ctx, book, state.State(DescribeStage)); err != nil { + return DescribeResult{}, err + } + + attempt, err := e.db.NextStageAttempt(ctx, book.ID, DescribeStage) + if err != nil { + return DescribeResult{}, fmt.Errorf("describe: number its stage run: %w", err) + } + runID, err := e.db.StartStageRun(ctx, book.ID, DescribeStage, attempt) + if err != nil { + return DescribeResult{}, fmt.Errorf("describe: open its stage run: %w", err) + } + m := map[string]any{"chapters": sel.K, "logical_chapters": sel.Logical, "basis": sel.Basis} + finish := func(ok bool, extra map[string]any) float64 { + for k, v := range extra { + m[k] = v + } + bg := context.WithoutCancel(ctx) + if ferr := e.db.FinishStageRun(bg, runID, ok, metrics(m)); ferr != nil { + e.log.Warn("describe: finish its stage run", "book", book.ID, "err", ferr) + } + cost, cerr := e.db.StageRunBudgetCost(bg, runID) + if cerr != nil { + e.log.Warn("describe: read its cost", "book", book.ID, "err", cerr) + } + return cost + } + // fail finishes the run failed, err recorded with extra. + fail := func(err error, extra map[string]any) float64 { + m["error"] = err.Error() + return finish(false, extra) + } + + st, err := agent.New(book.WorkDir, DescribeStage, attempt) + if err == nil { + err = st.WriteFile(openingNotesName, []byte(notes)) + } + if err != nil { + fail(err, nil) + return DescribeResult{}, fmt.Errorf("describe: stage the notes: %w", err) + } + + var composed []byte + var checked string + // checkErr is a local fault of the no-verbatim check (the disk, a transcript + // that can't be read): nothing the agent could fix, so the answer is taken and + // the run fails after it, never retried (and paid) with the Bridge's paths in + // the agent's prompt. + var checkErr error + validate := func(_ agent.Result, s *agent.Staging) error { + text, err := readDescriptionOut(s.OutDir()) + if err != nil { + return err + } + if problems := descriptionProblems(text, book.Title, book.Series); len(problems) > 0 { + return fmt.Errorf("out/%s: %s", descriptionFileName, strings.Join(problems, "; ")) + } + raw, err := composedDescription(chars, text) + if err != nil { + return err + } + findings, layers, err := descriptionNGram(book, raw) + if err != nil { + checkErr = err + return nil + } + if len(findings) > 0 { + return fmt.Errorf("out/%s copies the book word for word; rewrite these runs in your own words: %s", + descriptionFileName, joinFindings(findings)) + } + composed, checked = raw, checkedAgainst(layers) + return nil + } + data := descriptionPromptData{ + Title: book.Title, Authors: authors(book), Series: book.Series, SeriesPos: book.SeriesPos, + Chapters: sel.K, ChapterCount: sel.Logical, IsEbook: state.ParseKind(book.Kind) == state.KindEbook, + } + backoff := e.backoff + if noWait { + backoff = []time.Duration{} + } + usage, err := e.runAgentWith(ctx, book, state.State(DescribeStage), scheduler.StageReport{}, st, + "description.md", data, false, validate, backoff) + usageMetrics := usage.metricsMap() + if err == nil && checkErr != nil { + err = fmt.Errorf("describe: the no-verbatim check could not run: %w", checkErr) + } + if err != nil { + return DescribeResult{CostUSD: fail(err, usageMetrics)}, err + } + path := FinalDescriptionPath(book.WorkDir) + if err := fsutil.WriteFileAtomic(path, composed, 0o644); err != nil { + return DescribeResult{CostUSD: fail(err, usageMetrics)}, fmt.Errorf("describe: keep the description: %w", err) + } + usageMetrics["checked_against"] = checked + cost := finish(true, usageMetrics) + return DescribeResult{Path: path, Chapters: sel.K, CheckedAgainst: checked, CostUSD: cost, + GeneratedAt: time.Now().UTC()}, nil +} + +// describeAtContributing runs the description step at the start of the +// contributing stage, while the book's source text is still there: an ebook's text +// goes as the book reaches done, before the Bridge's job can ask, so a description +// first written then is checked against nothing. The job's own pass later finds it +// in the work dir and reuses it (re-composed with the work contributing resolved, +// nothing spent), keeping this check's checked_against from the run's metrics. +// +// Best effort, and only for a book a sidecars.generate job follows (the one reader +// of a description): once per book (a describing run already recorded, ok or not, +// is never paid for again, since a contributing park runs the stage again), with no +// rate-limit wait, under the stage's heartbeat, and any error only logged. The +// book is the scheduler's in-flight book here, so no hold is taken. +func (e *Executor) describeAtContributing(ctx context.Context, book store.Book, r scheduler.StageReport) { + if e.db == nil { + return + } + followed, err := e.db.GenerateBookFollowed(ctx, book.ID) + if err != nil || !followed { + if err != nil { + e.log.Warn("contributing: no description (the generate job couldn't be looked up)", "book", book.ID, "err", err) + } + return + } + if n, err := e.db.CountStageRuns(ctx, book.ID, DescribeStage); err != nil || n > 0 { + return + } + var res DescribeResult + err = runWithStageHeartbeat(ctx, r, nil, func() error { + var derr error + res, derr = e.Describe(ctx, book.ID, true) + return derr + }) + if err != nil { + e.log.Warn("contributing: no description; the stage goes on without one", "book", book.ID, "err", err) + return + } + e.log.Info("contributing: wrote the description", "book", book.ID, "checked_against", res.CheckedAgainst) +} + +// describeBasis reads the run's validated characters.json: the description +// carries its work and sources. Missing is ErrNoNotes (no finished run here). +func describeBasis(workDir string) (*model.Characters, error) { + path, _ := FinalSidecarsPaths(workDir) + var chars model.Characters + if err := decodeSidecarFile(path, &chars); err != nil { + if errors.Is(err, os.ErrNotExist) { + return nil, fmt.Errorf("%w (no characters.json)", ErrNoNotes) + } + return nil, fmt.Errorf("describe: read characters.json: %w", err) + } + return &chars, nil +} + +// composeDescription is the description file: the text with the run's work and +// sources and the share-alike license. +func composeDescription(chars *model.Characters, text string) model.Description { + return model.Description{Work: chars.Work, Text: text, License: sidecarLicenseContent, Sources: chars.Sources} +} + +// composedDescription composes the description and checks it against meta's +// description schema, returning its canonical JSON. +func composedDescription(chars *model.Characters, text string) ([]byte, error) { + raw, err := json.Marshal(composeDescription(chars, text)) + if err != nil { + return nil, err + } + if raw, err = canonical.Format(raw); err != nil { + return nil, err + } + schemas, err := sidecarSchemas() + if err != nil { + return nil, err + } + if v, _ := schemaViolation(schemas["description"], raw); v != "" { + return nil, fmt.Errorf("the description fails the description schema: %s", v) + } + return raw, nil +} + +// checkComposable says whether the run's work and sources can carry a description +// at all, by composing one around a placeholder text that satisfies the schema. +func checkComposable(chars *model.Characters) error { + if _, err := composedDescription(chars, strings.Repeat("placeholder ", 20)); err != nil { + return fmt.Errorf("describe: the run's characters.json can't carry a description: %w", err) + } + return nil +} + +// reuseDescription returns the description already in the work dir, re-composed +// with the run's current work and sources (a later stage may have changed them), +// when a successful describing run recorded how it was made. Anything less is +// written again. +func (e *Executor) reuseDescription(ctx context.Context, book store.Book, chars *model.Characters) (DescribeResult, bool) { + path := FinalDescriptionPath(book.WorkDir) + cur, err := os.ReadFile(path) //nolint:gosec // the book's work dir + if err != nil { + return DescribeResult{}, false + } + var d model.Description + if err := decodeSidecarBytes(cur, &d); err != nil || len(descriptionProblems(d.Text, book.Title, book.Series)) > 0 { + return DescribeResult{}, false + } + run, err := e.db.LatestOkStageRun(ctx, book.ID, DescribeStage) + if err != nil { + return DescribeResult{}, false + } + raw, err := composedDescription(chars, d.Text) + if err != nil { + return DescribeResult{}, false + } + if !bytes.Equal(cur, raw) { + if err := fsutil.WriteFileAtomic(path, raw, 0o644); err != nil { + return DescribeResult{}, false + } + } + var m struct { + Chapters int `json:"chapters"` + CheckedAgainst string `json:"checked_against"` + } + _ = json.Unmarshal(run.Metrics, &m) + at, _ := time.Parse(time.RFC3339Nano, run.FinishedAt) + return DescribeResult{Path: path, Chapters: m.Chapters, CheckedAgainst: cmp.Or(m.CheckedAgainst, CheckedNone), + GeneratedAt: at.UTC(), Reused: true}, true +} + +// openingSelection is which audio files' (or ebook chapters') notes the agent +// sees: the files whose LOGICAL position is 0 (front matter) or among the book's +// first K story positions, K = max(2, ceil(logical/4)) capped at logical-1 (never +// the last chapter), logical the count of story positions the files hold (capped +// at the classifier's count). +type openingSelection struct { + Files []int // manifest chapter numbers, ascending + K int + Logical int + // Basis is where the positions came from: "edges" (classified now), "saved" + // (classified while the transcripts were there), "files" (neither: file + // numbers, which never sit below their story position, so the bound holds). + Basis string +} + +// openingWindow is K for a book of logical story chapters: a quarter of them, +// two at least, and never the last one (its ending), so a book of two gets its +// first only and a book of one none. +func openingWindow(logical int) int { + return max(0, min(max(2, (logical+3)/4), logical-1)) +} + +// openingSelection classifies the book's chapters as the sidecar stages did +// (classifyBookEdges), so the bound is the same logical chapters the recaps use. +// The classification reads the transcripts, which the retention deletes; the +// positions are kept beside the notes each time they are classified, and a book +// whose transcripts are gone uses those, else its file numbers (a file's number +// is never below its story chapter, so that bound is safe, only less generous). +func (e *Executor) openingSelection(book store.Book) (openingSelection, error) { + m, err := audio.ReadManifest(book.WorkDir) + var fp string + if err == nil { + // The kept positions are keyed by the pipeline's one meaning of "the + // chapter layout changed". + fp, err = manifestFingerprint(book.WorkDir) + } + if err != nil { + return openingSelection{}, fmt.Errorf("%w (no manifest: %v)", ErrNoNotes, err) + } + chapters := make([]int, 0, len(m.Chapters)) + for _, c := range m.Chapters { + chapters = append(chapters, c.Chapter) + } + ebook := state.ParseKind(book.Kind) == state.KindEbook + switch { + case ebook || transcriptsPresent(book.WorkDir, chapters): + class, err := classifyBookEdges(book) + if err != nil { + return openingSelection{}, err + } + if !ebook { + if err := savePositions(book.WorkDir, fp, class); err != nil { + e.log.Warn("describe: keeping the chapter positions failed", "book", book.ID, "err", err) + } + } + return selectOpening(chapters, class, "edges"), nil + default: + if class, ok := loadPositions(book.WorkDir, fp); ok { + return selectOpening(chapters, class, "saved"), nil + } + return selectOpening(chapters, fileClassification(book.WorkDir, len(chapters)), "files"), nil + } +} + +// selectOpening picks the opening files from a classification: front matter +// (position 0) and the files at the book's first K story positions. Excluded edge +// files (an Audible intro, closing credits) are never story; end matter has no +// position. K counts the story positions the files hold, not the classifier's +// LogicalCount alone: narrated numbering takes that from the highest announcement +// it believed, so one late or misheard announcement ("Forty." opening the last of +// twelve files) would widen the window far past the opening, and a sequel's +// continued numbering (chapters 45 to 56) would leave no position inside 1..K. The +// logical count still caps it: the "files" basis's positions are file numbers, +// which can run past the story's count. +func selectOpening(chapters []int, class edgeClassification, basis string) openingSelection { + excluded := map[int]bool{} + for _, c := range slices.Concat(class.ExcludedLeading, class.ExcludedTrailing) { + excluded[c] = true + } + positions := map[int]int{} + var story []int // the story positions (1 on) the retained files hold, distinct + for _, c := range chapters { + pos, ok := class.PositionOf(c) + if excluded[c] || !ok || pos < 0 { + continue + } + positions[c] = pos + if pos >= 1 && !slices.Contains(story, pos) { + story = append(story, pos) + } + } + slices.Sort(story) + logical := min(len(story), class.LogicalCount) + sel := openingSelection{Logical: logical, K: openingWindow(logical), Basis: basis} + if sel.K < 1 { + return sel + } + last := story[sel.K-1] + for _, c := range chapters { + if pos, ok := positions[c]; ok && pos <= last { + sel.Files = append(sel.Files, c) + } + } + slices.Sort(sel.Files) + return sel +} + +// transcriptsPresent reports whether any chapter still has a transcript (the +// retention deletes them all at once). +func transcriptsPresent(workDir string, chapters []int) bool { + for _, c := range chapters { + if _, ok := transcript.ChapterTextPath(workDir, c); ok { + return true + } + } + return false +} + +// savedPositions is chapter_positions.json: the part of a classification the +// opening selection needs, with the manifest it was made from. +type savedPositions struct { + Manifest string `json:"manifest"` + LogicalCount int `json:"logical_count"` + Excluded []int `json:"excluded,omitempty"` + Offset int `json:"chapter_offset"` + Positions map[int]int `json:"positions,omitempty"` +} + +// savePositions keeps a classification of the manifest fingerprinted fp beside the +// work dir's other durables (the transcript retention leaves it); an unchanged one +// isn't written again. +func savePositions(workDir, fp string, class edgeClassification) error { + raw, err := json.Marshal(savedPositions{Manifest: fp, LogicalCount: class.LogicalCount, + Excluded: slices.Concat(class.ExcludedLeading, class.ExcludedTrailing), Offset: class.ChapterOffset, + Positions: class.Positions}) + if err != nil { + return err + } + path := filepath.Join(workDir, chapterPositionsName) + if cur, err := os.ReadFile(path); err == nil && bytes.Equal(cur, raw) { //nolint:gosec // the book's work dir + return nil + } + return fsutil.WriteFileAtomic(path, raw, 0o644) +} + +// loadPositions reads the kept classification, when it was made from the +// manifest the work dir holds now (fingerprinted fp). +func loadPositions(workDir, fp string) (edgeClassification, bool) { + raw, err := os.ReadFile(filepath.Join(workDir, chapterPositionsName)) //nolint:gosec // the book's work dir + if err != nil { + return edgeClassification{}, false + } + var s savedPositions + if err := json.Unmarshal(raw, &s); err != nil || s.LogicalCount < 1 { + return edgeClassification{}, false + } + if fp != s.Manifest { + return edgeClassification{}, false + } + return edgeClassification{LogicalCount: s.LogicalCount, ExcludedLeading: s.Excluded, ChapterOffset: s.Offset, + Positions: s.Positions}, true +} + +// fileClassification is the fallback with no transcripts and no kept positions: +// every file at its own number, the logical count from the recaps (their last +// chaptered entry runs through the last story chapter), else the file count. +func fileClassification(workDir string, files int) edgeClassification { + logical := files + _, recapsPath := FinalSidecarsPaths(workDir) + var recs model.Recaps + if err := decodeSidecarFile(recapsPath, &recs); err == nil { + last := 0 + for _, r := range recs.Recaps { + last = max(last, r.Through.Chapter) + } + if last > 0 && last < files { + logical = last + } + } + return edgeClassification{LogicalCount: logical} +} + +// sectionBreakRe ends a fact section: any level-1 or level-2 heading. +var sectionBreakRe = regexp.MustCompile(`^#{1,2}(\s|$)`) + +// buildOpeningNotes is opening.md: the `## Chapter N` sections of the chosen files, +// copied mechanically from the fact chunks with their file numbering kept, in +// chapter order. The chunks are the fact pass's own (chunk_plan.json, named by +// factsChunkName), and only those covering one of the files are read; nothing +// else from facts/ (knowledge-final.md, the spelling sheets, a chunk's preamble) +// is copied. "" when none of the files has notes. +func buildOpeningNotes(workDir string, files []int) (string, error) { + want := map[int]bool{} + for _, f := range files { + want[f] = true + } + plan, err := loadChunkPlan(workDir) + if errors.Is(err, os.ErrNotExist) { + return "", nil + } + if err != nil { + return "", err + } + sections := map[int]string{} + for _, c := range plan.Chunks { + if !slices.ContainsFunc(files, func(f int) bool { return f >= c.From && f <= c.To }) { + continue + } + raw, err := os.ReadFile(filepath.Join(factsDirPath(workDir), factsChunkName(c.From, c.To))) //nolint:gosec // the book's work dir + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return "", err + } + for ch, text := range factSections(string(raw)) { + if want[ch] && sections[ch] == "" { + sections[ch] = text + } + } + } + var b strings.Builder + for _, f := range files { + if s := sections[f]; s != "" { + if b.Len() > 0 { + b.WriteString("\n") + } + b.WriteString(s) + } + } + return b.String(), nil +} + +// factSections splits a fact chunk into its `## Chapter N` sections (heading +// included), each running to the next level-1 or level-2 heading. A manifest may +// number its files from 0 (a "Chapter 0" prologue), so 0 is a section like any +// other; -1 is "in none". +func factSections(text string) map[int]string { + out := map[int]string{} + cur := -1 + var b strings.Builder + flush := func() { + if cur >= 0 { + if s := strings.TrimRight(b.String(), " \t\r\n"); s != "" { + out[cur] = s + "\n" + } + } + cur = -1 + b.Reset() + } + for _, line := range strings.SplitAfter(text, "\n") { + trimmed := strings.TrimRight(line, "\r\n") + if sectionBreakRe.MatchString(trimmed) { + flush() + if m := factsHeadingRe.FindStringSubmatch(trimmed); m != nil { + if n, err := strconv.Atoi(m[1]); err == nil { + cur = n + } + } + } + if cur >= 0 { + b.WriteString(line) + } + } + flush() + return out +} + +// readDescriptionOut reads the agent's out/description.json: exactly {"text": ...} +// (no other key, nothing after it), its text trimmed. +func readDescriptionOut(outDir string) (string, error) { + path := filepath.Join(outDir, descriptionFileName) + f, err := os.Open(path) //nolint:gosec // the staged out/ dir + if err != nil { + return "", fmt.Errorf("out/%s: %v", descriptionFileName, err) + } + defer f.Close() //nolint:errcheck // read-only + raw, err := io.ReadAll(io.LimitReader(f, descriptionOutCap+1)) + if err != nil { + return "", fmt.Errorf("out/%s: %v", descriptionFileName, err) + } + if len(raw) > descriptionOutCap { + return "", fmt.Errorf("out/%s is larger than %d bytes", descriptionFileName, descriptionOutCap) + } + var out struct { + Text *string `json:"text"` + } + shape := `write exactly {"text": "..."} and nothing else` + if err := decodeSidecarBytes(raw, &out); err != nil { + return "", fmt.Errorf("out/%s: %v (%s)", descriptionFileName, err, shape) + } + if out.Text == nil { + return "", fmt.Errorf("out/%s has no text (%s)", descriptionFileName, shape) + } + return strings.TrimSpace(*out.Text), nil +} + +// The voice rules a description is held to (AUTHORING.md's `description` +// contract): no second person, no rhetorical question, no sales words. +var ( + // Word edges by Unicode letter, mark and digit: RE2's \b is ASCII only, so + // "Xīyou" would hold a "you". + secondPersonRe = regexp.MustCompile(`(?i)(?:^|[^\p{L}\p{M}\p{N}_])(you|your|yours|yourself|yourselves)(?:[^\p{L}\p{M}\p{N}_]|$)`) + // A question mark ending a sentence: followed (after any closing quote or + // bracket) by a space or the end of the text. + questionRe = regexp.MustCompile(`\?["'\x{201D}\x{2019})\]]*(\s|$)`) + salesWordRe = regexp.MustCompile(`(?i)\b(gripping|unputdownable|page-turners?|must-reads?|masterpieces?|breathtaking|captivating|riveting|thrilling|unforgettable)\b`) + // A blank line, a no-break space (\p{Zs}) on it included. + paragraphBreakRe = regexp.MustCompile(`\n[\s\p{Zs}]*\n`) +) + +// descriptionProblems lists what is wrong with a (trimmed) description text, none +// when it passes. Each is worded for the agent's retry. names are the book's own +// (its title, its series), which the voice rules skip: "Me Before You" is no second +// person, "Do Androids Dream of Electric Sheep?" no rhetorical question and "The +// Masterpiece" no sales word, and a description may name its book. +func descriptionProblems(text string, names ...string) []string { + var out []string + // The line and paragraph separators too: only a plain newline separates. + if strings.ContainsFunc(text, func(r rune) bool { + return unicode.IsControl(r) && r != '\n' || r == '\u2028' || r == '\u2029' + }) { + out = append(out, "the text holds a control character (only plain newlines between paragraphs are allowed)") + } + if hasEmDash(text) { + out = append(out, "the text uses an em dash; use a hyphen or restructure the sentence") + } + if n := utf8.RuneCountInString(text); n < descriptionMinRunes || n > descriptionMaxRunes { + out = append(out, fmt.Sprintf("the text is %d characters; it must be %d to %d (aim for 80 to 150 words)", + n, descriptionMinRunes, descriptionMaxRunes)) + } + if n := len(paragraphBreakRe.Split(text, -1)); n > descriptionMaxParagraphs { + out = append(out, fmt.Sprintf("the text has %d paragraphs; write one, two at most", n)) + } + voice := withoutNames(text, names) + if m := secondPersonRe.FindStringSubmatch(voice); m != nil { + out = append(out, fmt.Sprintf("the text addresses the reader (%q); write in the third person", m[1])) + } + if questionRe.MatchString(voice) { + out = append(out, "a sentence ends in a question mark; state the premise instead of asking") + } + if words := salesWordRe.FindAllString(voice, -1); len(words) > 0 { + out = append(out, fmt.Sprintf("the text uses sales language (%s); keep a neutral reference-guide voice", + strings.Join(words, ", "))) + } + return out +} + +// withoutNames is text with each of names, and a name's part before a subtitle +// colon ("Me Before You: A Novel"), replaced by a space (case-insensitively, as +// whole words where a name starts or ends with one), for the voice rules. +func withoutNames(text string, names []string) string { + for _, name := range names { + short, _, _ := strings.Cut(name, ":") + for _, n := range []string{name, short} { + if n = strings.TrimSpace(n); n == "" { + continue + } + pat := regexp.QuoteMeta(n) + if r, _ := utf8.DecodeRuneInString(n); isASCIIWordRune(r) { + pat = `\b` + pat + } + if r, _ := utf8.DecodeLastRuneInString(n); isASCIIWordRune(r) { + pat += `\b` + } + text = regexp.MustCompile(`(?i)`+pat).ReplaceAllString(text, " ") + } + } + return text +} + +// isASCIIWordRune reports whether r is a word character to RE2's \b. +func isASCIIWordRune(r rune) bool { + return r == '_' || r < utf8.RuneSelf && (unicode.IsLetter(r) || unicode.IsDigit(r)) +} + +// descriptionNGram runs the no-verbatim check over a composed description against +// the book's source layers, returning the findings and the layers measured. +func descriptionNGram(book store.Book, raw []byte) (findings, layers []string, err error) { + dir, err := os.MkdirTemp(book.WorkDir, ".describe-check-") + if err != nil { + return nil, nil, err + } + defer func() { _ = os.RemoveAll(dir) }() + path := filepath.Join(dir, descriptionFileName) + if err := os.WriteFile(path, raw, 0o600); err != nil { + return nil, nil, err + } + return ngramScan(book, []string{path}) +} + +// checkedAgainst words the layers a check measured. +func checkedAgainst(layers []string) string { + switch { + case len(layers) == 0: + return CheckedNone + case slices.Contains(layers, CheckedBookText): + return CheckedBookText + default: + return CheckedTranscripts + } +} diff --git a/internal/pipeline/describe_stage_test.go b/internal/pipeline/describe_stage_test.go new file mode 100644 index 0000000..9a35e65 --- /dev/null +++ b/internal/pipeline/describe_stage_test.go @@ -0,0 +1,759 @@ +package pipeline + +import ( + "context" + "encoding/json" + "errors" + "os" + "path/filepath" + "reflect" + "slices" + "strings" + "testing" + "time" + + "github.com/kodestar/audiosilo-meta/pkg/model" + + "github.com/kodestar/audiosilo-bridge/internal/agent" + "github.com/kodestar/audiosilo-bridge/internal/audio" + "github.com/kodestar/audiosilo-bridge/internal/scheduler" + "github.com/kodestar/audiosilo-bridge/internal/spelling" + "github.com/kodestar/audiosilo-bridge/internal/state" + "github.com/kodestar/audiosilo-bridge/internal/store" + "github.com/kodestar/audiosilo-bridge/internal/transcript" +) + +// cleanDescription passes every rule: 200-1500 characters, one paragraph, neutral +// voice, nothing a transcript of "word word word" could share. +const cleanDescription = "A miser who keeps a counting house in a cold city refuses every claim on his " + + "money or his time, including his nephew's dinner and two collectors for the poor. On the night " + + "before the holiday his dead partner returns in chains to warn him that three spirits will visit, " + + "and the book follows whether a man that settled can still be moved." + +// Opening selection: front matter (position 0) and the first K logical chapters, +// never an excluded edge file, never past K, and the file numbers kept. +func TestSelectOpening(t *testing.T) { + t.Parallel() + chapters := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10} + cases := []struct { + name string + class edgeClassification + want []int + k int + }{ + { + // An Audible intro (file 1, excluded), an unnumbered prologue (file 2, + // front matter at position 0), chapters 1-8 in files 3-10. + name: "intro excluded, front matter kept", + class: edgeClassification{LogicalCount: 8, ExcludedLeading: []int{1}, FrontMatter: []int{2}, ChapterOffset: 2, + Positions: map[int]int{2: 0, 3: 1, 4: 2, 5: 3, 6: 4, 7: 5, 8: 6, 9: 7, 10: 8}}, + want: []int{2, 3, 4}, k: 2, + }, + { + // No narrated numbering: the offset alone, the excluded intro at "position 0". + name: "offset only", + class: edgeClassification{LogicalCount: 9, ExcludedLeading: []int{1}, ChapterOffset: 1}, + want: []int{2, 3, 4}, k: 3, + }, + { + // Never the last chapter: a book of two gets its first. + name: "end matter has no position", + class: edgeClassification{LogicalCount: 2, ExcludedTrailing: []int{10}, EndMatter: []int{3}, Positions: map[int]int{1: 1, 2: 2}}, + want: []int{1}, k: 1, + }, + { + // A book of one has no opening that leaves out its ending. + name: "a single chapter", + class: edgeClassification{LogicalCount: 1, FrontMatter: []int{1}, Positions: map[int]int{1: 0, 2: 1}}, + want: nil, k: 0, + }, + { + name: "a whole book of ten", + class: edgeClassification{LogicalCount: 10}, + want: []int{1, 2, 3}, k: 3, + }, + { + // One late announcement believed ("Forty." opening the last file): the + // window counts the ten positions the files hold, not the forty. + name: "an inflated logical count", + class: edgeClassification{LogicalCount: 40, + Positions: map[int]int{1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 40}}, + want: []int{1, 2, 3}, k: 3, + }, + { + // A sequel numbering its chapters on from the last book's (45 to 54). + name: "continued numbering", + class: edgeClassification{LogicalCount: 54, + Positions: map[int]int{1: 45, 2: 46, 3: 47, 4: 48, 5: 49, 6: 50, 7: 51, 8: 52, 9: 53, 10: 54}}, + want: []int{1, 2, 3}, k: 3, + }, + { + // The "files" basis: file numbers, the logical count from the recaps. + name: "file numbers past the story's count", + class: edgeClassification{LogicalCount: 6}, + want: []int{1, 2}, k: 2, + }, + } + for _, tc := range cases { + sel := selectOpening(chapters, tc.class, "edges") + if !reflect.DeepEqual(sel.Files, tc.want) || sel.K != tc.k { + t.Errorf("%s: files %v K %d, want %v K %d", tc.name, sel.Files, sel.K, tc.want, tc.k) + } + } +} + +// The window counts the story positions the files hold, whatever the narrated +// numbering believed: one late or misheard announcement ("Forty." opening the last +// of twelve files, a year, "Sixty" heard for "Sixteen", a slip in the middle) +// never widens it past the opening, and a sequel numbering on (45 to 56) still +// gets its opening files. +func TestOpeningSelectionCountsTheFilesPositions(t *testing.T) { + t.Parallel() + ones := []string{"", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine", "Ten", + "Eleven", "Twelve", "Thirteen", "Fourteen", "Fifteen", "Sixteen", "Seventeen", "Eighteen", "Nineteen"} + tens := []string{"", "", "Twenty", "Thirty", "Forty", "Fifty", "Sixty"} + words := func(n int) string { + switch { + case n < 20: + return ones[n] + case n%10 == 0: + return tens[n/10] + } + return tens[n/10] + "-" + ones[n%10] + } + book := func(from, n int, swap map[int]string) []string { + var out []string + for i := range n { + o := "Chapter " + words(from+i) + ". The road went on and on and on." + if s, ok := swap[i+1]; ok { + o = s + } + out = append(out, o) + } + return out + } + for name, tc := range map[string]struct { + openings []string + want []int + }{ + "twelve chapters": {book(1, 12, nil), []int{1, 2, 3}}, + "a late Forty": {book(1, 12, map[int]string{12: "Forty. That was how many had died."}), []int{1, 2, 3}}, + "a year": {book(1, 12, map[int]string{12: "1944. The war had been over for a year."}), []int{1, 2, 3}}, + "Sixty heard for Sixteen": {book(1, 16, map[int]string{16: "Chapter Sixty. The road went on and on and on."}), []int{1, 2, 3, 4}}, + "a slip in the middle": {book(1, 20, map[int]string{14: "Forty. The road went on and on and on."}), []int{1, 2, 3, 4, 5}}, + "a sequel numbering on": {book(45, 12, nil), []int{1, 2, 3}}, + } { + var chs []edgeChapter + var chapters []int + for i, o := range tc.openings { + chs = append(chs, edgeChapter{Chapter: i + 1, Words: 4000, HasTranscript: true, Probed: true, DurationSec: 3000, Opening: o}) + chapters = append(chapters, i+1) + } + if sel := selectOpening(chapters, classifyEdgeChapters(chs), "edges"); !reflect.DeepEqual(sel.Files, tc.want) { + t.Errorf("%s: files %v (K %d of %d), want %v", name, sel.Files, sel.K, sel.Logical, tc.want) + } + } +} + +func TestOpeningWindow(t *testing.T) { + t.Parallel() + for logical, want := range map[int]int{1: 0, 2: 1, 3: 2, 4: 2, 8: 2, 9: 3, 12: 3, 13: 4, 40: 10, 49: 13} { + if got := openingWindow(logical); got != want { + t.Errorf("openingWindow(%d) = %d, want %d", logical, got, want) + } + } +} + +// writeFacts writes facts/ under work. +func writeFacts(t *testing.T, work, name, text string) { + t.Helper() + dir := filepath.Join(work, spelling.FactsDir) + if err := os.MkdirAll(dir, 0o750); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, name), []byte(text), 0o644); err != nil { //nolint:gosec // test artifact + t.Fatal(err) + } +} + +// seedDescribeFacts writes the fact pass's chunk plan and its two chunks over +// files 1-6 (each section carrying a marker naming its file), the whole-book sheet +// and a spelling sheet. +func seedDescribeFacts(t *testing.T, work string) { + t.Helper() + if err := writeChunkPlan(work, chunkPlan{Chunks: []chunkRange{{From: 1, To: 3}, {From: 4, To: 6}}, ChunkEnds: []int{3, 6}}); err != nil { + t.Fatal(err) + } + writeFacts(t, work, "facts-ch1-3.md", "# Facts, audio files 1-3\n\nNote: PREAMBLE-NOTE.\n\n"+ + "## Chapter 1\n- Non-narrative file (opening announcement). MARK-F1\n\n"+ + "## Chapter 2\nEVENTS\n- A miser refuses his nephew. MARK-F2\n### Sub\n- detail MARK-F2B\n\n"+ + "## Chapter 3\n- His partner's ghost warns him. MARK-F3\n") + writeFacts(t, work, "facts-ch4-6.md", "# Facts, audio files 4-6\n\n"+ + "## Chapter 4\n- The first spirit. MARK-F4\n\n## Chapter 5\n- MARK-F5\n\n## Chapter 6\n- He changes. MARK-F6\n") + writeFacts(t, work, knowledgeFinalName, "# Final\n## ENDING\nENDING-SECRET\n") + writeFacts(t, work, "spellings-through-ch6.md", "SPELLING-SHEET\n") +} + +// The notes are copied mechanically: only the chosen sections, headings kept as +// file numbers, no preamble, no other file. +func TestBuildOpeningNotes(t *testing.T) { + t.Parallel() + work := t.TempDir() + seedDescribeFacts(t, work) + notes, err := buildOpeningNotes(work, []int{2, 3}) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"## Chapter 2\n", "MARK-F2", "### Sub", "MARK-F2B", "## Chapter 3\n", "MARK-F3"} { + if !strings.Contains(notes, want) { + t.Errorf("notes lack %q:\n%s", want, notes) + } + } + for _, leak := range []string{"MARK-F1", "MARK-F4", "MARK-F5", "MARK-F6", "PREAMBLE", "ENDING-SECRET", "SPELLING-SHEET", "## Chapter 1", "## Chapter 4", "# Facts"} { + if strings.Contains(notes, leak) { + t.Errorf("notes leak %q:\n%s", leak, notes) + } + } + if strings.Index(notes, "MARK-F2") > strings.Index(notes, "MARK-F3") { + t.Errorf("sections out of order:\n%s", notes) + } + if empty, err := buildOpeningNotes(t.TempDir(), []int{1}); err != nil || empty != "" { + t.Fatalf("no facts dir = %q, %v", empty, err) + } + + // A manifest numbered from 0: its "Chapter 0" (a prologue) is a section too. + zero := t.TempDir() + if err := writeChunkPlan(zero, chunkPlan{Chunks: []chunkRange{{From: 0, To: 2}}, ChunkEnds: []int{2}}); err != nil { + t.Fatal(err) + } + writeFacts(t, zero, "facts-ch0-2.md", "# Facts, audio files 0-2\n\n## Chapter 0\n- The prologue. MARK-Z0\n\n"+ + "## Chapter 1\n- MARK-Z1\n\n## Chapter 2\n- MARK-Z2\n") + notes, err = buildOpeningNotes(zero, []int{0, 1}) + if err != nil || !strings.Contains(notes, "## Chapter 0\n- The prologue. MARK-Z0") || !strings.Contains(notes, "MARK-Z1") || + strings.Contains(notes, "MARK-Z2") || strings.Index(notes, "MARK-Z0") > strings.Index(notes, "MARK-Z1") { + t.Fatalf("a chapter-0 book's notes = %q, %v", notes, err) + } +} + +// describeBook seeds a finished audio book: 6 files (file 1 a short Audible-style +// intro, 2-6 story chapters), their transcripts, the fact notes and the run's +// characters.json and recaps.json. It returns the executor (fake agent, real +// store) and the book row. +func describeBook(t *testing.T, fake *fakeRunner, budget float64) (*Executor, store.Book, *store.DB) { + t.Helper() + dir := t.TempDir() + work := filepath.Join(dir, "work", "book") + if err := os.MkdirAll(work, 0o750); err != nil { + t.Fatal(err) + } + seedWordsManifest(t, work, audio.StyleFiles, []int{3, 3000, 3000, 3000, 3000, 3000}) + seedDescribeFacts(t, work) + chars := baseChars("the-carol") + chars.Sources = []model.Source{{Type: sourceTypeCommunity, Ref: "audiosilo-meta:work:the-carol"}} + seedWorkSidecars(t, work, chars, baseRecaps("the-carol")) + db, err := store.Open(context.Background(), filepath.Join(dir, store.FileName)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + book, err := db.CreateBook(context.Background(), store.NewBook{SourcePath: filepath.Join(dir, "carol"), WorkDir: work, + Title: "A Carol", Authors: []string{"An Author"}}) + if err != nil { + t.Fatal(err) + } + cfg := withAgentConfig(dir, fake) + cfg.DB = db + cfg.BookBudgetUSD = budget + e := NewExecutor(cfg) + e.backoff = []time.Duration{time.Millisecond} + return e, book, db +} + +// writeDescription scripts the agent's answer. +func writeDescription(t *testing.T, req agent.Request, text string) { + t.Helper() + writeOut(t, req, descriptionFileName, map[string]string{"text": text}) +} + +// The happy path, and the spoiler bound by construction: the staged dir holds +// only opening.md (and out/), and opening.md holds only the opening chapters' +// notes: no transcript, no knowledge-final.md, no later chapter, no excluded intro. +func TestDescribeStagesOnlyTheOpeningNotes(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, _ int) (agent.Result, error) { + writeDescription(t, req, " "+cleanDescription+"\n") + return agent.Result{Usage: agent.Usage{Model: "sonnet", Input: 900, Output: 120, CostUSD: 0.07, CostReported: true}}, nil + } + e, book, db := describeBook(t, fake, 0) + res, err := e.Describe(context.Background(), book.ID, false) + if err != nil { + t.Fatalf("Describe: %v", err) + } + req, ok := fake.lastRequest(DescribeStage) + if !ok || req.Web || req.MaxTurns != 10 { + t.Fatalf("request = %+v", req) + } + var staged []string + _ = filepath.WalkDir(req.Dir, func(p string, d os.DirEntry, err error) error { + if err == nil && p != req.Dir { + rel, _ := filepath.Rel(req.Dir, p) + if rel != filepath.Join(agent.OutDirName, descriptionFileName) { + staged = append(staged, rel) + } + } + return nil + }) + slices.Sort(staged) + if !reflect.DeepEqual(staged, []string{openingNotesName, agent.OutDirName}) { + t.Fatalf("staged dir holds %v, want only opening.md and out/", staged) + } + notes, _ := os.ReadFile(filepath.Join(req.Dir, openingNotesName)) + // 5 logical chapters (file 1 is the excluded intro) -> K = 2 -> files 2 and 3. + for _, want := range []string{"## Chapter 2", "MARK-F2", "## Chapter 3", "MARK-F3"} { + if !strings.Contains(string(notes), want) { + t.Errorf("opening.md lacks %q:\n%s", want, notes) + } + } + for _, leak := range []string{"MARK-F1", "MARK-F4", "MARK-F5", "MARK-F6", "ENDING-SECRET", "SPELLING-SHEET", "word word"} { + if strings.Contains(string(notes), leak) { + t.Errorf("opening.md leaks %q", leak) + } + } + if !strings.Contains(req.Prompt, "first\n 2 of 5 chapters") && !strings.Contains(req.Prompt, "first 2 of 5 chapters") { + t.Errorf("prompt doesn't state the bound:\n%s", req.Prompt) + } + + if res.Chapters != 2 || res.CheckedAgainst != CheckedTranscripts || res.Reused || res.CostUSD != 0.07 { + t.Fatalf("result = %+v", res) + } + var d model.Description + if err := decodeSidecarFile(FinalDescriptionPath(book.WorkDir), &d); err != nil { + t.Fatal(err) + } + if d.Work != "the-carol" || d.Text != cleanDescription || d.License != "CC-BY-SA-4.0" || + len(d.Sources) != 1 || d.Sources[0].Ref != "audiosilo-meta:work:the-carol" { + t.Fatalf("description = %+v", d) + } + // The spend sits on a describing stage run of the book, which the budgets sum. + runs, _ := db.ListStageRuns(context.Background(), book.ID) + if len(runs) != 1 || runs[0].Stage != DescribeStage || runs[0].Ok == nil || !*runs[0].Ok || runs[0].CostUSD != 0.07 { + t.Fatalf("runs = %+v", runs) + } + var m map[string]any + if err := json.Unmarshal(runs[0].Metrics, &m); err != nil || m["chapters"] != float64(2) || m["logical_chapters"] != float64(5) || + m["basis"] != "edges" || m["checked_against"] != CheckedTranscripts { + t.Fatalf("metrics = %s, %v", runs[0].Metrics, err) + } + if spent, _, _ := db.SumStageRunBudgetCost(context.Background(), book.ID); spent != 0.07 { + t.Fatalf("book spend = %v", spent) + } + if _, err := os.Stat(filepath.Join(book.WorkDir, chapterPositionsName)); err != nil { + t.Fatalf("positions not kept: %v", err) + } + + // Again: the description is reused, nothing is spent. + again, err := e.Describe(context.Background(), book.ID, false) + if err != nil || !again.Reused || again.CostUSD != 0 || again.Chapters != 2 || again.CheckedAgainst != CheckedTranscripts || + fake.count(DescribeStage) != 1 || again.GeneratedAt.IsZero() { + t.Fatalf("again = %+v, %v (calls %d)", again, err, fake.count(DescribeStage)) + } +} + +// The step runs on the model its stage name routes to (config's agent.claude +// "describing"), never the CLI's default. +func TestDescribeRoutesItsModel(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, _ int) (agent.Result, error) { + writeDescription(t, req, cleanDescription) + return agent.Result{}, nil + } + e, book, _ := describeBook(t, fake, 0) + e.agentModels.Claude[DescribeStage] = "haiku" + if _, err := e.Describe(context.Background(), book.ID, false); err != nil { + t.Fatal(err) + } + if req, ok := fake.lastRequest(DescribeStage); !ok || req.Model != "haiku" { + t.Fatalf("request = %+v", req) + } +} + +// Each describing run takes the next attempt number, so a second run never reuses +// (and wipes) the first one's staged dir; before, the first two were both attempt 1. +func TestDescribeRunsTakeTheNextAttempt(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, _ int) (agent.Result, error) { + writeDescription(t, req, "Too short.") + return agent.Result{}, nil + } + e, book, db := describeBook(t, fake, 0) + for range 2 { + var park *scheduler.ParkError + if _, err := e.Describe(context.Background(), book.ID, false); !errors.As(err, &park) { + t.Fatalf("err = %v", err) + } + } + runs, _ := db.ListStageRuns(context.Background(), book.ID) + if len(runs) != 2 || runs[0].Attempt != 1 || runs[1].Attempt != 2 { + t.Fatalf("runs = %+v", runs) + } + for _, dir := range []string{"describing-a01", "describing-a02"} { + if _, err := os.Stat(filepath.Join(book.WorkDir, agent.RunsDir, dir, openingNotesName)); err != nil { + t.Errorf("%s: %v", dir, err) + } + } +} + +// A copied run is a validation failure: the retry's prompt carries the overlap. +func TestDescribeNGramHitRetries(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + copied := "the quick brown fox jumps over the lazy dog near the river bank" + fake.act = func(_ *fakeRunner, req agent.Request, attempt int) (agent.Result, error) { + text := cleanDescription + if attempt == 1 { + text = cleanDescription + " It opens: " + copied + "." + } + writeDescription(t, req, text) + return agent.Result{}, nil + } + e, book, _ := describeBook(t, fake, 0) + if err := transcript.WriteText(filepath.Join(book.WorkDir, transcript.TextDir), 2, + strings.Repeat("filler ", 200)+copied+strings.Repeat(" filler", 200)); err != nil { + t.Fatal(err) + } + res, err := e.Describe(context.Background(), book.ID, false) + if err != nil { + t.Fatalf("Describe: %v", err) + } + if fake.count(DescribeStage) != 2 || res.CheckedAgainst != CheckedTranscripts { + t.Fatalf("calls %d, result %+v", fake.count(DescribeStage), res) + } + if p := fake.lastPrompt(DescribeStage); !strings.Contains(p, "copies the book word for word") || !strings.Contains(p, "quick brown fox") { + t.Fatalf("retry prompt lacks the overlap:\n%s", p) + } +} + +// A local fault in the no-verbatim check (a transcript that can't be read) is not +// the agent's to fix: the step takes its one answer and fails with a plain error, +// never re-prompting (and paying for) the agent, and keeps no description. +func TestDescribeCheckFaultIsNotTheAgents(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, _ int) (agent.Result, error) { + writeDescription(t, req, cleanDescription) + return agent.Result{}, nil + } + e, book, db := describeBook(t, fake, 0) + if err := os.Symlink(filepath.Join(t.TempDir(), "gone"), filepath.Join(book.WorkDir, transcript.TextDir, "ch099.txt")); err != nil { + t.Fatal(err) + } + _, err := e.Describe(context.Background(), book.ID, false) + var park *scheduler.ParkError + if err == nil || errors.As(err, &park) || !strings.Contains(err.Error(), "no-verbatim check could not run") || + fake.count(DescribeStage) != 1 { + t.Fatalf("err = %v, calls %d", err, fake.count(DescribeStage)) + } + if _, statErr := os.Stat(FinalDescriptionPath(book.WorkDir)); !os.IsNotExist(statErr) { + t.Fatal("a description the check couldn't measure was kept") + } + if runs, _ := db.ListStageRuns(context.Background(), book.ID); len(runs) != 1 || runs[0].Ok == nil || *runs[0].Ok { + t.Fatalf("runs = %+v", runs) + } +} + +// With every source layer gone (the retention purged the transcripts), the check +// is skipped and says so, and the opening still comes from the kept positions. +func TestDescribeAfterTheTranscriptsAreGone(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, _ int) (agent.Result, error) { + writeDescription(t, req, cleanDescription) + return agent.Result{}, nil + } + e, book, _ := describeBook(t, fake, 0) + live, err := e.openingSelection(book) // classifies and keeps the positions + if err != nil || live.Basis != "edges" || !reflect.DeepEqual(live.Files, []int{2, 3}) { + t.Fatalf("live = %+v, %v", live, err) + } + if err := os.RemoveAll(filepath.Join(book.WorkDir, transcript.TextDir)); err != nil { + t.Fatal(err) + } + saved, err := e.openingSelection(book) + if err != nil || saved.Basis != "saved" || !reflect.DeepEqual(saved.Files, live.Files) || saved.K != live.K { + t.Fatalf("saved = %+v, %v", saved, err) + } + res, err := e.Describe(context.Background(), book.ID, false) + if err != nil || res.CheckedAgainst != CheckedNone { + t.Fatalf("Describe = %+v, %v", res, err) + } + + // No kept positions either: file numbers, the logical count from the recaps + // (through chapter 3 here), so files 1-2 only (never past a story position). + if err := os.Remove(filepath.Join(book.WorkDir, chapterPositionsName)); err != nil { + t.Fatal(err) + } + files, err := e.openingSelection(book) + if err != nil || files.Basis != "files" || files.Logical != 3 || !reflect.DeepEqual(files.Files, []int{1, 2}) { + t.Fatalf("files = %+v, %v", files, err) + } +} + +// Validation exhausted, a budget park before any spend, and a rate limit that +// parks at once with NoWait: each comes back as the driver's park. +func TestDescribeParks(t *testing.T) { + t.Parallel() + t.Run("validation exhausted", func(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(_ *fakeRunner, req agent.Request, _ int) (agent.Result, error) { + writeDescription(t, req, "Too short.") + return agent.Result{}, nil + } + e, book, db := describeBook(t, fake, 0) + _, err := e.Describe(context.Background(), book.ID, false) + var park *scheduler.ParkError + if !errors.As(err, &park) || park.Code != state.ParkAgentValidationExhausted { + t.Fatalf("err = %v", err) + } + if _, statErr := os.Stat(FinalDescriptionPath(book.WorkDir)); !os.IsNotExist(statErr) { + t.Fatal("a failed description was kept") + } + runs, _ := db.ListStageRuns(context.Background(), book.ID) + if len(runs) != 1 || runs[0].Ok == nil || *runs[0].Ok { + t.Fatalf("runs = %+v", runs) + } + }) + t.Run("budget", func(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + e, book, db := describeBook(t, fake, 1) + ctx := context.Background() + id, _ := db.StartStageRun(ctx, book.ID, "fact_pass", 1) + _ = db.AddOpenStageRunUsage(ctx, book.ID, "fact_pass", "opus", 1, 1, 2) + _ = db.FinishStageRun(ctx, id, true, nil) + _, err := e.Describe(ctx, book.ID, false) + var park *scheduler.ParkError + if !errors.As(err, &park) || park.Code != state.ParkBudgetExceeded || fake.count(DescribeStage) != 0 { + t.Fatalf("err = %v, calls %d", err, fake.count(DescribeStage)) + } + // The (free) preparation ran first, so the positions are kept for a later + // try after the retention removed the transcripts. + if _, err := os.Stat(filepath.Join(book.WorkDir, chapterPositionsName)); err != nil { + t.Fatalf("a budget-parked call kept no positions: %v", err) + } + }) + t.Run("rate limit, no wait", func(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + fake.act = func(*fakeRunner, agent.Request, int) (agent.Result, error) { + return agent.Result{}, &agent.RateLimitError{Detail: "usage limit"} + } + e, book, _ := describeBook(t, fake, 0) + e.backoff = []time.Duration{time.Hour} // NoWait must not sleep it out + _, err := e.Describe(context.Background(), book.ID, true) + var park *scheduler.ParkError + if !errors.As(err, &park) || park.Code != state.ParkAgentRateLimited || park.RetryAfter.IsZero() || fake.count(DescribeStage) != 1 { + t.Fatalf("err = %v, calls %d", err, fake.count(DescribeStage)) + } + }) +} + +// Nothing to describe from: no characters.json, or no notes for the opening. +func TestDescribeNoNotes(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + e, book, _ := describeBook(t, fake, 0) + if err := os.RemoveAll(filepath.Join(book.WorkDir, spelling.FactsDir)); err != nil { + t.Fatal(err) + } + if _, err := e.Describe(context.Background(), book.ID, false); !errors.Is(err, ErrNoNotes) { + t.Fatalf("no facts: %v", err) + } + if err := os.RemoveAll(filepath.Join(book.WorkDir, sidecarsDir)); err != nil { + t.Fatal(err) + } + if _, err := e.Describe(context.Background(), book.ID, false); !errors.Is(err, ErrNoNotes) { + t.Fatalf("no characters.json: %v", err) + } + if fake.count(DescribeStage) != 0 { + t.Fatal("the agent ran with nothing to read") + } +} + +// A book of one story chapter gets no description: its opening is the whole book, +// ending included. Refused before any spend. +func TestDescribeRefusesASingleChapter(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + e, book, _ := describeBook(t, fake, 0) + // No transcripts and no kept positions: the file numbers, the logical count + // from the recaps, which run through chapter 1 only. + if err := os.RemoveAll(filepath.Join(book.WorkDir, transcript.TextDir)); err != nil { + t.Fatal(err) + } + chars := baseChars("the-carol") + chars.Sources = []model.Source{{Type: sourceTypeCommunity, Ref: "audiosilo-meta:work:the-carol"}} + recs := baseRecaps("the-carol") + recs.Recaps[0].Through.Chapter = 1 + seedWorkSidecars(t, book.WorkDir, chars, recs) + if _, err := e.Describe(context.Background(), book.ID, false); !errors.Is(err, ErrOneChapter) || fake.count(DescribeStage) != 0 { + t.Fatalf("err = %v, calls %d", err, fake.count(DescribeStage)) + } +} + +// A run whose characters.json can't carry a description (no sources) fails before +// any spend: no answer could fix it. +func TestDescribeRefusesAnUncomposableRun(t *testing.T) { + t.Parallel() + fake := newFakeRunner() + e, book, _ := describeBook(t, fake, 0) + chars := baseChars("the-carol") + chars.Sources = nil + seedWorkSidecars(t, book.WorkDir, chars, baseRecaps("the-carol")) + _, err := e.Describe(context.Background(), book.ID, false) + if err == nil || errors.Is(err, ErrNoNotes) || !strings.Contains(err.Error(), "can't carry a description") || fake.count(DescribeStage) != 0 { + t.Fatalf("err = %v, calls %d", err, fake.count(DescribeStage)) + } +} + +// Each voice and shape rule, allowed and denied. +func TestDescriptionProblems(t *testing.T) { + t.Parallel() + if p := descriptionProblems(cleanDescription); len(p) != 0 { + t.Fatalf("clean text refused: %v", p) + } + two := cleanDescription + "\n\nA second paragraph widens the frame to the clerk and his family at home." + if p := descriptionProblems(two); len(p) != 0 { + t.Fatalf("two paragraphs refused: %v", p) + } + // "Your" inside a word, a question mark inside a title, a hyphen: all fine. + fine := cleanDescription + " Its youthful clerk asks whether the work ever ends (\"Why?\"), in a well-known story." + if p := descriptionProblems(fine); len(p) != 0 { + t.Fatalf("allowed text refused: %v", p) + } + denied := map[string]string{ + "control character": cleanDescription + "\tTabbed.", + "em dash": strings.Replace(cleanDescription, ", including", " \u2014 including", 1), + "characters; it must be 200 to 1500": "Too short to say anything.", + "paragraphs": two + "\n\nAnd a third paragraph is one too many for a description.", + "addresses the reader": cleanDescription + " You will not forget it.", + "question mark": cleanDescription + " Can he change?", + "sales language": cleanDescription + " A gripping masterpiece.", + } + for want, text := range denied { + p := descriptionProblems(text) + if len(p) == 0 || !strings.Contains(strings.Join(p, "; "), want) { + t.Errorf("%q: problems %v, want one naming %q", want, p, want) + } + } + if p := descriptionProblems(strings.Repeat("word ", 400)); len(p) == 0 { + t.Error("an over-long text passed") + } + + // The book's own names (its title, its series) are not the description's + // voice: naming the book is allowed, the rules still hold around it. + for title, rule := range map[string]string{ + "Do Androids Dream of Electric Sheep?": "question mark", + "Me Before You": "addresses the reader", + "The Masterpiece": "sales language", + } { + named := title + " follows " + cleanDescription + if p := descriptionProblems(named); len(p) == 0 || !strings.Contains(strings.Join(p, "; "), rule) { + t.Errorf("%q without its names: problems %v, want one naming %q", title, p, rule) + } + if p := descriptionProblems(named, title, ""); len(p) != 0 { + t.Errorf("%q named in its own description: %v", title, p) + } + if p := descriptionProblems(named+" You will not forget it. Can he change?", title); len(p) < 2 { + t.Errorf("%q: the rules stopped holding beside the title: %v", title, p) + } + } + if p := descriptionProblems(cleanDescription+" The Gripping Hand is its sequel.", "", "The Gripping Hand"); len(p) != 0 { + t.Errorf("a series name refused: %v", p) + } + if p := descriptionProblems("Me Before You follows "+cleanDescription, "Me Before You: A Novel"); len(p) != 0 { + t.Errorf("a title without its subtitle refused: %v", p) + } + + // Word edges are Unicode's: a "you" inside "Xīyou" is no second person, one + // after an apostrophe still is. + if p := descriptionProblems(cleanDescription + " The clerk reads Xīyou Ji by the fire."); len(p) != 0 { + t.Errorf("a non-ASCII word holding \"you\" refused: %v", p) + } + if p := descriptionProblems(cleanDescription + " D'you think so."); len(p) == 0 { + t.Error("an elided \"you\" passed") + } + // Only a plain newline separates: the Unicode separators are refused, and a + // line holding a no-break space still breaks a paragraph. + if p := descriptionProblems(strings.Replace(cleanDescription, ". On", ".\u2029On", 1)); len(p) == 0 { + t.Error("a paragraph separator passed") + } + nbsp := two + "\n\u00a0\nAnd a third paragraph, after a line holding a no-break space, is one too many." + if p := descriptionProblems(nbsp); len(p) == 0 || !strings.Contains(strings.Join(p, "; "), "paragraphs") { + t.Errorf("three paragraphs (one break a no-break space) passed: %v", p) + } +} + +// The agent's answer is exactly {"text": ...}. +func TestReadDescriptionOut(t *testing.T) { + t.Parallel() + dir := t.TempDir() + write := func(s string) { + if err := os.WriteFile(filepath.Join(dir, descriptionFileName), []byte(s), 0o644); err != nil { //nolint:gosec // test artifact + t.Fatal(err) + } + } + write(`{"text": " A paragraph. "}`) + if text, err := readDescriptionOut(dir); err != nil || text != "A paragraph." { + t.Fatalf("allowed: %q, %v", text, err) + } + for _, bad := range []string{`{"text": "x", "title": "y"}`, `{"text": "x"} {}`, `{}`, `["x"]`, `not json`, `{"text": 3}`} { + write(bad) + if _, err := readDescriptionOut(dir); err == nil { + t.Errorf("%s was accepted", bad) + } + } + write(`{"text": "` + strings.Repeat("x", descriptionOutCap) + `"}`) + if _, err := readDescriptionOut(dir); err == nil { + t.Error("an oversized answer was accepted") + } +} + +// The composed file passes meta's description schema; a bad work slug doesn't. +func TestComposedDescriptionSchema(t *testing.T) { + t.Parallel() + chars := baseChars("the-carol") + raw, err := composedDescription(chars, cleanDescription) + if err != nil { + t.Fatal(err) + } + var d map[string]any + if err := json.Unmarshal(raw, &d); err != nil || d["work"] != "the-carol" || d["license"] != "CC-BY-SA-4.0" { + t.Fatalf("composed = %s, %v", raw, err) + } + if _, err := composedDescription(baseChars("Not A Slug"), cleanDescription); err == nil { + t.Fatal("a bad slug passed the schema") + } +} + +// The ebook path: chapters are the manifest's own, no intro, nothing kept. +func TestDescribeEbookOpening(t *testing.T) { + t.Parallel() + work := t.TempDir() + m := audio.Manifest{Source: "/x/b.epub", Style: audio.StyleEbook, ChapterCount: 12} + for i := 1; i <= 12; i++ { + m.Chapters = append(m.Chapters, audio.Chapter{Chapter: i}) + } + writeManifestStruct(t, work, m) + e := NewExecutor(Config{}) + sel, err := e.openingSelection(store.Book{WorkDir: work, Kind: string(state.KindEbook)}) + if err != nil || sel.Basis != "edges" || !reflect.DeepEqual(sel.Files, []int{1, 2, 3}) { + t.Fatalf("ebook = %+v, %v", sel, err) + } + if _, err := os.Stat(filepath.Join(work, chapterPositionsName)); !os.IsNotExist(err) { + t.Fatal("an ebook kept positions it can always derive") + } +} diff --git a/internal/pipeline/schema_drift_test.go b/internal/pipeline/schema_drift_test.go index 307780c..58d4ee7 100644 --- a/internal/pipeline/schema_drift_test.go +++ b/internal/pipeline/schema_drift_test.go @@ -29,6 +29,13 @@ func TestSidecarConstantsMatchUpstreamSchema(t *testing.T) { assertMaxLength(t, "recaps.in_short", topProps(t, recaps)["in_short"], capInShort) assertMaxLength(t, "recaps.ending", topProps(t, recaps)["ending"], capEnding) + // The description's bounds (both ends: the schema has a floor too). + descText := topProps(t, loadSchema(t, "schema/description.schema.json"))["text"] + assertMaxLength(t, "description.text", descText, descriptionMaxRunes) + if got, ok := asObj(t, descText)["minLength"].(float64); !ok || int(got) != descriptionMinRunes { + t.Errorf("description.text minLength = %v, local floor = %d", asObj(t, descText)["minLength"], descriptionMinRunes) + } + // Enums (schema enum <-> the local validRoles/validScopes sets). assertEnumSet(t, "characters.role", charProps["role"], validRoles) assertEnumSet(t, "recaps.scope", recapProps["scope"], validScopes) diff --git a/internal/pipeline/sidecars.go b/internal/pipeline/sidecars.go index aa678f1..a11bb83 100644 --- a/internal/pipeline/sidecars.go +++ b/internal/pipeline/sidecars.go @@ -81,6 +81,11 @@ func decodeSidecarFile(path string, v any) error { if err != nil { return err } + return decodeSidecarBytes(raw, v) +} + +// decodeSidecarBytes is decodeSidecarFile's strict decode, for bytes already read. +func decodeSidecarBytes(raw []byte, v any) error { dec := json.NewDecoder(bytes.NewReader(raw)) dec.DisallowUnknownFields() if err := dec.Decode(v); err != nil { diff --git a/internal/pipeline/sidecarschema.go b/internal/pipeline/sidecarschema.go index ee08bd9..ec2167a 100644 --- a/internal/pipeline/sidecarschema.go +++ b/internal/pipeline/sidecarschema.go @@ -14,15 +14,16 @@ import ( meta "github.com/kodestar/audiosilo-meta" ) -// sidecarSchemas compiles meta's embedded characters/recaps schemas (plus the -// common.schema.json they $ref) once per process, keyed by sidecar kind. Each is -// registered under its own `$id`, which is what their `$ref`s resolve against. +// sidecarSchemas compiles meta's embedded characters/recaps/description schemas +// (plus the common.schema.json they $ref) once per process, keyed by sidecar kind. +// Each is registered under its own `$id`, which is what their `$ref`s resolve +// against. var sidecarSchemas = sync.OnceValues(func() (map[string]*jsonschema.Schema, error) { c := jsonschema.NewCompiler() // Assert "format" as metacheck does, so this gate and the intake agree. c.AssertFormat() ids := map[string]string{} - for _, f := range []string{"common", "characters", "recaps"} { + for _, f := range []string{"common", "characters", "recaps", "description"} { name := f + ".schema.json" raw, err := meta.SchemaFS.ReadFile("schema/" + name) if err != nil { @@ -43,7 +44,7 @@ var sidecarSchemas = sync.OnceValues(func() (map[string]*jsonschema.Schema, erro ids[f] = id } out := map[string]*jsonschema.Schema{} - for _, k := range []string{"characters", "recaps"} { + for _, k := range []string{"characters", "recaps", "description"} { sch, err := c.Compile(ids[k]) if err != nil { return nil, fmt.Errorf("compile %s schema: %w", k, err) diff --git a/internal/pipeline/validate_stage.go b/internal/pipeline/validate_stage.go index fbf9f2c..dba0aaa 100644 --- a/internal/pipeline/validate_stage.go +++ b/internal/pipeline/validate_stage.go @@ -181,10 +181,30 @@ func ngramGate(charsPath, recapsPath string) (scannable, findings []string, err // both the transcripts-text/ and transcripts-corrected/ layers, per source file. Each // overlap is a finding naming the locus, the source layer, and the offending run. A // layer that does not exist (or holds no .txt files) is skipped; a genuine read -// failure inside the check is returned as an error. +// failure inside the check is returned as an error, and so is a check that found no +// layer at all. func ngramCheck(book store.Book, sidecars []string) ([]string, error) { + findings, checked, err := ngramScan(book, sidecars) + if err != nil { + return nil, err + } + // A check that measured nothing is not a clean check. Silently skipping every + // layer would report "no verbatim overlap" for a book whose source text was never + // compared - exactly the reassuring-but-empty result the copyright rule cannot + // afford, since nothing downstream re-runs it. + if len(checked) == 0 { + return nil, fmt.Errorf("no source text layer to check the sidecars against; the n-gram check would pass vacuously") + } + return findings, nil +} + +// ngramScan is ngramCheck's scan: the findings, and the labels of the layers it +// measured against (none when every layer is gone). The description step calls it +// directly, since a book whose source text is gone (the transcripts the retention +// purged; an ebook's text, deleted at done) still gets a description (its agent +// never saw it) and says it wasn't checked. +func ngramScan(book store.Book, sidecars []string) (findings, checked []string, err error) { workDir := book.WorkDir - var findings []string sources := []ngramSource{ {"transcripts-text", filepath.Join(workDir, transcript.TextDir)}, {"transcripts-corrected", filepath.Join(workDir, spelling.CorrectedDir)}, @@ -198,17 +218,16 @@ func ngramCheck(book store.Book, sidecars []string) ([]string, error) { // The directory comes from chapterTextDir, not a local join: that helper is // the single answer to "where is this book's final chapter text", and a second // spelling of it here would be the first place to drift. - sources = []ngramSource{{"book text", filepath.Join(workDir, chapterTextDir(book))}} + sources = []ngramSource{{CheckedBookText, filepath.Join(workDir, chapterTextDir(book))}} } - checked := 0 for _, src := range sources { if !hasTxtFiles(src.dir) { continue } - checked++ + checked = append(checked, src.label) hits, err := extract.NGram(src.dir, sidecars, ngramShingle) if err != nil { - return nil, fmt.Errorf("%s: %w", src.label, err) + return nil, nil, fmt.Errorf("%s: %w", src.label, err) } for _, h := range hits { findings = append(findings, fmt.Sprintf( @@ -216,14 +235,7 @@ func ngramCheck(book store.Book, sidecars []string) ([]string, error) { h.Words, h.Locus, filepath.Base(h.File), src.label, h.Text)) } } - // A check that measured nothing is not a clean check. Silently skipping every - // layer would report "no verbatim overlap" for a book whose source text was never - // compared - exactly the reassuring-but-empty result the copyright rule cannot - // afford, since nothing downstream re-runs it. - if checked == 0 { - return nil, fmt.Errorf("no source text layer to check the sidecars against; the n-gram check would pass vacuously") - } - return findings, nil + return findings, checked, nil } // hasTxtFiles reports whether dir exists and contains at least one .txt file (the diff --git a/internal/scheduler/control.go b/internal/scheduler/control.go index c271d63..8d5efab 100644 --- a/internal/scheduler/control.go +++ b/internal/scheduler/control.go @@ -6,6 +6,7 @@ import ( "log/slog" "os" "path/filepath" + "sync" "github.com/kodestar/audiosilo-bridge/internal/scratch" "github.com/kodestar/audiosilo-bridge/internal/state" @@ -314,6 +315,21 @@ func (s *Scheduler) reserve(id int64) bool { return true } +// Hold reserves a book for work outside the scheduler (the Bridge's description +// step reads a finished book's notes and writes in its work dir) through the same +// reservation PurgeScratch takes: while it is held no stage is dispatched for it, +// Delete and PurgeScratch see it busy, the timed and parked re-admits and the +// startup GC skip it, and a supervisor terminate finds no worker to stop. ok is +// false when the book is busy already (a running stage, or another reservation). +// release ends the hold; calling it again does nothing. +func (s *Scheduler) Hold(id int64) (release func(), ok bool) { + if !s.reserve(id) { + return func() {}, false + } + var once sync.Once + return func() { once.Do(func() { s.unreserve(id) }) }, true +} + // unreserve drops a reservation and wakes the dispatch loop, so a book made // dispatchable during the reservation (e.g. a Resume that landed mid-purge) is // picked up now that the slot is free. diff --git a/internal/scheduler/hold_test.go b/internal/scheduler/hold_test.go new file mode 100644 index 0000000..90552ef --- /dev/null +++ b/internal/scheduler/hold_test.go @@ -0,0 +1,70 @@ +package scheduler + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/kodestar/audiosilo-bridge/internal/state" +) + +// A hold reserves a book for work outside the scheduler: a second hold is refused, +// Delete and PurgeScratch see the book busy, no stage is dispatched for it, and +// once released (twice is harmless) the book is the scheduler's again. +func TestHold(t *testing.T) { + h := newHarness(t) + db := h.openDB(t) + ctx := context.Background() + done := h.addBook(t, db, "done-book", "", "") + _ = db.SetBookState(ctx, done.ID, string(state.Done), "", "", "") + queued := h.addBook(t, db, "queued-book", "", "") + + sched := New(db, h.hub, NewStubExecutor(time.Millisecond, 2*time.Millisecond), 2, h.workRoot, false) + release, ok := sched.Hold(done.ID) + if !ok { + t.Fatal("a free book's hold was refused") + } + if _, again := sched.Hold(done.ID); again { + t.Fatal("a held book was held twice") + } + if err := sched.Delete(ctx, done.ID); !errors.Is(err, ErrBusy) { + t.Fatalf("Delete of a held book = %v, want ErrBusy", err) + } + if err := sched.PurgeScratch(ctx, done.ID); !errors.Is(err, ErrBusy) { + t.Fatalf("PurgeScratch of a held book = %v, want ErrBusy", err) + } + + // A held queued book runs no stage until the hold ends. + releaseQueued, ok := sched.Hold(queued.ID) + if !ok { + t.Fatal("the queued book's hold was refused") + } + runCtx, cancel := context.WithCancel(ctx) + stopped := make(chan struct{}) + go func() { _ = sched.Start(runCtx); close(stopped) }() + defer func() { cancel(); <-stopped }() + sched.Notify() + time.Sleep(50 * time.Millisecond) + // (The waypoint promotion out of queued runs nothing.) + if runs, _ := db.ListStageRuns(ctx, queued.ID); len(runs) != 0 { + t.Fatalf("a held book ran %+v", runs) + } + releaseQueued() + deadline := time.Now().Add(5 * time.Second) + for { + if b, _ := db.GetBook(ctx, queued.ID); b.State == string(state.Done) { + break + } + if time.Now().After(deadline) { + t.Fatal("the released book never ran") + } + time.Sleep(5 * time.Millisecond) + } + + release() + release() + if err := sched.Delete(ctx, done.ID); err != nil { + t.Fatalf("Delete after the release = %v", err) + } +} diff --git a/internal/scheduler/reconcile_describe_test.go b/internal/scheduler/reconcile_describe_test.go new file mode 100644 index 0000000..6829315 --- /dev/null +++ b/internal/scheduler/reconcile_describe_test.go @@ -0,0 +1,48 @@ +package scheduler + +import ( + "context" + "testing" + + "github.com/kodestar/audiosilo-bridge/internal/state" +) + +// The Bridge's description step records "describing" runs on a finished (or +// contributing-parked) book: a stage outside the state machine, with no sentinel. +// Reconcile must not read one as a completed stage whose sentinel went missing and +// rewind the book to it (state.Order of an unknown stage is 0, before everything). +func TestReconcileIgnoresARunOutsideTheStateMachine(t *testing.T) { + h := newHarness(t) + db := h.openDB(t) + ctx := context.Background() + b := h.addBook(t, db, "described", "", "") + _ = db.SetBookState(ctx, b.ID, string(state.Contributing), string(state.StatusNeedsAttention), "needs a core record", string(state.ParkCoreNeeded)) + insID, _ := db.StartStageRun(ctx, b.ID, string(state.Inspecting), 1) + _ = db.FinishStageRun(ctx, insID, true, nil) + _ = WriteSentinel(b.WorkDir, string(state.Inspecting), StageResult{}) + descID, _ := db.StartStageRun(ctx, b.ID, "describing", 1) + _ = db.FinishStageRun(ctx, descID, true, nil) + + sched := New(db, h.hub, NewStubExecutor(0, 0), 2, h.workRoot, false) + sched.ctx = ctx + if err := sched.Reconcile(ctx); err != nil { + t.Fatalf("reconcile: %v", err) + } + cur, _ := db.GetBook(ctx, b.ID) + if cur.State != string(state.Contributing) || cur.ParkCode != string(state.ParkCoreNeeded) { + t.Fatalf("book moved to %q (%q), want it left at contributing", cur.State, cur.ParkCode) + } + if succ, _ := db.SucceededStages(ctx, b.ID); !succ["describing"] || !succ[string(state.Inspecting)] { + t.Fatalf("successes changed: %v", succ) + } + + // Denied counterpart: an engine stage's success with no sentinel still rewinds. + asrID, _ := db.StartStageRun(ctx, b.ID, string(state.ASR), 1) + _ = db.FinishStageRun(ctx, asrID, true, nil) + if err := sched.Reconcile(ctx); err != nil { + t.Fatalf("reconcile: %v", err) + } + if cur, _ := db.GetBook(ctx, b.ID); cur.State != string(state.ASR) { + t.Fatalf("rewound to %q, want asr", cur.State) + } +} diff --git a/internal/scheduler/scheduler.go b/internal/scheduler/scheduler.go index 98cb86a..53dbcc6 100644 --- a/internal/scheduler/scheduler.go +++ b/internal/scheduler/scheduler.go @@ -353,9 +353,19 @@ func (s *Scheduler) reconcileBook(ctx context.Context, b store.Book, succeeded m if state.IsTerminal(state.State(b.State)) { return nil } + // A run outside the state machine (the description step's "describing", called + // by the Bridge's jobs after the engine's stages) writes no sentinel and has no + // place in the order: rewinding to it would park the book in a state nothing + // runs, so only the machine's stages count. + var stages []string + for stage := range succeeded { + if state.IsStage(state.State(stage)) { + stages = append(stages, stage) + } + } var rewind string haveRewind := false - for stage := range succeeded { + for _, stage := range stages { if SentinelExists(b.WorkDir, stage) { continue } @@ -369,7 +379,7 @@ func (s *Scheduler) reconcileBook(ctx context.Context, b store.Book, succeeded m // Supersede the DB success of the rewind stage and every later completed stage so // their counts stay honest when the book re-advances (the rows - and their cost - // are preserved, only their scheduling success is retired). - for stage := range succeeded { + for _, stage := range stages { if state.Order(state.State(stage)) >= state.Order(state.State(rewind)) { if err := s.db.SupersedeStageSuccesses(ctx, b.ID, stage); err != nil { return err @@ -756,11 +766,11 @@ func (s *Scheduler) runStage(ctx context.Context, b store.Book) { // An unreadable sentinel falls through to a fresh execution below. } - n, err := s.db.CountStageRuns(ctx, b.ID, stageName) + attempt, err := s.db.NextStageAttempt(ctx, b.ID, stageName) if err != nil { return } - runID, err := s.db.StartStageRun(ctx, b.ID, stageName, n+1) + runID, err := s.db.StartStageRun(ctx, b.ID, stageName, attempt) if err != nil { return } diff --git a/internal/state/state.go b/internal/state/state.go index 2c3bed0..3d6a453 100644 --- a/internal/state/state.go +++ b/internal/state/state.go @@ -91,6 +91,17 @@ const ( Done State = "done" ) +// Describing names the stage runs of the Bridge's description step (Phase 1c). +// It is NOT a state of this machine: it has no table row (IsStage, IsAgent and the +// other predicates are false, Order is 0), since the step runs on a finished book +// outside the scheduler. It is declared here so the packages that meet it (the +// pipeline that runs it, config's model maps that route it) share one spelling. +const Describing State = "describing" + +// RoutesModel reports whether a model map (agent.claude, agent.openai) may route +// s: every agent stage, and the description step, an agent run outside the machine. +func RoutesModel(s State) bool { return IsAgent(s) || s == Describing } + // Lane is the executor pool a stage runs in. The three lanes have different // resource profiles (GPU-bound ASR, rate-limited agents, cheap mechanical work) // and independent capacities, so they run concurrently. diff --git a/internal/store/generatebooks.go b/internal/store/generatebooks.go index fea114d..6145b04 100644 --- a/internal/store/generatebooks.go +++ b/internal/store/generatebooks.go @@ -112,6 +112,22 @@ func (db *DB) OpenGenerateJobsForBook(ctx context.Context, bookID, exceptJob int return out, rows.Err() } +// LatestGenerateBookAt returns the engine book of the newest sidecars.generate +// job for the book at (libraryID, relPath) whose book still exists (the path the +// job last followed it to), or ErrNotFound. sidecars.describe finds the run whose +// notes it reads with it when the book's source path no longer leads there. +func (db *DB) LatestGenerateBookAt(ctx context.Context, libraryID int64, relPath string) (int64, error) { + var id int64 + err := db.sql.QueryRowContext(ctx, + `SELECT book_id FROM generate_books + WHERE library_id = ? AND rel_path = ? AND book_id IS NOT NULL + ORDER BY created_at DESC, server_job_id DESC LIMIT 1`, libraryID, relPath).Scan(&id) + if errors.Is(err, sql.ErrNoRows) { + return 0, ErrNotFound + } + return id, err +} + // SweepBook is an engine book whose transcripts the retention sweep may delete: // only what the sweep needs. type SweepBook struct { diff --git a/internal/store/runs.go b/internal/store/runs.go index e1b5753..c91f8a2 100644 --- a/internal/store/runs.go +++ b/internal/store/runs.go @@ -4,6 +4,7 @@ import ( "context" "database/sql" "encoding/json" + "errors" "fmt" ) @@ -233,6 +234,28 @@ func (db *DB) SumStageRunBudgetCost(ctx context.Context, bookID int64) (cost flo return cost, unknown == 0, err } +// StageRunBudgetCost returns one stage run's spend on the budget basis (the +// description step reports what its own run cost). ErrNotFound for no such run. +func (db *DB) StageRunBudgetCost(ctx context.Context, runID int64) (float64, error) { + var cost float64 + err := db.sql.QueryRowContext(ctx, `SELECT `+budgetCostSQL+` FROM stage_runs WHERE id=?`, runID).Scan(&cost) + if errors.Is(err, sql.ErrNoRows) { + return 0, ErrNotFound + } + return cost, err +} + +// LatestOkStageRun returns the book's newest successful run of stage, or +// ErrNotFound when there is none. +func (db *DB) LatestOkStageRun(ctx context.Context, bookID int64, stage string) (StageRun, error) { + r, err := scanRun(db.sql.QueryRowContext(ctx, + `SELECT `+runCols+` FROM stage_runs WHERE book_id=? AND stage=? AND ok=1 ORDER BY id DESC LIMIT 1`, bookID, stage)) + if errors.Is(err, sql.ErrNoRows) { + return StageRun{}, ErrNotFound + } + return r, err +} + // StageRunStarts returns each book's earliest stage-run start (MIN(started_at)) // bucketed by book id, in ONE grouped query so the book-list endpoint attaches a // per-book "started at" without an N+1. started_at is the store's fixed-width UTC @@ -282,6 +305,14 @@ func (db *DB) CountStageRuns(ctx context.Context, bookID int64, stage string) (i return n, err } +// NextStageAttempt is the attempt number a new run of stage on a book takes: one +// past every run already recorded, so two runs never share one (an agent stage's +// staging dir is named by it, and a new one wipes a dir of the same name). +func (db *DB) NextStageAttempt(ctx context.Context, bookID int64, stage string) (int, error) { + n, err := db.CountStageRuns(ctx, bookID, stage) + return n + 1, err +} + // CountStageSuccesses returns how many runs of stage completed ok AND are not // superseded for a book. It is a SCHEDULING reader (round counters, the fix-loop // count): a Retry marks a stage's successes superseded to reset the count, so a diff --git a/internal/store/store_test.go b/internal/store/store_test.go index 6181c33..4cbabee 100644 --- a/internal/store/store_test.go +++ b/internal/store/store_test.go @@ -1106,3 +1106,30 @@ func TestListBookTrackingCarriesWorkID(t *testing.T) { t.Errorf("unresolved work id = %q, want empty", got) } } + +// The next attempt is one past every recorded run (open, failed and done alike), +// and the latest ok run is the newest success of that stage only. +func TestNextStageAttemptAndLatestOkStageRun(t *testing.T) { + db := open(t) + ctx := context.Background() + b, _ := db.CreateBook(ctx, NewBook{SourcePath: "/x", WorkDir: "/w", Title: "X"}) + if n, err := db.NextStageAttempt(ctx, b.ID, "describing"); err != nil || n != 1 { + t.Fatalf("first attempt = %d, %v", n, err) + } + if _, err := db.LatestOkStageRun(ctx, b.ID, "describing"); !errors.Is(err, ErrNotFound) { + t.Fatalf("no runs = %v", err) + } + first, _ := db.StartStageRun(ctx, b.ID, "describing", 1) + _ = db.FinishStageRun(ctx, first, true, json.RawMessage(`{"chapters":2}`)) + failed, _ := db.StartStageRun(ctx, b.ID, "describing", 2) + _ = db.FinishStageRun(ctx, failed, false, nil) + other, _ := db.StartStageRun(ctx, b.ID, "asr", 1) + _ = db.FinishStageRun(ctx, other, true, nil) + if n, _ := db.NextStageAttempt(ctx, b.ID, "describing"); n != 3 { + t.Fatalf("next attempt = %d, want 3", n) + } + r, err := db.LatestOkStageRun(ctx, b.ID, "describing") + if err != nil || r.ID != first || string(r.Metrics) != `{"chapters":2}` { + t.Fatalf("latest ok = %+v, %v", r, err) + } +} diff --git a/internal/supervisor/describing_test.go b/internal/supervisor/describing_test.go new file mode 100644 index 0000000..28ea9b4 --- /dev/null +++ b/internal/supervisor/describing_test.go @@ -0,0 +1,59 @@ +package supervisor + +import ( + "context" + "testing" + "time" + + "github.com/kodestar/audiosilo-bridge/internal/config" + "github.com/kodestar/audiosilo-bridge/internal/pricing" + "github.com/kodestar/audiosilo-bridge/internal/store" +) + +// The Bridge's description step runs on a finished book outside the scheduler: an +// open "describing" run has no scheduler worker, and a finished one no sentinel. +// The supervisor recovers engine stages only, so its run reads drop them and +// neither is an incident; the same open run under an engine stage still is. +func TestDescribingRunsAreNotIncidents(t *testing.T) { + t.Parallel() + ok := true + old := "2026-10-16T08:00:00.000000000Z" + runs := []store.StageRun{ + {ID: 1, Stage: "describing", StartedAt: old, FinishedAt: old, Ok: &ok}, + {ID: 2, Stage: "describing", StartedAt: old, HeartbeatAt: old, ProgressAt: old}, + } + db := supervisorDB(t) + s := New(db, config.Default().Supervisor, pricing.Table{Version: "test"}, nil, Hooks{}) + pass := passReads{runtime: Runtime{ActiveBooks: map[int64]bool{}}, runtimeAt: time.Now().UTC()} + book := store.Book{ID: 7, State: "done", WorkDir: t.TempDir()} + snap := s.buildSnapshot(book, engineRuns(runs), pass) + if len(snap.Runs) != 0 || len(snap.Artifacts) != 0 { + t.Fatalf("snapshot kept describing runs: runs %+v, artifacts %+v", snap.Runs, snap.Artifacts) + } + if got := Classify(snap, s.policy); len(got) != 0 { + t.Fatalf("describing runs classified as %+v", got) + } + + engine := []store.StageRun{{ID: 3, Stage: "fact_pass", StartedAt: old, HeartbeatAt: old, ProgressAt: old}} + if kept := engineRuns(engine); &kept[0] != &engine[0] { + t.Fatal("engineRuns copied a slice with nothing to drop") + } + if got := Classify(s.buildSnapshot(store.Book{ID: 8, State: "fact_pass", WorkDir: t.TempDir()}, engine, pass), s.policy); len(got) == 0 { + t.Fatal("an orphaned engine run was not an incident") + } + + // The reads drop them at the boundary, one book's and every book's. + ctx := context.Background() + b, err := db.CreateBook(ctx, store.NewBook{SourcePath: "/x", WorkDir: t.TempDir(), Title: "X"}) + if err != nil { + t.Fatal(err) + } + _, _ = db.StartStageRun(ctx, b.ID, "fact_pass", 1) + _, _ = db.StartStageRun(ctx, b.ID, "describing", 1) + if got, err := s.engineRunsOf(ctx, b.ID); err != nil || len(got) != 1 || got[0].Stage != "fact_pass" { + t.Fatalf("engineRunsOf = %+v, %v", got, err) + } + if all, err := s.engineRunsAll(ctx); err != nil || len(all[b.ID]) != 1 || all[b.ID][0].Stage != "fact_pass" { + t.Fatalf("engineRunsAll = %+v, %v", all, err) + } +} diff --git a/internal/supervisor/service.go b/internal/supervisor/service.go index b449dae..8772e37 100644 --- a/internal/supervisor/service.go +++ b/internal/supervisor/service.go @@ -7,6 +7,7 @@ import ( "fmt" "os" "path/filepath" + "slices" "sync" "time" @@ -14,6 +15,7 @@ import ( "github.com/kodestar/audiosilo-bridge/internal/config" "github.com/kodestar/audiosilo-bridge/internal/pricing" "github.com/kodestar/audiosilo-bridge/internal/scheduler" + "github.com/kodestar/audiosilo-bridge/internal/state" "github.com/kodestar/audiosilo-bridge/internal/store" ) @@ -152,7 +154,7 @@ func (s *Service) check(ctx context.Context, trigger string) { s.setCheck(err) return } - runsByBook, err := s.db.StageRunsAll(ctx) + runsByBook, err := s.engineRunsAll(ctx) if err != nil { s.setCheck(err) return @@ -251,10 +253,11 @@ func (s *Service) buildSnapshot(book store.Book, runs []store.StageRun, pass pas AgentInvocations: pass.runtime.AgentInvocations, InvocationCapacity: pass.runtime.InvocationCapacity, BookInvocations: pass.runtime.InvocationsByBook[book.ID], MaxAgentsPerBook: pass.runtime.MaxAgentsPerBook, RemainingWorkUnits: remaining} for i := range runs { - if runs[i].FinishedAt != "" { + run := &runs[i] // the roll-up below writes through to snap.Runs + if run.FinishedAt != "" { continue } - children := pass.invocationsByRun[runs[i].ID] + children := pass.invocationsByRun[run.ID] if len(children) > 0 { oldestHeartbeat := children[0].HeartbeatAt for _, child := range children[1:] { @@ -265,7 +268,7 @@ func (s *Service) buildSnapshot(book store.Book, runs []store.StageRun, pass pas // A fresh sibling must not hide a stale child behind the compatible // parent roll-up. Classification therefore uses the oldest active // invocation heartbeat for this fanned-out run. - runs[i].HeartbeatAt = oldestHeartbeat + run.HeartbeatAt = oldestHeartbeat alive := true checked := false for _, child := range children { @@ -287,8 +290,8 @@ func (s *Service) buildSnapshot(book store.Book, runs []store.StageRun, pass pas break } // Legacy/pre-migration active process fallback. - if runs[i].ProcessActive && runs[i].ProcessID > 0 { - alive := processAlive(runs[i].ProcessID) + if run.ProcessActive && run.ProcessID > 0 { + alive := processAlive(run.ProcessID) snap.ProcessAlive = &alive break } @@ -305,13 +308,38 @@ func (s *Service) freshSnapshot(ctx context.Context, bookID int64, pass passRead if err != nil { return Snapshot{}, nil, err } - runs, err := s.db.ListStageRuns(ctx, bookID) + runs, err := s.engineRunsOf(ctx, bookID) if err != nil { return Snapshot{}, nil, err } return s.buildSnapshot(book, runs, pass), runs, nil } +// engineRunsOf and engineRunsAll are the supervisor's reads of the stage runs +// (one book's, every book's), engine stages only: see engineRuns. +func (s *Service) engineRunsOf(ctx context.Context, bookID int64) ([]store.StageRun, error) { + runs, err := s.db.ListStageRuns(ctx, bookID) + return engineRuns(runs), err +} + +func (s *Service) engineRunsAll(ctx context.Context) (map[int64][]store.StageRun, error) { + byBook, err := s.db.StageRunsAll(ctx) + for id, runs := range byBook { + byBook[id] = engineRuns(runs) + } + return byBook, err +} + +// engineRuns drops the runs of a stage outside the engine's state machine (the +// description step's "describing", which the Bridge's jobs run on a finished book): +// the supervisor recovers engine stages only, and such a run, open with no +// scheduler worker or done with no sentinel, would read as a broken one. It +// filters in place (each read's slice is its own) and returns runs as it is when +// nothing is dropped. +func engineRuns(runs []store.StageRun) []store.StageRun { + return slices.DeleteFunc(runs, func(r store.StageRun) bool { return !state.IsStage(state.State(r.Stage)) }) +} + func primaryIncident(incidents []Incident) (Incident, bool) { if len(incidents) == 0 { return Incident{}, false @@ -689,7 +717,7 @@ func (s *Service) Ask(ctx context.Context, bookID int64) (store.SupervisorRun, e if err != nil { return store.SupervisorRun{}, err } - runs, err := s.db.ListStageRuns(ctx, bookID) + runs, err := s.engineRunsOf(ctx, bookID) if err != nil { return store.SupervisorRun{}, err } diff --git a/web/src/api/types.ts b/web/src/api/types.ts index c64e90b..b3f9739 100644 --- a/web/src/api/types.ts +++ b/web/src/api/types.ts @@ -122,7 +122,8 @@ export interface Park { export type GenerateSource = 'audio' | 'ebook'; // jobs.Progress, with the fields a kind adds beside them (Extra; sidecars.generate -// sends stage, eta_seconds, cost_usd, source and, while parked, park). +// sends stage, eta_seconds, cost_usd, source and, while parked, park; +// sidecars.describe the same but eta_seconds, its stage "describing"). export interface Progress { step?: string; message?: string; @@ -151,6 +152,25 @@ export interface GenerateResult { work_id: string; // Intake issue links, with contribute = issue only. contributed?: Record; + // Whether description.json was written with the others (absent from a Bridge + // before Phase 1c); when it wasn't, why, in a few plain words. + description?: boolean; + description_error?: string; +} + +// What a description's no-verbatim check measured against ("none": the source +// text was already removed: an audio book's transcripts, or an ebook's text). +export type CheckedAgainst = 'transcripts' | 'book text' | 'none'; + +// generate.DescribeJobResult (a sidecars.describe job's result). +export interface DescribeResult { + library_id: number; + path: string; + description: boolean; + // The names written in the book's .audiosilo/ folder. + files: string[]; + cost_usd: number; + checked_against: CheckedAgainst; } // store.ServerJob diff --git a/web/src/components/tabs/JobsTab.test.tsx b/web/src/components/tabs/JobsTab.test.tsx index 3a523e4..1470416 100644 --- a/web/src/components/tabs/JobsTab.test.tsx +++ b/web/src/components/tabs/JobsTab.test.tsx @@ -4,7 +4,7 @@ import userEvent from '@testing-library/user-event'; import { JobsTab } from './JobsTab'; import { ApiClient } from '@/api/client'; import { lastCall, mockFetch } from '@/test/fetchMock'; -import { generateJobFixture, jobFixture } from '@/test/fixtures'; +import { describeJobFixture, generateJobFixture, jobFixture } from '@/test/fixtures'; const api = () => new ApiClient({ getToken: () => 't', onSignedOut: vi.fn() }); @@ -178,6 +178,8 @@ describe('JobsTab', () => { contributed: { characters: 'https://github.com/KodeStar/audiosilo-meta-community/issues/12', }, + description: false, + description_error: 'the agent account hit its usage limit', }, }), ], @@ -192,6 +194,49 @@ describe('JobsTab', () => { ).toBeInTheDocument(); expect(within(card).getByText(/characters.json, recaps.json, bridge.json/)).toBeInTheDocument(); expect(within(card).getByRole('link', { name: /issues\/12/ })).toBeInTheDocument(); + expect( + within(card).getByText('No description: the agent account hit its usage limit'), + ).toBeInTheDocument(); + }); + + it('shows a description job running, then its result', async () => { + mockFetch({ 'GET /api/jobs': { body: { jobs: [describeJobFixture()] } } }); + const { unmount } = render(); + const card = await screen.findByRole('article', { name: 'Job 14' }); + expect(within(card).getByText('Description')).toBeInTheDocument(); + expect(within(card).getByText('Writing a description')).toBeInTheDocument(); + expect(within(card).getByText(/\$0.04 spent/)).toBeInTheDocument(); + unmount(); + + mockFetch({ + 'GET /api/jobs': { + body: { + jobs: [ + describeJobFixture({ + state: 'done', + finished_at: '2026-10-16T12:00:00Z', + result: { + library_id: 1, + path: 'Author/Book', + description: true, + files: ['description.json', 'bridge.json'], + cost_usd: 0.25, + checked_against: 'none', + }, + }), + ], + }, + }, + }); + render(); + const done = await screen.findByRole('article', { name: 'Job 14' }); + await userEvent.click(within(done).getByText('Details')); + expect( + within(done).getByText( + 'A description, $0.25, not checked: the source text was already removed', + ), + ).toBeInTheDocument(); + expect(within(done).getByText(/description.json, bridge.json/)).toBeInTheDocument(); }); it("opens one job from the console's link", async () => { diff --git a/web/src/components/tabs/JobsTab.tsx b/web/src/components/tabs/JobsTab.tsx index a4b4e96..2e7ac76 100644 --- a/web/src/components/tabs/JobsTab.tsx +++ b/web/src/components/tabs/JobsTab.tsx @@ -1,6 +1,6 @@ import { useState } from 'react'; import { ApiError, errorText, type ApiClient } from '@/api/client'; -import type { GenerateResult, ServerJob, VerifyResult } from '@/api/types'; +import type { DescribeResult, GenerateResult, ServerJob, VerifyResult } from '@/api/types'; import { useLoad } from '@/lib/useLoad'; import { bytes, elapsed } from '@/lib/time'; import { @@ -19,10 +19,16 @@ import { import { VERIFY_KIND, parseVerifyResult, prettyJSON } from '@/lib/verify'; import { canRetry, + descriptionText, + describeSummary, effectiveState, generateFacts, + isDescribe, isGenerate, + isSidecars, + kindLabel, parkOf, + parseDescribeResult, parseGenerateResult, resultSummary, } from '@/lib/generate'; @@ -164,9 +170,10 @@ function JobCard({ }) { const [busy, setBusy] = useState(false); const [note, setNote] = useState<{ kind: 'error' | 'info'; text: string } | null>(null); - const generate = isGenerate(job); + const sidecars = isSidecars(job); + const label = kindLabel(job.kind); const shown = effectiveState(job); - const progress = generate ? '' : progressText(job.progress); + const progress = sidecars ? '' : progressText(job.progress); const percent = progressPercent(job.progress); const report = reportText(job); const hint = stateHint(shown); @@ -210,6 +217,7 @@ function JobCard({ #{job.id} + {label && {label}} {job.kind} {stateLabel(shown)} @@ -237,7 +245,7 @@ function JobCard({ {target &&

{target}

} - {generate && job.state === 'running' && } + {sidecars && job.state === 'running' && } {job.state === 'running' && progress && (
@@ -315,6 +323,8 @@ function ResultView({ job }: { job: ServerJob }) { if (verify) return ; const generated = isGenerate(job) ? parseGenerateResult(job.result) : null; if (generated) return ; + const described = isDescribe(job) ? parseDescribeResult(job.result) : null; + if (described) return ; return (
       {prettyJSON(job.result)}
@@ -324,13 +334,12 @@ function ResultView({ job }: { job: ServerJob }) {
 
 function GenerateView({ result }: { result: GenerateResult }) {
   const links = Object.entries(result.contributed ?? {});
+  const description = descriptionText(result);
   return (
     

{resultSummary(result)}

-

- Saved with {result.path} in its{' '} - .audiosilo folder: {result.files.join(', ')} -

+ {description &&

{description}

} + {result.work_id && (

Work {result.work_id} @@ -352,6 +361,25 @@ function GenerateView({ result }: { result: GenerateResult }) { ); } +function DescribeView({ result }: { result: DescribeResult }) { + return ( +

+

{describeSummary(result)}

+ +
+ ); +} + +// SavedWith says where a sidecars job saved its files. +function SavedWith({ path, files }: { path: string; files: string[] }) { + return ( +

+ Saved with {path} in its .audiosilo{' '} + folder: {files.join(', ')} +

+ ); +} + function VerifyView({ result }: { result: VerifyResult }) { return (
diff --git a/web/src/lib/generate.test.ts b/web/src/lib/generate.test.ts index b952315..79beb9d 100644 --- a/web/src/lib/generate.test.ts +++ b/web/src/lib/generate.test.ts @@ -1,16 +1,22 @@ import { describe, expect, it } from 'vitest'; import { canRetry, + checkedText, + describeSummary, + descriptionText, effectiveState, etaText, generateFacts, + isSidecars, + kindLabel, parkOf, + parseDescribeResult, parseGenerateResult, resultSummary, sourceText, usd, } from './generate'; -import { generateJobFixture, jobFixture } from '@/test/fixtures'; +import { describeJobFixture, generateJobFixture, jobFixture } from '@/test/fixtures'; const parked = () => generateJobFixture({ @@ -98,3 +104,55 @@ describe('parseGenerateResult', () => { expect(parseGenerateResult('x')).toBeNull(); }); }); + +describe('descriptions', () => { + it('labels the kinds and retries a parked description job', () => { + expect(kindLabel('sidecars.describe')).toBe('Description'); + expect(kindLabel('sidecars.generate')).toBe('Characters and recaps'); + expect(kindLabel('bridge.verify')).toBe(''); + expect(kindLabel('constructor')).toBe(''); + expect(kindLabel('__proto__')).toBe(''); + expect(isSidecars(describeJobFixture())).toBe(true); + expect(isSidecars(jobFixture())).toBe(false); + expect(canRetry(describeJobFixture())).toBe(false); + const parked = describeJobFixture({ + progress: { step: 'describing', park: { code: 'budget_exceeded', hint: 'Raise it.' } }, + }); + expect(canRetry(parked)).toBe(true); + }); + it("words a generate result's description outcome", () => { + const base = parseGenerateResult({ path: 'A/B', files: [] })!; + expect(descriptionText(base)).toBe(''); + expect(descriptionText({ ...base, description: true })).toBe('With a description'); + expect( + descriptionText( + parseGenerateResult({ + path: 'A/B', + files: [], + description: false, + description_error: 'the agent budget is spent', + })!, + ), + ).toBe('No description: the agent budget is spent'); + expect(descriptionText({ ...base, description: false })).toBe('No description'); + }); + it('reads and words a description job result', () => { + const r = parseDescribeResult({ + library_id: 1, + path: 'Author/Book', + description: true, + files: ['description.json', 'bridge.json', 2], + cost_usd: 0.25, + checked_against: 'transcripts', + }); + expect(r?.files).toEqual(['description.json', 'bridge.json']); + expect(describeSummary(r!)).toBe('A description, $0.25, checked against the transcripts'); + expect( + parseDescribeResult({ path: 'x', files: [], checked_against: 'nope' })?.checked_against, + ).toBe('none'); + expect(checkedText('book text')).toBe('checked against the book text'); + expect(checkedText('none')).toBe('not checked: the source text was already removed'); + expect(parseDescribeResult(null)).toBeNull(); + expect(parseDescribeResult({ files: [] })).toBeNull(); + }); +}); diff --git a/web/src/lib/generate.ts b/web/src/lib/generate.ts index 48125d9..e5efeab 100644 --- a/web/src/lib/generate.ts +++ b/web/src/lib/generate.ts @@ -1,13 +1,42 @@ -import type { GenerateResult, Park, Progress, ServerJob } from '@/api/types'; +import type { + CheckedAgainst, + DescribeResult, + GenerateResult, + Park, + Progress, + ServerJob, +} from '@/api/types'; // The sidecars.generate job (internal/generate): its progress carries the // engine's stage, ETA, spend, source and park beside the step and the count. export const GENERATE_KIND = 'sidecars.generate'; +// The description-only job: the same progress, without an ETA. +export const DESCRIBE_KIND = 'sidecars.describe'; export function isGenerate(job: ServerJob): boolean { return job.kind === GENERATE_KIND; } +export function isDescribe(job: ServerJob): boolean { + return job.kind === DESCRIBE_KIND; +} + +// isSidecars: a job that follows the engine (its progress and parks look alike). +export function isSidecars(job: ServerJob): boolean { + return isGenerate(job) || isDescribe(job); +} + +const KIND_LABELS: Record = { + [GENERATE_KIND]: 'Characters and recaps', + [DESCRIBE_KIND]: 'Description', +}; + +// kindLabel words a job kind ("" for one with no label: the kind shows as it is). +// Own keys only: a kind like "constructor" must not find Object's. +export function kindLabel(kind: string): string { + return Object.hasOwn(KIND_LABELS, kind) ? KIND_LABELS[kind] : ''; +} + // parkOf is a running job's park (null when it isn't parked). The Bridge keeps // a parked job running under its claim, so the park lives in its progress. export function parkOf(job: ServerJob): Park | null { @@ -24,7 +53,7 @@ export function effectiveState(job: ServerJob): string { // canRetry: a parked sidecars job can be retried from this page. export function canRetry(job: ServerJob): boolean { - return isGenerate(job) && parkOf(job) !== null; + return isSidecars(job) && parkOf(job) !== null; } // etaText words the engine's ETA: "under a minute", "about 25 min", @@ -72,20 +101,35 @@ export function generateFacts(p: Progress | null | undefined, parked: boolean): return facts; } -// parseGenerateResult reads a done sidecars job's result, or null when it isn't -// one. -export function parseGenerateResult(r: unknown): GenerateResult | null { +// SavedFields are what every sidecars job's result carries: the library, the +// book's path, the names written in its .audiosilo folder and the spend. +type SavedFields = Pick; + +// parseSavedFields reads a done sidecars job's shared fields, or null when the +// result isn't one. +function parseSavedFields(r: unknown): SavedFields | null { if (!r || typeof r !== 'object') return null; - const o = r as Partial; + const o = r as Partial; if (typeof o.path !== 'string' || !Array.isArray(o.files)) return null; return { library_id: typeof o.library_id === 'number' ? o.library_id : 0, path: o.path, - source: o.source === 'ebook' ? 'ebook' : 'audio', files: o.files.filter((f): f is string => typeof f === 'string'), + cost_usd: typeof o.cost_usd === 'number' ? o.cost_usd : 0, + }; +} + +// parseGenerateResult reads a done sidecars job's result, or null when it isn't +// one. +export function parseGenerateResult(r: unknown): GenerateResult | null { + const saved = parseSavedFields(r); + if (!saved) return null; + const o = r as Partial; + return { + ...saved, + source: o.source === 'ebook' ? 'ebook' : 'audio', characters: typeof o.characters === 'number' ? o.characters : 0, recaps: typeof o.recaps === 'number' ? o.recaps : 0, - cost_usd: typeof o.cost_usd === 'number' ? o.cost_usd : 0, work_id: typeof o.work_id === 'string' ? o.work_id : '', contributed: o.contributed && typeof o.contributed === 'object' @@ -95,6 +139,8 @@ export function parseGenerateResult(r: unknown): GenerateResult | null { ), ) : undefined, + description: typeof o.description === 'boolean' ? o.description : undefined, + description_error: typeof o.description_error === 'string' ? o.description_error : undefined, }; } @@ -104,3 +150,38 @@ export function resultSummary(r: GenerateResult): string { const what = `${plural(r.characters, 'character', 'characters')} and ${plural(r.recaps, 'recap', 'recaps')}`; return `${what} from ${sourceText(r.source)}, ${usd(r.cost_usd)}`; } + +// descriptionText words a generate result's description outcome ("" from a +// Bridge before descriptions). +export function descriptionText(r: GenerateResult): string { + if (r.description === true) return 'With a description'; + if (r.description === false) { + return r.description_error ? `No description: ${r.description_error}` : 'No description'; + } + return ''; +} + +// checkedText words what a description's no-verbatim check measured against. +export function checkedText(c: CheckedAgainst): string { + if (c === 'transcripts') return 'checked against the transcripts'; + if (c === 'book text') return 'checked against the book text'; + return 'not checked: the source text was already removed'; +} + +// parseDescribeResult reads a done description-only job's result, or null when +// it isn't one. +export function parseDescribeResult(r: unknown): DescribeResult | null { + const saved = parseSavedFields(r); + if (!saved) return null; + const o = r as Partial; + const checked: CheckedAgainst = + o.checked_against === 'transcripts' || o.checked_against === 'book text' + ? o.checked_against + : 'none'; + return { ...saved, description: o.description === true, checked_against: checked }; +} + +// describeSummary is "A description, $0.25, checked against the transcripts". +export function describeSummary(r: DescribeResult): string { + return `A description, ${usd(r.cost_usd)}, ${checkedText(r.checked_against)}`; +} diff --git a/web/src/test/fixtures.ts b/web/src/test/fixtures.ts index 25bc606..d1762d7 100644 --- a/web/src/test/fixtures.ts +++ b/web/src/test/fixtures.ts @@ -101,6 +101,23 @@ export function generateJobFixture(over: Partial = {}): ServerJob { }); } +export function describeJobFixture(over: Partial = {}): ServerJob { + return jobFixture({ + id: 14, + kind: 'sidecars.describe', + params: {}, + progress: { + step: 'describing', + message: 'Writing a description', + total: 1, + stage: 'describing', + cost_usd: 0.04, + source: 'audio', + }, + ...over, + }); +} + export function engineFixture(over: Partial = {}): Engine { return { started: true,