diff --git a/.llm/harness/workflow/activation.md b/.llm/harness/workflow/activation.md index ef0ee6420..ae2bb14bc 100644 --- a/.llm/harness/workflow/activation.md +++ b/.llm/harness/workflow/activation.md @@ -23,6 +23,10 @@ Harness mode activates when the user says `use harness` or explicitly asks for a 10. For a **seed run** (planning-only: the deliverable is a GitHub board — epics, milestones, issues, design packs — not code), read `workflow/seed-run.md` and follow its stage contracts A–I. Seed runs are drafts-only until owner ratification. +11. For a **research/RFC run** (evaluation-only: the deliverable is a verdict on an external + technology proposal — an evidence corpus plus an RFC or a documented rejection), read + `workflow/research-rfc-run.md` and follow its stage contracts A–H. Research/RFC runs are + drafts-only until owner ratification. ## Run ID diff --git a/.llm/harness/workflow/research-rfc-run.md b/.llm/harness/workflow/research-rfc-run.md new file mode 100644 index 000000000..dad82651e --- /dev/null +++ b/.llm/harness/workflow/research-rfc-run.md @@ -0,0 +1,115 @@ +# Research/RFC Run — Technology Evaluation → Verdict → RFC + +Operating protocol for **research/RFC runs** — evaluation-only harness runs that take an external +technology proposal ("should NetScript integrate X?") from zero to a **verdict-bearing research +corpus**: an evidence-grade codebase baseline, a primary-source external dossier, a compatibility +matrix against every backend/runtime NetScript must preserve, an ecosystem projection onto +NetScript's own surfaces, and — only when the verdict is positive — an RFC plus any enabling issue +drafts. Like a seed run, a research/RFC run is **drafts-only until owner ratification**; unlike a +seed run, its deliverable is a *decision*, not a board. + +> **Provenance.** Promoted from `copilot-evaluate-proposal-and-documentation--glidemq-rfc` +> (GlideMQ evaluation, 2026-07-09), which produced a conditional-positive verdict, a three-track +> RFC, and a benchmark-prerequisite issue draft. As with `seed-run.md`, this file freezes the +> **stage contracts**, not the exemplar's folder tree. + +## When to use + +The owner brings an external library/technology/architecture proposal and asks for an evaluation +whose outcome is unknown in advance ("either a clear explanation of why it doesn't fit, or a +complete RFC"). Triggers: a candidate dependency, a competing architecture, a "we could replace our +X with Y" idea. **Do not** use it when integration is already decided (that's run-loop planning) or +when the deliverable is a board (that's a seed run). A ratified positive RFC typically *feeds* a +seed run or run-loop issues; the two shapes chain, they don't merge. + +## Run layout + +Run dir `.llm/runs//` with the standard mandatory artifacts (`supervisor.md` first, then +`research.md`, `plan.md`, `worklog.md`, `context-pack.md`, `drift.md`) plus the shape-specific +artifact classes: + +- `research/` — the evidence corpus, free-form layout; the exemplar used four numbered files + (internal baseline / external dossier / compatibility matrix / ecosystem mapping). +- `rfc-.md` — only if the verdict is positive; carries its DRAFT/pending-ratification + status in the header. +- `issue-draft-*.md` — enabling issues (prerequisites the RFC depends on), taxonomy-complete + (labels + milestone per the netscript-pr skill) but **never filed by the run itself**. + +## Stage contracts + +### A — Activation & framing + +Write `supervisor.md` (lane table, blocked-eval overrides). Extract the proposal's **explicit +deliverables** into the run checklist — owners often embed side-deliverables inside the prose +(the exemplar carried three: RFC-or-rejection, a benchmark issue, and this workflow doc). Missing +one is a contract failure. + +### B — Internal baseline (parallel with C) + +Map every NetScript surface the candidate touches, with verified paths/exports: the ports it would +sit behind, the adapters it would coexist with, the plugins that consume the seam, the doctrine +constraints (archetype, thinness, wrap-don't-reinvent), and any **precedent** for the integration +shape (the exemplar's key find: the sagas Garnet list-transport proves the +"portable floor + capable ceiling" adapter policy). Delegate to a Tier-B explore agent, but the +supervisor re-verifies load-bearing claims by direct inspection before they enter `research.md`. + +### C — External dossier (parallel with B) + +Primary sources only: the candidate's repo docs, source layout, package manifest, changelog, +examples, ecosystem repos. Required axes — value proposition and mechanism; full API surface; +**server/runtime hard requirements** (the compatibility pivot); AI/domain-specific primitives; +observability; license, maturity, release cadence, **bus factor**, fork/vendoring risks. If the +docs site is unreachable from the sandbox, find the source-of-truth markdown in the repo (docs +sites are usually generated) and record the mapping in `research.md`. Save the dossier into +`research/` — sub-agent output is otherwise lost with the session. + +### D — Compatibility matrix (the verdict pivot) + +Cross B×C: for **every** backend/runtime NetScript supports, a per-cell verdict (full / degraded / +hard blocker) with cited primary evidence. Verify the two or three pivotal cells independently of +the sub-agents (the exemplar: Garnet's missing FUNCTION+Streams, Deno NAPI uncertainty — each +double-checked against upstream docs before becoming findings). This stage decides the verdict +shape: a hard blocker on a must-keep backend converts "replace" proposals into "adapter or reject". + +### E — Ecosystem projection + +Feature-by-feature mapping of the candidate onto NetScript surfaces, each row classified: +**A** adapter concern (free with integration), **P** port-level concept NetScript should own +regardless of backend, **R** design reference only. The P and R rows are the run's durable value — +they survive even a negative verdict, and they connect the evaluation to open epics/issues (cite +issue numbers). + +### F — Verdict & RFC + +`research.md` states numbered, independently-verifiable findings + open questions (each marked +safe-to-defer or must-resolve). The verdict must be explicit: fit / no-fit / **conditional** (with +named gates — the exemplar gated on a runtime spike and a benchmark baseline). Positive verdict → +RFC with: summary, motivation, the *non-fit half* (what the candidate must not become), design with +preservation guarantees for existing backends, risk register with containment (e.g. "no candidate +type on a public surface"), phasing **with kill-switches**, alternatives considered, and the open +questions carried to PLAN-EVAL. + +### G — Side deliverables & workflow review + +Author any issue drafts the RFC depends on (drafts-only, taxonomy-complete). If the run followed a +shape not yet codified under `workflow/`, document it there (this file is that step for the +exemplar). + +### H — Close + +Update `worklog.md`, `context-pack.md`, `drift.md` (record blocked evaluator launches and +unreachable sources). Hand off: the RFC's next step is a **separate-session PLAN-EVAL** treating +the RFC as the plan under review, then owner ratification, then seed-run or run-loop issues. + +## Invariants + +- **Drafts-only.** The run does not file issues, opens no non-run PRs, adds no dependencies. Ratification + gates everything downstream. +- **Generator ≠ evaluator.** The run that authored the RFC never certifies it; the RFC header names + its pending-PLAN-EVAL status, and blocked evaluator launches are recorded per `lane-policy.md`. +- **Primary sources or it didn't happen.** Marketing claims are checked against the candidate's own + internal docs (the exemplar caught README-claimed Deno support contradicted by the project's + HANDOVER.md) and upstream third-party docs for backend capabilities. +- **Preservation is explicit.** The RFC must state, backend by backend, what remains unchanged. +- **Findings are falsifiable.** Every finding in `research.md` names its source so a reviewer can + re-verify it without the session. diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/context-pack.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/context-pack.md new file mode 100644 index 000000000..d788ac7f6 --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/context-pack.md @@ -0,0 +1,37 @@ +# Context Pack — copilot-evaluate-proposal-and-documentation--glidemq-rfc + +**One-paragraph resume:** Evaluation of GlideMQ (avifenesh/glide-mq, v0.15.4) for NetScript. +Verdict: **conditional-positive** — GlideMQ hard-requires Valkey/Redis 7+ Functions + Streams, so +it can never replace the `MessageQueue` seam (Garnet has neither; Deno KV/AMQP/Postgres are out), +but it fits as (A) an opt-in adapter gated on a Deno/NAPI spike, (B) the blueprint for +NetScript-owned AI-execution ports (usage/streaming/suspend/budget/failover — the layer +`plugins/ai` is missing), and (C) design harvest for epics #399 (OTel) and #400 (dashboard). RFC +drafted; benchmark-reintroduction issue drafted as a hard prerequisite; both pending owner +ratification + separate-session PLAN-EVAL (evaluator lanes blocked in this sandbox). + +## Artifact index + +| Artifact | State | +| --- | --- | +| `research.md` + `research/01..04` | complete — findings F1–F9, OQ1–OQ4 | +| `rfc-glidemq.md` | DRAFT, pending PLAN-EVAL + ratification | +| `issue-draft-benchmark.md` | draft, not filed | +| `.llm/harness/workflow/research-rfc-run.md` | landed (workflow promoted from this run) | +| plan/worklog/drift/supervisor | current | + +## Key facts a resuming session must not re-derive + +- GlideMQ: 44 Lua functions via FUNCTION LOAD/FCALL, streams+XAUTOCLAIM consumer model, + `@glidemq/speedkey` Rust NAPI (personal valkey-glide fork), CJS, Node>=20, Apache-2.0, bus + factor 1, Deno NAPI support claimed in README but flagged untested in the project's HANDOVER.md. +- Garnet: Lua EVAL only (no FUNCTION), no Streams → permanent GlideMQ blocker. +- NetScript precedent for the adapter policy: sagas Garnet list-transport + (`packages/plugin-sagas-core/src/transports/`). +- `packages/bench` = agent self-bench, NOT a transport benchmark; legacy + `rickylabs/netscript-start/benchmark` is 404. + +## Next actions (owner) + +1. PLAN-EVAL the RFC (OpenHands minimax-M3, separate session). +2. File the benchmark issue. +3. Phase-0 spike: `npm:glide-mq` under Deno 2.9 + Valkey container (OQ1 kill-switch for Track A). diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/drift.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/drift.md new file mode 100644 index 000000000..b106ce6eb --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/drift.md @@ -0,0 +1,20 @@ +# Drift Log — copilot-evaluate-proposal-and-documentation--glidemq-rfc + +Append-only. + +- **2026-07-09 · blocked-lane · significant** — OpenHands PLAN-EVAL/IMPL-EVAL launch surfaces are + unavailable in the Copilot cloud-agent sandbox. Recorded per `lane-policy.md` blocked-lane + handling; fallback is drafts-only output + explicit pending-PLAN-EVAL status on the RFC. + Mirrored in `supervisor.md`. +- **2026-07-09 · source-unreachable · minor** — `https://glidemq.dev/*` is DNS-blocked in the + sandbox. Substituted the site's source-of-truth markdown in `avifenesh/glide-mq/docs/` + (site is generated from it); mapping recorded in `research.md`. +- **2026-07-09 · source-unreachable · minor** — legacy benchmark repo `rickylabs/netscript-start` + returns 404 via the GitHub API (private or removed). The benchmark issue draft scopes the + successor from first principles instead of porting the legacy suite. +- **2026-07-09 · lane-binding · minor** — Tier-B research lanes ran on Copilot internal + explore/research agents (not Opus 4.8) — the only research surface available here. Load-bearing + claims (Garnet FUNCTION/Streams gaps, GlideMQ FCALL dependency, Deno NAPI uncertainty, queue/kv + adapter inventories) were re-verified first-hand by the supervisor before entering findings. +- **2026-07-09 · tooling · minor** — `rtk` prefix convention not applied: the runner sandbox has + no `rtk` binary on PATH; raw git/grep used via the platform's built-in tools instead. diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/issue-draft-benchmark.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/issue-draft-benchmark.md new file mode 100644 index 000000000..6bbcbe4d6 --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/issue-draft-benchmark.md @@ -0,0 +1,70 @@ +# Issue Draft — Reintroduce the seam performance benchmark suite + +> **Draft only.** Authored by harness run `copilot-evaluate-proposal-and-documentation--glidemq-rfc` +> per seed-run drafts-only discipline; the owner files it (or authorizes filing) after ratifying +> `rfc-glidemq.md`. Labels/milestone follow `.github/labels.yml` + the netscript-pr taxonomy. + +--- + +**Title:** `feat(bench): reintroduce the seam performance benchmark suite (queue/KV transports)` + +**Labels:** `type:feature` · `area:queue` (or nearest area label for the queue/KV seams) · +`priority:p2` · `status:plan` · `wave:v1` + +**Milestone:** `Backlog / Triage` (promote to the beta milestone that hosts the GlideMQ Phase-1 +work if the RFC is ratified) + +**Body:** + +## Problem + +When the queue/KV seams behind the plugins (workers, sagas, streams, triggers) were first designed, +adapter selection was informed by a benchmark suite in the legacy repo +(`rickylabs/netscript-start/tree/master/benchmark`). That repo is no longer accessible and the +suite has no successor here: `packages/bench` (`@netscript/bench`) is the **agent self-bench** +instrument (scores coding-agent effectiveness), not a transport benchmark. Today we cannot answer +"what does adapter X cost on a NetScript workload" with first-party numbers — including the +provider-tier claims already printed in `packages/queue/ports/options.ts` doc comments +(Deno KV "< 1000 msg/s", Redis "> 10,000 msg/s"). + +## Why now + +The GlideMQ RFC (see run dir `.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/`) +makes a benchmark a **hard acceptance prerequisite** for any new queue adapter: GlideMQ's headline +claim (+9%→+38% over BullMQ via 1-RTT completeAndFetchNext) is author-published and must be +re-derived on NetScript workloads before it can justify a dependency. Landing the benchmark +*before* the adapter gives an untainted baseline and a permanent way to measure the benefit of this +and every future backend. + +## Scope + +- A workload-driven benchmark harness exercising the **`MessageQueue` port** (and optionally + `WatchableKv` watch latency) across the shipped adapters: Deno KV, Redis, Garnet + (Redis-compatible path), RabbitMQ, Postgres — same workload, same measurement plane. +- Metrics: throughput (jobs/s) at fixed concurrency steps (e.g. c=1/5/10/20), enqueue→complete + latency percentiles (p50/p95/p99), RTTs per job cycle, dead-letter/retry overhead. +- Scenarios: fire-and-forget job, retry-with-backoff, delayed message, worker fan-out; sagas + transport scenario (redis-transport vs list-transport) as a stretch goal. +- Runs locally against Aspire-provisioned containers (compose parity with `scaffold.runtime` + infra); results emitted as a machine-readable report (JSON) + markdown summary, suitable for PR + comments and future CI trending. +- Home: extend `packages/bench` with a transport-bench module or a sibling `bench/` surface — + decide against doctrine (Archetype 6 tooling vs Archetype 3 runtime) during planning; **do not** + entangle it with the agent self-bench scoring pipeline. + +## Non-goals + +- Cross-cloud managed-service benchmarking (ElastiCache, Upstash) — local/containerized only. +- Load-testing the HTTP/oRPC layer (separate concern). +- Shipping the GlideMQ adapter itself (RFC Track A, separate issue once ratified). + +## Acceptance + +- `deno task bench:seams` (name TBD) runs the suite against at least Deno KV + Redis + Garnet and + produces the JSON + markdown reports. +- Baseline numbers for all current adapters are committed/published before any new queue adapter + merges; the GlideMQ adapter PR (if ratified) must include a before/after delta from this suite. +- README documents methodology honestly (local containers ≠ production networks; note that + GlideMQ's own numbers were measured over real network latency, where 1-RTT designs compound). + +Refs #301. Related: RFC run dir above; #399/#400 consume some of the same primitives. diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/plan.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/plan.md new file mode 100644 index 000000000..4f5dc1fb8 --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/plan.md @@ -0,0 +1,53 @@ +# Plan — copilot-evaluate-proposal-and-documentation--glidemq-rfc + +Research/RFC run (evaluation-only, no framework code). Shape documented in +`.llm/harness/workflow/research-rfc-run.md` (authored by this run per the owner's request). + +## Scope (from the owner's proposal, deliverables extracted) + +1. Evaluate GlideMQ (avifenesh/glide-mq + docs + examples + dashboard) against NetScript's + queue/KV seam architecture and plugins. +2. Map compatibility with backends NetScript must keep: Garnet, Deno KV, Redis, RabbitMQ, + Postgres — "adapter layer maybe? You tell me". +3. Produce an exhaustive research corpus in the run dir; verdict → RFC (if positive) or rejection + rationale. +4. Author a benchmark-reintroduction issue (legacy `netscript-start/benchmark` has no successor). +5. Review and document the naturally-followed workflow in `.llm/harness/workflow/`. + +## Archetype / overlays + +No package archetype is implemented; the evaluation *reasons about* Archetype 2 (queue/KV +integration seams) and Archetype 5 (plugins). Overlay: `SCOPE-docs.md` in spirit — all outputs are +run-dir markdown plus one workflow doc. + +## Locked decisions + +- **LD1** Evaluation verdict is **conditional-positive**: GlideMQ enters only as an adapter behind + `MessageQueue` + as port vocabulary + design harvest; never a seam replacement (F1–F3). +- **LD2** All GitHub-facing outputs (RFC, issue) are drafts pending owner ratification and a + separate-session PLAN-EVAL (blocked evaluator lane recorded). +- **LD3** The workflow doc freezes stage contracts, not the exemplar's folder tree (seed-run + precedent). + +## Open-decision sweep + +OQ1 (Deno NAPI spike) — must resolve before Track A; gates nothing in this run. OQ2 (Aspire Valkey +resource) — safe to defer. OQ3 (port home package) — must resolve before Track B implementation; +safe to defer past this run. OQ4 (span naming) — safe to defer into #399 T1. + +## Risk register + +| Risk | Mitigation | +| --- | --- | +| Sandbox cannot run OpenHands evaluators | recorded as blocked lane; RFC header carries pending-PLAN-EVAL status | +| glidemq.dev unreachable | source-of-truth repo docs read directly; mapping recorded in research.md | +| legacy benchmark repo 404 | recorded in drift; issue draft scopes the successor from first principles | + +## Debt implications + +None — no framework code touched. The RFC names future debt-avoidance rules (no GlideMQ types on +public surfaces). + +## Deferred scope + +Filing the issue; running the Deno spike; any adapter/port implementation; PLAN-EVAL of the RFC. diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research.md new file mode 100644 index 000000000..24274e625 --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research.md @@ -0,0 +1,66 @@ +# Research — GlideMQ integration evaluation + +Run: `copilot-evaluate-proposal-and-documentation--glidemq-rfc` · baseline `2779fb24` (2026-07-09) + +## Corpus layout + +| File | Content | +| --- | --- | +| `research/01-netscript-seams.md` | verified map of NetScript's queue/KV seams, plugin consumers, doctrine constraints, benchmark state | +| `research/02-glidemq-dossier.md` | exhaustive external dossier: GlideMQ value proposition, full API, internals, AI primitives, observability, Deno/NAPI analysis, maturity/bus-factor, ecosystem | +| `research/03-compatibility-matrix.md` | backend-by-backend compatibility verdicts (Garnet/Deno KV/Valkey/…), Deno runtime risk, supply-chain assessment | +| `research/04-ecosystem-mapping.md` | feature-by-feature projection onto workers/sagas/triggers/streams/ai/telemetry/dashboard + epics #399/#400 | + +## Re-baseline + +No carried-in plan. The proposal arrived as free-form owner text; all facts were derived fresh +against the current branch and primary sources (GlideMQ GitHub docs — glidemq.dev is DNS-blocked in +this sandbox, but the site is generated from `glide-mq/docs/*.md`, which were read directly; +mapping: guide/usage→USAGE.md, guide/architecture→ARCHITECTURE.md, guide/advanced→ADVANCED.md, +guide/observability→OBSERVABILITY.md, guide/ai-native→README+USAGE AI sections). + +## Findings (each verifiable) + +1. **F1 — GlideMQ hard-requires Valkey/Redis 7.0+ Functions and Streams.** All 44 server ops go + through `FCALL` on a persistent Lua library; workers consume via `XREADGROUP`/`XAUTOCLAIM`. + No EVAL fallback exists. (`glide-mq/docs/ARCHITECTURE.md`) +2. **F2 — Garnet cannot run GlideMQ.** Garnet has no FUNCTION/FCALL and no Streams. Deno KV, + RabbitMQ, Postgres are structurally out. Therefore GlideMQ is only viable as **an additional + adapter behind the `MessageQueue` port**, never a seam replacement. +3. **F3 — NetScript has precedent for exactly this shape.** `plugin-sagas-core` ships a + Garnet-compatible LIST transport alongside the Redis Streams transport + (`packages/plugin-sagas-core/src/transports/`), proving the "portable floor + capable ceiling" + adapter policy this evaluation recommends. +4. **F4 — Deno compatibility is claimed but untested upstream.** README says "Deno with NAPI + support"; `HANDOVER.md` says Bun/Deno NAPI testing is still pending (across 0.14→0.15.4). The + `@glidemq/speedkey` Rust NAPI binary is the risk unit. A spike must gate any adapter plan. The + documented cross-language `WIRE_PROTOCOL.md` (FCALL specs) provides a pure-TS fallback path. +5. **F5 — Bus factor is 1 with a forked native transport.** Single author across core, speedkey + (personal valkey-glide fork, self-declared temporary), dashboard, and all integrations; pre-1.0 + (0.15.4); Apache-2.0. Containment: GlideMQ types must never leak into a NetScript public + surface. +6. **F6 — The AI-native primitive set is the missing layer of `plugins/ai`.** NetScript's ai stack + has model adapters + registry but zero durable-execution semantics. GlideMQ's seven primitives + (usage recording, token streaming, suspend/resume, fallback chains, TPM limits, budgets, per-job + locks) are queue-level concepts portable to any backend as NetScript ports. +7. **F7 — Dashboard/OTel assets align with open epics.** The dashboard REST+SSE+authorize surface + is prior art for #400 S7–S10 and `/_netscript/*` (with its log-tail/metrics screens as the + documented anti-pattern); span conventions + BYO-tracer + server-side metric buckets inform + #399 TC/T6/T7. +8. **F8 — The seam benchmark has no successor.** `packages/bench` is the agent self-bench, not a + seam perf benchmark; the legacy `rickylabs/netscript-start` benchmark repo is inaccessible + (404). A benchmark must exist *before* a GlideMQ adapter ships or its 1-RTT claim cannot be + validated on NetScript workloads → `issue-draft-benchmark.md`. +9. **F9 — Claimed performance is credible but narrow.** +9%→+38% vs "leading alternative" (BullMQ) + on ElastiCache Valkey with TLS at c=5..20, converging at the Valkey single-thread ceiling. The + mechanism (completeAndFetchNext in 1 FCALL) is sound; numbers are author-published and must be + re-derived on NetScript's own benchmark (F8) before being cited. + +## Open questions + +| # | Question | Resolution path | +| --- | --- | --- | +| OQ1 | Does `@glidemq/speedkey` load under Deno 2.9 (`npm:` + nodeModulesDir + `--allow-ffi`)? | Phase 0 spike (RFC §7) | +| OQ2 | Valkey container resource in Aspire scaffold: reuse existing Redis resource with a Valkey image, or add a first-class provider? | CLI scaffold slice design at implementation time | +| OQ3 | Do the proposed AI ports (usage/stream/suspend/budget) live in `plugin-workers-core`, `plugin-ai-core`, or a shared execution-contract package? | doctrine decision, needs owner + PLAN-EVAL | +| OQ4 | Should NetScript adopt GlideMQ span names or keep OTel semconv `messaging.*`? | resolve in #399 T1 convention work (recommend semconv) | diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/01-netscript-seams.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/01-netscript-seams.md new file mode 100644 index 000000000..46ba737f4 --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/01-netscript-seams.md @@ -0,0 +1,93 @@ +# NetScript Seam & Plugin Map (baseline for the GlideMQ evaluation) + +Facts verified against the working tree at baseline `2779fb24` (2026-07-09). Every claim below was +checked by direct file inspection; paths are repo-relative. + +## 1. The queue seam — `@netscript/queue` (`packages/queue`, Archetype 2) + +**Contract:** `MessageQueue` (`packages/queue/ports/message-queue.ts`) — `enqueue`, `listen`, +ack/nack with `NackOptions` (requeue, terminal `DeadLetterReason`, errorCode/errorMessage). +Factories: `createQueue`, `createTypedQueue` (Zod-validated), `createParallelQueue` +(`packages/queue/factory/`). Wraps Fedify's battle-tested queue adapters with a unified interface. + +**Providers** (`packages/queue/ports/options.ts` `enum QueueProvider` + `packages/queue/adapters/`): + +| Provider | Adapter file | Positioning (from source docs) | +| --- | --- | --- | +| `deno-kv` | `deno-kv.adapter.ts`, `kv-polling.adapter.ts` | zero-config default, dev/single instance, < 1000 msg/s | +| `redis` | `redis.adapter.ts` | production, > 10,000 msg/s, pub/sub | +| RabbitMQ (AMQP) | `amqp.adapter.ts` | enterprise routing/reliability | +| Postgres | `postgres.adapter.ts` | DB-backed queue | + +**Dead-letter seam:** `DeadLetterStorePort` (`ports/dead-letter.ts`) with KV / Redis / Postgres +stores (`kv-dead-letter-store.ts`, `redis-dead-letter-store.ts`, `postgres-dead-letter-store.ts`). +This is also the co-requisite surface for dashboard S12 (#553, #555). + +**Discovery:** Aspire environment variables (`services__*`, `ConnectionStrings__`) resolved inside +`factory/create-queue.ts`; no hardcoded backends (composition-root law). + +## 2. The KV seam — `@netscript/kv` (`packages/kv`, Archetype 2) + +**Contract:** `WatchableKv` (CRUD + `watch`/`watchPrefix`). Adapters in `packages/kv/adapters/`: +`deno-kv.adapter.ts`, `denokv-bridge.ts`, `redis.adapter.ts` (+ `redis/` connection/types), +`memory.adapter.ts`, `kvdex.ts`. Redis gives pub/sub watch; Deno KV Connect falls back to polling. +Provider selection via `CACHE_PROVIDER` override then env/Aspire discovery +(`application/auto-detect.ts`). + +**Garnet** is a first-class cache/KV provider: `packages/cli/src/kernel/constants/providers.ts` +declares `GARNET: 'garnet'` with a `garnet://` URL scheme; `packages/kv` handles Garnet via the +Redis-compatible adapter path. + +## 3. Plugin consumers of the seams + +| Plugin | Core package | Seam usage | +| --- | --- | --- | +| `plugins/workers` | `packages/plugin-workers-core` | `defineJob`/`defineTask`/`defineWorkflow`/`defineJobHandler`, `createWorkersRuntime`, `startWorkers`; jobs dispatched over `MessageQueue`; cron via `packages/cron`; idempotency claims (`WorkerIdempotencyClaim`); telemetry reference implementation | +| `plugins/sagas` | `packages/plugin-sagas-core` | Durable sagas with **two transports** (`src/transports/`): `redis-transport*.ts` (Redis Streams) and `list-transport*.ts` — an explicitly **Garnet-compatible saga transport using Redis LIST operations** (`GarnetListTransportOptions`, `resolveGarnetListTransportOptions`). KV-backed `SagaStorePort`, idempotency store, clock port | +| `plugins/triggers` | `packages/plugin-triggers-core` | event/trigger firing over queue/KV seams; known W3C trace-parenting bug (fixed under #399 T4) | +| `plugins/streams` | `packages/plugin-streams-core` | KV-backed event state + pub/sub delivery; durable stream producer | +| `plugins/ai` | `packages/plugin-ai-core` + `packages/ai` | provider adapters (anthropic, openai-compatible, openrouter, ollama, mcp, embeddings), model registry (`ModelRef`/`ModelHandle`/`ModelDescriptor`), tools; **no queue/KV usage today** — no durable job semantics, cost tracking, token streaming, budgets, or failover chains | + +**The list-transport is the smoking gun for backend policy:** NetScript already pays an +extra-engineering cost (a second, feature-reduced transport) specifically to keep Garnet viable when +a backend lacks Redis Streams. Any new queue technology must respect the same law: capability lives +behind a port; degraded-but-working fallbacks stay available. + +## 4. Telemetry — `packages/telemetry` (+ epic #399) + +Ports/adapters package with `TracerProviderPort`/`MeterPort`/`PropagatorPort`, instrumentation +wrappers per seam (`src/instrumentation/queue.ts`, `worker.ts`, `scheduler.ts`, `saga.ts`, +`orpc.ts`), context propagation (`getParentContextFromHeaders`, `injectContext`), and subpath +exports `./otel`, `./query`, `./orpc`. Epic #399 (beta.6) lands the TC-1..14 convention, the +opt-in OTel SDK adapter (`NETSCRIPT_TELEMETRY_PROVIDER`), fan-in span-links, and the +`@netscript/telemetry/query` contract the dashboard consumes. + +## 5. Dev dashboard — epic #400 (beta.7) + +`plugins/dashboard` + `packages/plugin-dashboard-core` on `@netscript/fresh-ui`. Owns +NetScript-domain state only (acceptance line 1: never re-render OTLP; deep-link Aspire/Scalar). +Data plane: `/_netscript/*` introspection + `/_netscript/flows` SSE. Screens S7–S10 are the +Workers/Sagas/Triggers/Streams consoles (create → configure → monitor → act); S12 is Dead-Letter +Queues (deferred, gated on #554/#555). + +## 6. Benchmarks — what exists today + +- `packages/bench` (`@netscript/bench`) is the **agent self-bench instrument** — it scores how well + a *coding agent* builds a NetScript service (frozen HTTP suite, sandbox, scoring). It is **not** a + seam/queue performance benchmark. +- The seam performance benchmark that informed the original queue/KV seam design lived in the + legacy repo `rickylabs/netscript-start/tree/master/benchmark`. That repo returns **404 via the + GitHub API from this sandbox** (private or removed) — the benchmark suite has no successor in + this repo. This gap is the subject of the issue draft in `issue-draft-benchmark.md`. + +## 7. Doctrine constraints that bind this evaluation + +- **Archetype 2 (Integration):** provider-agnostic integration, multiple adapters behind stable + ports — the queue seam is the canonical example. A new backend enters as an adapter, never as a + contract rewrite. +- **Archetype 5 + thinness law** (`docs/architecture/doctrine/11-plugin-thinness-and-base-seams.md`): + capability lives in `-core` siblings; plugins stay thin glue. AI-native job features would live in + `plugin-workers-core`/`plugin-ai-core` port contracts, not in a plugin. +- **Wrap, don't reinvent** (AGENTS.md rule 3): prefer upstream APIs behind local ports. +- **Aspire-first discovery:** any new backend must be resolvable from Aspire env, and scaffold + generation (`packages/cli`) must know how to provision it (a Valkey container resource). diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/02-glidemq-dossier.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/02-glidemq-dossier.md new file mode 100644 index 000000000..c8b0d496d --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/02-glidemq-dossier.md @@ -0,0 +1,833 @@ + +# GlideMQ — Exhaustive Technical Dossier for NetScript Integration Evaluation + +--- + +## 1. What GlideMQ Is: Core Value Proposition & Backend Requirements + +### 1.1 Overview + +**GlideMQ** (`npm: glide-mq`) is a Node.js job/task-queue library purpose-built to replace BullMQ. Its defining architectural choice is the elimination of per-job round-trips: every "complete current job + fetch next" operation is collapsed into a **single FCALL** to a persistent Valkey function library, yielding 1 RTT per job cycle instead of BullMQ's 2–3. + +> *Source: `avifenesh/glide-mq:README.md` — "complete + fetch next in a single server-side function call (1 RTT per job)"* + +The other defining choice is its Rust-native client: the library communicates with Valkey/Redis through **`@glidemq/speedkey`**, a fork of [`valkey-glide`](https://github.com/valkey-io/valkey-glide) that uses **NAPI bindings** (i.e., a compiled Rust `.node` binary loaded by Node.js), bypassing the JavaScript RESP parser entirely. + +> *Source: `avifenesh/speedkey:README.md` — "Valkey/Redis client with direct NAPI bindings based on valkey-glide core. No IPC socket - Rust talks directly to Node.js via NAPI."* + +### 1.2 Differentiation from BullMQ and BeeQueue + +| Aspect | glide-mq | BullMQ | BeeQueue | +|---|---|---|---| +| Network per job | **1 RTT** (FCALL completes + fetches next) | 2–3 RTTs (separate complete, acknowledge, fetch) | 1–2 RTTs | +| Server logic | **44 named Valkey Functions** (FUNCTION LOAD / FCALL — persistent, named) | 53 ephemeral EVAL/EVALSHA scripts (NOSCRIPT errors possible) | Lua EVAL | +| Client layer | **Rust NAPI** (no JS protocol parsing) | ioredis (JS) | ioredis (JS) | +| Cluster | **Native hash-tags `{queueName}` zero-config** | Requires explicit slot management | Standalone only | +| AI-native primitives | 7 first-class (see §4) | None | None | + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:109` — "BullMQ uses 53 EVAL scripts. We use a single function library."* + +### 1.3 Backend Server Requirements — **Critical for NetScript** + +GlideMQ has a hard dependency on **Valkey Functions** (`FUNCTION LOAD` / `FCALL`). This is **not** optional — all 44 server-side operations go through `FCALL`, and the library is loaded on every Queue/Worker construction: + +``` +1. Check: FUNCTION LIST LIBRARYNAME glidemq +2. If missing or version mismatch: FUNCTION LOAD [source] REPLACE +3. In cluster: route to 'allPrimaries' +``` + +> *Source: `avifenesh/glide-mq:src/connection.ts` — `ensureFunctionLibrary()` called on every client init* + +Consequences by backend: + +| Backend | Support | Notes | +|---|---|---| +| **Valkey 7.0+** | ✅ Full | First-class target; tested in CI with Valkey 9.1.0 | +| **Redis OSS 7.0+** | ✅ Supported | Valkey Functions were introduced in Redis 7.0 | +| **Garnet** (Microsoft) | ❌ **Blocker** | Garnet does **not** implement `FUNCTION LOAD`/`FCALL`. XAUTOCLAIM and Streams consumer groups also unverified. GlideMQ cannot be used with Garnet without fundamental rewriting. | +| **Dragonfly** | ⚠️ Uncertain | Dragonfly has partial Lua/EVAL support but `FUNCTION LOAD` support is incomplete as of mid-2025. **Unverified.** | +| **Upstash** | ❌ Blocker | Upstash Redis does not support `FUNCTION LOAD`. | +| **ElastiCache Valkey** | ✅ Supported | Benchmarks run on ElastiCache Valkey 8.2 (TLS, r7g.large) | +| **AWS MemoryDB** | ✅ Supported | IAM auth (`IamCredentials` with `ServiceType.MemoryDB`) built in | + +> *Source: `avifenesh/glide-mq:src/connection.ts:4` — `import { GlideClient, GlideClusterClient, ServiceType } from '@glidemq/speedkey'`; connection.ts lines 18–25 show IAM service type mapping* + +Also: GlideMQ also requires **Redis Streams** (`XREADGROUP`, `XADD`, `XAUTOCLAIM`) and **Sorted Sets**. Garnet's Streams support is incomplete (consumer groups are partial). + +### 1.4 @glidemq/speedkey vs valkey-glide + +`@glidemq/speedkey` is a **personal fork** maintained by `avifenesh` (not a release from the upstream AWS/valkey-io team). The README explicitly states it is a bridge until the official `valkey-glide` ships NAPI support: + +> *"speedkey will be replaced by valkey-glide when glide completes its migration to NAPI. The API surface is designed to align with valkey-glide to minimize migration effort."* +> *Source: `avifenesh/speedkey:README.md`* + +The `speedkey` repository includes `glide-core/` as a subdirectory (a copy of the valkey-glide Rust core), meaning the Rust binary is compiled from a vendored snapshot of `valkey-io/valkey-glide`. This has significant maintenance implications (see §7). + +--- + +## 2. Full API Surface + +### 2.1 Queue + +```typescript +class Queue extends EventEmitter { + // Core + add(name: string, data: D, opts?: JobOptions): Promise | null> + addAndWait(name: string, data: D, opts?: AddAndWaitOptions): Promise // RPC pattern + addBulk(jobs: { name, data, opts? }[]): Promise[]> + getJob(id: string): Promise | null> + getJobs(type, start?, end?, opts?): Promise[]> + getJobCounts(): Promise // { waiting, active, delayed, completed, failed } + count(): Promise + pause() / resume() / isPaused() + revoke(jobId): Promise + obliterate(opts?: { force: boolean }): Promise + close(): Promise + // Bulk operations + clean(grace, limit, type): Promise + drain(delayed?: boolean): Promise + retryJobs(opts?): Promise + // Rate limiting + setGlobalConcurrency(n): Promise + setGlobalRateLimit(config): Promise + // Schedulers + upsertJobScheduler(name, schedule, template?): Promise + getRepeatableJobs(): Promise<...> + removeJobScheduler(name): Promise + // Observability + getMetrics(type, opts?): Promise + getJobLogs(id, start?, end?): Promise<{ logs: string[], count: number }> + getDeadLetterJobs(): Promise + searchJobs(opts): Promise + getWorkers(): Promise + // AI primitives + getFlowUsage(parentJobId): Promise + getFlowBudget(flowId): Promise + readStream(jobId, opts?): Promise + signal(jobId, name, data?): Promise + getSuspendInfo(jobId): Promise + getUsageSummary(opts?): Promise + rateLimitGroup(groupKey, duration, opts?): Promise + // Vector search + createJobIndex(opts?): Promise + vectorSearch(embedding, opts?): Promise +} +``` + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:227–288`* + +### 2.2 Worker + +```typescript +class Worker extends EventEmitter { + constructor( + name: string, + processor: Processor | string, // string = file path for sandbox mode + opts: WorkerOptions + ) +} + +interface WorkerOptions { + concurrency?: number // per-worker, default 1 + globalConcurrency?: number // cross-worker via Valkey + prefetch?: number // XREADGROUP COUNT + blockTimeout?: number // XREADGROUP BLOCK ms + lockDuration?: number // stall detection window (per-job override in JobOptions) + stalledInterval?: number // XAUTOCLAIM frequency + maxStalledCount?: number // max reclaims before fail + promotionInterval?: number // delayed job promotion interval + limiter?: { max: number; duration: number } // RPM sliding window + tokenLimiter?: { maxTokens: number; duration: number; scope?: 'queue' | 'worker' | 'both' } // TPM + backoffStrategies?: Record number> + sandbox?: SandboxOptions // run in child process + batch?: { size: number; timeout?: number } // bulk batch mode + events?: boolean // disable XADD event emission (for throughput) + metrics?: boolean // disable HINCRBY metrics (for throughput) +} +``` + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:312–328`* + +### 2.3 JobOptions + +```typescript +interface JobOptions { + jobId?: string // custom ID + delay?: number // ms before eligible + priority?: number // 0 = highest; max 2^21 + lifo?: boolean // LIFO list instead of stream + ordering?: { + key: string // per-key ordering + group concurrency + concurrency?: number + rateLimit?: RateLimitConfig // sliding window per group key + tokenBucket?: TokenBucketConfig + } + cost?: number // token cost for token-bucket rate limiting + attempts?: number // retry attempts + backoff?: { + type: 'fixed' | 'exponential' | string // 'string' = custom strategy name + delay: number + jitter?: number + } + timeout?: number // per-job timeout (throws on breach) + lockDuration?: number // override worker-level stall window for this job + removeOnComplete?: boolean | number | { age: number; count: number } + removeOnFail?: boolean | number | { age: number; count: number } + deduplication?: { + id: string + ttl?: number + mode?: 'simple' | 'throttle' | 'debounce' + } + parent?: { queue: string; id: string } // single-parent (tree flow) + parents?: Array<{ queue: string; id: string }> // multi-parent (DAG) + ttl?: number // job time-to-live + fallbacks?: Array<{ // AI: model fallback chain + model: string + provider?: string + metadata?: Record + }> +} +``` + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:403–426`* + +### 2.4 Flows / DAG / Workflows + +**FlowProducer** — parent/child trees where a parent only becomes runnable after all children succeed. Atomically enqueued via `glidemq_addFlow`. + +```typescript +const flow = new FlowProducer({ connection }); +await flow.add({ + name: 'aggregate', queueName: 'reports', data: {}, + children: [ + { name: 'fetch-sales', queueName: 'data', data: { region: 'eu' } }, + { name: 'fetch-returns', queueName: 'data', data: { region: 'eu' } }, + ], +}); +``` + +**DAG** — multi-parent dependency graphs. Jobs are submitted level-by-level (reverse topological order). Cycle detection via `CycleError`. + +```typescript +import { dag } from 'glide-mq'; +await dag([ + { name: 'A', queueName: 'tasks', data: {} }, + { name: 'B', queueName: 'tasks', data: {}, deps: ['A'] }, + { name: 'C', queueName: 'tasks', data: {}, deps: ['A'] }, + { name: 'D', queueName: 'tasks', data: {}, deps: ['B', 'C'] }, // diamond +], connection); +``` + +**Workflow helpers**: `chain(name, jobs, connection)`, `group(name, jobs, connection)`, `chord(name, groupJobs, callbackJob, connection)` + +**Dynamic children**: `job.moveToWaitingChildren()` — parent pauses itself until dynamically-spawned children complete; processor re-runs from top on re-entry. + +> *Source: `avifenesh/glide-mq:docs/WORKFLOWS.md`* + +### 2.5 Schedulers (Repeatable Jobs) + +5-field cron + timezone + fixed-interval, bounded schedulers: + +```typescript +await queue.upsertJobScheduler('daily-report', { + cron: '0 0 * * *', + tz: 'America/New_York', + // OR: every: 60000, // ms + // limit: 10, // bounded +}); +``` + +> *Source: `avifenesh/glide-mq:src/scheduler.ts`* + +### 2.6 Events + +```typescript +const events = new QueueEvents('tasks', { connection }); +events.on('completed', ({ jobId, returnvalue }) => { ... }); +events.on('failed', ({ jobId, failedReason }) => { ... }); +events.on('progress', ({ jobId, data }) => { ... }); +events.on('added' | 'stalled' | 'paused' | 'resumed', handler); +// AI events (via SSE proxy): 'usage', 'suspended', 'budget-exceeded' +``` + +`QueueEvents` internally uses `XREAD BLOCK` on the `glide:{queueName}:events` stream. + +### 2.7 Deduplication Modes + +Three modes: +- **simple** — block duplicate jobId for `ttl` ms; second add returns `null` +- **throttle** — accept first, block subsequent within window +- **debounce** — accept last, delay previous within window + +All enforced by `glidemq_dedup` Lua function atomically. + +### 2.8 Request-Reply (RPC) + +`queue.addAndWait(name, data, opts)` adds a job and polls until it completes/fails, returning the result. SSE-based variant available over HTTP proxy. + +### 2.9 Broadcast / Pub-Sub + +`Broadcast` + `BroadcastWorker` — NATS-style subject filtering, independent per-subscriber retries. Uses a dedicated stream per broadcast channel. Every subscriber receives every message. Not dependency-linked. + +--- + +## 3. Architecture Internals + +### 3.1 Data Structures + +The complete Redis/Valkey key schema (all hash-tagged to `{queueName}`): + +``` +glide:{queueName}:id String — auto-increment job ID counter +glide:{queueName}:stream Stream — ready jobs, consumed via XREADGROUP +glide:{queueName}:scheduled ZSet — delayed + priority staging (score = priority*2^42 + ts) +glide:{queueName}:job:{id} Hash — job data, opts, state, timestamps, return value, cost, usage:* +glide:{queueName}:completed ZSet — score = completed timestamp +glide:{queueName}:failed ZSet — score = failed timestamp +glide:{queueName}:events Stream — lifecycle events (XREAD BLOCK for QueueEvents) +glide:{queueName}:meta Hash — paused, concurrency, rate limiter state +glide:{queueName}:deps:{id} Set — child job IDs for parent (flow tracking) +glide:{queueName}:parents:{id} Set — parent refs for DAG multi-parent +glide:{queueName}:parent:{id} Hash — parent queue + job ID reference +glide:{queueName}:dedup Hash — deduplication entries +glide:{queueName}:rate Hash — global rate limiter counters +glide:{queueName}:schedulers Hash — scheduler entries (name → next_run_ts) +glide:{queueName}:ordering Hash — per-key sequence counters +glide:{queueName}:group:{key} Hash — group state (active count, TB tokens, etc.) +glide:{queueName}:groupq:{key} ZSet — group-limited job wait list +glide:{queueName}:lifo List — LIFO jobs (RPUSH/RPOP) +glide:{queueName}:jstream:{id} Stream — per-job LLM token output stream +glide:{queueName}:signals:{id} List — signals to suspended job +glide:{queueName}:suspended ZSet — suspended jobs (score = timeout deadline) +glide:{queueName}:budget:{flowId} Hash — flow-level budget state +glide:{queueName}:tpm Hash — token-per-minute limiter state +glide:{queueName}:ratelimited ZSet — rate-limited jobs waiting for window reset +``` + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:11–42`* + +### 3.2 Atomicity Mechanism — Valkey Functions, NOT EVAL + +**Critical design detail:** GlideMQ uses **`FUNCTION LOAD` + `FCALL`**, not per-call `EVAL`/`EVALSHA`. The entire Lua library (44 functions, ~42KB source in `src/functions/glidemq.lua`) is loaded once to the server and persists across restarts (stored in RDB/AOF). This is fundamentally different from BullMQ which uses 53 ephemeral EVAL scripts subject to NOSCRIPT errors. + +```lua +#!lua name=glidemq +-- glidemq.lua — loaded once via FUNCTION LOAD +redis.register_function('glidemq_addJob', function(keys, args) ... end) +redis.register_function('glidemq_completeAndFetchNext', function(keys, args) ... end) +-- ... 42 more functions +``` + +> *Source: `avifenesh/glide-mq:src/functions/glidemq.lua:1–5`* + +The key functions and what they do atomically: + +| Function | Keys | What it does | +|---|---|---| +| `glidemq_addJob` | 4 | INCR id, HSET job hash, XADD stream or ZADD scheduled, XADD event | +| `glidemq_completeAndFetchNext` | 5 | **1 RTT core**: XACK + ZADD completed + HSET + XADD event + XREADGROUP next | +| `glidemq_fail` | 6 | XACK + ZADD failed or ZADD scheduled (retry w/ backoff) + HSET + XADD event | +| `glidemq_reclaimStalled` | 2 | XAUTOCLAIM on stream, HSET stalledCount, move to failed if limit exceeded | +| `glidemq_addFlow` | N | Atomic parent+children creation + dep wiring | +| `glidemq_completeChild` | 4 | SREM from deps set, if empty → re-queue parent | +| `glidemq_dedup` | 5 | Check dedup hash, skip or add based on mode | +| `glidemq_suspend` | 4 | Move active → suspended ZSet, release group slot | +| `glidemq_signal` | 5 | Deliver signal to suspended job, re-queue it | +| `glidemq_recordUsageAndCheckBudget` | 1 | Increment usage counters, check budget limits | + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:133–178`* + +### 3.3 Blocking Reads + +The `Worker` uses two separate client connections: +- **`commandClient`** (non-blocking) — XACK, HSET, FCALL for complete/fail/promote +- **`blockingClient`** (blocking-only, dedicated) — `XREADGROUP BLOCK {timeout}` — never shared with other commands + +``` +Worker + ├── commandClient (XACK, HSET, FCALL, scheduler loop) + ├── blockingClient (XREADGROUP BLOCK only) + └── schedulerLoop (uses commandClient — promote delayed, reclaim stalled) +``` + +Consumer group: `glide:{queueName}:workers`; each Worker instance = consumer `worker-{uuid}`. + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:196–211`* + +### 3.4 Stalled-Job Recovery + +Uses **`XAUTOCLAIM`** (Redis 6.2+ / Valkey), not lock-renewal polling. The `schedulerLoop` calls `glidemq_reclaimStalled` periodically at `stalledInterval` ms: + +1. `XAUTOCLAIM` retrieves PEL entries older than `lockDuration` +2. `stalledCount` is incremented on the job hash +3. If `stalledCount <= maxStalledCount` (default 1) → redispatch to stream +4. If exceeded → `glidemq_fail` → moves to failed ZSet + +This design means one crash = one retry; chronic crashes = clean `failed` state. + +> *Source: `avifenesh/glide-mq:docs/DURABILITY.md`* + +### 3.5 Priority Encoding + +Score in scheduled ZSet: `(priority * 2^42) + timestamp_ms` + +Priority 0 (highest) always sorts before priority 1 regardless of timestamp. Non-delayed priority jobs get `timestamp = 0` so they promote immediately. Max priority: `2^21` (matches BullMQ range). + +> *Source: `avifenesh/glide-mq:docs/ARCHITECTURE.md:89–96`* + +--- + +## 4. AI-Native Features + +GlideMQ declares 7 "AI-native primitives" built directly into the core API. These are not an external plugin — they live in the `Job`, `Worker`, and `FlowProducer` classes: + +### 4.1 Cost Tracking (Usage Reporting) + +```typescript +await job.reportUsage({ + model: 'gpt-5.4', + provider: 'openai', + tokens: { input: 50, output: 200 }, + totalTokens: 250, + costs: { total: 0.003 }, + totalCost: 0.003, + costUnit: 'usd', + latencyMs: 1200, + cached: false, +}); +``` + +Usage is stored in the job hash as `usage:model`, `usage:tokens` (JSON), `usage:totalCost`, etc. Also updates per-minute rolling buckets for `getUsageSummary()`. + +Flow-level aggregation: `queue.getFlowUsage(parentJobId)` traverses the job tree and aggregates tokens/costs/models across all jobs in the flow. + +Cross-queue rolling summary (without scanning job hashes): `queue.getUsageSummary({ queues: ['llm-tasks'], windowMs: 3_600_000 })`. + +> *Source: `avifenesh/glide-mq:docs/OBSERVABILITY.md` — AI Usage Telemetry section* + +### 4.2 Token Streaming (LLM output in real time) + +```typescript +// Inside processor: +await job.stream({ type: 'token', content: 'Hello' }); +await job.streamChunk('token', 'world'); // thin wrapper + +// External consumer (long-poll): +const chunks = await queue.readStream(jobId, { block: 5000 }); +``` + +Chunks stored in `glide:{queueName}:jstream:{id}` (a dedicated Stream). SSE endpoint at `/queues/:name/jobs/:id/stream` in the HTTP proxy for browser consumption. + +### 4.3 Suspend / Resume (Human-in-the-Loop) + +```typescript +// Inside processor: +await job.suspend({ timeout: 86400000, onResume: async (signals) => { ... } }); +// throws SuspendError — worker catches it, sets job state to 'suspended' + +// From outside (webhook, approval, etc.): +await queue.signal(jobId, 'approve', { reviewedBy: 'alice' }); +// moves job back to stream, processor re-runs with job.signals populated +``` + +Suspended jobs stored in `glide:{queueName}:suspended` ZSet (score = timeout deadline). `glidemq_sweepSuspended` Lua function fails timed-out suspended jobs. + +### 4.4 Fallback Chains (Model Failover) + +```typescript +await queue.add('inference', { prompt: '...' }, { + fallbacks: [ + { model: 'gpt-5.4-mini', provider: 'openai' }, + { model: 'claude-3-opus', provider: 'anthropic' }, + ], + attempts: 3, +}); + +// In worker: +const model = job.currentFallback?.model ?? 'gpt-5.4'; // advances on each retry +``` + +`job.currentFallback` is computed from `job.fallbackIndex` (persisted in job hash) and `job.opts.fallbacks`. + +### 4.5 TPM Rate Limiting (Tokens Per Minute) + +```typescript +const worker = new Worker('ai', handler, { + connection, + tokenLimiter: { maxTokens: 100_000, duration: 60_000, scope: 'both' }, + limiter: { max: 100, duration: 60_000 }, // RPM (combine for dual-axis) +}); +``` + +Token consumption tracked via `job.reportTokens(n)`. State stored in `glide:{queueName}:tpm` (Valkey-side) and/or in-memory based on `scope`. + +### 4.6 Budget Caps + +```typescript +const flow = new FlowProducer({ connection }); +await flow.add(flowJob, { + budget: { + maxTotalTokens: 500_000, + maxTotalCost: 5.00, + costUnit: 'usd', + onExceeded: 'fail', // or 'pause' + }, +}); +``` + +Budget state in `glide:{queueName}:budget:{flowId}`. `glidemq_recordUsageAndCheckBudget` atomically increments and checks. When exceeded, subsequent jobs in the flow fail or pause (configurable). + +### 4.7 Per-Job Lock Duration + +```typescript +await queue.add('inference', { prompt: '...' }, { lockDuration: 120_000 }); // 2-min lock +``` + +Overrides `WorkerOptions.lockDuration` per job, allowing short locks for classifiers and long locks for multi-minute LLM calls. + +### 4.8 Vector Search + +Via `@glidemq/speedkey`'s `GlideFt` module (requires `valkey/valkey-bundle` with Valkey Search 1.1+): + +```typescript +await queue.createJobIndex({ /* FT.CREATE schema */ }); +await job.storeVector('embedding', float32Array); +const results = await queue.vectorSearch(queryEmbedding, { topK: 5 }); +``` + +> *Source: `avifenesh/speedkey:README.md` — GlideFt section* + +--- + +## 5. Observability + +### 5.1 Job Logs + +```typescript +await job.log('Step 1 done'); // appended to job hash (list field) +const { logs, count } = await queue.getJobLogs(jobId, 0, 49); // paginated +``` + +### 5.2 Metrics + +`queue.getMetrics('completed' | 'failed', opts?)` returns per-minute time-series with `{ count, avgDuration }` data points. Buckets retained 24 hours, auto-trimmed server-side with zero extra RTTs. + +### 5.3 OpenTelemetry Integration — Details + +OTel is a **zero-code-change optional integration** via `@opentelemetry/api` peer dep. The library detects it at runtime via `require('@opentelemetry/api')` in a try/catch: + +```typescript +// src/telemetry.ts — runtime detection +let otelApi: OTelApi | null = null; +try { + otelApi = require('@opentelemetry/api') as OTelApi; +} catch { /* no-op tracing */ } +``` + +**Install and it works** — no code changes needed. Or provide your own tracer: + +```typescript +import { setTracer } from 'glide-mq'; +setTracer(trace.getTracer('my-service', '1.0.0')); +``` + +**Instrumented operations and span attributes:** + +| Operation | Span name | Key attributes | +|---|---|---| +| `queue.add()` | `glide-mq.queue.add` | `glide-mq.queue`, `glide-mq.job.name`, `glide-mq.job.id`, `glide-mq.job.delay`, `glide-mq.job.priority` | +| `flowProducer.add()` | `glide-mq.flow.add` | `glide-mq.queue`, `glide-mq.flow.name`, `glide-mq.flow.childCount` | + +> *Source: `avifenesh/glide-mq:docs/OBSERVABILITY.md` — OpenTelemetry Integration section* +> *Source: `avifenesh/glide-mq:src/telemetry.ts`* + +**⚠️ Notable gaps:** +- **Only 2 span types** are currently instrumented (queue.add and flow.add). Worker processing, job completion, and failure do not emit spans. +- **No W3C Trace Context propagation** in job data — there is no mechanism to propagate `traceparent` from the producer to the worker across the queue boundary. Spans are purely local to each client process. +- Span types: `startSpan` → `setAttribute` → `setStatus({ code: OK|ERROR })` → `recordException` → `end()`. No `startActiveSpan` / context propagation. +- `isTracingEnabled()` is exported for conditionally initializing the tracer. + +### 5.4 AI Usage Telemetry + +Per-minute usage buckets in `glide:{queueName}:metrics:*` keys updated atomically via `glidemq_recordUsageAndCheckBudget`. `getUsageSummary()` aggregates across queues without scanning job hashes. SSE events include `usage`, `suspended`, `budget-exceeded` event types. + +> *Source: `avifenesh/glide-mq:docs/OBSERVABILITY.md` — AI Usage Telemetry section* + +### 5.5 Dashboard (`@glidemq/dashboard`) + +- **Package**: `@glidemq/dashboard` v0.4.1, `avifenesh/glidemq-dashboard` +- **Tech stack**: Express middleware, single self-contained HTML SPA (`src/dashboard-ui.html`, 81KB), SSE for live updates +- **How it works**: `createDashboard(queues, opts?) → Express Router` — drop-in mount, no frontend build required +- **Features**: queue counts/states, job details (with AI fields), pause/resume/drain/retry/obliterate, SSE real-time events, per-action authorization callbacks +- **AI endpoints exposed**: `/api/queues/:name/flows/:id/usage`, `/api/queues/:name/flows/:id/budget`, `/api/usage/summary`, `/api/queues/:name/jobs/:id/stream` +- **SSE events**: `added | completed | failed | progress | stalled | heartbeat | usage | suspended | budget-exceeded` + +```typescript +import express from 'express'; +import { createDashboard } from '@glidemq/dashboard'; +app.use('/dashboard', createDashboard([queue], { + readOnly: false, + authorize: (req, action) => action !== 'queue:obliterate' || isAdmin(req), +})); +``` + +> *Source: `avifenesh/glidemq-dashboard:README.md`* + +**⚠️ Limitations:** +- **Express-only** — no Hono, Fastify, or Koa adapter. Cannot be mounted on a Deno-native HTTP server. +- Middleware only, not a standalone server — requires an existing Express app. +- Does not connect to Valkey/Redis directly — requires live `Queue` instances passed in. + +--- + +## 6. Runtime Compatibility — Deno / NAPI Analysis + +### 6.1 Node.js Version Requirement + +`engines.node: ">=20"` — Node.js 20+ required. + +> *Source: `avifenesh/glide-mq:package.json:10`* + +### 6.2 Module Format + +`"type": "commonjs"` in package.json. The published package is **CJS only** (`dist/index.js`, `dist/testing.js`, `dist/proxy/index.js`). There is no `"exports"` ESM entry point. + +> *Source: `avifenesh/glide-mq:package.json:5`* + +### 6.3 Deno Compatibility — ⚠️ UNCERTAIN, Likely Problematic + +The README explicitly mentions Deno as a supported target: + +> *"The Rust NAPI client requires a server-side runtime (Node.js 20+, Bun, or **Deno with NAPI support**)."* +> *Source: `avifenesh/glide-mq:README.md`* + +However, the internal HANDOVER.md (a developer-facing file) states explicitly: + +> **"Bun/Deno NAPI compatibility testing: still pending from 0.14.0 handover."** +> *Source: `avifenesh/glide-mq:HANDOVER.md` — Open Threads section* + +The current version is 0.15.4. This means Deno/Bun NAPI testing has been **deferred across the entire 0.15.x release series**. Deno compatibility is stated as a goal but is **not verified** by the author. + +**Technical obstacles for Deno:** + +1. **NAPI `.node` binary**: `@glidemq/speedkey` ships a platform-specific compiled Rust `.node` binary. Deno supports `npm:` specifiers and NAPI, but NAPI modules must expose the correct binary path in their `package.json` `"node"` / platform-specific fields. Deno's NAPI support works for many packages but has edge cases with complex native module loading. + +2. **CJS-only output**: `glide-mq` itself is CommonJS. Deno can import CJS modules via `npm:glide-mq`, but the import path semantics differ and sub-path exports (`glide-mq/testing`, `glide-mq/proxy`) must be correctly resolved. + +3. **`require('@opentelemetry/api')` in telemetry.ts**: Uses Node.js `require()` dynamically. This works under Deno's Node.js compat layer, but is less robust than ESM dynamic import. + +4. **Process signal handling**: `process.on('SIGTERM', ...)` in examples — standard Node.js pattern, supported in Deno's Node compat. + +5. **Worker sandbox mode** (`src/sandbox/`) uses `child_process` or `worker_threads` — Deno supports these via Node compat but with restrictions. + +**Recommendation for NetScript**: Do **not** assume Deno compatibility. The NAPI binary for `@glidemq/speedkey` is the primary risk. Even if Deno's NAPI layer works, testing is needed. This is a **significant integration blocker** until verified. + +### 6.4 ESM / CJS + +The library is CJS. The hono-api example (`examples/hono-api/package.json`) uses `"type": "module"` (ESM) and imports `glide-mq` without issues under tsx/Node.js, so the CJS-as-dependency-in-ESM pattern works normally in Node. + +--- + +## 7. License, Maturity, Release Cadence, Bus Factor, Ecosystem + +### 7.1 License + +**Apache-2.0** — permissive, patent grant included, compatible with commercial use. + +> *Source: `avifenesh/glide-mq:package.json`, `avifenesh/glidemq-dashboard:package.json`* + +### 7.2 Version History & Release Cadence + +From CHANGELOG: + +| Version | Date | Highlights | +|---|---|---| +| 0.15.4 | 2026-06-04 | Interval scheduler drift fix, CI correctness, Valkey 9.1.0 CI images | +| 0.15.3 | ~2026-05 | DAG fixes, long-running job heartbeats, broadcast retry isolation, CVE deps | +| 0.15.2 | ~2026-05 | Batch priority/LIFO, `lockDuration`-aware stall reclaim (LIBRARY_VERSION 84) | +| 0.15.1 | ~2026-04 | Debounce+ordering deadlock fix (LIBRARY_VERSION 81) | +| 0.15.0 | ~2026-03 | HTTP proxy parity expansion, flow HTTP API, getUsageSummary | + +The 0.15.x series is recent (April–June 2026). The library is young and iterating rapidly — 5 minor versions in ~3 months. + +> *Source: `avifenesh/glide-mq:HANDOVER.md`* + +### 7.3 Maturity + +- **Early-stage commercial product** — described as "replacing BullMQ" but no evidence of production adoption at scale outside the author's own infrastructure +- 2,414 tests in the non-fuzzer test suite (as of 0.15.4) +- CI: green on main with Vitest +- No external contributors visible in the commit history fetched (single-author commits: `avifenesh / aviarchi1994@gmail.com`) +- Stars: Not directly queryable via these tools, but the `glidemq-hono` repo has 13 stars, other repos 2–5 stars → small ecosystem, very early discovery phase + +### 7.4 Bus Factor — ⚠️ HIGH RISK + +**Bus factor: 1.** All code, documentation, commit activity, and the critical `@glidemq/speedkey` NAPI bridge are maintained by a single developer. There is no org, no co-maintainer, no corporate backing visible. The HANDOVER.md file itself is consistent with a solo developer self-documenting for continuity. + +`@glidemq/speedkey` is especially concerning: it is a **personal fork** of the valkey-glide Rust core with a custom NAPI binding layer. If the author becomes unavailable, both the queue library and its Rust transport layer would be unmaintained. + +> *Source: `avifenesh/speedkey:README.md` — "speedkey will be replaced by valkey-glide when..."* + +### 7.5 Ecosystem Packages (All by avifenesh) + +| Package | Repo | Stars | Purpose | +|---|---|---|---| +| `glide-mq` | `avifenesh/glide-mq` | — | Core library (v0.15.4) | +| `@glidemq/speedkey` | `avifenesh/speedkey` | — | Rust NAPI client (valkey-glide fork) | +| `@glidemq/dashboard` | `avifenesh/glidemq-dashboard` | 3 | Express web UI | +| `@glidemq/hono` | `avifenesh/glidemq-hono` | 13 | Hono middleware + REST API + SSE | +| `@glidemq/fastify` | `avifenesh/glidemq-fastify` | 3 | Fastify plugin | +| `@glidemq/hapi` | `avifenesh/glidemq-hapi` | 3 | Hapi plugin | +| `@glidemq/nestjs` | `avifenesh/glidemq-nestjs` | 5 | NestJS decorators/DI module | + +There is no `@glidemq/deno` package. No JSR package. + +> *Source: `avifenesh/glidemq-dashboard:README.md` — links section; `github-mcp-server-search_repositories` results* + +--- + +## 8. The Hono-API Example + +**Source: `avifenesh/glidemq-examples:examples/hono-api/index.ts`** + +```typescript +import { Hono } from 'hono'; +import { serve } from '@hono/node-server'; +import { glideMQ, glideMQApi, QueueRegistryImpl } from '@glidemq/hono'; +import type { GlideMQEnv } from '@glidemq/hono'; +import type { Job } from 'glide-mq'; + +const connection = { addresses: [{ host: 'localhost', port: 6379 }] }; + +const registry = new QueueRegistryImpl({ + connection, + queues: { + emails: { processor: processEmail, concurrency: 5 }, + orders: { processor: processOrder, concurrency: 3 }, + }, +}); + +const app = new Hono(); +app.use(glideMQ(registry)); // injects registry into c.var.glideMQ +app.route('/api/queues', glideMQApi()); // typed REST API + SSE + +app.post('/send-email', async (c) => { + const { queue } = c.var.glideMQ.get('emails'); + const job = await queue.add('send', await c.req.json()); + return c.json({ jobId: job?.id ?? null }); +}); +``` + +**Integration pattern observations:** +1. Uses `@glidemq/hono` middleware package, not `glide-mq` directly in Hono routing +2. Runs on **`@hono/node-server`** (Node.js adapter) — not on Deno natively +3. `QueueRegistryImpl` manages queue + worker lifecycle with `registry.closeAll()` on `SIGTERM` +4. `glideMQApi()` exposes a full typed REST surface: job CRUD, pause/resume, schedulers, SSE, flow API, usage summary, broadcast + +The `@glidemq/hono` README states the middleware is "**multi-runtime — runs on Node, Bun, Deno, and edge runtimes**", but this claim is about the Hono middleware layer itself, not about the underlying `@glidemq/speedkey` NAPI binary. The example actually uses `@hono/node-server`, confirming Node.js as the runtime for this pattern. + +> *Source: `avifenesh/glidemq-examples:examples/hono-api/index.ts`, `avifenesh/glidemq-hono:README.md`* + +**For NetScript**: If NetScript's Hono integration is intended for Deno, the `@hono/node-server` adapter would need to be replaced with Deno.serve + the question of whether `@glidemq/speedkey`'s NAPI binary loads on Deno remains open (see §6.3). + +--- + +## 9. The Dashboard Repo + +### 9.1 Tech Stack + +- **Language**: TypeScript compiled to CJS (`tsconfig.json` → `dist/`) +- **Server layer**: Express 4 or 5 (peer dep) — pure Express `Router` +- **Frontend**: A single embedded HTML file (`src/dashboard-ui.html`, 81KB) served as a static asset. No React, Vue, or build step. Vanilla JS + inline CSS inside the HTML file. +- **Real-time**: SSE (`/api/events`) — browser-side `EventSource` polling +- **Test stack**: Vitest + supertest + +### 9.2 APIs It Consumes + +The dashboard calls `Queue` instance methods directly (not HTTP-to-Valkey): + +```typescript +createDashboard(queues: Queue[], opts?: DashboardOptions): Router +``` + +It calls: `queue.getJobCounts()`, `queue.getJobs()`, `queue.getJob()`, `queue.getJobLogs()`, `queue.pause()`, `queue.resume()`, `queue.drain()`, `queue.obliterate()`, `queue.retryJobs()`, `queue.getFlowUsage()`, `queue.getFlowBudget()`, `queue.getUsageSummary()`, `queue.readStream()` — all synchronously in the same Node.js process. + +REST API surface: + +| Method | Path | Description | +|---|---|---| +| GET | `/api/queues` | All queue counts + metrics | +| GET | `/api/queues/:name` | Queue details + recent jobs | +| GET | `/api/queues/:name/jobs/:id` | Single job details | +| POST | `/api/queues/:name/jobs` | Add job | +| POST | `/api/queues/:name/pause` | Pause | +| POST | `/api/queues/:name/resume` | Resume | +| POST | `/api/queues/:name/jobs/:id/retry` | Retry failed job | +| DELETE | `/api/queues/:name/jobs/:id` | Remove job | +| POST | `/api/queues/:name/drain` | Drain waiting jobs | +| POST | `/api/queues/:name/obliterate` | Obliterate queue | +| GET | `/api/events` | SSE lifecycle stream | +| GET | `/api/queues/:name/flows/:id/usage` | Flow usage | +| GET | `/api/queues/:name/flows/:id/budget` | Flow budget | +| GET | `/api/usage/summary` | Rolling usage summary | +| GET | `/api/queues/:name/jobs/:id/stream` | Job SSE stream | + +> *Source: `avifenesh/glidemq-dashboard:README.md`* + +### 9.3 Embeddability + +The dashboard is an Express middleware — **embeddable in any Express 4/5 application** with one line. Not embeddable in Hono, Fastify, Koa, or Deno's native HTTP server without wrapping. No standalone server mode (there's a `demo.ts` that creates a minimal Express server for dev use). + +**Authorization** is handled via an `authorize(req, action): boolean | Promise` callback that fires per-action. + +--- + +## Summary Assessment for NetScript Integration + +### Critical Blockers + +| Issue | Severity | Notes | +|---|---|---| +| **Garnet incompatibility** | 🔴 Hard blocker | `FUNCTION LOAD`/`FCALL` not supported by Garnet. NetScript's Garnet seam cannot use GlideMQ without replacing the transport layer. | +| **Deno NAPI unverified** | 🔴 Risk | `@glidemq/speedkey` NAPI binary on Deno is explicitly untested (HANDOVER: "still pending"). Must be verified before any Deno integration. | + +### Significant Concerns + +| Issue | Severity | Notes | +|---|---|---| +| **Bus factor: 1** | 🟠 High | Single developer, personal NAPI fork, no org backing. GlideMQ is a dependency risk for production infrastructure. | +| **CJS-only output** | 🟡 Medium | Works with Deno `npm:` CJS compat but suboptimal. No ESM entry point. | +| **Early maturity** | 🟡 Medium | 0.15.x series, fast iteration, small ecosystem (2–13 stars). Docs are excellent but adoption is nascent. | +| **Express-only dashboard** | 🟡 Medium | @glidemq/dashboard requires Express. NetScript would need Express as a host or a different dashboard approach. | +| **OTel span coverage** | 🟡 Medium | Only 2 span types (queue.add, flow.add). No worker-side spans, no W3C traceparent propagation across queue boundary. | + +### Genuine Strengths Relevant to NetScript + +| Strength | Relevance | +|---|---| +| **1 RTT per job** via `glidemq_completeAndFetchNext` | Direct throughput benefit in any queue seam | +| **Zero-config cluster** via hash-tag `{queueName}` | Compatible with ElastiCache/Valkey cluster | +| **44 Valkey Functions** (no NOSCRIPT errors, persistent) | More reliable than BullMQ's EVAL scripts under high concurrency | +| **Hono integration** (`@glidemq/hono`) | Direct alignment with NetScript's Hono-based HTTP layer | +| **At-least-once with `XAUTOCLAIM` recovery** | Reliable over BullMQ-style lock-based stall detection | +| **AI-native primitives** built into core | LLM pipeline orchestration without external tooling | +| **TestQueue/TestWorker** (in-memory) | Zero-Valkey dependency for unit tests | +| **Apache-2.0** | Permissive license, no LGPL/GPL concerns | +| **IAM auth** for ElastiCache/MemoryDB | AWS-native deployment path | + +--- + +## Research Limitations & Workarounds + +1. **`glidemq.dev` docs unreachable** — DNS resolution failed for `glidemq.dev` in this environment. All guide-page content was sourced from the GitHub docs/ directory which mirrors the website content. Assessed as complete. + +2. **NPM download stats** — npmjs.com unreachable. Cannot confirm weekly download figures. + +3. **Dragonfly FUNCTION LOAD** — whether Dragonfly's implementation of `FUNCTION LOAD` is sufficient for GlideMQ's Lua library is **unverified**. The Lua library makes heavy use of `redis.register_function()` and the Streams + consumer group API. Needs empirical testing. + +4. **Deno NAPI binary resolution** — The specific mechanism by which `@glidemq/speedkey`'s `node_modules/.../speedkey.*.node` binary would be found and loaded under Deno's `npm:` compatibility layer is untested. This is the single highest-risk item for Deno integration. + +5. **Star counts for `glide-mq` core repo** — could not retrieve directly; proxy signal via ecosystem repos (2–13 stars) suggests very early adoption. + +6. **`LIBRARY_VERSION` 93** (0.15.3) and **LIBRARY_VERSION** (0.15.4 — 94 or higher implied) — the Lua library has been revised frequently across 0.15.x, meaning clients must reload on version mismatch. This is handled automatically but means cluster rolling upgrades need coordination. \ No newline at end of file diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/03-compatibility-matrix.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/03-compatibility-matrix.md new file mode 100644 index 000000000..627e9266e --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/03-compatibility-matrix.md @@ -0,0 +1,94 @@ +# Compatibility Matrix — GlideMQ vs NetScript's supported backends + +The pivotal question: *can GlideMQ subsume or coexist with the backends NetScript supports today +(Deno KV, Redis, Garnet, RabbitMQ, Postgres), and does it run under Deno at all?* + +## 1. What GlideMQ demands from the server + +From `glide-mq/docs/ARCHITECTURE.md` (verified first-hand, 2026-07-09): + +1. **Valkey Functions** — `FUNCTION LOAD` / `FCALL`, a single Lua library (`#!lua name=glidemq`, + 44 functions, `LIBRARY_VERSION`-tracked, `FUNCTION LOAD REPLACE` on mismatch, routed to + `allPrimaries` in cluster mode). Introduced in Redis 7.0 / all Valkey versions. **Not EVAL** — + there is no script-based fallback path. +2. **Redis Streams with consumer groups** — `XADD`, `XREADGROUP ... BLOCK`, `XACK`, `XAUTOCLAIM` + (stalled-job reclaim), `XTRIM` (broadcast retention). +3. **Sorted sets, hashes, lists** — scheduled/priority ZSet with `(priority * 2^42) + ts` score + encoding, completed/failed ZSets, job hashes, LIFO list. +4. Optional: **Valkey Search module** (`FT.CREATE`, KNN vector search over job hashes). + +Client side: `@glidemq/speedkey` (Rust NAPI `.node` binary, personal fork of valkey-glide), CJS-only +package, `engines.node >= 20`. + +## 2. Backend-by-backend verdict + +| Backend (NetScript today) | FUNCTION | Streams + XAUTOCLAIM | Verdict for GlideMQ | +| --- | --- | --- | --- | +| **Valkey 7.0+** (incl. ElastiCache Valkey, MemoryDB) | ✅ | ✅ | ✅ full — GlideMQ's first-class target | +| **Redis OSS 7.0+** | ✅ | ✅ | ✅ full | +| **Garnet** | ❌ no FUNCTION/FCALL (Lua EVAL only, Redis ≤6 model) | ❌ no Streams at all | ❌ **hard blocker** — GlideMQ cannot run on Garnet, period | +| **Deno KV** | n/a (not Redis-protocol) | n/a | ❌ structurally impossible | +| **RabbitMQ / Postgres** (queue adapters) | n/a | n/a | ❌ different protocol family | +| Upstash / most serverless Redis | ❌ | partial | ❌ blocker | +| Dragonfly | ⚠️ incomplete FUNCTION | ⚠️ | ⚠️ unverified | + +Sources: Garnet docs `microsoft.github.io/garnet/docs/commands/scripting` (Lua EVAL/EVALSHA only, +no FUNCTION) and `docs/welcome/compatibility` (no Streams/XADD/XREADGROUP); microsoft/garnet#509. + +**Consequence:** GlideMQ can never *replace* NetScript's queue seam. It can only be **one more +adapter behind the `MessageQueue` port** — exactly the shape the sagas list-transport precedent +established for Garnet. The existing Deno KV / Garnet / RabbitMQ / Postgres paths are untouched; +they remain the portable floor while GlideMQ becomes the opt-in high-performance ceiling. + +## 3. Deno runtime compatibility — the gating unknown + +- GlideMQ's README claims "Node.js 20+, Bun, or Deno with NAPI support". +- The repo's own `HANDOVER.md` contradicts the marketing: **"Bun/Deno NAPI compatibility testing: + still pending from 0.14.0 handover"** — unresolved across the whole 0.15.x series (current + 0.15.4, 2026-06-04). +- Risk stack: platform-specific NAPI `.node` binary from a personal fork (`@glidemq/speedkey`), + CJS-only entry + subpaths (`glide-mq/testing`, `glide-mq/proxy`), dynamic + `require('@opentelemetry/api')`, `child_process`/`worker_threads` sandbox mode. +- Deno's Node-API layer loads many NAPI modules but is not universal; `--allow-ffi` and a local + `node_modules` (`nodeModulesDir`) are typically required. NetScript already manages npm deps via + the catalog (`catalog:` is npm-only — netscript-deno-toolchain), so the packaging mechanics are + available; the binary-load behavior is the untested part. + +**Verdict:** unproven, plausibly workable, must be settled by a ~1-day spike (Phase 0 of the RFC) +before any adapter work is planned. If the NAPI binding fails under Deno, the fallback is writing +the adapter against a pure-TS Redis client speaking `FCALL` to GlideMQ's *server-side function +library* (the wire protocol is documented in `docs/WIRE_PROTOCOL.md` precisely for cross-language +clients) — at the cost of losing the Rust-client performance half of GlideMQ's advantage while +keeping the 1-RTT server-side half. + +## 4. Supply-chain / maturity assessment + +| Axis | Finding | Risk | +| --- | --- | --- | +| License | Apache-2.0 across core + ecosystem | ✅ none | +| Version | 0.15.4 (pre-1.0), 5 minors in ~3 months | ⚠️ churning API, `LIBRARY_VERSION` bumps (81→84 within 0.15.x) | +| Bus factor | **1** — single author (avifenesh) for core, speedkey NAPI fork, dashboard, all framework integrations | 🔴 highest single risk | +| speedkey | personal fork of valkey-glide core, self-declared temporary ("will be replaced by valkey-glide when glide completes its NAPI migration") | 🔴 transport layer could be orphaned | +| Tests/CI | 2,414 tests, green CI, Valkey 9.1.0 images | ✅ strong for its age | +| Ecosystem | dashboard 3★, hono 13★ — pre-discovery phase | ⚠️ no community safety net | + +**Mitigation shape:** never expose GlideMQ types on a NetScript public surface. The +`MessageQueue`/port boundary is the containment vessel: if the project dies, NetScript swaps the +adapter (worst case: back to the plain Redis adapter) without breaking a single consumer. + +## 5. What survives even if we never ship the adapter + +Three GlideMQ assets are backend-agnostic **design references** with direct NetScript targets: + +1. **AI-native job primitives** (`reportUsage`, `job.stream`, suspend/resume signals, fallback + chains, TPM limiters, flow budgets) → port-level vocabulary for `plugin-workers-core` + + `plugin-ai-core` (the ai plugin currently has zero durable-job semantics). +2. **Dashboard REST/SSE surface** (`GET /api/queues`, per-job detail + logs, pause/resume/drain/ + retry/clean mutations, `/api/events` SSE, `readOnly` + per-action `authorize` callback, flow + usage/budget endpoints) → concrete prior art for #400 S7–S10 consoles and the `/_netscript/*` + introspection plane — including its mistakes (it *is* the log-tail/metrics-chart shape #400's + acceptance line 1 forbids, so it's both a positive and a negative reference). +3. **OTel span conventions** (`glide-mq.queue.add` spans, bring-your-own-tracer `setTracer()`, + optional `@opentelemetry/api` peer, server-side per-minute metrics buckets with zero extra + RTTs, `events:false`/`metrics:false` throughput switches) → input to #399 TC conventions and + the T6 oRPC/AI port. diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/04-ecosystem-mapping.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/04-ecosystem-mapping.md new file mode 100644 index 000000000..b8601708d --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/research/04-ecosystem-mapping.md @@ -0,0 +1,95 @@ +# Ecosystem Mapping — GlideMQ concepts → NetScript surfaces + +Feature-by-feature projection of the GlideMQ surface onto NetScript's plugins, epics, and ports. +Legend: **A** = adapter concern (comes for free if the GlideMQ adapter ships), **P** = port-level +concept NetScript should own regardless of backend, **R** = design reference only. + +## Workers (`plugins/workers` / `plugin-workers-core`) + +| GlideMQ | NetScript today | Mapping | +| --- | --- | --- | +| `Queue.add/addBulk`, `Worker` + processor, concurrency | `defineJob`/`defineJobHandler`, `createWorkersRuntime` over `MessageQueue` | **A** — adapter slots under the existing dispatch | +| retries + exponential/fixed/custom backoff, DLQ | `NackOptions` + `DeadLetterStorePort` | **A** (richer backoff strategies **P** — worth port vocabulary) | +| priorities, LIFO, delays | `EnqueueOptions` delayed execution; no priority concept | **P** — priority belongs in `EnqueueOptions` as optional capability with adapter capability-flags | +| global concurrency / rate limiting (sliding window, token bucket, per-group) | none at seam level | **P** — rate-limit port vocabulary; adapters implement or degrade | +| deduplication (simple/throttle/debounce) | worker idempotency claims (`WorkerIdempotencyClaim`) | **P** — dedup modes generalize the existing idempotency seam | +| stalled-job recovery via XAUTOCLAIM | adapter-specific redelivery | **A** | +| `addAndWait` request-reply | none | **R** — interesting for oRPC service layer, defer | +| batch processing, sandboxed processors | none | **R** | + +## Sagas (`plugins/sagas` / `plugin-sagas-core`) + +| GlideMQ | NetScript today | Mapping | +| --- | --- | --- | +| FlowProducer parent-child trees, DAG (multi-parent, Kahn ordering, cycle detection), chain/group/chord | saga definitions with cascaded effects over redis/list transports | **R** — NetScript's saga model is its own; DAG dependency-resolution + `waiting-children` state are prior art for saga fan-in | +| step jobs (`moveToDelayed` with nextStep) | saga steps | **R** | +| flow budgets (`maxTotalTokens`/`maxTotalCost`, onExceeded fail/pause) | none | **P** — see AI section | + +## Triggers / Cron (`plugins/triggers`, `packages/cron`) + +| GlideMQ | NetScript today | Mapping | +| --- | --- | --- | +| `upsertJobScheduler` — 5-field cron + timezone, intervals, bounded schedulers | `packages/cron` + workers `cron` helpers | **A/R** — scheduler state lives server-side in GlideMQ (`schedulers` hash); NetScript keeps its own cron seam | + +## Streams (`plugins/streams`) + +| GlideMQ | NetScript today | Mapping | +| --- | --- | --- | +| Broadcast fan-out (consumer-group-per-subscription, NATS-style subject filtering, per-subscription retries, no XDEL + XTRIM retention) | KV-watch based delivery, durable stream producer | **R** — the per-subscription consumer-group pattern is the strongest available design for a future Redis/Valkey streams adapter | + +## AI (`plugins/ai` / `plugin-ai-core` / `packages/ai`) — the highest-value mapping + +NetScript's ai plugin has model adapters and a registry but **no durable execution layer**: no cost +tracking, no token streaming persistence, no human-in-the-loop suspend, no failover, no budgets. +GlideMQ's seven AI primitives are exactly that missing layer, and they are queue-level, not +model-level — i.e. they belong at the seam between `plugin-ai-core` and `plugin-workers-core`: + +| GlideMQ primitive | Proposed NetScript port concept | +| --- | --- | +| `job.reportUsage()` + `getFlowUsage` + `getUsageSummary` (rolling per-minute buckets) | `UsageRecorderPort` — model/tokens/cost/latency per execution, aggregation across a workflow | +| `job.stream(chunk)` + `queue.readStream` (per-job stream channel, long-poll) | `ExecutionStreamPort` — durable token streaming that survives worker crash; feeds dashboard SSE | +| `job.suspend()` / `queue.signal()` (suspended ZSet + signals list + timeout sweep) | `SuspensionPort` — human-in-the-loop approval for agent workflows | +| `fallbacks` array + `job.currentFallback` | model failover policy in `plugin-ai-core` retry contract | +| `tokenLimiter` (TPM) dual-axis with RPM | `TokenRateLimiterPort` | +| flow `budget` (maxTotalTokens/maxTotalCost, fail-or-pause) | `BudgetPort` on workflow contracts | +| per-job `lockDuration` | execution lease duration on the job contract | +| vector search over job hashes (Valkey Search, KNN) | **R** only — NetScript has embeddings in `packages/ai`; job-hash vector search is backend-exclusive | + +These ports can be implemented on **any** backend (Deno KV/Garnet via KV counters + streams-lite), +with GlideMQ as the reference/high-performance implementation. This is the "keep them through an +adapter layer" answer to the proposal's question. + +## Telemetry (epic #399) + +- Span naming (`glide-mq.queue.add` + `glide-mq.*` attributes) → compare with TC-1..14 messaging + conventions; NetScript should stay on OTel semconv `messaging.*` where GlideMQ invented its own. +- `setTracer()` bring-your-own-tracer + optional `@opentelemetry/api` peer = same zero-dep default + philosophy as OF-5's opt-in SDK adapter. Validates the #399 design. +- Server-side minute-bucket metrics with zero extra RTTs → pattern for the T7 query/export surface + (per-queue counts/durations without client-side scanning). +- `events:false` / `metrics:false` throughput escape hatches → worth mirroring as telemetry config. + +## Dashboard (epic #400) + +`@glidemq/dashboard` + `@glidemq/hono` are working prior art for the S7 workers console and the +`/_netscript/*` plane: + +- REST: `GET /api/queues` (counts+metrics), `GET /api/queues/:name`, job detail incl. logs, + mutations (pause/resume/drain/retry/clean/obliterate, job remove/retry/promote/changePriority/ + changeDelay, scheduler upsert/remove). +- SSE: `/api/events` lifecycle events incl. `usage`, `suspended`, `budget-exceeded`; per-job + output stream with `?lastId=` resumption → same replay shape as `/_netscript/flows`. +- Governance: `readOnly` flag + per-action `authorize(req, action)` callback with namespaced + action strings (`queue:pause`, `job:retry`, `scheduler:upsert`) → directly reusable authz + vocabulary for #400's gated write-back (acceptance line 2, one-generator-two-callers). +- Anti-pattern flag: the dashboard renders metrics charts and log tails — precisely what #400 + acceptance line 1 assigns to Aspire. Reuse the API/action design, not the screen inventory. + +## Hono example → oRPC/service layer + +`examples/hono-api/index.ts` shows the pattern: a `QueueRegistryImpl` composition root (connection + +queues + processors + concurrency), middleware injecting the registry into request context, a +mounted queue HTTP API + SSE, and graceful shutdown via `registry.closeAll()`. NetScript's +equivalent: plugin registry generation already builds the registry; a `MessageQueue`-level HTTP +surface in the service layer (oRPC contract routes) would be the port of `glideMQApi()` — and is +effectively what #423 (`/_netscript/*` introspection) plus the S7–S10 thin API slices already plan. diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/rfc-glidemq.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/rfc-glidemq.md new file mode 100644 index 000000000..e902d80be --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/rfc-glidemq.md @@ -0,0 +1,131 @@ +# RFC — GlideMQ as an opt-in high-performance backend + AI-execution port vocabulary + +- **Status:** DRAFT — produced by harness run + `copilot-evaluate-proposal-and-documentation--glidemq-rfc`; pending owner ratification and a + separate-session PLAN-EVAL before any issue is filed. +- **Verdict that motivates this RFC:** **conditional-positive.** GlideMQ fits NetScript — but only + in the adapter shape, never as a seam replacement, and only after a Deno compatibility spike + passes. Full evidence: `research.md` + `research/`. + +## 1. Summary + +Adopt GlideMQ in three decoupled tracks: + +1. **Track A — adapter:** an experimental `QueueProvider.GlideMq` adapter behind the existing + `MessageQueue` port in `@netscript/queue`, requiring Valkey/Redis 7.0+, gated on a Deno/NAPI + compatibility spike. All current backends (Deno KV, Garnet-via-Redis-adapter, RabbitMQ, + Postgres) are preserved untouched as the portable floor. +2. **Track B — AI-execution ports:** NetScript-owned port vocabulary for durable AI execution + (usage recording, token streaming, suspend/resume, fallback chains, TPM limits, budgets) in the + `-core` packages, implementable on **any** backend, with GlideMQ as the reference + high-performance implementation. This is the layer `plugins/ai` is missing today. +3. **Track C — design harvest (no dependency):** apply GlideMQ's dashboard API/authz surface and + OTel/metrics patterns as prior art inside the already-planned epics #400 and #399. Track C + costs nothing and is valuable even if Tracks A/B never ship. + +A seam performance benchmark (issue draft `issue-draft-benchmark.md`) is a **hard prerequisite** +for Track A acceptance: no adapter merges without before/after numbers on NetScript workloads. + +## 2. Motivation + +- NetScript's queue seam positions Redis at "> 10,000 msg/s"; GlideMQ's completeAndFetchNext + single-FCALL design (author-benchmarked +9%→+38% over BullMQ on ElastiCache Valkey under real + network latency) is the strongest currently available implementation of that tier, with + cluster-native hash-tagging for free. +- `plugins/ai` has model adapters but no durable-execution semantics; agentic workloads need cost + tracking, budget caps, human-in-the-loop suspension, streaming persistence, and model failover + *as queue-level guarantees*, not per-request middleware. +- Epics #399/#400 are actively designing surfaces GlideMQ has already shipped working versions of; + harvesting them de-risks both epics. + +## 3. Why not "just adopt GlideMQ" (the non-fit half of the verdict) + +1. **Backend exclusivity.** Hard dependency on Valkey Functions + Streams excludes Garnet + (no FUNCTION, no Streams), Deno KV, RabbitMQ, Postgres, Upstash. NetScript's doctrine (Archetype + 2, sagas list-transport precedent) requires the portable floor to survive. +2. **Runtime risk.** Rust NAPI client from a personal valkey-glide fork, CJS-only, with upstream's + own Deno testing still pending. Unverified ≠ incompatible, but it gates everything. +3. **Supply chain.** Bus factor 1, pre-1.0, fast-churning server library versions. Containment via + the port boundary is mandatory: **no GlideMQ type ever appears on a NetScript public surface.** + +## 4. Design + +### 4.1 Track A — `glide-mq` adapter (packages/queue, Archetype 2) + +- New `QueueProvider.GlideMq = 'glide-mq'` + `adapters/glide-mq.adapter.ts` implementing + `MessageQueue` (enqueue → `queue.add`, listen → `Worker` processor, ack/nack → complete/fail + with retry/backoff mapping, `NackOptions.reason` → DLQ). +- Dead-letter: map `getDeadLetterJobs` to `DeadLetterStorePort` (feeds dashboard S12 / #553/#555). +- Discovery: Aspire env (`ConnectionStrings__valkey` / `services__valkey__*`); CLI scaffold gains a + Valkey container resource option (OQ2). +- Capability degradation is explicit: priorities/rate-limits/dedup surface as optional + `EnqueueOptions` capabilities the adapter advertises; other adapters no-op or degrade, mirroring + how KV watch degrades to polling on Deno KV Connect. +- npm dependency enters via the catalog (`catalog:` is npm-only per the netscript-deno-toolchain skill) and stays **internal to the + adapter module**; the adapter is lazily imported so non-users never load the NAPI binary. + +### 4.2 Track B — AI-execution ports (plugin-workers-core / plugin-ai-core) + +Port set (names indicative; OQ3 decides the home package): +`UsageRecorderPort`, `ExecutionStreamPort`, `SuspensionPort`, `TokenRateLimiterPort`, `BudgetPort`, +plus fallback-chain policy on the ai retry contract and execution-lease (`lockDuration`) on the job +contract. Reference implementations: KV-backed (portable floor, works on Deno KV/Garnet) and +GlideMQ-native (ceiling, if Track A ships). Contract-first per doctrine: schema/type contract → +implementation → tests. + +### 4.3 Track C — design harvest into #399/#400 + +- #400: adopt the `authorize(req, action)` namespaced action-string model for gated write-back; + treat `@glidemq/dashboard`'s REST/SSE inventory as the S7 console prior art and its + log-tail/metrics screens as the documented anti-pattern (acceptance line 1). +- #399: keep OTel semconv `messaging.*` naming (recommendation for OQ4); mirror the BYO-tracer + zero-dep default; evaluate server-side minute-bucket metrics for the T7 query surface; add + `events`/`metrics` style throughput escape hatches to telemetry config. + +## 5. Compatibility & preservation guarantees + +| Backend | After this RFC | +| --- | --- | +| Deno KV | unchanged default | +| Garnet | unchanged (Redis adapter for queue/KV; sagas list-transport) | +| Redis / Valkey (plain) | unchanged `redis` provider | +| RabbitMQ / Postgres | unchanged | +| Valkey/Redis 7+ | **new opt-in** `glide-mq` provider | + +## 6. Risks & mitigations + +| Risk | Severity | Mitigation | +| --- | --- | --- | +| speedkey NAPI fails under Deno | high | Phase 0 spike gates everything; fallback: pure-TS client speaking the documented FCALL wire protocol (keeps 1-RTT server half, loses Rust-client half) | +| upstream abandonment (bus factor 1) | high | port containment; worst-case swap back to `redis` provider is invisible to consumers | +| pre-1.0 API churn / LIBRARY_VERSION bumps | medium | pin exact version; adapter-internal types only | +| perf claims don't reproduce on NetScript workloads | medium | benchmark prerequisite (F8) — adapter acceptance requires before/after numbers | +| scope creep into seam redesign | medium | Tracks are independent; Track B ports are backend-agnostic by construction | + +## 7. Phasing + +- **Phase 0 (spike, ~1 day):** `deno run` loading `npm:glide-mq` + speedkey against a Valkey + container; enqueue/process/complete round-trip; record verdict. **Kill-switch:** NAPI failure → + Track A paused (re-evaluate on upstream valkey-glide NAPI migration or wire-protocol client), + Tracks B/C proceed unaffected. +- **Phase 1:** benchmark reintroduction (issue draft) — baseline current adapters. +- **Phase 2:** Track A adapter + Aspire/CLI provisioning + benchmark delta; experimental flag. +- **Phase 3:** Track B ports with KV reference implementation; GlideMQ implementation if Phase 2 + landed. +- **Continuous:** Track C inside #399/#400 design work (no new issues needed; cite this RFC in + those epics). + +## 8. Alternatives considered + +- **Adopt GlideMQ as the queue seam** — rejected: breaks Garnet/Deno KV/AMQP/Postgres, violates + Archetype 2 and the portable-floor precedent. +- **BullMQ adapter instead** — weaker: 53 EVAL scripts (NOSCRIPT churn), no AI primitives, same + Redis-only exclusivity without the 1-RTT design; ioredis works in Deno though — it remains the + fallback if Phase 0 fails and the wire-protocol route is unwanted. +- **Do nothing** — leaves `plugins/ai` without durable execution and cedes the AI-queue category; + Track C harvest would still be worth doing. + +## 9. Unresolved questions carried to PLAN-EVAL + +OQ1–OQ4 in `research.md`. OQ3 (port home package) is the only one that must be resolved before +Track B implementation; OQ1 gates Track A; OQ2/OQ4 are safe to defer. diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/supervisor.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/supervisor.md new file mode 100644 index 000000000..064c276f6 --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/supervisor.md @@ -0,0 +1,41 @@ +# Supervisor Identity — copilot-evaluate-proposal-and-documentation--glidemq-rfc + +Written at run start per `workflow/lane-policy.md` § Supervisor identity. + +| Field | Value | +| --- | --- | +| Model | GitHub Copilot cloud coding agent (Claude-family supervisor) | +| Session | GitHub Copilot cloud agent session (task: evaluate GlideMQ proposal) | +| Host | GitHub Actions runner (ubuntu, /home/runner/work/netscript/netscript) | +| Checkout | /home/runner/work/netscript/netscript | +| Worktree | same (single sandbox checkout) | +| Branch | copilot/evaluate-proposal-and-documentation | +| Baseline | 2779fb249d990fbac283e98158525bc87bb699d7 (branch base, 2026-07-09) | +| Run ID | `copilot-evaluate-proposal-and-documentation--glidemq-rfc` | + +## Run shape + +Research/evaluation run (RFC-producing). Deliverables are **artifacts, not code**: an exhaustive +research corpus, an integration verdict, an RFC draft (verdict was positive — see `verdict.md`), a +benchmark-reintroduction issue draft, and a new workflow doc +(`.llm/harness/workflow/research-rfc-run.md`) documenting this run shape. No `packages/` or +`plugins/` source is touched. + +## Lane table in force + +| Tier | Binding | Role in this run | +| --- | --- | --- | +| A | Copilot cloud agent (this session) | supervision, synthesis, RFC authoring, run-dir upkeep | +| B | Copilot internal sub-agents (explore/research) | codebase seam mapping, external GlideMQ dossier | +| E | OpenHands (minimax-M3 / qwen-3.7-max) | **BLOCKED in this sandbox** — see overrides | + +## Recorded lane/eval overrides + +- **PLAN-EVAL / IMPL-EVAL blocked launch.** This sandboxed cloud-agent session has no OpenHands + launch surface. Per lane-policy blocked-lane handling, the blocked launch is recorded here and in + `drift.md`. The RFC and issue draft are **drafts pending owner ratification** (seed-run drafts-only + discipline); a follow-up OpenHands PLAN-EVAL on the RFC is the recommended next step before any + implementation issue is filed. +- **Tier B binding override.** Research sub-agents are Copilot cloud internal agents (Haiku/Sonnet + class), not Opus 4.8 — the only research lane available in this environment. Authorized by the + task assignment itself (owner dispatched this task to the Copilot cloud agent). diff --git a/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/worklog.md b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/worklog.md new file mode 100644 index 000000000..982d9efe8 --- /dev/null +++ b/.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/worklog.md @@ -0,0 +1,51 @@ +# Worklog — copilot-evaluate-proposal-and-documentation--glidemq-rfc + +## Design + +Evaluation-only run; the "design" is the artifact architecture: + +1. **Public surface (of the run):** `research.md` (findings index), `research/01..04` (corpus), + `rfc-glidemq.md` (verdict-bearing RFC, DRAFT), `issue-draft-benchmark.md`, + `.llm/harness/workflow/research-rfc-run.md` (+ activation.md step 11 pointer). +2. **Domain vocabulary:** findings F1–F9; tracks A (adapter) / B (AI-execution ports) / C (design + harvest); A/P/R classification in the ecosystem mapping; portable floor vs capable ceiling. +3. **Ports:** none created — the run *proposes* ports (UsageRecorder/ExecutionStream/Suspension/ + TokenRateLimiter/Budget) without implementing them. +4. **Commit slices:** (1) activation + supervisor.md; (2) research corpus + RFC; + (3) issue draft + workflow doc + close artifacts. +5. **Deferred scope:** see plan.md. +6. **Contributor path:** start at `research.md`, drill into `research/03` for the verdict pivot, + then `rfc-glidemq.md` §7 for phasing. + +## Evidence + +- **Slice 1** (commit "harness: activate glidemq-rfc research run"): supervisor.md with blocked + evaluator lanes recorded. +- **Slice 2** (commit "glidemq-rfc: research corpus + RFC draft"): + - Internal baseline verified by direct inspection: `packages/queue/ports/options.ts` + (`QueueProvider` enum), `packages/queue/adapters/` (deno-kv/redis/amqp/postgres + 3 DLQ + stores), `packages/kv/adapters/`, `packages/plugin-sagas-core/src/transports/` (redis + + Garnet list transports), `packages/plugin-workers-core/mod.ts`, `packages/bench/README.md` + (agent self-bench, not a transport bench), `packages/cli/.../providers.ts` (`garnet://`). + - External corpus from primary sources: `glide-mq` README/ARCHITECTURE/OBSERVABILITY/ + package.json (v0.15.4, CJS, `engines.node>=20`, `@glidemq/speedkey` sole dep), + `glidemq-examples/hono-api/index.ts`, `glidemq-dashboard` README. Third-party verification: + Garnet docs (Lua-only scripting, no Streams), Deno Node/NAPI compat docs. + - Verdict: conditional-positive; RFC drafted with preservation guarantees and kill-switches. +- **Slice 3** (this commit): benchmark issue draft (taxonomy-complete, drafts-only), workflow doc + `research-rfc-run.md` promoted from this run, activation.md step 11, close artifacts. + +## Gates + +No code gates apply (markdown-only changes; no `packages/`/`plugins/` sources touched — verified +via git status). Formatting: prose markdown under `.llm/` is outside the package-quality fmt gate +per netscript-tools rules. + +## Verdict handoff + +IMPL-EVAL blocked in this sandbox (see drift.md). Recommended next steps for the owner: + +1. Ratify or amend `rfc-glidemq.md`; run a separate-session PLAN-EVAL (OpenHands minimax-M3) + treating the RFC as the plan. +2. File `issue-draft-benchmark.md` (Phase 1 prerequisite). +3. Schedule the Phase-0 Deno/NAPI spike (OQ1) before any Track-A issue is filed.