Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .llm/harness/workflow/activation.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ Harness mode activates when the user says `use harness` or explicitly asks for a
10. For a **seed run** (planning-only: the deliverable is a GitHub board — epics, milestones,
issues, design packs — not code), read `workflow/seed-run.md` and follow its stage contracts
A–I. Seed runs are drafts-only until owner ratification.
11. For a **research/RFC run** (evaluation-only: the deliverable is a verdict on an external
technology proposal — an evidence corpus plus an RFC or a documented rejection), read
`workflow/research-rfc-run.md` and follow its stage contracts A–H. Research/RFC runs are
drafts-only until owner ratification.

## Run ID

Expand Down
115 changes: 115 additions & 0 deletions .llm/harness/workflow/research-rfc-run.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Research/RFC Run — Technology Evaluation → Verdict → RFC

Operating protocol for **research/RFC runs** — evaluation-only harness runs that take an external
technology proposal ("should NetScript integrate X?") from zero to a **verdict-bearing research
corpus**: an evidence-grade codebase baseline, a primary-source external dossier, a compatibility
matrix against every backend/runtime NetScript must preserve, an ecosystem projection onto
NetScript's own surfaces, and — only when the verdict is positive — an RFC plus any enabling issue
drafts. Like a seed run, a research/RFC run is **drafts-only until owner ratification**; unlike a
seed run, its deliverable is a *decision*, not a board.

> **Provenance.** Promoted from `copilot-evaluate-proposal-and-documentation--glidemq-rfc`
> (GlideMQ evaluation, 2026-07-09), which produced a conditional-positive verdict, a three-track
> RFC, and a benchmark-prerequisite issue draft. As with `seed-run.md`, this file freezes the
> **stage contracts**, not the exemplar's folder tree.

## When to use

The owner brings an external library/technology/architecture proposal and asks for an evaluation
whose outcome is unknown in advance ("either a clear explanation of why it doesn't fit, or a
complete RFC"). Triggers: a candidate dependency, a competing architecture, a "we could replace our
X with Y" idea. **Do not** use it when integration is already decided (that's run-loop planning) or
when the deliverable is a board (that's a seed run). A ratified positive RFC typically *feeds* a
seed run or run-loop issues; the two shapes chain, they don't merge.

## Run layout

Run dir `.llm/runs/<run-id>/` with the standard mandatory artifacts (`supervisor.md` first, then
`research.md`, `plan.md`, `worklog.md`, `context-pack.md`, `drift.md`) plus the shape-specific
artifact classes:

- `research/` — the evidence corpus, free-form layout; the exemplar used four numbered files
(internal baseline / external dossier / compatibility matrix / ecosystem mapping).
- `rfc-<subject>.md` — only if the verdict is positive; carries its DRAFT/pending-ratification
status in the header.
- `issue-draft-*.md` — enabling issues (prerequisites the RFC depends on), taxonomy-complete
(labels + milestone per the netscript-pr skill) but **never filed by the run itself**.

## Stage contracts

### A — Activation & framing

Write `supervisor.md` (lane table, blocked-eval overrides). Extract the proposal's **explicit
deliverables** into the run checklist — owners often embed side-deliverables inside the prose
(the exemplar carried three: RFC-or-rejection, a benchmark issue, and this workflow doc). Missing
one is a contract failure.

### B — Internal baseline (parallel with C)

Map every NetScript surface the candidate touches, with verified paths/exports: the ports it would
sit behind, the adapters it would coexist with, the plugins that consume the seam, the doctrine
constraints (archetype, thinness, wrap-don't-reinvent), and any **precedent** for the integration
shape (the exemplar's key find: the sagas Garnet list-transport proves the
"portable floor + capable ceiling" adapter policy). Delegate to a Tier-B explore agent, but the
supervisor re-verifies load-bearing claims by direct inspection before they enter `research.md`.

### C — External dossier (parallel with B)

Primary sources only: the candidate's repo docs, source layout, package manifest, changelog,
examples, ecosystem repos. Required axes — value proposition and mechanism; full API surface;
**server/runtime hard requirements** (the compatibility pivot); AI/domain-specific primitives;
observability; license, maturity, release cadence, **bus factor**, fork/vendoring risks. If the
docs site is unreachable from the sandbox, find the source-of-truth markdown in the repo (docs
sites are usually generated) and record the mapping in `research.md`. Save the dossier into
`research/` — sub-agent output is otherwise lost with the session.

### D — Compatibility matrix (the verdict pivot)

Cross B×C: for **every** backend/runtime NetScript supports, a per-cell verdict (full / degraded /
hard blocker) with cited primary evidence. Verify the two or three pivotal cells independently of
the sub-agents (the exemplar: Garnet's missing FUNCTION+Streams, Deno NAPI uncertainty — each
double-checked against upstream docs before becoming findings). This stage decides the verdict
shape: a hard blocker on a must-keep backend converts "replace" proposals into "adapter or reject".

### E — Ecosystem projection

Feature-by-feature mapping of the candidate onto NetScript surfaces, each row classified:
**A** adapter concern (free with integration), **P** port-level concept NetScript should own
regardless of backend, **R** design reference only. The P and R rows are the run's durable value —
they survive even a negative verdict, and they connect the evaluation to open epics/issues (cite
issue numbers).

### F — Verdict & RFC

`research.md` states numbered, independently-verifiable findings + open questions (each marked
safe-to-defer or must-resolve). The verdict must be explicit: fit / no-fit / **conditional** (with
named gates — the exemplar gated on a runtime spike and a benchmark baseline). Positive verdict →
RFC with: summary, motivation, the *non-fit half* (what the candidate must not become), design with
preservation guarantees for existing backends, risk register with containment (e.g. "no candidate
type on a public surface"), phasing **with kill-switches**, alternatives considered, and the open
questions carried to PLAN-EVAL.

### G — Side deliverables & workflow review

Author any issue drafts the RFC depends on (drafts-only, taxonomy-complete). If the run followed a
shape not yet codified under `workflow/`, document it there (this file is that step for the
exemplar).

### H — Close

Update `worklog.md`, `context-pack.md`, `drift.md` (record blocked evaluator launches and
unreachable sources). Hand off: the RFC's next step is a **separate-session PLAN-EVAL** treating
the RFC as the plan under review, then owner ratification, then seed-run or run-loop issues.

## Invariants

- **Drafts-only.** The run does not file issues, opens no non-run PRs, adds no dependencies. Ratification
gates everything downstream.
- **Generator ≠ evaluator.** The run that authored the RFC never certifies it; the RFC header names
its pending-PLAN-EVAL status, and blocked evaluator launches are recorded per `lane-policy.md`.
- **Primary sources or it didn't happen.** Marketing claims are checked against the candidate's own
internal docs (the exemplar caught README-claimed Deno support contradicted by the project's
HANDOVER.md) and upstream third-party docs for backend capabilities.
- **Preservation is explicit.** The RFC must state, backend by backend, what remains unchanged.
- **Findings are falsifiable.** Every finding in `research.md` names its source so a reviewer can
re-verify it without the session.
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Context Pack — copilot-evaluate-proposal-and-documentation--glidemq-rfc

**One-paragraph resume:** Evaluation of GlideMQ (avifenesh/glide-mq, v0.15.4) for NetScript.
Verdict: **conditional-positive** — GlideMQ hard-requires Valkey/Redis 7+ Functions + Streams, so
it can never replace the `MessageQueue` seam (Garnet has neither; Deno KV/AMQP/Postgres are out),
but it fits as (A) an opt-in adapter gated on a Deno/NAPI spike, (B) the blueprint for
NetScript-owned AI-execution ports (usage/streaming/suspend/budget/failover — the layer
`plugins/ai` is missing), and (C) design harvest for epics #399 (OTel) and #400 (dashboard). RFC
drafted; benchmark-reintroduction issue drafted as a hard prerequisite; both pending owner
ratification + separate-session PLAN-EVAL (evaluator lanes blocked in this sandbox).

## Artifact index

| Artifact | State |
| --- | --- |
| `research.md` + `research/01..04` | complete — findings F1–F9, OQ1–OQ4 |
| `rfc-glidemq.md` | DRAFT, pending PLAN-EVAL + ratification |
| `issue-draft-benchmark.md` | draft, not filed |
| `.llm/harness/workflow/research-rfc-run.md` | landed (workflow promoted from this run) |
| plan/worklog/drift/supervisor | current |

## Key facts a resuming session must not re-derive

- GlideMQ: 44 Lua functions via FUNCTION LOAD/FCALL, streams+XAUTOCLAIM consumer model,
`@glidemq/speedkey` Rust NAPI (personal valkey-glide fork), CJS, Node>=20, Apache-2.0, bus
factor 1, Deno NAPI support claimed in README but flagged untested in the project's HANDOVER.md.
- Garnet: Lua EVAL only (no FUNCTION), no Streams → permanent GlideMQ blocker.
- NetScript precedent for the adapter policy: sagas Garnet list-transport
(`packages/plugin-sagas-core/src/transports/`).
- `packages/bench` = agent self-bench, NOT a transport benchmark; legacy
`rickylabs/netscript-start/benchmark` is 404.

## Next actions (owner)

1. PLAN-EVAL the RFC (OpenHands minimax-M3, separate session).
2. File the benchmark issue.
3. Phase-0 spike: `npm:glide-mq` under Deno 2.9 + Valkey container (OQ1 kill-switch for Track A).
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Drift Log — copilot-evaluate-proposal-and-documentation--glidemq-rfc

Append-only.

- **2026-07-09 · blocked-lane · significant** — OpenHands PLAN-EVAL/IMPL-EVAL launch surfaces are
unavailable in the Copilot cloud-agent sandbox. Recorded per `lane-policy.md` blocked-lane
handling; fallback is drafts-only output + explicit pending-PLAN-EVAL status on the RFC.
Mirrored in `supervisor.md`.
- **2026-07-09 · source-unreachable · minor** — `https://glidemq.dev/*` is DNS-blocked in the
sandbox. Substituted the site's source-of-truth markdown in `avifenesh/glide-mq/docs/`
(site is generated from it); mapping recorded in `research.md`.
- **2026-07-09 · source-unreachable · minor** — legacy benchmark repo `rickylabs/netscript-start`
returns 404 via the GitHub API (private or removed). The benchmark issue draft scopes the
successor from first principles instead of porting the legacy suite.
- **2026-07-09 · lane-binding · minor** — Tier-B research lanes ran on Copilot internal
explore/research agents (not Opus 4.8) — the only research surface available here. Load-bearing
claims (Garnet FUNCTION/Streams gaps, GlideMQ FCALL dependency, Deno NAPI uncertainty, queue/kv
adapter inventories) were re-verified first-hand by the supervisor before entering findings.
- **2026-07-09 · tooling · minor** — `rtk` prefix convention not applied: the runner sandbox has
no `rtk` binary on PATH; raw git/grep used via the platform's built-in tools instead.
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Issue Draft — Reintroduce the seam performance benchmark suite

> **Draft only.** Authored by harness run `copilot-evaluate-proposal-and-documentation--glidemq-rfc`
> per seed-run drafts-only discipline; the owner files it (or authorizes filing) after ratifying
> `rfc-glidemq.md`. Labels/milestone follow `.github/labels.yml` + the netscript-pr taxonomy.

---

**Title:** `feat(bench): reintroduce the seam performance benchmark suite (queue/KV transports)`

**Labels:** `type:feature` · `area:queue` (or nearest area label for the queue/KV seams) ·
`priority:p2` · `status:plan` · `wave:v1`

**Milestone:** `Backlog / Triage` (promote to the beta milestone that hosts the GlideMQ Phase-1
work if the RFC is ratified)

**Body:**

## Problem

When the queue/KV seams behind the plugins (workers, sagas, streams, triggers) were first designed,
adapter selection was informed by a benchmark suite in the legacy repo
(`rickylabs/netscript-start/tree/master/benchmark`). That repo is no longer accessible and the
suite has no successor here: `packages/bench` (`@netscript/bench`) is the **agent self-bench**
instrument (scores coding-agent effectiveness), not a transport benchmark. Today we cannot answer
"what does adapter X cost on a NetScript workload" with first-party numbers — including the
provider-tier claims already printed in `packages/queue/ports/options.ts` doc comments
(Deno KV "< 1000 msg/s", Redis "> 10,000 msg/s").

## Why now

The GlideMQ RFC (see run dir `.llm/runs/copilot-evaluate-proposal-and-documentation--glidemq-rfc/`)
makes a benchmark a **hard acceptance prerequisite** for any new queue adapter: GlideMQ's headline
claim (+9%→+38% over BullMQ via 1-RTT completeAndFetchNext) is author-published and must be
re-derived on NetScript workloads before it can justify a dependency. Landing the benchmark
*before* the adapter gives an untainted baseline and a permanent way to measure the benefit of this
and every future backend.

## Scope

- A workload-driven benchmark harness exercising the **`MessageQueue` port** (and optionally
`WatchableKv` watch latency) across the shipped adapters: Deno KV, Redis, Garnet
(Redis-compatible path), RabbitMQ, Postgres — same workload, same measurement plane.
- Metrics: throughput (jobs/s) at fixed concurrency steps (e.g. c=1/5/10/20), enqueue→complete
latency percentiles (p50/p95/p99), RTTs per job cycle, dead-letter/retry overhead.
- Scenarios: fire-and-forget job, retry-with-backoff, delayed message, worker fan-out; sagas
transport scenario (redis-transport vs list-transport) as a stretch goal.
- Runs locally against Aspire-provisioned containers (compose parity with `scaffold.runtime`
infra); results emitted as a machine-readable report (JSON) + markdown summary, suitable for PR
comments and future CI trending.
- Home: extend `packages/bench` with a transport-bench module or a sibling `bench/` surface —
decide against doctrine (Archetype 6 tooling vs Archetype 3 runtime) during planning; **do not**
entangle it with the agent self-bench scoring pipeline.

## Non-goals

- Cross-cloud managed-service benchmarking (ElastiCache, Upstash) — local/containerized only.
- Load-testing the HTTP/oRPC layer (separate concern).
- Shipping the GlideMQ adapter itself (RFC Track A, separate issue once ratified).

## Acceptance

- `deno task bench:seams` (name TBD) runs the suite against at least Deno KV + Redis + Garnet and
produces the JSON + markdown reports.
- Baseline numbers for all current adapters are committed/published before any new queue adapter
merges; the GlideMQ adapter PR (if ratified) must include a before/after delta from this suite.
- README documents methodology honestly (local containers ≠ production networks; note that
GlideMQ's own numbers were measured over real network latency, where 1-RTT designs compound).

Refs #301. Related: RFC run dir above; #399/#400 consume some of the same primitives.
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Plan — copilot-evaluate-proposal-and-documentation--glidemq-rfc

Research/RFC run (evaluation-only, no framework code). Shape documented in
`.llm/harness/workflow/research-rfc-run.md` (authored by this run per the owner's request).

## Scope (from the owner's proposal, deliverables extracted)

1. Evaluate GlideMQ (avifenesh/glide-mq + docs + examples + dashboard) against NetScript's
queue/KV seam architecture and plugins.
2. Map compatibility with backends NetScript must keep: Garnet, Deno KV, Redis, RabbitMQ,
Postgres — "adapter layer maybe? You tell me".
3. Produce an exhaustive research corpus in the run dir; verdict → RFC (if positive) or rejection
rationale.
4. Author a benchmark-reintroduction issue (legacy `netscript-start/benchmark` has no successor).
5. Review and document the naturally-followed workflow in `.llm/harness/workflow/`.

## Archetype / overlays

No package archetype is implemented; the evaluation *reasons about* Archetype 2 (queue/KV
integration seams) and Archetype 5 (plugins). Overlay: `SCOPE-docs.md` in spirit — all outputs are
run-dir markdown plus one workflow doc.

## Locked decisions

- **LD1** Evaluation verdict is **conditional-positive**: GlideMQ enters only as an adapter behind
`MessageQueue` + as port vocabulary + design harvest; never a seam replacement (F1–F3).
- **LD2** All GitHub-facing outputs (RFC, issue) are drafts pending owner ratification and a
separate-session PLAN-EVAL (blocked evaluator lane recorded).
- **LD3** The workflow doc freezes stage contracts, not the exemplar's folder tree (seed-run
precedent).

## Open-decision sweep

OQ1 (Deno NAPI spike) — must resolve before Track A; gates nothing in this run. OQ2 (Aspire Valkey
resource) — safe to defer. OQ3 (port home package) — must resolve before Track B implementation;
safe to defer past this run. OQ4 (span naming) — safe to defer into #399 T1.

## Risk register

| Risk | Mitigation |
| --- | --- |
| Sandbox cannot run OpenHands evaluators | recorded as blocked lane; RFC header carries pending-PLAN-EVAL status |
| glidemq.dev unreachable | source-of-truth repo docs read directly; mapping recorded in research.md |
| legacy benchmark repo 404 | recorded in drift; issue draft scopes the successor from first principles |

## Debt implications

None — no framework code touched. The RFC names future debt-avoidance rules (no GlideMQ types on
public surfaces).

## Deferred scope

Filing the issue; running the Deno spike; any adapter/port implementation; PLAN-EVAL of the RFC.
Loading