Skip to content

Record objects' HTTP traffic as typeId-keyed cassettes - #12

Open
andreBurnt wants to merge 3 commits into
mempko:mainfrom
andreBurnt:ratchet/c1-traffic-recorder
Open

Record objects' HTTP traffic as typeId-keyed cassettes#12
andreBurnt wants to merge 3 commits into
mempko:mainfrom
andreBurnt:ratchet/c1-traffic-recorder

Conversation

@andreBurnt

Copy link
Copy Markdown
Contributor

Record objects' HTTP traffic as typeId-keyed cassettes

Piece 1 of the #11 series, per the order we agreed there: the recorder first, so evidence exists before anything judges it. No judging in this PR, no manifest declarations, no new dependencies. 3 commits, 4 files, +385/-5.

What this adds

Every HTTP request an object makes through HttpClient now leaves a trace: a cassette under cassettes:<typeId> in Storage, written by a CassetteRecorder abject spawned permanent at boot. Restart the object and the evidence is still there, because it is keyed by the durable typeId, not the ephemeral AbjectId.

The seam

I asked in #11 whether you wanted an open exchange event or a scoped subscribe message. Reading the dependents protocol answered it: changed() already IS both. HttpClient emits changed('httpExchange', ...), and only objects that sent addDependent receive it. Anything on the bus may subscribe (your everything-on-the-bus spirit), nothing sees traffic without subscribing (my scoping concern). If you want a different shape, the emission is one method and moves cheaply.

Everything crosses the bus as messages: the subscription (addDependent), the exchanges (events), the persistence (Storage requests), the attribution lookups (Registry requests). HttpClient and the recorder are both workerEligible and may sit on different threads. There is no module state anywhere in the path.

One line lands in core/abject.ts: a protected hasDependents accessor, so an unobserved HttpClient skips building event payloads entirely. Zero work when nobody subscribed.

What the redaction does, and does not, promise

Redaction is stem-based on NAMES, applied before emission inside HttpClient: headers, query params, and JSON/form body fields whose name contains a secret stem (key, token, secret, passw, credential, session, signature, cookie, auth) are replaced with REDACTED before anything crosses the bus. That catches authorization, client_secret, refresh_token, x-amz-security-token, X-Amz-Signature, an access_token in an OAuth response body, a password in a form post - and whatever similarly-named header a generated object invents. A false positive redacts something harmless; a false negative persists a live credential, so the matcher errs toward matching.

What it does NOT do: find a secret stored under an innocent name, or scan free text. It is a name-pattern scrub, not a secret scanner. If you want value-based redaction against what SecretsVault actually holds, that is a conversation I would have before building it - it means the vault's values reach HttpClient for comparison, which has its own threat model. The request handed to the network is untouched either way; only the emitted copy is scrubbed.

Two more honest boundaries: bodies are capped at 64K characters (not bytes) with a truncated flag, and requests that throw before a response exists (timeout, DNS failure, your SSRF guard) produce no cassette - only resolved responses are evidence, including 4xx/5xx. If you want transport-failure evidence too, that is an event-shape question and I would rather design it with you than guess.

Design choices you should check

  1. Attribution is registry-verified. The event carries msg.routing.from, and the recorder resolves it through resolveCallerIdentity - your existing anti-spoofing helper - rather than trusting the payload. A caller with no durable typeId is not recorded: a cassette that cannot be tied to a type is evidence about nothing.

  2. Eviction is bucketed by endpoint (method + path, query stripped): FIFO cap 5 per bucket, 50 per typeId, and global overflow always comes out of the LARGEST bucket. Your Fitness gate: judge generated objects by evidence they cannot edit #11 bucketing suggestion, landed here where _http volume makes it real - a hot parameterized endpoint cannot evict the one recording of a rarely-hit one, no matter how old that recording is.

  3. The recorder never silently gives up. HttpClient discovery retries with capped backoff for as long as the recorder lives, and the subscription does not wait on Storage. While Storage is missing, recording continues in memory; persistence catches up when it appears, merging what the store already held so a restart never clobbers evidence. Both behaviors exist because scripted verification caught the failure modes, not because I thought of them.

  4. Discovery is by registered name (discoverDep('HttpClient') / discoverDep('Storage')), the same pattern HealthMonitor and the Abject base class already use. I know your rule about naming specific objects - the sandbox hardcoding you flagged in Fitness gate: judge generated objects by evidence they cannot edit #11 special-cased behavior per name, which this does not. If you want capability-based discovery here instead, say so and I will follow whatever HealthMonitor migrates to.

  5. The cassette shape is contributor-authored infrastructure, not generator-authored evidence. Your Fitness gate: judge generated objects by evidence they cannot edit #11 concern was the generator declaring the standards it is judged by. Nothing here is written by a generator: the recorder records what actually crossed the wire, and a judged object cannot influence what gets recorded about it. The shape itself is yours to review like any other code.

  6. fetchBase64 does not emit. Base64 image fetches are not replay evidence and would blow the body cap for nothing. Say the word if you want it covered.

Verification

tsc --noEmit clean. Three one-shot scripts (below, not committed - your convention), 16 checks total, run with npx tsx --test <file>:

  1. recorder-unit.ts - recorder behaviors against the real bus and real dependents protocol: typeId keying, no-typeId skip, merge-not-clobber on restart, per-bucket eviction, largest-bucket overflow eviction, late-registering HttpClient, recording through a Storage outage.
  2. http-emit-unit.ts - emission behaviors with only makeRequest stubbed (your SSRF guard blocks loopback, so no live socket): caller attribution, header/query/body redaction incl. the stem cases above, truncation, all four entry points, zero work when nobody subscribes.
  3. recorder-live.ts - end to end with production classes: real Registry, real Storage, real HttpClient, real recorder. Boot-style spawn and register, discovery, subscription, a caller's request, cassette read back from Storage with the secret gone.
lab/recorder-unit.ts
/**
 * One-shot verification for the CassetteRecorder abject (PR-A, abject#11 series).
 * Not committed — repo convention keeps tests out of the tree; this script goes
 * in the PR description. Run: npx tsx --test lab/recorder-unit.ts
 *
 * Real plumbing: a real MessageBus, real Abject subclasses, real dependents
 * protocol. Only Registry and Storage are test doubles (in-memory), speaking
 * the exact message protocol of the real ones.
 */
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { MessageBus } from '../src/runtime/message-bus.js';
import { Abject } from '../src/core/abject.js';
import { request } from '../src/core/message.js';
import type { AbjectId, AbjectMessage, TypeId } from '../src/core/types.js';
import { CassetteRecorder } from '../src/objects/cassette-recorder.js';

const iface = (name: string) => ({
  name,
  description: `test ${name}`,
  version: '0.0.1',
  interface: { id: `test:${name.toLowerCase()}`, name, description: name, methods: [] },
});

/** In-memory Registry double: discover-by-name + lookup(objectId) -> {name, typeId}. */
class FakeRegistry extends Abject {
  private byName = new Map<string, AbjectId>();
  private identities = new Map<AbjectId, { name: string; typeId?: TypeId }>();
  constructor() { super({ manifest: iface('Registry') }); }
  addEntry(id: AbjectId, name: string, typeId?: TypeId): void {
    this.byName.set(name, id);
    this.identities.set(id, { name, typeId });
  }
  protected async onInit(): Promise<void> {
    this.on('discover', (msg: AbjectMessage) => {
      const { name } = msg.payload as { name: string };
      const id = this.byName.get(name);
      return id ? [{ id }] : [];
    });
    this.on('lookup', (msg: AbjectMessage) => {
      const { objectId } = msg.payload as { objectId: AbjectId };
      return this.identities.get(objectId) ?? null;
    });
  }
}

/** In-memory Storage double speaking the real get/set protocol. */
class FakeStorage extends Abject {
  readonly data = new Map<string, unknown>();
  constructor() { super({ manifest: iface('Storage') }); }
  protected async onInit(): Promise<void> {
    this.on('get', (msg: AbjectMessage) => {
      const { key } = msg.payload as { key: string };
      return this.data.get(key) ?? null;
    });
    this.on('set', (msg: AbjectMessage) => {
      const { key, value } = msg.payload as { key: string; value: unknown };
      this.data.set(key, value);
      return true;
    });
  }
}

/** Stands in for HttpClient: accepts addDependent (universal) and lets the
 *  test fire a real changed('httpExchange') through the real dependents path. */
class FakeHttpClient extends Abject {
  constructor() { super({ manifest: iface('HttpClient') }); }
  protected async onInit(): Promise<void> {
    this.on('emitExchange', (msg: AbjectMessage) => {
      this.changed('httpExchange', msg.payload);
      return true;
    });
  }
}

/** Test driver: an initialized abject the test uses to send real requests. */
class Driver extends Abject {
  constructor() { super({ manifest: iface('Driver') }); }
  protected async onInit(): Promise<void> {}
  async call<T>(to: AbjectId, method: string, payload: unknown): Promise<T> {
    return this.request<T>(request(this.id, to, method, payload));
  }
}

async function until(cond: () => boolean, ms = 3000): Promise<void> {
  const deadline = Date.now() + ms;
  while (!cond()) {
    if (Date.now() > deadline) throw new Error('condition not met in time');
    await new Promise(r => setTimeout(r, 25));
  }
}

interface Rig {
  registry: FakeRegistry;
  storage: FakeStorage;
  http: FakeHttpClient;
  driver: Driver;
  recorder: CassetteRecorder;
}

async function rig(opts: { registerHttp?: boolean; registerStorage?: boolean } = {}): Promise<Rig> {
  const bus = new MessageBus();
  const registry = new FakeRegistry();
  await registry.init(bus);
  const storage = new FakeStorage();
  const http = new FakeHttpClient();
  const driver = new Driver();
  const recorder = new CassetteRecorder({ flushMs: 10, discoverRetryMs: 20 });
  for (const a of [storage, http, driver, recorder]) await a.init(bus, undefined, registry.id);
  if (opts.registerHttp !== false) registry.addEntry(http.id, 'HttpClient');
  if (opts.registerStorage !== false) registry.addEntry(storage.id, 'Storage');
  if (opts.registerHttp !== false) await recorder.ready();
  return { registry, storage, http, driver, recorder };
}

const CALLER = 'caller-1111-2222-3333-444444444444' as AbjectId;
const CALLER_TYPE = 'peer/ws/user/Fetcher' as TypeId;

function exchange(over: Record<string, unknown> = {}) {
  return {
    caller: CALLER,
    request: { method: 'GET', url: 'https://api.example.com/v1/items?page=2', headers: {} },
    response: { status: 200, headers: {}, bodyText: '{"items":[1,2]}' },
    durationMs: 12,
    at: 1756400000000,
    ...over,
  };
}

test('records an httpExchange as a cassette under the caller typeId', async () => {
  const r = await rig();
  r.registry.addEntry(CALLER, 'Fetcher', CALLER_TYPE);

  await r.driver.call(r.http.id, 'emitExchange', exchange());
  await until(() => r.storage.data.has(`cassettes:${CALLER_TYPE}`));

  const stored = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as Array<Record<string, any>>;
  assert.equal(stored.length, 1);
  const c = stored[0];
  assert.equal(c.method, '_http');
  assert.equal(c.request.method, 'GET');
  assert.equal(c.request.url, 'https://api.example.com/v1/items?page=2');
  assert.equal(c.response.status, 200);
  assert.equal(c.response.bodyText, '{"items":[1,2]}');
  assert.equal(c.at, 1756400000000);
});

test('skips exchanges whose caller has no durable typeId', async () => {
  const r = await rig();
  // CALLER registered with a name but NO typeId
  r.registry.addEntry(CALLER, 'Fetcher');

  await r.driver.call(r.http.id, 'emitExchange', exchange());
  await new Promise(res => setTimeout(res, 150));

  assert.equal(r.storage.data.size, 0);
});

test('merges with cassettes already in Storage instead of clobbering', async () => {
  const r = await rig();
  r.registry.addEntry(CALLER, 'Fetcher', CALLER_TYPE);
  const preexisting = [{
    method: '_http',
    request: { method: 'GET', url: 'https://api.example.com/v1/old', headers: {} },
    response: { status: 200, bodyText: 'old' },
    at: 1756300000000,
  }];
  r.storage.data.set(`cassettes:${CALLER_TYPE}`, preexisting);

  await r.driver.call(r.http.id, 'emitExchange', exchange());
  await until(() => {
    const s = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as unknown[];
    return Array.isArray(s) && s.length === 2;
  });

  const stored = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as Array<Record<string, any>>;
  assert.equal(stored[0].request.url, 'https://api.example.com/v1/old');
  assert.equal(stored[1].request.url, 'https://api.example.com/v1/items?page=2');
});

test('evicts FIFO per endpoint bucket, capped at 5 per bucket', async () => {
  const r = await rig();
  r.registry.addEntry(CALLER, 'Fetcher', CALLER_TYPE);

  for (let i = 0; i < 7; i++) {
    await r.driver.call(r.http.id, 'emitExchange', exchange({
      request: { method: 'GET', url: `https://api.example.com/v1/items?page=${i}`, headers: {} },
      at: 1756400000000 + i,
    }));
  }
  // one exchange on a DIFFERENT endpoint must survive the churn above
  await r.driver.call(r.http.id, 'emitExchange', exchange({
    request: { method: 'GET', url: 'https://api.example.com/v1/users', headers: {} },
    at: 1756400009999,
  }));

  await until(() => {
    const s = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as unknown[];
    return Array.isArray(s) && s.length === 6;
  });

  const stored = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as Array<Record<string, any>>;
  const items = stored.filter(c => (c.request.url as string).includes('/v1/items'));
  const users = stored.filter(c => (c.request.url as string).includes('/v1/users'));
  assert.equal(items.length, 5, 'items bucket capped at 5');
  assert.equal(users.length, 1, 'other bucket untouched');
  // FIFO: the two OLDEST items exchanges are gone
  assert.ok(items.every(c => (c.at as number) >= 1756400000002));
});

test('global cap evicts from the LARGEST bucket, never a rare endpoint\'s only recording', async () => {
  const r = await rig();
  r.registry.addEntry(CALLER, 'Fetcher', CALLER_TYPE);

  // the rare endpoint recorded FIRST (chronologically oldest)
  await r.driver.call(r.http.id, 'emitExchange', exchange({
    request: { method: 'GET', url: 'https://api.example.com/rare', headers: {} }, at: 1756400000000,
  }));
  // one hot bucket at its per-bucket cap of 5
  for (let i = 0; i < 5; i++) {
    await r.driver.call(r.http.id, 'emitExchange', exchange({
      request: { method: 'GET', url: `https://api.example.com/hot?p=${i}`, headers: {} },
      at: 1756400001000 + i,
    }));
  }
  // 45 more singleton endpoints -> total 51, one over the global 50 cap
  for (let i = 0; i < 45; i++) {
    await r.driver.call(r.http.id, 'emitExchange', exchange({
      request: { method: 'GET', url: `https://api.example.com/one-off-${i}`, headers: {} },
      at: 1756400002000 + i,
    }));
  }

  await until(() => {
    const s = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as unknown[];
    return Array.isArray(s) && s.length === 50;
  });
  const stored = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as Array<Record<string, any>>;
  assert.ok(stored.some(c => c.request.url.endsWith('/rare')), 'rare survives despite being oldest');
  assert.equal(stored.filter(c => (c.request.url as string).includes('/hot')).length, 4,
    'overflow came out of the largest bucket');
});

test('subscribes to HttpClient even when it registers late, and keeps retrying', async () => {
  const r = await rig({ registerHttp: false });
  r.registry.addEntry(CALLER, 'Fetcher', CALLER_TYPE);

  await new Promise(res => setTimeout(res, 200)); // recorder is retrying discovery
  r.registry.addEntry(r.http.id, 'HttpClient');
  await r.recorder.ready();

  await r.driver.call(r.http.id, 'emitExchange', exchange());
  await until(() => r.storage.data.has(`cassettes:${CALLER_TYPE}`));
});

test('records while Storage is missing and persists once it appears, merging what the store held', async () => {
  const r = await rig({ registerStorage: false });
  r.registry.addEntry(CALLER, 'Fetcher', CALLER_TYPE);
  // the store already holds evidence from a previous run - must not be clobbered
  r.storage.data.set(`cassettes:${CALLER_TYPE}`, [{
    method: '_http',
    request: { method: 'GET', url: 'https://api.example.com/v1/old', headers: {} },
    response: { status: 200, bodyText: 'old' }, at: 1756300000000,
  }]);

  await r.driver.call(r.http.id, 'emitExchange', exchange());
  await new Promise(res => setTimeout(res, 100)); // recorded in memory, nowhere to persist yet

  r.registry.addEntry(r.storage.id, 'Storage');
  await until(() => {
    const s = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as unknown[];
    return Array.isArray(s) && s.length === 2;
  });
  const stored = r.storage.data.get(`cassettes:${CALLER_TYPE}`) as Array<Record<string, any>>;
  assert.equal(stored[0].request.url, 'https://api.example.com/v1/old');
});
lab/http-emit-unit.ts
/**
 * One-shot verification for HttpClient's httpExchange emission (PR-A).
 * Not committed — goes in the PR description. Run: npx tsx --test lab/http-emit-unit.ts
 *
 * The network edge (makeRequest) is stubbed via subclass because validateDomain
 * blocks loopback (SSRF guard) — everything else is real: real bus, real
 * handlers, real dependents protocol, real redaction.
 */
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { MessageBus } from '../src/runtime/message-bus.js';
import { Abject } from '../src/core/abject.js';
import { request } from '../src/core/message.js';
import type { AbjectId, AbjectMessage } from '../src/core/types.js';
import { HttpClient, HttpRequest, HttpResponse } from '../src/objects/capabilities/http-client.js';

class TestHttpClient extends HttpClient {
  canned: HttpResponse = { status: 200, statusText: 'OK', headers: { 'content-type': 'application/json' }, body: '{"ok":true}', ok: true };
  seen: HttpRequest[] = [];
  override async makeRequest(req: HttpRequest): Promise<HttpResponse> {
    this.seen.push(req);
    return this.canned;
  }
}

const iface = (name: string) => ({
  name,
  description: `test ${name}`,
  version: '0.0.1',
  interface: { id: `test:${name.toLowerCase()}`, name, description: name, methods: [] },
});

class Listener extends Abject {
  readonly events: Array<Record<string, any>> = [];
  constructor() { super({ manifest: iface('Listener') }); }
  protected async onInit(): Promise<void> {
    this.on('httpExchange', (msg: AbjectMessage) => {
      this.events.push(msg.payload as Record<string, any>);
      return true;
    });
  }
  async subscribeTo(target: AbjectId): Promise<void> {
    await this.request(request(this.id, target, 'addDependent', {}));
  }
}

class Driver extends Abject {
  constructor() { super({ manifest: iface('Driver') }); }
  protected async onInit(): Promise<void> {}
  async call<T>(to: AbjectId, method: string, payload: unknown): Promise<T> {
    return this.request<T>(request(this.id, to, method, payload));
  }
}

async function until(cond: () => boolean, ms = 3000): Promise<void> {
  const deadline = Date.now() + ms;
  while (!cond()) {
    if (Date.now() > deadline) throw new Error('condition not met in time');
    await new Promise(r => setTimeout(r, 25));
  }
}

async function rig() {
  const bus = new MessageBus();
  const http = new TestHttpClient();
  const listener = new Listener();
  const driver = new Driver();
  for (const a of [http, listener, driver]) await a.init(bus);
  await listener.subscribeTo(http.id);
  return { http, listener, driver };
}

test('a completed request is emitted to dependents with caller attribution', async () => {
  const r = await rig();
  const reply = await r.driver.call<HttpResponse>(r.http.id, 'request', {
    method: 'GET', url: 'https://api.example.com/v1/items?page=2',
  });
  assert.equal(reply.status, 200);

  await until(() => r.listener.events.length === 1);
  const e = r.listener.events[0];
  assert.equal(e.caller, r.driver.id);
  assert.equal(e.request.method, 'GET');
  assert.equal(e.request.url, 'https://api.example.com/v1/items?page=2');
  assert.equal(e.response.status, 200);
  assert.equal(e.response.bodyText, '{"ok":true}');
  assert.equal(typeof e.at, 'number');
  assert.equal(typeof e.durationMs, 'number');
});

test('secret query params and auth headers are redacted before emission', async () => {
  const r = await rig();
  await r.driver.call(r.http.id, 'request', {
    method: 'GET',
    url: 'https://api.example.com/v1/items?api_key=sk-SECRET&page=2&TOKEN=abc',
    headers: { authorization: 'Bearer sk-SECRET', cookie: 'session=SECRET', accept: 'application/json' },
  });

  await until(() => r.listener.events.length === 1);
  const e = r.listener.events[0];
  assert.ok(!JSON.stringify(e).includes('SECRET'), 'no secret anywhere in the event');
  assert.equal(e.request.url, 'https://api.example.com/v1/items?api_key=REDACTED&page=2&TOKEN=REDACTED');
  assert.equal(e.request.headers.authorization, 'REDACTED');
  assert.equal(e.request.headers.cookie, 'REDACTED');
  assert.equal(e.request.headers.accept, 'application/json');
  // the request handed to the network is NOT redacted
  assert.ok((r.http.seen[0].url).includes('sk-SECRET'));
});

test('response set-cookie is redacted before emission', async () => {
  const r = await rig();
  r.http.canned = { status: 200, statusText: 'OK', headers: { 'set-cookie': 'sid=SECRET', 'content-type': 'text/plain' }, body: 'hi', ok: true };
  await r.driver.call(r.http.id, 'get', { url: 'https://api.example.com/hello' });

  await until(() => r.listener.events.length === 1);
  const e = r.listener.events[0];
  assert.equal(e.response.headers['set-cookie'], 'REDACTED');
  assert.equal(e.response.headers['content-type'], 'text/plain');
});

test('bodies over the cap are truncated with a flag', async () => {
  const r = await rig();
  r.http.canned = { status: 200, statusText: 'OK', headers: {}, body: 'x'.repeat(100_000), ok: true };
  await r.driver.call(r.http.id, 'get', { url: 'https://api.example.com/big' });

  await until(() => r.listener.events.length === 1);
  const e = r.listener.events[0];
  assert.equal(e.response.bodyText.length, 64 * 1024);
  assert.equal(e.response.truncated, true);
});

test('post and postJson entry points emit too, with request body as text', async () => {
  const r = await rig();
  await r.driver.call(r.http.id, 'postJson', {
    url: 'https://api.example.com/v1/items', data: { name: 'widget' },
  });

  await until(() => r.listener.events.length === 1);
  const e = r.listener.events[0];
  assert.equal(e.request.method, 'POST');
  assert.equal(e.request.bodyText, '{"name":"widget"}');
});

test('secret stems are redacted: client_secret, refresh_token, X-Amz-Signature, x-amz-security-token', async () => {
  const r = await rig();
  await r.driver.call(r.http.id, 'request', {
    method: 'GET',
    url: 'https://api.example.com/cb?client_secret=SECRET1&refresh_token=SECRET2&X-Amz-Signature=SECRET3&page=1',
    headers: { 'x-amz-security-token': 'SECRET4', 'x-auth-token': 'SECRET5', accept: 'text/plain' },
  });

  await until(() => r.listener.events.length === 1);
  const e = r.listener.events[0];
  assert.ok(!JSON.stringify(e).match(/SECRET[0-9]/), 'no stem-named secret survives');
  assert.ok(e.request.url.includes('page=1'), 'non-secret params survive');
  assert.equal(e.request.headers.accept, 'text/plain');
});

test('secret-bearing fields inside body text are redacted, JSON and form-encoded', async () => {
  const r = await rig();
  r.http.canned = {
    status: 200, statusText: 'OK', headers: {},
    body: '{"access_token":"sk-BODYSECRET","expires_in":3600,"scope":"read"}', ok: true,
  };
  await r.driver.call(r.http.id, 'request', {
    method: 'POST', url: 'https://api.example.com/oauth/token',
    body: 'grant_type=refresh_token&refresh_token=BODYSECRET2&client_id=pub123',
  });

  await until(() => r.listener.events.length === 1);
  const e = r.listener.events[0];
  assert.ok(!JSON.stringify(e).includes('BODYSECRET'), 'no body secret crosses the bus');
  assert.ok(e.response.bodyText.includes('"expires_in":3600'), 'non-secret JSON fields survive');
  assert.ok(e.request.bodyText.includes('client_id=pub123'), 'non-secret form fields survive');
});

test('nothing is emitted (or serialized) when no dependents are subscribed', async () => {
  const bus2 = new MessageBus();
  const http2 = new TestHttpClient();
  const driver2 = new Driver();
  for (const a of [http2, driver2]) await a.init(bus2);
  let stringified = 0;
  const big = { toJSON() { stringified++; return { x: 1 }; } };
  await driver2.call(http2.id, 'request', { method: 'POST', url: 'https://api.example.com/v1', body: big });
  await new Promise(res => setTimeout(res, 100));
  // makeRequest is stubbed here, so the only serialization that could run is
  // the emission path's - and with no dependents it must not run at all.
  assert.equal(stringified, 0, 'no event serialization when nobody is subscribed');
});
lab/recorder-live.ts
/**
 * End-to-end evidence for the PR-A recorder (mempko/abject#11 series).
 * Not committed — goes in the PR description. Run: npx tsx --test lab/recorder-live.ts
 *
 * Production classes throughout: real Registry, real Storage, real HttpClient
 * (network edge stubbed — validateDomain blocks loopback, so no live socket),
 * real CassetteRecorder, real MessageBus, real dependents protocol. Proves the
 * full lifecycle: boot-style spawn/register -> discovery -> subscription ->
 * a caller's HTTP request -> typeId-keyed cassette readable back from Storage.
 */
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { MessageBus } from '../src/runtime/message-bus.js';
import { Abject } from '../src/core/abject.js';
import { request } from '../src/core/message.js';
import type { AbjectId, TypeId } from '../src/core/types.js';
import { Registry } from '../src/objects/registry.js';
import { Storage } from '../src/objects/capabilities/storage.js';
import { HttpClient, HttpRequest, HttpResponse } from '../src/objects/capabilities/http-client.js';
import { CassetteRecorder } from '../src/objects/cassette-recorder.js';

class StubNetHttpClient extends HttpClient {
  override async makeRequest(_req: HttpRequest): Promise<HttpResponse> {
    return { status: 200, statusText: 'OK', headers: { 'content-type': 'application/json' }, body: '{"price":42}', ok: true };
  }
}

class Caller extends Abject {
  constructor() {
    super({ manifest: { name: 'PriceFetcher', description: 'test caller', version: '0.0.1',
      interface: { id: 'test:pricefetcher', name: 'PriceFetcher', description: 'x', methods: [] },
      requiredCapabilities: [], providedCapabilities: [], tags: [] } });
  }
  protected async onInit(): Promise<void> {}
  async fetchPrice(httpId: AbjectId): Promise<HttpResponse> {
    return this.request<HttpResponse>(request(this.id, httpId, 'get',
      { url: 'https://api.example.com/price?api_key=sk-LIVE-SECRET' }));
  }
  async readStorage<T>(storageId: AbjectId, key: string): Promise<T> {
    return this.request<T>(request(this.id, storageId, 'get', { key }));
  }
}

async function until(cond: () => Promise<boolean>, ms = 4000): Promise<void> {
  const deadline = Date.now() + ms;
  while (!(await cond())) {
    if (Date.now() > deadline) throw new Error('condition not met in time');
    await new Promise(r => setTimeout(r, 50));
  }
}

test('full lifecycle: spawn, subscribe, record, persist, read back', async () => {
  const bus = new MessageBus();
  const registry = new Registry();
  await registry.init(bus);
  const storage = new Storage('lab-recorder-live');
  const http = new StubNetHttpClient();
  const caller = new Caller();
  const recorder = new CassetteRecorder({ flushMs: 20 });
  for (const a of [storage, http, caller, recorder]) await a.init(bus, undefined, registry.id);

  // Boot-style registration, typeIds the way WorkspaceManager stamps them.
  const reg = async (a: Abject, typeId: string) => {
    await (caller as any).request(request(caller.id, registry.id, 'register', {
      objectId: a.id, manifest: (a as any).manifest, typeId: typeId as TypeId,
    }));
  };
  await reg(http, 'peer/system/HttpClient');
  await reg(storage, 'peer/system/Storage');
  await reg(recorder, 'peer/system/CassetteRecorder');
  await reg(caller, 'peer/ws/user/PriceFetcher');

  // Recorder discovers deps and subscribes through the REAL registry.
  await recorder.ready();

  const reply = await caller.fetchPrice(http.id);
  assert.equal(reply.status, 200);

  let stored: any;
  await until(async () => {
    stored = await caller.readStorage(storage.id, 'cassettes:peer/ws/user/PriceFetcher');
    return Array.isArray(stored) && stored.length === 1;
  });

  const c = stored[0];
  assert.equal(c.method, '_http');
  assert.equal(c.request.url, 'https://api.example.com/price?api_key=REDACTED');
  assert.equal(c.response.bodyText, '{"price":42}');
  assert.ok(!JSON.stringify(stored).includes('sk-LIVE-SECRET'));
});

What comes next in the series

The blocklist PR (Atomics/SharedArrayBuffer, its own reasoning), then the judge as an abject, evaluated inside the deploy ops. Once cassettes accumulate here, the learned-schema conversation from #11 has real data to stand on.

andreBurnt and others added 3 commits August 29, 2026 13:52
HttpClient's four message entry points now emit a changed('httpExchange')
event after each completed request, carrying the caller's id
(msg.routing.from), the request, and the response as text. Redaction is
stem-based on NAMES at the emission boundary: headers, query params, and
JSON/form body fields whose name contains a secret stem (key, token,
secret, passw, credential, session, signature, cookie, auth) are replaced
with REDACTED before anything crosses the bus. A false positive redacts
something harmless; a false negative persists a live credential - so the
matcher errs toward matching. Bodies are capped at 64K characters with a
truncated flag.

Nothing on this path parses JSON. With no dependents subscribed the
emission is skipped entirely (new hasDependents accessor on Abject), so
an unobserved HttpClient does no serialization work at all. Emission
failures are logged and never break the reply to the caller. Requests
that throw before a response exists (timeout, DNS, the SSRF guard)
produce no event - evidence of failed transport is a different shape
and a deliberate non-goal here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q23ProuK7YuZ4wtBbBmQGQ
A CassetteRecorder abject subscribes to HttpClient's httpExchange events
(addDependent, the universal dependents protocol) and persists each
exchange under cassettes:<typeId> in Storage. The caller's identity is
resolved against the registry via resolveCallerIdentity - never trusted
from the payload - and exchanges whose caller has no durable typeId are
not recorded. A live AbjectId dies with its object; the typeId survives
restarts, so the evidence does too.

Retention is per endpoint bucket (method + path, query stripped, FIFO
cap 5) with an overall cap of 50 per typeId. Global overflow always
comes out of the LARGEST bucket, so a rare endpoint's only recording
survives no matter how old it is.

Failure independence: HttpClient discovery retries with capped backoff
for as long as the recorder lives - a permanent recorder that silently
gives up has failed its one job - and the subscription never waits on
Storage. While Storage is missing, recording continues in memory;
persistence catches up once it appears, merging what the store already
held so a restart never clobbers accumulated evidence.

Everything crosses the bus as messages. HttpClient and this recorder may
be hosted on different worker threads (both are workerEligible), which
is why no in-process seam could work. See mempko#11.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q23ProuK7YuZ4wtBbBmQGQ
Same shape as HealthMonitor: registered constructor, workerEligible,
supervisedSpawn permanent with a system typeId. Recording is on from
boot - evidence accumulates without anyone asking for it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q23ProuK7YuZ4wtBbBmQGQ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant