Scheduled sync repeatedly reparses large Codex logs twice, causing sustained high CPU

[NubsCarson]

Problem

The five-minute background service can spend minutes at high CPU repeatedly reparsing large Codex rollout logs. This is not a leaked or overlapping job: each scheduled run succeeds, exits, and then the next scheduled run repeats the same expensive work.

The cost is amplified because a sync invokes ccusage twice per source, sequentially:

1. ccusage <source> daily --json --breakdown ...
2. ccusage <source> session --json ...

For Codex, both reports scan the same JSONL corpus. --since <today> limits the reported period but does not make a large session file cheap to parse.

Evidence from a real installation

Environment:

• macOS arm64
• Tokenmaxxing CLI/service runner 0.6.0
• ccusage 20.0.19
• launchd service template 5, StartInterval = 300
• scheduler healthy, no stale lock, no overlapping Tokenmaxxing process

Sanitized local corpus metrics:

• ~/.codex/sessions: 1,043 JSONL files, 83 GB total
• current date directory: 5 JSONL files, 4.8 GB total
• four current-date session files are approximately 1.2 GB each

Recent successful scheduled-run durations as the current-date corpus grew:

101,653 ms
 95,572 ms
 88,165 ms
 95,878 ms
195,223 ms
191,953 ms
174,798 ms
149,221 ms

During a run, Activity Monitor/btop showed the native ccusage codex daily --json --breakdown ... process at approximately:

CPU: 128% (about 1.28 cores)
RSS: 414 MiB

Each run uploaded only two daily aggregate rows, so the bulk of the work was repeated local parsing rather than network/upload volume.

Large Codex JSONL files are realistic for long-lived, tool-heavy, compacted, or forked threads. Upstream ccusage has related reports:

• Performance: statusline/daily commands timeout with large data directories (~750 files, 4GB) ccusage/ccusage#821
• [Bug] Large Codex session JSONL files are silently skipped, causing historical days to disappear from reports ccusage/ccusage#952

Current code path

• The service cadence is hard-coded to five minutes:
  

  tokenmaxxing/apps/cli/src/commands/service.ts

  Lines 73 to 74 in 3097e0e

  	const SERVICE_INTERVAL_MINUTES = 5;
  	const SERVICE_INTERVAL_SECONDS = SERVICE_INTERVAL_MINUTES * 60;

• Daily and session reports are separate ccusage subprocesses:
  

  tokenmaxxing/apps/cli/src/ccusage/runner.ts

  Lines 113 to 136 in 3097e0e

  	function dailyCcusageCommand(source: CcusageSource, options: RunOptions = {}): string[] {
  	return [CCUSAGE_SPEC, ...dailyCcusageArgs(source, options)];
  	}
  	function sessionCcusageCommand(source: CcusageSource, options: RunOptions = {}): string[] {
  	return [CCUSAGE_SPEC, ...sessionCcusageArgs(source, options)];
  	}
  	function dailyCcusageArgs(source: CcusageSource, options: RunOptions = {}): string[] {
  	const args = [source.subcommand, "daily", "--json", "--breakdown", "--mode", "calculate"];
  	if (options.since !== undefined) {
  	args.push("--since", options.since.replaceAll("-", ""));
  	}
  	return args;
  	}
  	function sessionCcusageArgs(source: CcusageSource, options: RunOptions = {}): string[] {
  	const args = [source.subcommand, "session", "--json", "--mode", "calculate"];
  	if (options.since !== undefined) {
  	args.push("--since", options.since.replaceAll("-", ""));
  	}
  	return args;

• Sync runs the daily report, then the session report, for every source:
  

  tokenmaxxing/apps/cli/src/commands/sync.ts

  Lines 273 to 327 in 3097e0e

  	for (const source of sources) {
  	const spinner = yield* humanSpinner(`Syncing ${source.source}`, options);
  	const dailyResult = yield* runDailyReport(source, { since: options.since }).pipe(
  	Effect.match({
  	onFailure: (error) => ({ error, _tag: "failure" as const }),
  	onSuccess: (report) => ({ report, _tag: "success" as const }),
  	}),
  	);
  	if (dailyResult._tag === "failure") {
  	const result = {
  	issue: syncSourceIssue(dailyResult.error),
  	source: source.source,
  	status: "failed" as const,
  	summary: null,
  	};
  	sourceSummaries[source.source] = null;
  	sourceResults.push(result);
  	spinner.error(
  	renderInlineResults ? renderSyncSourceResult(result) : `Failed syncing ${source.source}`,
  	);
  	continue;
  	}
  	const dailyReport = dailyResult.report;
  	if (dailyReport.daily.length === 0) {
  	const result = {
  	reason: "no_data" as const,
  	source: source.source,
  	status: "skipped" as const,
  	summary: null,
  	};
  	sourceSummaries[source.source] = result.summary;
  	sourceResults.push(result);
  	spinner.stop(renderInlineResults ? renderSyncSourceResult(result) : undefined);
  	continue;
  	}
  	const sourceRows = aggregateDays(source.source, dailyReport.daily);
  	rawReports.push({
  	command: dailyCcusageCommand(source, { since: options.since }),
  	payload: dailyReport,
  	reportKind: "daily",
  	source: source.source,
  	});
  	const sessionResult = yield* runSessionReport(source, { since: options.since }).pipe(
  	Effect.match({
  	onFailure: (error) => ({ error, _tag: "failure" as const }),
  	onSuccess: (report) => ({ report, _tag: "success" as const }),
  	}),
  	);
  	const sessionCount =
  	sessionResult._tag === "success" ? sessionResult.report.sessions.length : null;
  	const summary = { ...summarize(sourceRows), sessions: sessionCount };

• Each individual ccusage invocation has a 180-second timeout, which this installation is already approaching:
  

  tokenmaxxing/apps/cli/src/ccusage/runner.ts

  Lines 15 to 16 in 3097e0e

  	const CCUSAGE_SPEC = "ccusage@^20.0.19";
  	const RUN_TIMEOUT_MS = 180_000;

Expected behavior

An unchanged or lightly changed local corpus should make subsequent scheduled syncs cheap. The background service should not repeatedly scan multiple gigabytes twice merely to refresh daily aggregates and a session count.

Suggested fix

Safe short-term changes in Tokenmaxxing

1. Avoid the second scan in the scheduled path. Upload the daily report without recomputing the session count every five minutes; retain the last successful count, make it nullable, or refresh it at a much lower cadence. Foreground tokenmaxxing sync can preserve the current full behavior.
2. Make the service interval configurable and/or apply adaptive backoff when a run consumes a significant fraction of the interval. A successful 2–3 minute scan does not need to run again five minutes later.
3. Record duration per source and per report (daily versus session) in service metadata. The current whole-run duration makes this diagnosable only by observing child processes.

Longer-term optimization

Prefer one of:

• an upstream ccusage combined report that produces daily aggregates and session cardinality from one parse; or
• an upstream incremental/cache layer keyed by file identity, size/mtime, parser version, pricing mode/snapshot, and date window.

Tokenmaxxing should not independently implement a naive byte-offset parser unless it can preserve ccusage's cross-file replay deduplication and handle truncation, rewrites, archives, and pricing changes. Those correctness rules belong closest to the ccusage parser.

Acceptance criteria

• A scheduled Codex sync does not invoke two full-corpus scans back-to-back.
• Repeated scheduled syncs over an unchanged corpus are materially cheaper than the first run.
• Slow runs do not produce near-continuous recurring CPU load.
• Foreground sync behavior and daily-row idempotency remain intact.
• Scheduled JSON/log output remains machine-readable and reports per-source/per-report durations.
• Tests cover the scheduled/full-sync distinction without committing a multi-gigabyte fixture (a generated temporary JSONL corpus is sufficient).