Track and report malformed line count in Claude Code JSONL parser

[kannupriyakalra]

Problem

parseClaudeCodeJsonl() in src/token-usage.ts silently skips malformed JSON lines. If a JSONL file is corrupt or partially written, the parser drops data without informing the user.

Solution

Return a { calls, skippedLines } object from parseClaudeCodeJsonl (or add a second return value). When skippedLines > 0, log a warning with the filename and count so users know data may be incomplete.

Acceptance criteria

• Malformed lines increment a counter rather than being silently dropped
• A warning is logged when skipped lines exceed 0
• A unit test verifies the counter increments on bad input

[Necmttn]

I looked at the current parser/call-site shape, and I think the smallest clean version is to keep the line counter in the parser but emit file-aware warnings from the collector.

Today parseClaudeCodeJsonl(content) only receives raw text, so it cannot name the file in the warning. collectClaudeCode() has the file path and already loops one JSONL file at a time:

const content = fs.readFileSync(f, "utf-8");
for (const c of parseClaudeCodeJsonl(content)) calls.push(c);

One low-risk shape would be:

interface ClaudeCodeParseResult {
  calls: TokenCall[];
  skippedLines: number;
  partialTailLines: number;
}

export function parseClaudeCodeJsonlDetailed(content: string): ClaudeCodeParseResult
export function parseClaudeCodeJsonl(content: string): TokenCall[] {
  return parseClaudeCodeJsonlDetailed(content).calls;
}

That preserves the existing exported parseClaudeCodeJsonl() contract for tests/importers, while the collector can switch to the detailed version and do:

const parsed = parseClaudeCodeJsonlDetailed(content);
if (parsed.skippedLines > 0) {
  console.warn(`Skipped ${parsed.skippedLines} malformed Claude Code JSONL line(s): ${f}`);
}
calls.push(...parsed.calls);

I would separate partialTailLines from true malformed lines if possible. Active Claude Code files can be observed mid-write; warning every refresh for one incomplete last line will look like corruption even when the next read would parse cleanly. A pragmatic rule is: if the final non-empty line fails JSON parse and the file does not end with \n, count it as partialTailLines; malformed middle lines increment skippedLines.

Suggested tests to lock the behavior:

• a malformed middle line increments skippedLines and valid assistant usage rows before/after still parse;
• a final incomplete line without trailing newline increments partialTailLines, not skippedLines;
• the compatibility wrapper still returns just TokenCall[];
• collectClaudeCode() emits one warning per affected file, not one warning per bad line.

There are similar silent JSONL skips in claude-code-sessions.ts and sessions.ts, but I would keep this issue scoped to token usage unless you want a repo-wide parser diagnostic pass.

---

Generated with ax.

[kannupriyakalra]

I looked at the current parser/call-site shape, and I think the smallest clean version is to keep the line counter in the parser but emit file-aware warnings from the collector.

Today parseClaudeCodeJsonl(content) only receives raw text, so it cannot name the file in the warning. collectClaudeCode() has the file path and already loops one JSONL file at a time:

const content = fs.readFileSync(f, "utf-8");
for (const c of parseClaudeCodeJsonl(content)) calls.push(c);
One low-risk shape would be:

interface ClaudeCodeParseResult {
calls: TokenCall[];
skippedLines: number;
partialTailLines: number;
}

export function parseClaudeCodeJsonlDetailed(content: string): ClaudeCodeParseResult
export function parseClaudeCodeJsonl(content: string): TokenCall[] {
return parseClaudeCodeJsonlDetailed(content).calls;
}
That preserves the existing exported parseClaudeCodeJsonl() contract for tests/importers, while the collector can switch to the detailed version and do:

const parsed = parseClaudeCodeJsonlDetailed(content);
if (parsed.skippedLines > 0) {
console.warn(Skipped ${parsed.skippedLines} malformed Claude Code JSONL line(s): ${f});
}
calls.push(...parsed.calls);
I would separate partialTailLines from true malformed lines if possible. Active Claude Code files can be observed mid-write; warning every refresh for one incomplete last line will look like corruption even when the next read would parse cleanly. A pragmatic rule is: if the final non-empty line fails JSON parse and the file does not end with \n, count it as partialTailLines; malformed middle lines increment skippedLines.

Suggested tests to lock the behavior:

• a malformed middle line increments skippedLines and valid assistant usage rows before/after still parse;
• a final incomplete line without trailing newline increments partialTailLines, not skippedLines;
• the compatibility wrapper still returns just TokenCall[];
• collectClaudeCode() emits one warning per affected file, not one warning per bad line.

There are similar silent JSONL skips in claude-code-sessions.ts and sessions.ts, but I would keep this issue scoped to token usage unless you want a repo-wide parser diagnostic pass.

Generated with ax.

Thanks @Necmttn for the thoughtful breakdown. I agree with this approach.

Please go ahead with the backward-compatible parseClaudeCodeJsonlDetailed() implementation, keep parseClaudeCodeJsonl() returning TokenCall[], and add the collector warning plus the tests you suggested.

Let’s keep this issue scoped to token usage for now; we can open a separate follow-up for the other JSONL parsers.