Skip to content

PROMPTFOO_JSONL.md

Latest commit

 

History

History
97 lines (77 loc) · 6.21 KB

PROMPTFOO_JSONL.md

File metadata and controls

97 lines (77 loc) · 6.21 KB
Raw

Promptfoo JSONL: audit the large-eval export directly

Research snapshot: 2026-08-12.

The reproduced gap

Promptfoo documents promptfoo eval --output results.jsonl for very large evaluations and specifically recommends JSONL when its ordinary JSON export fails with memory errors. Each physical line is one current EvaluateResult. EvalInt v0.2.24 claimed Promptfoo ingestion but understood only its JSON envelopes.

A four-row artifact was then generated by the real npx promptfoo@0.122.0 eval entry point with Promptfoo's local echo provider and a second local provider. No model API or cloud share was used. Publicly installed EvalInt v0.2.24 failed on that artifact through both CLI routes:

  • automatic detection returned exit 1 because the generic flattener treated the result id and nested provider id as conflicting generic identities;
  • --format promptfoo returned exit 1 because it tried to decode the whole JSONL file as one JSON document.

No report was emitted, so the defect was availability and compatibility rather than silent score corruption. The eval data is a deliberately synthetic local fixture and carries no claim about users, adoption, or either project's performance.

Evidence:

The current JSON v3 shape was tested separately and already worked. It was therefore rejected as the target gap rather than changed on the strength of an outdated or ambiguous example.

Reuse and alternatives considered

Repository, npm, and package metadata were checked on 2026-08-12.

Approach Maintenance, license, dependency and platform fit Cost and migration
Emit Promptfoo JSON as a second output Promptfoo 0.122.0 is active and MIT; its npm package requires Node 22.22+ and declares about 80 direct runtime plus 42 optional dependencies Sensible for future runs, but it does not repair an existing JSONL-only artifact. Re-running an eval can repeat provider calls and their cost.
Convert with jq Maintained native CLI under the jq license; available across major desktop platforms --slurp only makes an array. Users must still write and maintain a Promptfoo-specific projection for item, provider, and score semantics.
Add jsonlines 4.0.0 BSD-3-Clause with one attrs runtime dependency; compatible license, but it only supplies line I/O It does not recognize Promptfoo or decide identities. Adding it would expand EvalInt's install and security surface without solving the domain mapping.
Parse each line with Python's standard library and reuse EvalInt's Promptfoo mapper Python 3.9+, MIT-compatible, zero new dependencies, offline, and cross-platform Selected. Existing files work directly; users keep their current producer and do not need an account, service, conversion script, or rerun.

No Promptfoo source code is copied. The implementation consumes its public artifact shape and retains EvalInt's existing MIT license and zero-runtime- dependency package.

After the change, separately installed wheel/Python 3.13 and sdist/Python 3.9 builds each read the actual four-row Promptfoo artifact through automatic detection with exit 0, format promptfoo, two items, two systems, four observations, and empty stderr. A truncated second row exited 1 through both automatic and forced-format routes with empty stdout, its physical line and column, and no traceback.

Resulting contract

  • Automatic detection requires the current non-boolean integer testIdx and promptIdx plus testCase, provider, success, and score fields.
  • Every nonblank physical line must be one complete object with that shape. Malformed JSON, duplicate member names, and non-result rows exit 1 at their physical line before any report is printed.
  • The existing Promptfoo JSON v3 and wrapped-results paths remain supported.
  • Test variables remain the preferred item identity. Prompt text is the next fallback. If Promptfoo's documented strip settings remove both, EvalInt uses promptfoo:test:<testIdx> so the rows in one export can still be compared.
  • Provider id, then provider label, remains the system identity. Score remains the numeric score with success as the existing fallback when score is null.

Limits and falsifiable boundaries

Detection has a possible false positive: unrelated JSONL using every current Promptfoo discriminator will be routed to this reader. --format jsonl overrides that choice. It also has a possible false negative: a future Promptfoo release can rename or omit a discriminator. --format promptfoo does not promise compatibility with an unknown future row schema.

testIdx is an export-local fallback, not a stable identifier across separate runs. A stripped export can therefore be audited internally but should not be merged across runs unless the producer's ordering is known to match. Prompt variants are still grouped by provider under EvalInt's existing Promptfoo semantics; this change does not claim to distinguish every upstream execution configuration.

EvalInt decodes rows one at a time after the file has been read, but its current file API still loads the complete text into memory. This release does not claim streaming, a safe maximum file size, denial-of-service resistance, completeness, authenticity, or semantic correctness. A clean parse and audit only show that the observed artifact matched the supported structure and supplied enough identifiable comparison data.