Skip to content

Standardize reading/writing of error files #84

Description

@yoonspark

Problem

Error file handling is scattered across multiple task implementations (LocalTask, LocalAsyncTask, SlurmTask) and the output aggregation layer (PipelineOutput). Writing is done via a standalone write_error_file utility in tasks/utils.py; reading is done inline with raw json.loads calls and manual field access. This split means the on-disk schema is implicitly defined in two places with no shared contract.

As a result, the write and read paths can drift independently — field names are duplicated as string literals, error handling diverges (callers catch different exception types), and adding or changing a field requires coordinated edits across unrelated modules. The file attribution field added recently exposed this: the reader had to fall back to a stem-derived name because there was no guarantee the field was present, and no single place enforced that it should be.

Proposal

Introduce an ErrorRecord dataclass that owns the on-disk schema and exposes from_exception(), write(), and read() class methods. Replace all write_error_file call sites with ErrorRecord.from_exception(...).write(path) and all inline json.loads reads with ErrorRecord.read(path). Remove write_error_file from tasks/utils.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

refactorMake code cleaner, clearer, and easier to understand

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions