Skip to content

DocRepoIngester: avoid double file read in process_file #87

New issue
New issue
Closed
Closed
DocRepoIngester: avoid double file read in process_file#87
Assignees
itdove
Labels
phase:4-ingestersPhase 4 — Ingester Framework

Description

@itdove
itdove
opened on Jul 27, 2026

Summary

DocRepoIngester.process_file() reads the file twice:

  1. file_path.read_bytes() for SHA-256 content hash (line 70)
  2. chunk_document(file_path) internally calls file_path.read_text() (chunker.py:233)

Every doc file is read from disk twice per sync.

Fix

Add optional text parameter to chunk_document() so callers can pass pre-read content. Read bytes once, decode to text, hash the bytes, pass the text to chunker.

Acceptance Criteria

  • chunk_document accepts optional text parameter
  • DocRepoIngester.process_file reads file once
  • DocIngester.extract_delta unchanged (still uses file path)
  • All existing chunker tests pass
  • No behavior change

Discovered during /simplify review of #37.

Activity

  1. itdove
    added
    phase:4-ingestersPhase 4 — Ingester Framework
    on Jul 27, 2026
  2. itdove
    self-assigned this
    on Sep 8, 2026

  3. itdove commented on Sep 8, 2026

    @itdove
    itdove
    on Sep 8, 2026
    OwnerAuthor

    Pull request created: #97

  4. itdove commented on Sep 8, 2026

    @itdove
    itdove
    on Sep 8, 2026
    OwnerAuthor

    ✅ Session Complete

    Project: lore
    Goal: #87: DocRepoIngester: avoid double file read in process_file
    Time Spent:
    0h 26m total

    • itdove: 0h 26m
    • unknown: 0h 0m
      Branch: 87

    No significant activity detected in this session.

    Session Notes:

    Session Notes: itdove-lore-87

    JIRA: #87

    2026-09-08 12:22

    • Implemented issue DocRepoIngester: avoid double file read in process_file #87: chunk_document accepts optional pre-read text; DocRepoIngester hashes one byte read and reuses decoded text; DocIngester.extract_delta remains unchanged. Focused tests (56) pass. Full suite: 654 passed, 3 unrelated cache-path failures under read-only /home/itdove/.cache/lore/pr-workdirs.

    Generated by DevAIFlow

  5. itdove
    closed this as completedon Sep 8, 2026
  6. itdove
    added a commit that references this issue on Sep 8, 2026

    Merge pull request #97 from itdove/87

    9183c1c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

phase:4-ingestersPhase 4 — Ingester Framework

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions