Skip to content

Evaluate replacing MD5 with SHA-256 for file hashing #2236

Description

@Darkroom4364

Problem

We use MD5 for file hashing in the processing pipeline:

  • src/file-processor/file-queue-processor.provider.ts (line 121)
  • src/file-processor/helper/drive-helper.ts (line 29)
  • src/fileCleanup/file-cleanup-queue-processor.provider.ts (line 126)

MD5 collisions are practical, two different files can produce the same hash. If we rely on these hashes for dedup or integrity checks, that's a problem.

The concern: speed

These hashes run on large files (multi-GB). MD5 is roughly 30-40% faster than SHA-256 in software, which adds up on a 15 GB download.

That said:

  • Modern CPUs (Intel Ice Lake+, AMD Zen+) have hardware SHA-256 instructions. Node's OpenSSL bindings pick these up automatically, such that the gap shrinks to near-zero on supported hardware.
  • The hash probably isn't the bottleneck anyway. Network and disk I/O on a 15 GB file will dominate wall-clock time regardless of algorithm.

Before switching

  • Run openssl speed md5 sha256 on the deployment target to see actual throughput
  • Figure out what these hashes are actually used for, if it's just cache keys, urgency is lower
  • Check if the hash is even visible in profiling during large file processing

Fix

If perf is fine: createHash('md5') -> createHash('sha256'). Same streaming API, same hex output, just 64 chars instead of 32.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions