Skip to content

Evaluate replacing MD5 with SHA-256 for file hashing #2236

Description

@Darkroom4364

Problem

We use MD5 for file hashing in the processing pipeline:

  • src/file-processor/file-queue-processor.provider.ts (line 121)
  • src/file-processor/helper/drive-helper.ts (line 29)
  • src/fileCleanup/file-cleanup-queue-processor.provider.ts (line 126)

MD5 collisions are practical, two different files can produce the same hash. If we rely on these hashes for dedup or integrity checks, that's a problem.

The concern: speed

These hashes run on large files (multi-GB). MD5 is roughly 30-40% faster than SHA-256 in software, which adds up on a 15 GB download.

That said:

  • Modern CPUs (Intel Ice Lake+, AMD Zen+) have hardware SHA-256 instructions. Node's OpenSSL bindings pick these up automatically, such that the gap shrinks to near-zero on supported hardware.
  • The hash probably isn't the bottleneck anyway. Network and disk I/O on a 15 GB file will dominate wall-clock time regardless of algorithm.

Before switching

  • Run openssl speed md5 sha256 on the deployment target to see actual throughput
  • Figure out what these hashes are actually used for, if it's just cache keys, urgency is lower
  • Check if the hash is even visible in profiling during large file processing

Fix

If perf is fine: createHash('md5') -> createHash('sha256'). Same streaming API, same hex output, just 64 chars instead of 32.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions