fix(upload): send a whole-file SHA1 checksum with every upload - #3531
Open
Mortimer-RR wants to merge 2 commits into
Open
Mortimer-RR wants to merge 2 commits into
Mortimer-RR wants to merge 2 commits into
Conversation
Browser uploads carried no checksum, so the server could not tell a damaged upload from a good one: overlapping or spliced chunks were stored silently. tus-js-client can also resume a different file's partial upload when name, type, size, mtime and endpoint all match, splicing the two files together. Add an Uppy pre-processor that computes the SHA1 of the bytes that are actually sent (after vault encryption) in a web worker, reading the file in 8 MiB slices with hash-wasm since crypto.subtle cannot hash incrementally. The checksum goes into the tus Upload-Metadata as `checksum: sha1 <hex>`, and into the `OC-Checksum: SHA1:<hex>` header for plain PUT uploads. The server verifies it and rejects mismatching uploads. If a file cannot be hashed, that file fails instead of being uploaded without a checksum. The upload info shows "Calculating checksum..." for a file while it is hashed.
hash-wasm needs 'wasm-unsafe-eval' in the Content-Security-Policy's script-src to compile its WebAssembly module. OpenCloud's default CSP does not allow it, so computing the checksum failed and every browser upload was rejected before it started. The checksum worker now tries hash-wasm once and, if WebAssembly cannot be compiled, uses the pure JavaScript js-sha1 implementation instead. Both produce the same SHA1; the JavaScript one is about 7 times slower. Deployments that add 'wasm-unsafe-eval' to script-src get the fast path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Browser uploads now carry a SHA1 of the bytes that are actually sent, so the server rejects
damaged or spliced uploads instead of storing them.
UploadChecksumPlugin(packages/web-pkg/src/services/uppy/checksum/),registered by
UppyServicefor the tus and the XHR uploader. It hashesfile.datain aweb worker, reading 8 MiB
Blob.slices, and setsfile.meta.checksum = 'sha1 <hex>'.Folders, remote (companion) files and files that already have a checksum (retries) are
skipped. If hashing fails, that file fails (
upload-error) instead of being uploadedwithout a checksum; other files continue.
hash-wasm(incremental;crypto.subtle.digestcan't hash incrementally).Compiling its WebAssembly needs
'wasm-unsafe-eval'in the CSP'sscript-src, whichOpenCloud's default CSP doesn't allow, so the worker falls back to the pure JavaScript
js-sha1when WebAssembly can't be compiled. Same SHA1; JavaScript is about 7脳 slower.checksumis added toTUS_ALLOWED_META_FIELDS(a hash of the transmitted bytes revealsno path or cleartext). The XHR fallback sends
OC-Checksum: SHA1:<hex>, the desktopclient's format.
HandleUpload.applyVaultEncryptionreplacesfile.datawith theciphertext before
uppy.upload()runs the pre-processors, so the hash covers theciphertext that is sent.
UploadInfo.vueshows "Calculating checksum..." for a file while it is hashed.New dependencies:
hash-wasm,js-sha1.Related Issue
How Has This Been Tested?
@vitest/web-worker); headless Chromium against anOpenCloud server with the matching server fix (460 for checksum mismatches,
fix(tus): return 460 for checksum mismatches instead of 500聽reva#843), default CSP
checksum.spec.ts: known SHA1s, slice-by-slice hashing, the meta field is setbefore the upload, ciphertext is hashed when
file.datais replaced, folders skipped, anunreadable file fails alone, and the JavaScript fallback when WebAssembly is blocked
uppyService.spec.ts: the tusUpload-Metadatafields are exactlyname,mtime,checksum, with no path fieldsuseUpload.spec.ts: XHR uploads sendOC-ChecksumUploadInfo.spec.ts: "Calculating checksum..." while hashingan upload with a tampered checksum and the same-fingerprint resume splice were both
rejected (460) and stored nothing
per 10 GB, UI fully responsive; Firefox 155 (Linux container): 5.1 s / 50.6 s. JavaScript
fallback in the same browser: about 7脳 slower
pnpm check:types,pnpm lintand Prettier pass; 6 unit tests unrelated to this changefail on
mainas well (CreateFolderModal,ResourcePreview,conflictDialog,resourcesTransfer)Types of changes