Deduplicate CAD assets and stop re-fingerprinting undo history during .skf autosave - #101
Open
davidsteinmeyer wants to merge 1 commit into
Open
Deduplicate CAD assets and stop re-fingerprinting undo history during .skf autosave#101davidsteinmeyer wants to merge 1 commit into
davidsteinmeyer wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Autosave generates a full .skf archive containing the complete undo history. Export was encoding and SHA-256-hashing every resource in every history state, deduplicating only after hashing, inlining the large
cadDisplayEdgescache into every node of every state, and re-fingerprinting the whole history twice. On a real 6.9 MB project this produced a 30.8 MB project.json and a ~3.8 s export on the main thread. This PR makes asset handling content-addressed before the expensive work, stores display edges once as a shared asset, and reuses history fingerprints. History behavior is fully preserved.Root cause
SkfArchiveBuilder.addAssetcomputedsha256Hex(bytes)before checking itskind:sha256dedup map, so identical resources were re-encoded and re-hashed once per state.addDerivedMeshcached only by object identity, andcadBrep,brepStep, and image data URLs had no pre-hash cache at all.cadDisplayEdgeswas left in the node definition and inlined per node per state. In the real project 230 occurrences held only 2 unique payloads (0.21 MB unique -> 24.5 MB in project.json).canonicalizeShapealways allocated new nested arrays, so the fingerprint WeakMap missed, andexportSkfProjectfingerprinted the entire history twice.Changes
recordByResourceKeyandaddEncodedAsset(kind, resourceKey, encode, options)toSkfArchiveBuilder; it returns a cached promise on a key hit and only invokes theencode()thunk on a miss, so encoding and hashing are skipped for repeated content.addDerivedMeshnow keys onmesh.storageResourceId ?? immutableResourceFingerprint(mesh)(content-based) instead of object identity.serializeShapeNoderoutesimagePlate,sketchProfile.images,brepStep, andcadBrepthroughaddEncodedAssetwith content keys.cadDisplayEdgesis now extracted from the node definition, stored once in the asset table (kindbrep, media typeapplication/vnd.sketchforge.display-edges), and referenced by the new optionalcadDisplayEdgesAssetIdnode field. The importer restores it and also shares one array per distinct payload for legacy inline packages.exportSkfProjectreuses the fingerprint already computed byhydrateEditorHistoryStateand only re-fingerprints entries whose shapes were actually repaired.repairDuplicateGroupedObjectIdsreturns the original array when nothing changed.canonicalizeShapeis now identity-preserving when a shape is already canonical, so resource fingerprints cached by object identity survive repeated canonicalization.Why this fixes the issue
cadDisplayEdgesstorage drops from 24.5 MB to 0.21 MB, shrinking the real project's project.json from 30.8 MB to ~6.5 MB and the archive from 7.8 MB to 1.9 MB.Testing
npm run typecheckNotes
New unit tests:
Performance impact