feat(cli): upload hash caching and progress tracking #19855
+414
−103
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces several improvements to the CLI upload functionality: 1. Performance Improvements: - Implement file hash caching with auto-save every 30s - Set default concurrency to (CPU cores - 1) - Add proper file stats caching to avoid redundant stat calls 2. Progress Bar UI Improvements: - Add human-readable file sizes in progress bars - Add human-readable ETA time format - Improve ETA calculation with larger sample buffer - Add proper cursor cleanup on SIGINT - Show total size of files being processed - Fix progress bar formatting 3. Album Name Formatting: - Add --format-album-names flag to clean up album names - Remove leading non-alphabetic characters - Convert underscores and dashes to spaces - Capitalize first letter - Show album name changes in console 4. Code Organization & Fixes: - Add @types/picomatch dependency - Fix watch mode implementation with proper file matching - Change -h flag to -s for skip-hash to avoid conflict - Improve error handling and cleanup in hash operations The main focus of these changes is to improve upload performance through caching and better resource utilization, especially beneficial for large uploads, while also providing a more informative progress display and cleaner album names.
|
Label error. Requires exactly 1 of: changelog:.*. Found: cli. A maintainer will add the required label. |
jrasm91
reviewed
Oct 8, 2025
Comment on lines
+87
to
+94
| // Merge program options with command options | ||
| const mergedOptions = { ...program.opts(), ...options }; | ||
| upload(paths, program.opts(), mergedOptions); |
There was a problem hiding this comment.
why are you merging them? they are passed separately to the upload() command and the implementation can handle with merging them if needed, no?
| const content = await readFile(this.cacheFile, 'utf-8'); | ||
| this.cache = JSON.parse(content); | ||
| } catch (error) { | ||
| console.warn('Failed to load hash cache, starting fresh'); |
There was a problem hiding this comment.
we probably should log the error information as well
| async save() { | ||
| if (this.isDirty) { | ||
| try { | ||
| await writeFile(this.cacheFile, JSON.stringify(this.cache, null, 2)); |
There was a problem hiding this comment.
this file is going to be significantly bigger with pretty-print, and it's not meant to be read by humans, so I would probably avoid stringifying it.
- Implement persistent hash caching to avoid unnecessary file rehashing - Switch progress bars to show processed file size instead of asset count - Fix concurrency handling based on CPU count
…m names Fixes an issue where uploading photos with the same directory name would cause albums to be merged unintentionally. The CLI now properly handles duplicate album names by: - Generating unique album names when duplicates are detected - Adding parent directory names to create distinct album name - Detecting duplicates both on the server and in the current upload batch This fix is especially useful for directory structure like: photos/ ├── 2023/ │ ├── Bretagne/ -> Album: 2023 Bretagne │ └── summer/ │ └── beach/ -> Album: 2023 summer beach │ └── city/ -> Album: 2023 summer city ├── 2024/ │ ├── Bretagne/ -> Album: 2024 Bretagne │ └── summer/ │ └── beach/ -> Album: 2024 summer beach │ └── city/ -> Album: 2024 summer city
Nykri
force-pushed
the
feat/cli-upload-improvements
branch
from
1776d96 to
98f261f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds some small improvements to the
immich uploadcommand. It is particularly useful for big uploads of large libraries:It adds hash caching :
~/.config/immich/hash-cache.jsonProgress bar has been adapted to display file size instead of assets (which were hardly readable and usable for large uploads)
Concurrency number is automatically set to CPU cores - 1
Generates a unique album name when several directories have the same name. It does that using the parent folders names.
This fix is especially useful for directory structure like:
photos/
├── 2023/
│ ├── Bretagne/ -> Album: 2023 Bretagne
│ └── summer/
│ └── beach/ -> Album: 2023 summer beach
│ └── city/ -> Album: 2023 summer city
├── 2024/
│ ├── Bretagne/ -> Album: 2024 Bretagne
│ └── summer/
│ └── beach/ -> Album: 2024 summer beach
│ └── city/ -> Album: 2024 summer city
How Has This Been Tested?
Tested on a dummy file structure and on a real >1TB library upload.
Checklist:
src/services/uses repositories implementations for database calls, filesystem operations, etc.src/repositories/is pretty basic/simple and does not have any immich specific logic (that belongs insrc/services/)