[Feature] Prioritized Thumbnail Generation for New Uploads

[coltenkrauter]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

As a user with a large media library, I want new uploads to have their thumbnails generated promptly even when a full-library thumbnail generation job is running, so that new media does not show "Error Loading Image" placeholders.

Context

I occasionally need to run a full-library thumbnail generation job (if I've changed thumbnail settings, etc). With a library of approximately 5TB and 170,000 images/videos on TrueNAS hardware, this job takes 1-2 weeks, using around 50 concurrent tasks. During this time, new uploads have no thumbnails generated, leading to a poor user experience for me and my family. We actively use Immich, and this problem will become more significant as more family members join the platform and as my library grows.

The current lack of task prioritization for new uploads results in a temporary "outage" for thumbnail generation, causing inconvenience and confusion. Ensuring that new uploads can be processed first, or providing a way to prioritize jobs, would improve the user experience significantly.

Related GitHub issue #12578 – To gain traction, I started this feature request.

Platform

• Server
• Web
• Mobile

[coltenkrauter]

Another idea is to handle "ALL" jobs as separate jobs from regular jobs. This could apply not just to thumbnail generation but to other tasks such as video transcoding.

By isolating these large-scale "ALL" jobs, Immich could allow them to run independently with separate job limits, preventing an "outage" of services like thumbnail generation or transcoding for new uploads. This approach would enhance scalability and flexibility for large media collections.

[mertalev]

With a library of approximately 5TB and 170,000 images/videos on TrueNAS hardware, this job takes 1-2 weeks, using around 50 concurrent tasks.

That seems like a very long amount of time for that number of assets. Can you share more details about your hardware? How many of these are videos?

Also, running 50 concurrent tasks is only going to hurt thumbnail generation speed. It should be much, much lower than that unless the server is a monster (and it isn't by the sound of it).

[coltenkrauter]

That seems like a very long amount of time for that number of assets.

That is good to know. I don't have much to compare to. What are some estimates that you can share for different-sized libraries (and how many vids would those have)?

Can you share more details about your hardware?

TrueNAS Mini X+ (Compact ZFS Storage Server)

• 32GB ECC RAM
• 8-Core 2.2GHz Intel Atom CPU
• 10 Gigabit Network (helps for local stuff...)
• 2 HDDs mirrored – Western Digital 20TB WD Red Pro NAS Internal Hard Drive
  • 7200 RPM, SATA 6 Gb/s, CMR, 512 MB Cache

How many of these are videos?

16,150 Videos
166,734 Photos

Also, running 50 concurrent tasks is only going to hurt thumbnail generation speed.

That makes sense. Based on my hardware... What would be a reasonable concurrency?

---

@mertalev, what are your thoughts on the broader issue–long-running Immich jobs can delay some types of processing for new uploads, causing users to encounter degraded functionality for possibly hours or days.

[coltenkrauter]

We don't expect admins to regularly re-run these jobs as you describe, so optimizing for this doesn't make much sense to me.

Yes, this isn't common, so I would expect any improvement would likely be low priority.

Thank you for considering.

[mertalev]

That is good to know. I don't have much to compare to. What are some estimates that you can share for different-sized libraries (and how many vids would those have)?

Oops, I missed this question. I can say that a system with 32GB RAM, a 7800x3D processor and an NVMe SSD can process 30k ~1080p JPG images in maybe 5 minutes. I'd expect a 30k library of 4K images to take roughly 15 minutes, or 1.5 hours for 180k.

[coltenkrauter]

I can say that a system with 32GB RAM, a 7800x3D processor and an NVMe SSD can process 30k ~1080p JPG images in maybe 5 minutes. I'd expect a 30k library of 4K images to take roughly 15 minutes, or 1.5 hours for 180k.

Ah, good to know.
Do you think that the thumbnail settings (size/type/etc) would have much of an impact on this?

[mertalev]

I can say that a system with 32GB RAM, a 7800x3D processor and an NVMe SSD can process 30k ~1080p JPG images in maybe 5 minutes. I'd expect a 30k library of 4K images to take roughly 15 minutes, or 1.5 hours for 180k.

Ah, good to know. Do you think that the thumbnail settings (size/type/etc) would have much of an impact on this?

The biggest difference will be the format and resolution of the original images. 4K RAW is much more intensive than 4K HEIC, which is more intensive than 4K JPEG. For faster source formats like JPEG, the encoding settings will have a bigger impact than for slower formats like RAW since more of the time is spent in the encoder vs. the decoder. Of the encoding settings, resolution makes more of a difference than format.

[coltenkrauter]

The biggest difference will be the format and resolution of the original images. 4K RAW is much more intensive than 4K HEIC, which is more intensive than 4K JPEG. For faster source formats like JPEG, the encoding settings will have a bigger impact than for slower formats like RAW since more of the time is spent in the encoder vs. the decoder. Of the encoding settings, resolution makes more of a difference than format.

That makes sense. I do have many thousands of images in RAW format, so I may have a unique situation that led to such long processing times.