Fixing Magento's catalog image difficulties

[colinmollenhour]

be me: Merchant adds thousands of new products every year and has a trigger-happy photographer with a 50MP camera.. Disk usage alerts.. Time to upgrade server disks again!

be me: Merchant updates watermark or other image config during peak hours and server gets pummeled trying to suddenly resize thousands of images..

I'm querying to see how much interest there is in a solution to these problems. I'm sure with Magento being as old as it is many of you have already worked around this to some degree but I am looking to fix it fix it and I have an idea to propose.

To me the crux of the problem is that even if you use S3, Magento requires every file to be present on the server, not just the originals but also the resizes, of which there could be many variations and cleaning them up is only simple if your system can handle a sudden wipe of the image cache but they can still take up much space.

Questions:

1. How does Magento 2 solve these issues, or does it?
2. Do you have any particularly clever fixes yourself that you'd like to share?
3. What do you think of my proposal and are you interested in helping test it and develop it?

My proposed solution is to make minimal changes such that the catalog images are not stored on the local disk at all and are only stored in S3. Resized images are handled by a "resizer" service (microservice if you like) which sits behind a caching CDN and generates the resized images on the fly. As long as the cache is effective at maintaining a high hit rate, the resizer service should theoretically not get hit too hard, but it can also be deployed to a serverless environment so you don't have to worry about provisioning large capacity for it.

I've built a proof of concept for this which appears to work great for my specific use case. However, it is not quite ready for general use as it lacks:

1. Support for watermarks. (I propose these are staged in a well-known location and the resizer service fetches the watermark image on the fly and caches it locally so it can apply it as needed using a short identifier in the url like "wm=g3xve4cb")
2. Fallback for users that don't want to use the resizer service (currently resizing is just not supported without it so original images are served for all sizes). I think it would be fairly easy to add fallback to the old behavior.
3. Heavy testing. :)

The resizer service is written in Node.js using https://github.com/lovell/sharp and is already fairly configurable. The image library is very mature and well-supported and blazing fast with excellent image quality output. As a bonus this allows you to take advantage of webp with zero additional effort as the resizer converts to webp by default (if you wanted to support really old browsers you'd want to change the default). It can be deployed to any serverless environment or just run on the same server for small deployments. It is easy to test with docker-compose by launching minio locally as an S3 stand-in.

I've been improving a fork of https://github.com/thaiphan/magento-s3/ - started 5 years ago! 🤣 If I remember correctly the author is happy to allow OpenMage to absorb this extension into the core but I don't think it would be completely necessary and it could remain a separate extension although without a designated maintainer I think I'd prefer it merged into core.

What do you think?

[fballiano]

I offset resizing and watermarking to a CDN, I've also modified every project in order not to have different copies of all the images on a per-store bases, which in most cases doesn't make any sense.

[fballiano]

sure, in case of an openmage-extension that works as a bridge between openmage and the thumbnailer there's no hard lock-in ;-)

I've always used bunny (and other CDNs) in "pull" mode so all the original contents are primarily on the customer' server, but it would be possible to store them in the CDN itself as a storage container. It also depends on how the users of the backend upload the images... if you've an automated script that there's no problem at all, if they use the backend then you need another piece of software to offload the files to the storage container.

[colinmollenhour]

Yes, the solution I'm proposing uses the core/file_storage interface for uploading to S3 so this is mostly already supported by the core code throughout - so you could set Bunny Storage as the storage container (assuming it supports S3 API) or any other S3-compatible service and have Bunny Optimizer pull from your bucket.

[fballiano]

it should be possible although I've never done it since I still prefer (if possible) to have the files locally

[colinmollenhour]

You don't trust S3 or some other reason?

[fballiano]

I didn't have the need for it, local storage was cheaper, I didn't have to modify anything in openmage ;-)

[empiricompany]

Also on our end, the media folder is always a problem, especially when the theme generates many thumbnails (often unnecessary ones), and the cache folder grows rapidly. I was also looking for a systematic solution to move it out of the nvme, which costs me a lot compared to an S3 bucket. Ideally, it would be to reserve the NVMe disks exclusively for handling dynamic files.