Model the CPU impact of request preprocessing

[willb]

Some multimodal models incorporate nontrivial preprocessing pipelines. Even relatively simple cases like image-to-text models are loading, decoding, converting representations or color spaces, and scaling before tokenizing, and many of these steps are unaccelerated. An imprecise or static napkin-math model of performance here (e.g., "Pillow can downscale images at a rate of 30-60 megapixels/second/core on an Intel Xeon") could be useful to guide resource limits for deployments.

[tianhaox]

Hi, it's a good suggestion. I will first suggest moving to leverage GPU resources using nvjpeg/dali/cv-cuda/nvpyvideocodec to mitigate the impact. But indeed, there will be resource convention and perf degradation on gpu for multi-modal use cases. For Dynamo, we encourage user to do EPD with pre-processing on dedicated E instance. Currently I don't have a clear idea in my mind. Perhaps an empirical correction factor, to degrade the prefill latency as well as the prefill throughput. We exposed this argument to user in yaml config. prefill_latency_correction_scale. BTW, multi-modal model is not supported yet, need to add the modeling for vision encoder.