Cache pipeline by model+device only#499
Conversation
top_k, function_to_apply, and batch_size are call-time parameters passed at pipe() invocation, not pipeline construction config. Including them in the cache key created duplicate pipeline instances in memory for each unique parameter combination. Now a single pipeline is reused regardless of top_k/function_to_apply values — these are passed per-call at inference time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Hugging Face audio classification pipeline caching mechanism to use only the model identity and device as the cache key, treating parameters like top_k and function_to_apply as call-time arguments. Feedback highlights that top_k, function_to_apply, and batch_size are now unused in the method signature and that batch_size is currently ignored in the pipeline invocation, suggesting a cleanup and fix.
| # Cache by model identity + device only. top_k, function_to_apply, | ||
| # and batch_size are call-time parameters passed at pipe() invocation. | ||
| key = f"{model.path_or_uri}-{model.revision}-{device.value}" |
There was a problem hiding this comment.
The comment states that batch_size is a call-time parameter passed at pipe() invocation. However, in classify_audios_with_transformers (line 149), batch_size is not actually passed to the pipeline call, meaning the batch_size argument is currently ignored. Additionally, since top_k, function_to_apply, and batch_size are no longer used to construct the cache key or initialize the pipeline, they are now unused parameters in this method. Consider removing them from the signature of _get_hf_audio_classification_pipeline and updating the caller at line 108.
1 participant
top_k/function_to_apply/batch_size are call-time params, not pipeline config. Single pipeline reused regardless.