LLM: release plugin once pipeline is removed and WA for GPU #1846
Conversation
github-actions
bot
added
category: continuous batching
| template<class... Ts> overloaded(Ts...) -> overloaded<Ts...>; | ||
| ContinuousBatchingPipeline::IContinuousBatchingPipeline::~IContinuousBatchingPipeline() { | ||
| m_tokenizer = {}; | ||
| } |
There was a problem hiding this comment.
IContinuousBatchingPipeline is ContinuousBatchingImpl's parent. The order of destructor calls is reverse:
- ~ContinuousBatchingImpl()
- ~IContinuousBatchingPipeline()
By the time ~IContinuousBatchingPipeline() is called, utils::release_core_plugin(m_device) from ContinuousBatchingImpl() had already been executed. Given that the solution is satisfactory, you could have defined ~IContinuousBatchingPipeline() = default; and get the same result. But maybe a better solution would be to move utils::release_core_plugin(m_device) to IContinuousBatchingPipeline() (m_device is already there). This would fix the call order, remove the need to manually clear ContinuousBatchingImpl's members and enable clearing for other children: PromptLookupImpl and SpeculativeDecodingImpl.
Based on #1627 and WA for GPU oneDNN cache clean added