Skip to content

Update GetPodScores of Indexer to get tokens as an input instead of preprocessing.RenderJinjaTemplateRequest #244

@mayabar

Description

@mayabar

This change will help the pd profile handler to be able to make decision about disaggregation based on the precise prefix scorer's logic.
Tokenization will be done in the scorer's code before calling the GetPodScores function.
Relevant issue is in llm-d-inference-scheduler project.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions