Skip to content

Make blocksize configurable #111

@Gregory-Pereira

Description

@Gregory-Pereira

This was particularly relevant when @oglok was testing the prefixAware caching demo. It would be nice to be able to configure block size but have it able to fallback to 256 as default.

Coderef: https://github.com/llm-d/llm-d-inference-scheduler/blob/main/pkg/scheduling/plugins/scorer/prefix_store.go#L19

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions