Skip to content

Respect Context Length of Models #175

@timlenardo

Description

@timlenardo

Between web search, autocontext, and general context uploading, we're getting to a point where there can be quite a lot of text in the context window. For some models, particularly local models, it's possible to run out of context. We should add in the context length for each model, and find ways to shorten the context when sending our requests, if it's too long. This can be part of the RAG system that we eventually integrate.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions