feat(conversation): add prompt cache + usage metrics to test caching #4154
+979
−225
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… works Signed-off-by: Samantha Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]>
7 tasks
Signed-off-by: Samantha Coyle <[email protected]>
JoshVanL
requested changes
Jan 7, 2026
| // NOTE: These are all translations due to langchaingo data types. | ||
|
|
||
| // extractInt64FromGenInfo extracts an int64 value from genInfo map to extract usage data from langchaingo's GenerationInfo map in the choices response. | ||
| func extractInt64FromGenInfo(genInfo map[string]any, key string) int64 { |
Contributor
There was a problem hiding this comment.
Should we not return an error here when the value is not a number? Do we not expect other number types (uint)?
Contributor
Author
There was a problem hiding this comment.
I initially defined this as int64 only in my original PR. However, after splitting this effort and what I'm assuming is a subsequent version bump, my conformance tests were no longer returning usage metrics, which required loosening the handling here. This change resolved the issue.
TLDR: Yes, I can add support for uint as a defensive measure and return an error for any other types.
conversation/converse.go
Outdated
| Metadata map[string]string `json:"metadata"` | ||
| Model *string `json:"model"` | ||
|
|
||
| PromptCacheRetention time.Duration `json:"promptCacheRetention"` |
Contributor
There was a problem hiding this comment.
Should this be optional?
Suggested change
| PromptCacheRetention time.Duration `json:"promptCacheRetention"` | |
| PromptCacheRetention *time.Duration `json:"promptCacheRetention"` |
10 tasks
Signed-off-by: Samantha Coyle <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR breaks out work from a larger contrib PR into a smaller, more focused change: #4129
It introduces response caching support via metadata passed to LLM providers. This serves as a workaround for LangChain’s WithPromptCaching(true) option, which currently sets a boolean value that fails Dapr conformance tests because OpenAI-based providers expect a duration string rather than a boolean.
To properly validate this workaround, usage metrics support was added, which required additional data type translations within LangChain.
The PR also renames CacheTTL to ResponseCacheTTL to more accurately reflect its behavior. Backward compatibility is maintained by continuing to support the original JSON tag.
Finally, the LangChain dependency was updated to support the newly required options that I am using.
Reconfirmed things work as expected:
Issue reference
We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.
Please reference the issue this PR will close: #[issue number]
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:
Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.