Commit 4b6a891
Cache pipeline by model+device only, not call-time params (#499)
top_k, function_to_apply, and batch_size are call-time parameters
passed at pipe() invocation, not pipeline construction config.
Including them in the cache key created duplicate pipeline instances
in memory for each unique parameter combination.
Now a single pipeline is reused regardless of top_k/function_to_apply
values — these are passed per-call at inference time.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent ae773d5 commit 4b6a891
1 file changed
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
48 | 50 | | |
49 | 51 | | |
50 | 52 | | |
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
54 | 56 | | |
55 | | - | |
56 | | - | |
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| |||
0 commit comments