Fix multi hypothesis streaming #1496

zphoenixrises · 2025-05-21T20:33:01Z

No description provided.

…llm-server

- Cleanup CSV generation to produce only median values - Switch from streaming to non streaming output

…llm-server

…o-protect-server

For greedy selection, or if already in a `softmax`/`log_softmax` form don't take temperature, Otherwise only take temperature on the values that are getting converted to `softmax`

…o_int and int_to_float conversions

…m/nod-ai/shark-ai

…llm-server

…isolation

…o shortfin-llm-multi-workers-fibers

… instead of synchronizing device

…o shortfin-llm-no-device-syncs

…aione/SHARK-Platform into chai_demo

…ecode-allocs into https://github.com/nod-ai/shark-ai/tree/chai_demo

…culated value from `prefill`

… that `check_host_array` operates properly

…o ensure that `check_host_array` operates properly" This reverts commit 01523de.

[WARNING] This can break stuff. Just meant for parallel iteration.

sfnp.argmax uses XTensor's implementation of argmax underneath, which is slower than how numpy implements it. This commit changes the argmax calls in favor of the numpy implementation.

zphoenixrises and others added 30 commits April 10, 2025 17:11

Added benchmarking code

37bc3d4

Add configurable inputlengths

e437ae4

Modify benchmark to match chai benchmarks

fb4e41b

minor modification

4e8f38c

Merge remote-tracking branch 'origin/main' into add-benchmarking-for-…

5d85844

…llm-server

Refactor benhmark code

7e11852

- Cleanup CSV generation to produce only median values - Switch from streaming to non streaming output

Computmaximum batch size with latency less than 4.2s

a41a984

Merge branch 'main' into add-benchmarking-for-llm-server

6356bcc

Reject requests to protect server

3c9538b

Merge remote-tracking branch 'origin/main' into add-benchmarking-for-…

0ddad40

…llm-server

Merge branch 'add-benchmarking-for-llm-server' into reject-requests-t…

2b1ad08

…o-protect-server

Only take temperature when we need it and on the logits that need it,

a5829f7

For greedy selection, or if already in a `softmax`/`log_softmax` form don't take temperature, Otherwise only take temperature on the values that are getting converted to `softmax`

Fix beam_group bug

063f27d

Add faster-temperature goodness to beam_search

65ecd45

Merge branch 'main' into faster-temperature

70cf1af

Add a TODO and link to issue to clean up how we're doing to float_t…

a76b53f

…o_int and int_to_float conversions

Update #TODO comment

1e79644

Enable multiple workers and multiple fibers per worker

a95e4f1

Merge branch 'main' into shortfin-llm-multi-workers-fibers

745faa8

Merge branch 'main' into shortfin-llm-multi-workers-fibers

6c03052

Fix typo

c0be5ca

Merge branch 'reject-requests-to-protect-server' of https://github.co…

77d1e89

…m/nod-ai/shark-ai

Merge remote-tracking branch 'origin/main' into add-benchmarking-for-…

7855e10

…llm-server

Add a check for multiple fibers_per_worker, when using per_fiber …

3a2f557

…isolation

Merge branch 'main' of https://github.com/stbaione/SHARK-Platform int…

a119dc9

…o shortfin-llm-multi-workers-fibers

Fix _check_for_per_fiber_bug and lint server.py

92325a2

Fix direct_to_batcher tests

57adc6b

Modify benchmark client to provide sustained lod on server

c16671a

Merge remote-tracking branch 'origin/main' into chai_demo

9a9fffd

Merge branch 'add-benchmarking-for-llm-server' into chai_demo

abb6d38

Zeeshan Haque and others added 14 commits April 21, 2025 16:38

Add exponential backoff

a02716d

Use check_host_array hack to determine if logits have been updated,…

e9a1c86

… instead of synchronizing device

Merge branch 'main' of https://github.com/stbaione/SHARK-Platform int…

cba57a5

…o shortfin-llm-no-device-syncs

Merge branch 'main' of https://github.com/nod-ai/shark-ai into chai_demo

5ad8d6c

Merge branch 'shortfin-llm-no-device-syncs' of https://github.com/stb…

156c90d

…aione/SHARK-Platform into chai_demo

Merge https://github.com/vinayakdsci/SHARK-Platform/tree/fix-device-d…

33361af

…ecode-allocs into https://github.com/nod-ai/shark-ai/tree/chai_demo

Reallocate seq_block_ids when block_count surpasses initially cal…

92d1edf

…culated value from `prefill`

Add multi instance llm inference

87ece03

Ensure that decode_logits is cleared out prior to reuse to ensure…

01523de

… that `check_host_array` operates properly

Revert "Ensure that decode_logits is cleared out prior to reuse t…

31ee40f

…o ensure that `check_host_array` operates properly" This reverts commit 01523de.

Validate accuracy with re-usable device allocations.

5ca8067

[WARNING] This can break stuff. Just meant for parallel iteration.

Replace sfnp.argmax with numpy.argmax

52890aa

sfnp.argmax uses XTensor's implementation of argmax underneath, which is slower than how numpy implements it. This commit changes the argmax calls in favor of the numpy implementation.

Add streaming to multi_greedy

2a930b8

Fix Multi-hypothesis Streaming

cef06c6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix multi hypothesis streaming #1496

Fix multi hypothesis streaming #1496

zphoenixrises commented May 21, 2025

Uh oh!

Uh oh!

Fix multi hypothesis streaming #1496

Are you sure you want to change the base?

Fix multi hypothesis streaming #1496

Conversation

zphoenixrises commented May 21, 2025

Uh oh!

Uh oh!