Error: Backend(Warmup(Generation("Hidden size mismatch"))) when launch Mixtral-8x22B-v0.1

[alexhegit]

System Info

System Info

TGI version: 2.2.1-dev0

LLM: mistralai/Mixtral-8x22B-v0.1 (https://huggingface.co/mistralai/Mixtral-8x22B-v0.1)

GPU: AMD MI300X

ROCm v6.2.0

Commands:
text-generation-launcher --model-id /data/llm/Mixtral-8x22B-v0.1/ --num-shard 8

Logs:

AssertionError: Hidden size mismatch
2024-09-21T02:24:10.881143Z ERROR warmup{max_input_length=4095 max_prefill_tokens=4145 max_total_tokens=4096 max_batch_size=None}:warmup: text_generation_router_v3::client: backends/v3/src/client/mod.rs:54: Server error: Hidden size mismatch
Error: Backend(Warmup(Generation("Hidden size mismatch")))
2024-09-21T02:24:10.923425Z ERROR text_generation_launcher: Webserver Crashed
2024-09-21T02:24:10.923441Z  INFO text_generation_launcher: Shutting down shards
2024-09-21T02:24:10.928558Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=0
2024-09-21T02:24:10.928558Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=1
2024-09-21T02:24:10.928579Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=6
2024-09-21T02:24:10.928596Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=1
2024-09-21T02:24:10.928561Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=4
2024-09-21T02:24:10.928585Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=0
2024-09-21T02:24:10.928615Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=4
2024-09-21T02:24:10.928561Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=2
2024-09-21T02:24:10.928602Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=6
2024-09-21T02:24:10.928635Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=2
2024-09-21T02:24:10.928682Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=7
2024-09-21T02:24:10.928706Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=7
2024-09-21T02:24:10.928912Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=5
2024-09-21T02:24:10.928933Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=5
2024-09-21T02:24:10.934859Z  INFO shard-manager: text_generation_launcher: Terminating shard rank=3
2024-09-21T02:24:10.934879Z  INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=3
2024-09-21T02:24:15.831422Z  INFO shard-manager: text_generation_launcher: shard terminated rank=2
2024-09-21T02:24:15.831607Z  INFO shard-manager: text_generation_launcher: shard terminated rank=5
2024-09-21T02:24:16.131589Z  INFO shard-manager: text_generation_launcher: shard terminated rank=6
2024-09-21T02:24:16.231667Z  INFO shard-manager: text_generation_launcher: shard terminated rank=0
2024-09-21T02:24:16.231667Z  INFO shard-manager: text_generation_launcher: shard terminated rank=4
2024-09-21T02:24:16.331713Z  INFO shard-manager: text_generation_launcher: shard terminated rank=7
2024-09-21T02:24:16.338006Z  INFO shard-manager: text_generation_launcher: shard terminated rank=3
2024-09-21T02:24:16.431761Z  INFO shard-manager: text_generation_launcher: shard terminated rank=1
Error: WebserverFailed

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Steps1:
Download the model

Steps2:
text-generation-launcher --model-id /data/llm/Mixtral-8x22B-v0.1/ --num-shard 8

Failed with error

2024-09-21T02:24:10.881143Z ERROR warmup{max_input_length=4095 max_prefill_tokens=4145 max_total_tokens=4096 max_batch_size=None}:warmup: text_generation_router_v3::client: backends/v3/src/client/mod.rs:54: Server error: Hidden size mismatch Error: Backend(Warmup(Generation("Hidden size mismatch"))) 2024-09-21T02:24:10.923425Z ERROR text_generation_launcher: Webserver Crashed 2024-09-21T02:24:10.923441Z INFO text_generation_launcher: Shutting down shards 2024-09-21T02:24:10.928558Z INFO shard-manager: text_generation_launcher: Terminating shard rank=0 2024-09-21T02:24:10.928558Z INFO shard-manager: text_generation_launcher: Terminating shard rank=1 2024-09-21T02:24:10.928579Z INFO shard-manager: text_generation_launcher: Terminating shard rank=6 2024-09-21T02:24:10.928596Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=1 2024-09-21T02:24:10.928561Z INFO shard-manager: text_generation_launcher: Terminating shard rank=4 2024-09-21T02:24:10.928585Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=0 2024-09-21T02:24:10.928615Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=4 2024-09-21T02:24:10.928561Z INFO shard-manager: text_generation_launcher: Terminating shard rank=2 2024-09-21T02:24:10.928602Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=6 2024-09-21T02:24:10.928635Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=2 2024-09-21T02:24:10.928682Z INFO shard-manager: text_generation_launcher: Terminating shard rank=7 2024-09-21T02:24:10.928706Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=7 2024-09-21T02:24:10.928912Z INFO shard-manager: text_generation_launcher: Terminating shard rank=5 2024-09-21T02:24:10.928933Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=5 2024-09-21T02:24:10.934859Z INFO shard-manager: text_generation_launcher: Terminating shard rank=3 2024-09-21T02:24:10.934879Z INFO shard-manager: text_generation_launcher: Waiting for shard to gracefully shutdown rank=3 2024-09-21T02:24:15.831422Z INFO shard-manager: text_generation_launcher: shard terminated rank=2 2024-09-21T02:24:15.831607Z INFO shard-manager: text_generation_launcher: shard terminated rank=5 2024-09-21T02:24:16.131589Z INFO shard-manager: text_generation_launcher: shard terminated rank=6 2024-09-21T02:24:16.231667Z INFO shard-manager: text_generation_launcher: shard terminated rank=0 2024-09-21T02:24:16.231667Z INFO shard-manager: text_generation_launcher: shard terminated rank=4 2024-09-21T02:24:16.331713Z INFO shard-manager: text_generation_launcher: shard terminated rank=7 2024-09-21T02:24:16.338006Z INFO shard-manager: text_generation_launcher: shard terminated rank=3 2024-09-21T02:24:16.431761Z INFO shard-manager: text_generation_launcher: shard terminated rank=1 Error: WebserverFailed

Expected behavior

Model server launcher successfully.