Endpoint failed to start due to ShardFailed on hugging face inference endpoint

[relativity-codes]

System Info

When starting the endpoint, the server crashes with ShardFailed. The backend process exits with code 1 before serving requests.

[Server message]Endpoint failed to start
Exit code: 1. Reason: :"backends/v3/src/client/mod.rs","line_number":45,"span":{"batch_id":"Some(69)","name":"clear_cache"},"spans":[{"batch_size":1,"name":"batch"},{"name":"prefill"},{"batch_id":"Some(69)","name":"clear_cache"},{"batch_id":"Some(69)","name":"clear_cache"}]}
{"timestamp":"2025-08-29T14:40:17.288163Z","level":"ERROR","message":"Request failed during generation: Server error: error trying to connect: No such file or directory (os error 2)","target":"text_generation_router_v3::backend","filename":"backends/v3/src/backend.rs","line_number":546,"span":{"name":"send_error"},"spans":[{"parameters":"GenerateParameters { best_of: None, temperature: None, repetition_penalty: None, frequency_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: true, max_new_tokens: Some(4800), return_full_text: None, stop: [], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None }","name":"chat_completions"},{"name":"generate"},{"name":"generate_stream"},{"name":"schedule"},{"name":"infer"},{"name":"send_error"}]}
{"timestamp":"2025-08-29T14:40:17.300833Z","level":"ERROR","fields":{"message":"Shard 0 crashed"},"target":"text_generation_launcher"}
{"timestamp":"2025-08-29T14:40:17.300875Z","level":"INFO","fields":{"message":"Terminating webserver"},"target":"text_generation_launcher"}
{"timestamp":"2025-08-29T14:40:17.300901Z","level":"INFO","fields":{"message":"Waiting for webserver to gracefully shutdown"},"target":"text_generation_launcher"}
{"timestamp":"2025-08-29T14:40:17.301003Z","level":"INFO","message":"signal received, starting graceful shutdown","target":"text_generation_router::server","filename":"router/src/server.rs","line_number":2395}
{"timestamp":"2025-08-29T14:40:17.501249Z","level":"INFO","fields":{"message":"webserver terminated"},"target":"text_generation_launcher"}
{"timestamp":"2025-08-29T14:40:17.501293Z","level":"INFO","fields":{"message":"Shutting down shards"},"target":"text_generation_launcher"}
Error: ShardFailed

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

happens on hugging face inference endpoint

Expected behavior

this related to deprecation of a dependency