feat: auto-detect model download hub between Hugging Face and ModelScope

doc/source/getting_started/environments.rst

@@ -11,8 +11,15 @@ Default value is http://127.0.0.1:9997 , you can get it through logs.
 
 XINFERENCE_MODEL_SRC
 ~~~~~~~~~~~~~~~~~~~~~
-Modelhub used for downloading models. Default is "huggingface", or you
-can set "modelscope" as downloading source.
+Modelhub used for downloading models. Options are "huggingface", "modelscope"
+and "auto". When unset or set to "auto", Xinference probes whether the Hugging
+Face endpoint is reachable and picks "huggingface" if so, otherwise falls back
+to "modelscope". Set an explicit value to pin the download source.
+
+XINFERENCE_HUB_DETECT_TIMEOUT
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Timeout in seconds for probing hub connectivity when the download hub is
+automatically detected. Default is 3.
 
 .. _environments_xinference_home:
 

doc/source/getting_started/using_docker_compose.rst

@@ -61,7 +61,7 @@ Available variables:
 
 * ``XINFERENCE_IMAGE``: image to run, defaults to ``xprobe/xinference:latest``. Pin a release tag such as ``xprobe/xinference:v<version>`` for production.
 * ``XINFERENCE_PORT``: host port of the RESTful API / Web UI, defaults to ``9997``.
-* ``XINFERENCE_MODEL_SRC``: model download source, ``huggingface`` (default) or ``modelscope``.
+* ``XINFERENCE_MODEL_SRC``: model download source, ``auto`` (default, probes Hugging Face reachability and falls back to ModelScope), ``huggingface`` or ``modelscope``.
 * ``XINFERENCE_SHM_SIZE``: shared memory size, defaults to ``8gb``. Increase for multi-GPU inference.
 * ``XINFERENCE_LOG_LEVEL``: log level, defaults to ``info``.
 * ``XINFERENCE_HOME_DIR`` / ``XINFERENCE_HF_CACHE_DIR`` / ``XINFERENCE_MODELSCOPE_CACHE_DIR``: persistence locations. They default to named Docker volumes; point them at absolute host paths to reuse existing model caches, in the same way as described in :ref:`using_docker_image`.

doc/source/models/sources/sources.rst

@@ -6,9 +6,31 @@ Download Sources
 
 Xinference supports downloading various models from different sources.
 
+Automatic Detection
+^^^^^^^^^^^^^^^^^^^
+
+By default, Xinference automatically decides between Hugging Face and ModelScope when
+launching a model: it probes whether the Hugging Face endpoint is reachable (mirrors set
+via ``HF_ENDPOINT`` and proxies set via ``HTTP_PROXY`` / ``HTTPS_PROXY`` are honored).
+If it is reachable, models are downloaded from Hugging Face; otherwise Xinference falls
+back to ModelScope. An HTTP error response (for example ``403``/``407`` from a blocking
+corporate proxy, or a ``5xx`` from a broken mirror) counts as unreachable, since
+downloads would fail anyway. When Hugging Face offline mode is enabled
+(``HF_HUB_OFFLINE=1`` or ``TRANSFORMERS_OFFLINE=1``), no probe runs and Hugging Face is
+selected directly, so air-gapped deployments keep reading from their pre-populated local
+Hugging Face cache. The detection result is cached, so the probe runs at most once per
+process, and its timeout can be tuned via the ``XINFERENCE_HUB_DETECT_TIMEOUT``
+environment variable (default: 3 seconds).
+
+You can also request the detection explicitly by passing ``--download_hub auto`` when
+launching a model, or setting ``XINFERENCE_MODEL_SRC=auto``.
+
+To pin a download source instead of relying on detection, set ``XINFERENCE_MODEL_SRC``
+to ``huggingface`` or ``modelscope``, or pass ``--download_hub`` when launching a model.
+
 HuggingFace
 ^^^^^^^^^^^^^^
-Xinference directly downloads the required models from the official `Hugging Face model repository <https://huggingface.co/models>`_ by default.
+Xinference downloads the required models from the official `Hugging Face model repository <https://huggingface.co/models>`_ when it is reachable.
 
 .. note::
    If you have trouble connecting to Huggingface, you can use a mirror website to download with setting the environment variable ``HF_ENDPOINT=https://hf-mirror.com``.
@@ -17,10 +39,11 @@ Xinference directly downloads the required models from the official `Hugging Fac
 ModelScope
 ^^^^^^^^^^^^^^
 
-When Xinference detects that the system's language is set to Simplified Chinese, it will automatically
-set the model download source to `ModelScope <https://modelscope.cn/models>`_.
+When the Hugging Face endpoint is not reachable (for example, no proxy is available),
+Xinference automatically falls back to downloading models from
+`ModelScope <https://modelscope.cn/models>`_.
 
-You can also achieve this by manually setting an environment variable ``XINFERENCE_MODEL_SRC=modelscope``.
+You can also force this by manually setting an environment variable ``XINFERENCE_MODEL_SRC=modelscope``.
 
 Please check the detail page of a model to confirm whether the model supports downloading from ModelScope.
 If a model spec supports downloading from ModelScope, the "Model Hubs" section in the spec information will

frontend/src/components/pages/launch-model/launch-dialog/launch-dialog.tsx

@@ -318,7 +318,11 @@ export default function LaunchDialog({
   }, [gpuAvailable, modelType, form]);
 
   const downloadHubOptions = useMemo(
-    () => ['none', ...(model?.download_hubs || [])].map((item) => ({ label: item, value: item })),
+    () =>
+      ['auto', 'none', ...(model?.download_hubs || [])].map((item) => ({
+        label: item,
+        value: item,
+      })),
     [model?.download_hubs]
   );
 

xinference/constants.py

@@ -37,6 +37,7 @@
 XINFERENCE_ENV_DISABLE_HEALTH_CHECK = "XINFERENCE_DISABLE_HEALTH_CHECK"
 XINFERENCE_ENV_DISABLE_METRICS = "XINFERENCE_DISABLE_METRICS"
 XINFERENCE_ENV_DOWNLOAD_MAX_ATTEMPTS = "XINFERENCE_DOWNLOAD_MAX_ATTEMPTS"
+XINFERENCE_ENV_HUB_DETECT_TIMEOUT = "XINFERENCE_HUB_DETECT_TIMEOUT"
 XINFERENCE_ENV_TEXT_TO_IMAGE_BATCHING_SIZE = "XINFERENCE_TEXT_TO_IMAGE_BATCHING_SIZE"
 XINFERENCE_ENV_VIRTUAL_ENV = "XINFERENCE_ENABLE_VIRTUAL_ENV"
 XINFERENCE_ENV_VIRTUAL_ENV_SKIP_INSTALLED = "XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED"
@@ -340,6 +341,10 @@ def is_metrics_disabled() -> bool:
 XINFERENCE_DOWNLOAD_MAX_ATTEMPTS = int(
     os.environ.get(XINFERENCE_ENV_DOWNLOAD_MAX_ATTEMPTS, 3)
 )
+# Timeout (seconds) for probing hub connectivity when download_hub is "auto"
+XINFERENCE_HUB_DETECT_TIMEOUT = float(
+    os.environ.get(XINFERENCE_ENV_HUB_DETECT_TIMEOUT, 3)
+)
 XINFERENCE_TEXT_TO_IMAGE_BATCHING_SIZE = os.environ.get(
     XINFERENCE_ENV_TEXT_TO_IMAGE_BATCHING_SIZE, None
 )

xinference/core/supervisor.py

@@ -1956,7 +1956,9 @@ async def launch_builtin_model(
         peft_model_config: Optional[PeftModelConfig] = None,
         worker_ip: Optional[str] = None,
         gpu_idx: Optional[Union[int, List[int]]] = None,
-        download_hub: Optional[Literal["huggingface", "modelscope", "csghub"]] = None,
+        download_hub: Optional[
+            Literal["auto", "huggingface", "modelscope", "csghub"]
+        ] = None,
         model_path: Optional[str] = None,
         enable_virtual_env: Optional[bool] = None,
         virtual_env_packages: Optional[List[str]] = None,
@@ -2437,7 +2439,9 @@ async def _launch_builtin_sharded_model(
         peft_model_config: Optional[PeftModelConfig] = None,
         worker_ip: Optional[str] = None,
         gpu_idx: Optional[Union[int, List[int]]] = None,
-        download_hub: Optional[Literal["huggingface", "modelscope", "csghub"]] = None,
+        download_hub: Optional[
+            Literal["auto", "huggingface", "modelscope", "csghub"]
+        ] = None,
         model_path: Optional[str] = None,
         enable_virtual_env: Optional[bool] = None,
         virtual_env_packages: Optional[List[str]] = None,

xinference/core/worker.py

@@ -3208,7 +3208,7 @@ async def launch_builtin_model(
         request_limits: Optional[int] = None,
         gpu_idx: Optional[Union[int, List[int]]] = None,
         download_hub: Optional[
-            Literal["huggingface", "modelscope", "openmind_hub", "csghub"]
+            Literal["auto", "huggingface", "modelscope", "openmind_hub", "csghub"]
         ] = None,
         model_path: Optional[str] = None,
         enable_virtual_env: Optional[bool] = None,

xinference/deploy/docker/.env.example

@@ -9,8 +9,9 @@
 # Host port for the RESTful API / Web UI.
 # XINFERENCE_PORT=9997
 
-# Model download source: huggingface or modelscope.
-# XINFERENCE_MODEL_SRC=huggingface
+# Model download source: auto (default, probes Hugging Face reachability and
+# falls back to modelscope), huggingface, or modelscope.
+# XINFERENCE_MODEL_SRC=auto
 
 # Shared memory size; increase for multi-GPU inference.
 # XINFERENCE_SHM_SIZE=8gb

xinference/deploy/docker/docker-compose.yml

@@ -30,8 +30,9 @@ services:
       - "${XINFERENCE_PORT:-9997}:9997"
     command: xinference-local -H 0.0.0.0 --port 9997 --log-level ${XINFERENCE_LOG_LEVEL:-info}
     environment:
-      # Model download source: huggingface (default) or modelscope.
-      XINFERENCE_MODEL_SRC: ${XINFERENCE_MODEL_SRC:-huggingface}
+      # Model download source: auto (default, probes Hugging Face reachability
+      # and falls back to modelscope), huggingface, or modelscope.
+      XINFERENCE_MODEL_SRC: ${XINFERENCE_MODEL_SRC:-auto}
     env_file:
       # Only present in offline deployments (cp offline.env.example offline.env).
       # Silently skipped when the file does not exist.

xinference/deploy/docker/offline.env.example

@@ -38,7 +38,12 @@ UV_INSECURE_HOST=xinference-pypiserver:8080
 PIP_INDEX_URL=http://xinference-pypiserver:8080/simple
 PIP_TRUSTED_HOST=xinference-pypiserver
 
-# Model weights are read from the local cache volumes only.
+# Model weights are read from the local cache volumes only. With offline
+# mode enabled, the default XINFERENCE_MODEL_SRC=auto resolves to huggingface
+# without probing the network, so launches read the pre-populated Hugging Face
+# cache volume. If your weights live in the ModelScope cache instead, set
+# XINFERENCE_MODEL_SRC=modelscope in .env (not here: the compose `environment:`
+# entry takes precedence over this env_file).
 HF_HUB_OFFLINE=1
 TRANSFORMERS_OFFLINE=1
 

xinference/model/core.py

@@ -11,7 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from typing import Any, List, Literal, Optional, Union
+from typing import Any, List, Literal, Optional, Union, cast
 
 from .._compat import BaseModel
 from ..types import PeftModelConfig
@@ -27,7 +27,7 @@ def create_model_instance(
     quantization: Optional[str] = None,
     peft_model_config: Optional[PeftModelConfig] = None,
     download_hub: Optional[
-        Literal["huggingface", "modelscope", "openmind_hub", "csghub"]
+        Literal["auto", "huggingface", "modelscope", "openmind_hub", "csghub"]
     ] = None,
     model_path: Optional[str] = None,
     **kwargs,
@@ -38,8 +38,15 @@ def create_model_instance(
     from .image.core import create_image_model_instance
     from .llm.core import create_llm_model_instance
     from .rerank.core import create_rerank_model_instance
+    from .utils import resolve_download_hub
     from .video.core import create_video_model_instance
 
+    # resolve_download_hub never returns "auto", so the narrowing cast is safe
+    resolved_download_hub = cast(
+        Optional[Literal["huggingface", "modelscope", "openmind_hub", "csghub"]],
+        resolve_download_hub(download_hub, model_path),
+    )
+
     # enable_thinking is only meaningful for LLMs; drop it for other model types.
     if model_type != "LLM":
         kwargs.pop("enable_thinking", None)
@@ -53,7 +60,7 @@ def create_model_instance(
             model_size_in_billions,
             quantization,
             peft_model_config,
-            download_hub,
+            resolved_download_hub,
             model_path,
             **kwargs,
         )
@@ -67,7 +74,7 @@ def create_model_instance(
             model_engine,
             model_format,
             quantization,
-            download_hub,
+            resolved_download_hub,
             model_path,
             **kwargs,
         )
@@ -77,7 +84,7 @@ def create_model_instance(
             model_uid,
             model_name,
             peft_model_config,
-            download_hub,
+            resolved_download_hub,
             model_path,
             model_engine,
             model_format,
@@ -92,7 +99,7 @@ def create_model_instance(
             model_engine,
             model_format,
             quantization,
-            download_hub,
+            resolved_download_hub,
             model_path,
             **kwargs,
         )
@@ -101,7 +108,7 @@ def create_model_instance(
         return create_audio_model_instance(
             model_uid,
             model_name,
-            download_hub,
+            resolved_download_hub,
             model_path,
             model_engine=model_engine,
             **kwargs,
@@ -111,7 +118,7 @@ def create_model_instance(
         return create_video_model_instance(
             model_uid,
             model_name,
-            download_hub,
+            resolved_download_hub,
             model_path,
             **kwargs,
         )

xinference/model/embedding/embed_family.py

@@ -58,7 +58,9 @@ def match_embedding(
             f"models: {BUILTIN_EMBEDDING_MODELS.keys()}"
         )
 
-    if download_hub == "modelscope" or download_from_modelscope():
+    if download_hub == "modelscope" or (
+        download_hub is None and download_from_modelscope()
+    ):
         specs = [
             x for x in target_family.model_specs if x.model_hub == "modelscope"
         ] + [x for x in target_family.model_specs if x.model_hub == "huggingface"]

xinference/model/image/core.py

@@ -129,7 +129,9 @@ def match_diffusion(
             return model_spec
 
     if model_name in BUILTIN_IMAGE_MODELS:
-        if download_hub == "modelscope" or download_from_modelscope():
+        if download_hub == "modelscope" or (
+            download_hub is None and download_from_modelscope()
+        ):
             return (
                 [
                     x
@@ -378,7 +380,9 @@ def _select_ocr_model_family(
             f"model list: {BUILTIN_IMAGE_MODELS.keys()}"
         )
 
-    prefer_modelscope = download_hub == "modelscope" or download_from_modelscope()
+    prefer_modelscope = download_hub == "modelscope" or (
+        download_hub is None and download_from_modelscope()
+    )
     preferred_hubs = (
         ["modelscope", "huggingface"]
         if prefer_modelscope

xinference/model/rerank/rerank_family.py

@@ -59,7 +59,9 @@ def match_rerank(
             f"Rerank model {model_name} not found, available models: {BUILTIN_RERANK_MODELS.keys()}"
         )
 
-    if download_hub == "modelscope" or download_from_modelscope():
+    if download_hub == "modelscope" or (
+        download_hub is None and download_from_modelscope()
+    ):
         specs = [
             x for x in target_family.model_specs if x.model_hub == "modelscope"
         ] + [x for x in target_family.model_specs if x.model_hub == "huggingface"]