worker-cpu cannot startup

[leslie2046]

System Info / 系統信息

ubuntu-22.04

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

3.1.0-cpu

The command used to start Xinference / 用以启动 xinference 的命令

services:
  xinference-worker1:
    image: "${IMAGE}"
    container_name: xinference-worker1
    restart: always
    network_mode: host
    environment:
      - XINFERENCE_HOME=${XINFERENCE_HOME}
      - XINFERENCE_MODEL_SRC=${XINFERENCE_MODEL_SRC}
      - XINFERENCE_HEALTH_CHECK_TIMEOUT=${XINFERENCE_HEALTH_CHECK_TIMEOUT}
      - XINFERENCE_HEALTH_CHECK_FAILURE_THRESHOLD=${XINFERENCE_HEALTH_CHECK_FAILURE_THRESHOLD}
      - XINFERENCE_HEALTH_CHECK_INTERVAL=${XINFERENCE_HEALTH_CHECK_INTERVAL}
      - XINFERENCE_LOG_FORMAT=json
      - XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED=${XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED}
      - XINFERENCE_SUBPOOL_LAUNCH_TIMEOUT=600
      - XINFERENCE_MAX_CONCURRENT_LAUNCHES=1
    volumes:
      - ${CACHE_PATH}:/data
    command: >
      xinference-worker -e "${ENDPOINT}" -H "${HOST}" --metrics-exporter-host "${HOST}"  --metrics-exporter-port 31001 --log-level ${LOG_LEVEL}
    stdin_open: true
    tty: true
  xinference-worker2:
    image: "${IMAGE}"
    container_name: xinference-worker2
    restart: always
    network_mode: host
    environment:
      - XINFERENCE_HOME=${XINFERENCE_HOME}
      - XINFERENCE_MODEL_SRC=${XINFERENCE_MODEL_SRC}
      - XINFERENCE_HEALTH_CHECK_TIMEOUT=${XINFERENCE_HEALTH_CHECK_TIMEOUT}
      - XINFERENCE_HEALTH_CHECK_FAILURE_THRESHOLD=${XINFERENCE_HEALTH_CHECK_FAILURE_THRESHOLD}
      - XINFERENCE_HEALTH_CHECK_INTERVAL=${XINFERENCE_HEALTH_CHECK_INTERVAL}
      - XINFERENCE_LOG_FORMAT=json
      - XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED=${XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED}
      - XINFERENCE_SUBPOOL_LAUNCH_TIMEOUT=600
      - XINFERENCE_MAX_CONCURRENT_LAUNCHES=1
    volumes:
      - ${CACHE_PATH}:/data
    command: >
      xinference-worker -e "${ENDPOINT}" -H "${HOST}" --metrics-exporter-host "${HOST}"  --metrics-exporter-port 31002 --log-level ${LOG_LEVEL}
    stdin_open: true
    tty: true

docker compose -f docker-compose.worker-cpu.yaml up -d

Reproduction / 复现过程

docker compose -f docker-compose.worker-cpu.yaml up -d

Expected behavior / 期待表现

Failed to load /opt/conda/lib/python3.11/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /opt/conda/lib/python3.11/site-packages/torchao/_C_cutlass_90a.abi3.so
Failed to load /opt/conda/lib/python3.11/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /opt/conda/lib/python3.11/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so
W0801 11:05:03.345000 1 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0801 11:05:03.384000 1 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
Failed to load /opt/conda/lib/python3.11/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /opt/conda/lib/python3.11/site-packages/torchao/_C_cutlass_90a.abi3.so
Failed to load /opt/conda/lib/python3.11/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /opt/conda/lib/python3.11/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so
W0801 11:05:22.726000 1 site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
W0801 11:05:22.765000 1 site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.

[OliverBryant]

I looked into this. The v3.1.0-cpu image itself starts fine as a worker in a clean environment, so the torchao lines in your log are harmless import noise (same as #5237), not the failure itself.

What matches your log signature (repeated import noise, no worker logs at all) is the supervisor endpoint being unreachable at worker startup: the worker's very first action is a REST call to ${ENDPOINT} made before any log line, with no retry, so with restart: always it turns into a crash loop that prints almost nothing. Please check:

• that the supervisor at ${ENDPOINT} is up and reachable from the worker host (curl ${ENDPOINT}/v1/address should return the internal address), and
• that ENDPOINT is actually set in your .env — if it is empty the worker falls back to http://127.0.0.1:9997, which under network_mode: host only works if the supervisor runs on the same machine.

Also run docker logs xinference-worker1 and scroll past the torchao lines — with the current image there should be a ConnectionError traceback between the repeated blocks.

#5278 improves this: the worker now logs the supervisor endpoint immediately at startup and retries the connection for up to 5 minutes with explicit warnings, joining automatically once the supervisor becomes reachable. If your supervisor endpoint checks out fine and the full log shows something other than a connection error, please share it and I'll dig further.

[leslie2046]

@OliverBryant xprobe/xinference:v3.1.0.rc1-cpu
But when i use last version,it normally startup.And my supervisor is up and reachable.