How to depends multi services that on other nodes?

[llc-kc]

In a condition we have one router service deploy on one node, and multiple workers deployed on other nodes, so how to connect the router with the multiple workers? Thank you very much.

[aarnphm]

@bentoml.service
class Service
  child = bentoml.Depends(ChildCls, url="address_to_other_node")

this will ensure type + correctness based on your usecase here.

[llc-kc]

Thanks, but how to handle many other child instances? For example, current survice is a router that responds to route requests to many other services that on different nodes?

[frostming]

How do you deploy your services? If you use bentoml start-http-server --service-name <NAME> to start single services, there is --depends option to provide the depenency urls maps.

The format is --depends FooService=http://1.2.3.4:5678

[llc-kc]

OK, I will learn it.

[xXMrNidaXx]

For multi-node router + workers architecture, you have a few options:

Option 1: HTTP-based routing

import httpx

@bentoml.service
class Router:
    def __init__(self):
        self.workers = [
            "http://worker-1:3000",
            "http://worker-2:3000",
            "http://worker-3:3000",
        ]
        self.current = 0
    
    @bentoml.api
    async def predict(self, data):
        # Round-robin load balancing
        worker = self.workers[self.current % len(self.workers)]
        self.current += 1
        
        async with httpx.AsyncClient() as client:
            response = await client.post(f"{worker}/predict", json=data)
            return response.json()

Option 2: Service discovery via environment

import os

@bentoml.service
class Router:
    def __init__(self):
        # Workers registered via env vars or config
        self.workers = os.getenv("WORKER_URLS", "").split(",")

Option 3: Use a load balancer (recommended for production)

# kubernetes/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bento-workers
spec:
  rules:
    - host: workers.internal
      http:
        paths:
          - path: /
            backend:
              service:
                name: bento-worker-pool
                port: 3000

Then router just calls http://workers.internal.

Option 4: BentoCloud distributed deployment

BentoCloud handles multi-node orchestration automatically with bentoml.depends() across deployments.

We run distributed ML inference at Revolution AI — for production, putting workers behind a load balancer (K8s Service or nginx) is cleanest.