What separates connected accelerators from a trustworthy distributed system?

[RyanX0198]

Hi maintainers — I’m Rui from Psync.labs. Collective-communication work raises a question about evaluating local multi-device systems.

Peak all-reduce bandwidth does not show whether topology discovery, timeout behavior, fault isolation and recovery remain understandable during long-running workloads. What failure or observability test best separates a connected demo from a system you would trust?

I’m looking for the decisive validation condition, including a reason this framing might be incomplete. @sjeaugey

— Rui, Psync.labs

[xiaofanl-nvidia]

@RyanX0198 I am not sure if this is a translation issue but we don't really understand your question. If it's helpful, feel free to post your question in Chinese and we can try to help.

[RyanX0198]

Hi Xiaofan，

感谢回复。不是翻译问题，确实是我之前的问题太抽象了。我把场景具体化一下：

我们正在评估由 4–8 台 DGX Spark / GB10 组成的本地 AI 系统，用于长期运行的大模型推理、长上下文任务和多 Agent 工作流。我们想了解的不是峰值带宽，而是这种多节点系统什么时候才可以被认为“能够长期可靠运行”。

具体有两个问题：

1. 如果某个节点、rank 或网络链路变慢或失效，上层调度系统最应该监测哪些 NCCL 信号，才能区分暂时的通信抖动和必须重启或重新调度的故障？

2. 在投入长期任务之前，你认为最基本的可靠性验证应该包括哪些测试？例如链路故障注入、拓扑配置错误、某个 rank 卡住，或者长时间 collective 压力测试。

我们希望最终让上层系统能够发现故障、保留任务状态并恢复运行，而不是只知道设备已经连接成功。

如果方便，只回答你认为最关键的一点也非常有帮助。谢谢！

Best,
Rui

[sylvesterkaczmarek]

If I had to choose one decisive validation test, I'd make it deliberate fault injection during sustained collective traffic.

Run collectives continuously, then kill/SIGSTOP one rank or break one communication path. The acceptance condition should be that the failure becomes observable within a bounded time, identifies the affected process/rank, and the application either follows its supported recovery path or terminates cleanly. An indefinite collective hang would be the failure condition.

For NCCL specifically, I'd capture ncclras monitoring/JSON output during the test and verify that the observed rank/host/PID matches the fault you injected.

I think that gives you something a peak-bandwidth benchmark cannot: bounded failure detection plus attributable diagnostics. The only caveat is that NCCL is one layer of the system. Scheduler-level restart, checkpointing and application recovery still have to be tested separately.

[RyanX0198]

Hi Sylvester, Thank you again for your concrete response in the NCCL discussion: #2330 (comment) Your suggestion to validate deliberate fault injection during sustained collective traffic gave us a very useful reliability boundary: bounded failure detection, attributable diagnostics, and a clearly defined recovery or clean-termination path. We are currently at an early stage of developing an Agent Computer for long-running local AI workloads. The product direction focuses on local execution of large models, long-context tasks, and multi-step agent workflows, with particular attention to memory capacity, unattended operation, observability, and recovery. At this stage, we are not presenting a finished product or making performance claims. We are trying to find a small number of technically experienced early collaborators who can help us challenge our assumptions, identify important failure modes, and provide feedback as the architecture develops. Our hardware details are still under development, so I am not asking you to evaluate a product or make a performance judgment. Instead, we would like to understand how experienced AI and autonomy practitioners use local or edge compute systems in practice. If you are comfortable sharing, could you briefly help us with the following? 1. What is your current role, and which projects or systems best represent your work in safe AI, autonomy, robotics, or edge deployment? 2. In your own work, do you currently operate or evaluate local, edge, or multi-node AI systems? If so, what hardware and software stack do you use? 3. What is the most important workload you run locally or at the edge? 4. What does a typical run look like in terms of input, output, duration, concurrency, and whether it needs to run unattended? 5. Apart from the NCCL failure-injection case, what reliability or operational problems have you encountered in practice? 6. What capability or measurable result would make a local AI system genuinely useful for your work? Approximate values are completely fine, and please feel free to skip anything you would rather not discuss. A short reply to only one or two questions would already be very helpful. If this direction is relevant to your work, we would also be glad to stay in touch asynchronously. As the project becomes more concrete, I can share updates and technical artifacts with you for feedback. There is no commitment expected at this stage. Anything you share would be treated as your personal technical experience and would not be presented as the position of your employer. Best regards, Rui Psync.labs ---- Replied Message ---- FromSylvester ***@***.***>Date8/22/2026 ***@***.***>***@***.***>, ***@***.***>SubjectRe: [NVIDIA/nccl] What separates connected accelerators from a trustworthy distributed system? (Discussion #2330) If I had to choose one decisive validation test, I'd make it deliberate fault injection during sustained collective traffic. Run collectives continuously, then kill/SIGSTOP one rank or break one communication path. The acceptance condition should be that the failure becomes observable within a bounded time, identifies the affected process/rank, and the application either follows its supported recovery path or terminates cleanly. An indefinite collective hang would be the failure condition. For NCCL specifically, I'd capture ncclras monitoring/JSON output during the test and verify that the observed rank/host/PID matches the fault you injected. I think that gives you something a peak-bandwidth benchmark cannot: bounded failure detection plus attributable diagnostics. The only caveat is that NCCL is one layer of the system. Scheduler-level restart, checkpointing and application recovery still have to be tested separately. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[RyanX0198]

Hi Sylvester and Xiaofan, Thank you for the very practical response. The recommendation to perform deliberate fault injection during sustained collective traffic gives us a much clearer starting point than a peak-bandwidth benchmark. The bounded detection time, affected-rank attribution, and clean recovery or termination criteria are exactly the kind of acceptance conditions we want to study. We are currently in the early stage of developing an Agent Computer for long-running local AI and multi-agent workloads. We are looking for early technical contributors who may be interested in sharing experience and helping us evaluate the reliability assumptions behind this type of system. If you are open to sharing, I would be interested in learning: 1. What distributed or local AI workloads do you currently run? 2. What hardware and software stack do you use? 3. Apart from NCCL fault detection, what problem most limits your ability to run long-lived workloads reliably? There is no need to arrange a call. We would be happy to stay in touch through email and return with a concrete test specification or prototype when we have one ready for review. Best, Rui