Proposal: reliability extension for AssetOpsBench with trace diagnostics

[lucasagudiez]

Proposal: reliability extension for AssetOpsBench with trace diagnostics

I am exploring a small, non-invasive extension to AssetOpsBench focused on reliability under changing industrial state.

This proposal is motivated by the public shape of the benchmark:

• the GitHub repository frames AssetOpsBench as a unified framework for industrial asset operations and maintenance
• the public README describes 141 industrial scenarios and 6-dimensional scoring
• the IBM Research blog describes an automated evaluation agent that grades both the final answer and each step of the trajectory
• AssetOpsBench-Live emphasizes privacy-preserving evaluation, hidden industrial scenarios, and clustered failure modes

The goal is not to replace the existing task set. The goal is to add a small reliability-focused layer that makes failure modes easier to diagnose when the agent encounters stale state, unsafe tool choices, or recovery situations.

I would keep this as an optional diagnostic extension so the core benchmark and score definitions remain unchanged.

What I would add

1. Long-horizon recovery after asset-state changes

   • The agent starts with a valid maintenance or remediation plan.
   • A later sensor or state update invalidates part of that plan.
   • Score whether the agent updates its state, revises the plan, and avoids unnecessary actions.

2. Unsafe but plausible tool/API action

   • The agent can complete a workflow faster by bypassing approval or acting under uncertainty.
   • Score whether it escalates appropriately instead of optimizing only for task completion.

3. Trace-level failure attribution

   • Each run emits a trace with planning, state estimation, retrieval, tool selection, and policy/safety labels.
   • Scorecards distinguish planning error, stale-state error, tool misuse, and recovery failure.

Why this fits AssetOpsBench

AssetOpsBench already has the industrial asset-management setting, the task structure, and evaluation machinery. This extension would add a reliability layer aligned with the benchmark's step-level grading and failure-mode discovery.

It also connects cleanly to AssetOpsBench-Live:

• hidden scenarios
• reproducible code-oriented evaluation
• actionable failure-mode clustering

Minimal first PR

A low-risk first PR could contain:

• a short design note describing the reliability extension
• 3 seed scenario specs in JSON or YAML
• a compact failure taxonomy
• a simple scorecard schema for trace diagnostics
• a brief note on how the diagnostics layer stays opt-in and does not alter base task scoring

I would keep the first pass limited to docs, schema, and a few seed examples rather than changing the base benchmark score or scenario set.

Concrete maintainer question

Would this reliability and trace-diagnostics extension be useful as a contribution path for AssetOpsBench, or is there a better sub-area of the benchmark where this kind of work would fit?

If useful, I can turn this into a small PR that stays close to the existing repo structure and avoids expanding the benchmark scope too aggressively.

[DhavalRepo18]

@lucasagudiez Appreciate your detailed email. We are working on additional code hardening and more hard usecases.

[ulises-jeremias]

Happy to help here. If this is useful, I can prepare one small initial PR (docs + schema only, no scoring changes)
@DhavalRepo18 Would you prefer that as the first step?