Operation getting stuck at "running" state

[aadarsh79]

I have created a HealthCare APT with around ~1k abilities but this is getting stuck in "running" state for days after completing 40-50 abilities everytime. The Agent is live and trusted, allocated 8GB container memory, no error logs regarding operation on caldera server logs.
Please help me understand this issue and what could be the workaround ?

[github-actions]

Looks like your first issue -- we aim to respond to issues as quickly as possible. In the meantime, check out our documentation here: http://caldera.readthedocs.io/

[uruwhy]

Is your operation set to auto-close or to run forever? My initial guess is that not all of the abilities were able to execute due to unfulfilled facts / requirements / etc, so the operation performed all it could and is waiting for any additional ad-hoc or manual instructions (unless set to auto-close)

[aadarsh79]

    {
        "name": "APT28",
        "autonomous": 1,
        "use_learning_parsers": true,
        "auto_close": false,
        "jitter": "5/10",
        "state": "running",
        "visibility": 51,
        "obfuscator": "plain-text",
        "source": { "id": "ed32b9c3-9593-4c33-b0db-e2007315096b" },
        "planner": { "id": "aaa7c857-37a0-4c4a-85f7-4e9f7f30e31a" },
        "adversary": { "adversary_id": "5e83f3af-4c6e-4053-8e73-d9f9d58dda09" }
    }

This is payload I am using to create operation.

[uruwhy]

Right, so your operation is set to stay running even after exhausting all possible abilities (auto_close is set to false). As for why it only executes 40-50 abilities before "stopping", that's because your agents can't execute the remaining abilities due to missing facts or other requirements or incompatibilities (e.g. adversary has linux abilities, but there are no linux agents. Or the ability requires an elevated agent, but none are elevated). Most common reason is a missing fact - a subsequent ability requires information that hasn't been discovered or satisfied by a previously executed ability

[aadarsh79]

Thanks for your quick response.

Okay so I need to do auto_close: true so that it finishes after executing all abilities.
What if it is getting stuck at some ability due to whatever reason - is there any timeout I can set for ability so that It can move to next and finally reaches finished state ?

FYI, I am not using Caldera UI - I know default timeout is set to 60. I am confused how it is not moving to next ?

[uruwhy]

ability timeouts are 60 seconds by default, so agents will stop executing a command after 60 seconds and then move on to the next one, if available. If no more commands are available for agents to run then the operation will either wait in an open state for manual / ad-hoc instructions (if not set to auto-close), or it will automatically enter the cleanup phase and close (if auto-close is set to true)

[uruwhy]

I just did a couple small tests to see if the operation would hang if the agent dies in the middle of running a command, and the operation seems to handle that case just fine. If all agents are deemed untrusted in the midst of an operation, the operation will stop and enter the cleanup/finished state if was set to auto-close