Skip to content

lxd/operations: Return failed operation state from wait endpoint - #18830

Open
IzzaldinSamir wants to merge 3 commits into
canonical:mainfrom
IzzaldinSamir:fix-network-api-v2
Open

lxd/operations: Return failed operation state from wait endpoint#18830
IzzaldinSamir wants to merge 3 commits into
canonical:mainfrom
IzzaldinSamir:fix-network-api-v2

Conversation

@IzzaldinSamir

@IzzaldinSamir IzzaldinSamir commented Jul 31, 2026

Copy link
Copy Markdown

Root Cause

The wait endpoint commits to an HTTP 200 OK before blocking on the operation. When op.Wait returned an operation error, the handler attempted to render a second HTTP error response instead of returning the operation state. As a result, clients (like lxc network) interpreted failed asynchronous operations as successful because they received a 200 OK without the failure payload.

The Fix

Instead of checking the error from op.Wait(ctx), we now call op.Render() after the wait returns to fetch the actual final state of the operation (which includes failures and timeouts) and return it to the client via a SyncResponse.

Added regression coverage in test/suites/operations.sh to ensure duplicate network creations properly return the failure state.

Fixes #18715

Fixes canonical#18715

Signed-off-by: Izzaldin <izzaldinsamir@gmail.com>
@tomponline

tomponline commented Jul 31, 2026

Copy link
Copy Markdown
Member

Hi @IzzaldinSamir I suspect that changing the semantics of operationWaitGet as you have done here needs to be considered very carefully as could be considered an API break.

I'd want @markylaing to take a look at this as he has been focussing on operations lately.

Also in the bug report the OP mentions the operation is failing with:

2026-07-09T18:15:24.424346-05:00 terra lxd.daemon[2962]: time="2026-07-09T18:15:24-05:00" level=warning msg="Failure for operation" class=task description="Creating network" err="The network already exists" operation=019f492a-1c59-7c61-999f-668635bc0b69 project=prod

Do you know why this is not resulting in an error being returned from operationWaitGet because according to the docblock of the function:

// Returns non-nil error if operation failed or context was cancelled.

So from what I can see piecing the info together:

  1. The operation is failing with an error.
  2. op.Wait() should return a non-nil error for a failed operation.
  3. This should be returned to the caller in operationWaitGet?

Is this occurring and where is it being converted into a "success" response in the CLI?

Thanks

@IzzaldinSamir

Copy link
Copy Markdown
Author

@tomponline Thanks for the review! The core issue is that w.WriteHeader(http.StatusOK) and f.Flush() are called before op.Wait(ctx) blocks.

If the operation fails later, SmartError can't change the HTTP status code because the 200 OK was already flushed. The CLI sees the 200 OK without a proper Failure payload and assumes it succeeded.

By using op.Render(), we fetch the final Operation object (which contains the Failure status and error message) so the CLI can correctly parse it and exit with code 1. Happy to hear @markylaing's thoughts on this!

@tomponline

Copy link
Copy Markdown
Member

w.WriteHeader(http.StatusOK)

Got you, makes sense.

As long as we are happy to define operationWaitGet endpoint's 200 HTTP return code to mean "the operation finished, but you need to inspect the metadata to see if it was successful" then this change is OK.

@tomponline

tomponline commented Jul 31, 2026

Copy link
Copy Markdown
Member

And what ever we do here also needs to be mirrored in the lxd-agent operationWaitGet function too.

Although I notice they are not the same right now, so this is a good opportunity to align them.

@IzzaldinSamir

Copy link
Copy Markdown
Author

@tomponline That makes perfect sense. I will go ahead and update the lxd-agent implementation of operationWaitGet right now so they are perfectly aligned. I'll push the commit to this PR shortly!

Signed-off-by: Izzaldin <izzaldinsamir@gmail.com>
@github-actions github-actions Bot added the d/cli label Jul 31, 2026
@tomponline

Copy link
Copy Markdown
Member

@tomponline That makes perfect sense. I will go ahead and update the lxd-agent implementation of operationWaitGet right now so they are perfectly aligned. I'll push the commit to this PR shortly!

I'm still not quite following why response.SmartError() that will generate an api.ResponseRaw JSON response to the client, is causing the error to be lost in the client (even if the HTTP response code is 200)?

@IzzaldinSamir

Copy link
Copy Markdown
Author

@tomponline Good question. The issue is how the client handles the response after a 200 OK has already been sent.

Once the status is 200, the client treats the request as successful and tries to decode the body as a SyncResponse containing an Operation in metadata.

SmartError(err) returns:

{
  "type": "error",
  "error": "The network already exists",
  "error_code": 500
}

Because there is no metadata field, the client ends up with an almost empty Operation. Since op.Status is not explicitly Failure, the CLI assumes the command succeeded.

Using op.Render() returns the expected sync response format:

{
  "type": "sync",
  "metadata": { ... }
}

The client can then read the operation status and error message correctly and report the failure to the user.

@tomponline

tomponline commented Jul 31, 2026

Copy link
Copy Markdown
Member

@tomponline Good question. The issue is how the client handles the response after a 200 OK has already been sent.

Once the status is 200, the client treats the request as successful and tries to decode the body as a SyncResponse containing an Operation in metadata.

SmartError(err) returns:

{
  "type": "error",
  "error": "The network already exists",
  "error_code": 500
}

Because there is no metadata field, the client ends up with an almost empty Operation. Since op.Status is not explicitly Failure, the CLI assumes the command succeeded.

Using op.Render() returns the expected sync response format:

{
  "type": "sync",
  "metadata": { ... }
}

The client can then read the operation status and error message correctly and report the failure to the user.

OK this makes sense, thanks.

So really we want this operationWaitGet function to align with operationGet (as you have done) so it functions like a synchronous "wait for operation to finish, and then get":

Which also uses SyncResponse:

// First check if the query is for a local operation from this node
op, err := operations.Get(id)
if err != nil {
return response.SmartError(err)
}
_, body = op.Render()
return response.SyncResponse(true, body)

@tomponline

Copy link
Copy Markdown
Member

In which case I am in agreement with your approach - although could you add a comment to the op.Wait lines explaining why you are ignoring the error.

@IzzaldinSamir

Copy link
Copy Markdown
Author

@tomponline Absolutely. I will add a comment above both _ = op.Wait(ctx) calls explaining that the error is intentionally ignored so we can render the final failure state to the client. I'll push that up right now!

Signed-off-by: Izzaldin <izzaldinsamir@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lxc network gets 200 from API when action failed

2 participants