fix(guest-agent): answer 501, not 400, when the image cannot attest a GPU

`AttestGpu` reported "GPU attestation is not available in this image" as an
uncoded error, which `dispatch_prpc` turns into the generic 400. That tells a
client its request was malformed. It was not: the request is well-formed, and
no other request would succeed either, because the image ships no nvattest and
will not grow one at runtime. A client branching on the status retries with
different arguments forever instead of falling back.

`ra_rpc::ErrorExt::with_code` already carries a chosen status through the
transport, and `code_of` walks the whole error chain, so the code survives the
handler's `.context("GPU attestation failed")`. 501 rather than 503 because the
capability is absent for the lifetime of the CVM, not temporarily unavailable.

A malformed nonce keeps the default 400 -- that one really is the caller's
fault, and the two failures must not be indistinguishable.

Safe to change now: `AttestGpu` is v1-only and never shipped in a release, so
no deployed client is reading the old 400. `rpc_service_v1.rs` already asserts
it is absent from both frozen surfaces.

The availability probe moves from `nvattest::available()` to a `GpuAttestor`
field holding the binary's path. Without that, a test for the unavailable
answer would pass only on a host with no nvattest installed and would spawn a
real collection against the host's GPUs anywhere else. The test fixture pins it
to a path that cannot exist, so every guest-agent test sees the same answer.

Documented in the proto comment, the spec's Errors and Status codes sections,
and the curl API reference, which listed only 400 and 500.

Author: kvinwang

docs/guest-api-v1.md

@@ -103,10 +103,12 @@ returns JSON.
 | 404 | the server's own 404 page | No such mount: this agent has no surface at that path |
 | 404 | `{"error": "Service not found: <Method>"}` | The surface is mounted; it has no such method |
 | 400 | `{"error": "<message>"}` | The method ran and failed |
+| 501 | `{"error": "<message>"}` | `AttestGpu` on an image that ships no GPU attestation |
 | other | `{"error": "<message>"}` | A handler chose the status; the message says why |
 
-A handler failure is a 400 with the error text in the body. v1 does not default,
-coerce, or truncate a malformed request into a well-formed one.
+A handler failure is a 400 with the error text in the body unless the handler
+chose otherwise, and `AttestGpu` is the only v1 method that does. v1 does not
+default, coerce, or truncate a malformed request into a well-formed one.
 
 ### Detecting an agent without v1
 
@@ -640,19 +642,20 @@ which is the authoritative description.
 | Derived secp256k1 scalar out of range | Error; the caller picks another domain |
 | `report_data` longer than 64 bytes | Error |
 | `nonce` not exactly 32 bytes | Error naming the required length |
-| GPU attestation unavailable in this image | Error; `AttestGpu` has nothing to collect |
+| GPU attestation unavailable in this image | HTTP 501; `AttestGpu` has nothing to collect |
 | `not_before` not earlier than `not_after` | Error |
 | Unknown method on a mounted surface | HTTP 404, `Service not found: <Method>` |
 | `/v1/...` on an agent that predates v1 | HTTP 404, no such mount |
 
-Everything above the last two rows is an HTTP 400 with the message in the body.
-See [Status codes](#status-codes) for the full mapping. v1 does not default,
-coerce, or truncate a malformed request into a well-formed one.
+Every row that does not name a status is an HTTP 400 with the message in the
+body. See [Status codes](#status-codes) for the full mapping. v1 does not
+default, coerce, or truncate a malformed request into a well-formed one.
 
-A 400 here means "the method ran and failed", not "your request was malformed".
-The GPU-unavailable row is a property of the image, not of the call: a client
-that gets it should stop rather than retry with different arguments, and the
-message in the body is the only thing that separates the two cases.
+The GPU row is a 501 rather than a 400 because it is a property of the image,
+not of the call: the request is well-formed, and no other request would succeed
+either. A client that reads 400 there retries with different arguments forever;
+one that reads 501 stops and falls back to whatever it does without a GPU. A
+malformed nonce is still the caller's fault and still a 400.
 
 ## Migration from the unversioned API
 

dstack/guest-agent/rpc/proto/agent_rpc_v1.proto

@@ -108,6 +108,11 @@ service DstackGuest {
   // Returns vendor-native evidence, not a local verdict, so a relying party
   // can appraise it with its own verifier. Evidence still does not bind the
   // GPU to this TD; see `AttestGpuResponse.bundles`.
+  //
+  // An image that ships no GPU attestation answers HTTP 501, not the 400 every
+  // other v1 failure uses. The request is well-formed there and no other
+  // request would succeed either, so a caller can tell "fall back" from "fix
+  // your arguments" on the status alone. A malformed nonce stays a 400.
   rpc AttestGpu(AttestGpuRequest) returns (AttestGpuResponse) {}
 
   // Return this application's identity and measurements.

dstack/guest-agent/src/gpu_attest.rs

@@ -8,9 +8,11 @@
 //! process and talks to the GPU through the driver. The gate below serialises
 //! collection so concurrent callers do not compete for the same devices.
 
+use std::path::{Path, PathBuf};
 use std::time::Duration;
 
-use anyhow::{bail, Result};
+use anyhow::{anyhow, bail, Result};
+use ra_rpc::ErrorExt;
 use tokio::sync::Mutex;
 
 /// Serialises GPU evidence collection.
@@ -20,13 +22,28 @@ use tokio::sync::Mutex;
 /// challenge is exactly the confusion this API exists to avoid.
 pub struct GpuAttestor {
     timeout: Duration,
+    /// The binary whose presence says this image can attest a GPU.
+    ///
+    /// A field rather than a call to `nvattest::available()` so a test can pin
+    /// the unavailable answer on a host that happens to have nvattest
+    /// installed -- otherwise that test would spawn a real collection against
+    /// whatever GPUs the host has.
+    nvattest: PathBuf,
     run_lock: Mutex<()>,
 }
 
 impl GpuAttestor {
     pub fn new() -> Self {
+        Self::with_nvattest_path(nvattest::NVATTEST)
+    }
+
+    /// `pub(crate)` so the handler tests can build a state that is
+    /// deterministically without GPU support, whatever the host running
+    /// them has installed.
+    pub(crate) fn with_nvattest_path(path: impl AsRef<Path>) -> Self {
         Self {
             timeout: nvattest::DEFAULT_TIMEOUT,
+            nvattest: path.as_ref().to_path_buf(),
             run_lock: Mutex::new(()),
         }
     }
@@ -40,8 +57,16 @@ impl GpuAttestor {
                 nonce.len()
             );
         }
-        if !nvattest::available() {
-            bail!("GPU attestation is not available in this image");
+        if !self.nvattest.exists() {
+            // 501, not the 400 an uncoded error would report. The request is
+            // well-formed and no other request would succeed: this image ships
+            // no nvattest, so the capability is absent for the lifetime of the
+            // CVM. A caller that reads 400 retries with different arguments
+            // forever; one that reads 501 stops and falls back.
+            //
+            // Safe to say now because `AttestGpu` is v1-only and has never
+            // shipped in a release -- no client is reading 400 here today.
+            return Err(anyhow!("GPU attestation is not available in this image").with_code(501));
         }
         // Held across the whole run, so a second caller waits rather than
         // starting a competing nvattest against the same devices.
@@ -61,4 +86,23 @@ mod tests {
         let err = attestor.attest(&[0u8; 16]).await.unwrap_err().to_string();
         assert!(err.contains("exactly 32 bytes"), "{err}");
     }
+
+    /// A malformed nonce is the caller's fault; an image without nvattest is
+    /// not. The two must not answer with the same status.
+    ///
+    /// The nonce here is well-formed, so this reaches the availability check
+    /// rather than stopping at the length check above.
+    #[tokio::test]
+    async fn an_image_without_nvattest_answers_not_implemented() {
+        let attestor = GpuAttestor::with_nvattest_path("/nonexistent/nvattest");
+        let err = attestor.attest(&[0u8; 32]).await.unwrap_err();
+        assert!(
+            err.to_string().contains("not available in this image"),
+            "{err}"
+        );
+        assert_eq!(ra_rpc::code_of(&err), Some(501));
+
+        let malformed = attestor.attest(&[0u8; 16]).await.unwrap_err();
+        assert_eq!(ra_rpc::code_of(&malformed), None, "a bad nonce stays a 400");
+    }
 }

dstack/guest-agent/src/rpc_service.rs

@@ -1325,7 +1325,12 @@ pNs85uhOZE8z2jr8Pg==
                 probe,
             }),
             health: None,
-            gpu_attestor: crate::gpu_attest::GpuAttestor::new(),
+            // Pinned to a path that cannot exist, so no test ever spawns a
+            // real collection against a host GPU and every test sees the
+            // same "this image cannot attest a GPU" answer.
+            gpu_attestor: crate::gpu_attest::GpuAttestor::with_nvattest_path(
+                "/nonexistent/nvattest",
+            ),
             app_root_signing_key: SigningKey::from_slice(&DUMMY_K256_KEY).ok(),
             identity: RwLock::new(None),
             identity_last_failure: Mutex::new(None),

dstack/guest-agent/src/rpc_service_v1.rs

@@ -826,10 +826,35 @@ mod tests {
             })
             .await
             .unwrap_err();
+        assert_eq!(
+            ra_rpc::code_of(&err),
+            None,
+            "a malformed nonce is the caller's fault, so it keeps the default 400"
+        );
         let err = format!("{err:#}");
         assert!(err.contains("exactly 32 bytes"), "{err}");
     }
 
+    /// An image that ships no nvattest must not answer like a malformed
+    /// request: the call is well-formed and no other call would succeed, so
+    /// the caller has to be able to tell "stop" from "try something else".
+    ///
+    /// Asserted through the handler because that is where `.context(..)` wraps
+    /// the attestor's error. `code_of` walks the whole chain for exactly this
+    /// reason -- a context layer that buried the code would silently put the
+    /// answer back to 400, and the message would still read correctly.
+    #[tokio::test]
+    async fn attest_gpu_reports_an_image_without_gpu_support_as_not_implemented() {
+        let (state, _guard) = state().await;
+        let err = V1RpcHandler::new(state)
+            .attest_gpu(AttestGpuRequest {
+                nonce: vec![0u8; 32],
+            })
+            .await
+            .unwrap_err();
+        assert_eq!(ra_rpc::code_of(&err), Some(501), "{err:#}");
+    }
+
     /// The on-demand format tag, pinned like its boot-time counterpart.
     ///
     /// A consumer selects its verifier on `(vendor, format)`, so these two

sdk/curl/api.md

@@ -331,6 +331,11 @@ the evidence signature, certificate chain, measurements, and embedded nonce. The
 agent does not appraise the evidence. Evidence does not by itself bind the GPU to this
 CVM.
 
+> An image that ships no GPU attestation answers `501 Not Implemented`, not
+> `400`. Retrying with a different nonce will not help; fall back to whatever
+> your application does without a GPU. A nonce that is not exactly 32 bytes is
+> still a `400`.
+
 ### 8. Boot-time GPU evidence *(v1)*
 
 Returns the complete output NVIDIA `nvattest` produced during boot, as part of
@@ -415,6 +420,8 @@ All endpoints may return the following HTTP status codes:
 
 - `200 OK`: Request successful
 - `400 Bad Request`: Invalid request parameters
+- `501 Not Implemented`: The agent cannot serve this method in this image; only
+  `/v1/AttestGpu` answers this, and only when the image ships no GPU attestation
 - `500 Internal Server Error`: Server-side error
 
 Error responses will include a JSON body with error details: