Skip to content

feat(dtls): add CustomSigner trait for external signing providers#94

Draft
twizansk wants to merge 1 commit into
webrtc-rs:masterfrom
twizansk:custom-signer
Draft

feat(dtls): add CustomSigner trait for external signing providers#94
twizansk wants to merge 1 commit into
webrtc-rs:masterfrom
twizansk:custom-signer

Conversation

@twizansk
Copy link
Copy Markdown

Summary

Add a CustomSigner trait to CryptoPrivateKeyKind that allows delegating DTLS signing operations to an external provider instead of requiring raw private key bytes in memory.

Motivation

Hardware security modules (HSMs), TPMs, and cloud KMS services expose signing APIs but do not allow extracting the raw private key. The current CryptoPrivateKeyKind enum requires a ring key pair constructed from raw key bytes, making it impossible to use these external signing providers for DTLS.

This is a common requirement for IoT devices with TPM-backed device certificates, cloud services using AWS KMS or GCP Cloud HSM, and any environment where private keys must remain in hardware.

Changes

  • New CustomSigner trait in rtc-dtls/src/crypto/mod.rs with sign() and clone_box() methods
  • New Custom variant in CryptoPrivateKeyKind enum
  • Match arms added in generate_key_signature and generate_certificate_verify to delegate signing to the custom signer
  • Config validation updated to accept Custom key type
  • Signature scheme compatibility returns true for Custom (the signer is responsible for producing the correct format)

Usage

```rust
use rtc_dtls::crypto::{CustomSigner, Certificate, CryptoPrivateKey, CryptoPrivateKeyKind};

#[derive(Debug)]
struct MySigner { /* HSM handle, API client, etc. */ }

impl CustomSigner for MySigner {
fn sign(&self, message: &[u8]) -> Result<Vec, String> {
// Call your HSM/TPM/KMS signing API
// Return signature in the format expected by the negotiated algorithm
// (e.g., ASN.1 DER for ECDSA)
todo!()
}

fn clone_box(&self) -> Box<dyn CustomSigner> {
    Box::new(MySigner { /* ... */ })
}

}

let cert = Certificate {
certificate: vec![/* DER-encoded cert chain */],
private_key: CryptoPrivateKey {
kind: CryptoPrivateKeyKind::Custom(Box::new(MySigner {})),
serialized_der: Vec::new(), // No key material needed
},
};
```

Notes

  • The Custom signer must return signatures in the wire format expected by the peer (e.g., ASN.1 DER for ECDSA, PKCS#1 for RSA). The library does not perform format conversion.
  • clone_box() is required because the DTLS listener clones the certificate config for each incoming connection.
  • is_compatible() returns true for all signature schemes when using a Custom signer — the caller is responsible for ensuring their signer supports the negotiated algorithm.

Add a CustomSigner trait to CryptoPrivateKeyKind that allows delegating
DTLS signing operations to an external provider (HSM, TPM, cloud KMS)
instead of requiring raw private key bytes in memory.

This enables use cases where the private key cannot be extracted from
hardware — for example, TPM-backed device certificates where only a
signing API is available. The caller implements CustomSigner with their
signing logic and passes it as a CryptoPrivateKeyKind::Custom variant.

Changes:
- Add CustomSigner trait with sign() and clone_box() methods
- Add Custom variant to CryptoPrivateKeyKind enum
- Add Custom match arms in generate_key_signature and
  generate_certificate_verify
- Allow Custom keys in config validation and signature scheme
  compatibility checks
@rainliu
Copy link
Copy Markdown
Member

rainliu commented May 15, 2026

please fix fmt and clippy issue

@twizansk twizansk marked this pull request as draft May 15, 2026 21:40
@codecov
Copy link
Copy Markdown

codecov Bot commented May 15, 2026

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 71.23%. Comparing base (aae46b7) to head (564e8d4).

Files with missing lines Patch % Lines
rtc-dtls/src/config.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master      #94      +/-   ##
==========================================
- Coverage   71.30%   71.23%   -0.07%     
==========================================
  Files         442      442              
  Lines       67361    67362       +1     
==========================================
- Hits        48030    47985      -45     
- Misses      19331    19377      +46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants