suggestion: add modern algo as obfuscation method

[trim21]

Add AES-CTR as MSE stream cipher alongside RC4

Motivation

RC4 has no hardware acceleration. On modern CPUs, AES-NI is dramatically faster. Measured on AMD Ryzen 7 255 (Zen 5, vaes + avx512):

Implementation	Throughput (single core)
RC4 asm (current)	~900 MB/s
AES-128-CTR (AES-NI)	~13,500 MB/s

~15× throughput improvement. Even on older AES-NI-only CPUs (without VAES), the gap remains 3-5×.

At 100 MB/s download + 10 MB/s upload, the CPU cost of encryption alone:

Cipher	Download (100 MB/s)	Upload (10 MB/s)	Total
RC4 asm	11.1% of one core	1.1%	12.2%
AES-CTR	0.74%	0.07%	0.81%

AES-CTR saves ~11.4% single-core CPU at these speeds, effectively eliminating encryption as a visible cost.

The change is limited to MSE stream encryption only — the peer wire protocol is untouched.

Background

MSE uses RC4 purely as obfuscation, not for confidentiality or authentication. Since RC4 offers no meaningful cryptographic security here, replacing it with AES-CTR — which is both faster (via AES-NI) and harder to crack — is a straightforward win.

Design

Interface

rc4_handler already implements the crypto_plugin interface (extensions.hpp):

struct crypto_plugin {
    virtual void set_incoming_key(span<char const> key) = 0;
    virtual void set_outgoing_key(span<char const> key) = 0;
    virtual std::tuple<int, span<span<char const>>> encrypt(span<span<char>>) = 0;
    virtual std::tuple<int, int, int> decrypt(span<span<char>>) = 0;
};

A new aes_ctr_handler implements the same interface — same call sites in bt_peer_connection.cpp, no control flow changes.

Why no IV in the handshake

RC4 has no IV. MSE sets up a fresh key per connection (via DH), so each key is unique. AES-CTR's uniqueness requirement is "same key + same nonce must not repeat." Since the key itself is per-connection unique, the nonce can be derived from key material without exchanging any additional bytes over the wire.

Key mapping

MSE always derives 20-byte keys via SHA-1 (sha1_hash), regardless of v1 or v2 torrents. For AES-128-CTR this maps cleanly:

SHA-1 output: [16 bytes AES-128 key][4 bytes nonce = 32-bit]
              0..15                  16..19

All 20 bytes are used, no overlap, no waste. Same key derivation as RC4 — init_pe_rc4_handler() stays.

Counter scheme

AES block: [nonce 4B][counter 12B] = 128 bit
counter starts at 0, increments per block

Discard-1024-bytes becomes counter += 64 (O(1) vs. RC4's 1024-byte loop).

Internal buffering

AES-CTR produces keystream in 16-byte blocks. An internal 16-byte buffer caches unused output:

struct aes_ctr_state {
    uint8_t  buf[16];        // cached keystream
    uint8_t  available;      // 0..16 (bytes remaining in buf)
    uint32_t nonce;          // from SHA-1 bytes 16..19
    uint64_t counter_lo;     // lower 64 bits of 96-bit counter
    uint32_t counter_hi;     // upper 32 bits, starts at 0
};

Most BT traffic is 16 KB piece data (exactly 16-byte aligned), so the bulk path hits full AES blocks with no buffer overhead. Small protocol messages (< 16 bytes) go through the cached path — one if (available == 0) check every 16 bytes, branch predictor hits ~100%.

Protocol changes

crypto_field bits

Currently:

Bit	Meaning
1	pe_plaintext
2	pe_rc4

Which bit to use for AES-CTR is open for discussion. For example, using bit 4 (pe_aes_ctr = 4) would keep it visually distinct from existing bits. pe_both remains pe_plaintext | pe_rc4 for backward compatibility. A new prefer_aes_ctr setting controls negotiation priority when both AES-CTR and RC4 are advertised.

settings_pack additions

enum enc_level : std::uint8_t {
    pe_plaintext = 1,
    pe_rc4       = 2,
    pe_both      = 3,   // pe_plaintext | pe_rc4
    pe_aes_ctr   = 4,   // new
};

// new setting
bool prefer_aes_ctr;    // prioritize AES-CTR when both sides support it

If both peers support AES-CTR, the stream uses AES-CTR. If one only supports RC4, it falls back to RC4. The negotiation is already handled by the existing crypto_provide/crypto_select logic in read_pe_cryptofield().

No handshake structure change

The MSE handshake order is:

1. DH key exchange (plaintext) → shared secret
2. Derive RC4 keys from shared secret
3. Exchange crypto_provide / crypto_select encrypted with RC4 (pe3/pe4 messages)
4. Switch to the negotiated stream cipher for the bulk data

Step 3 must use RC4 because the negotiation hasn't happened yet — there's no way to agree on a different cipher for the handshake without first doing the handshake. Since handshake bytes (a few hundred) are negligible compared to the data stream, keeping RC4 for the handshake and switching to AES-CTR for the bulk stream is both correct and sufficient.

Implementation scope

File	Change
include/libtorrent/pe_crypto.hpp	Add aes_handler struct (like rc4_handler)
src/pe_crypto.cpp	Add aes_handler implementation with AES-NI
include/libtorrent/settings_pack.hpp	Add pe_aes_ctr = 4, prefer_aes_ctr
src/settings_pack.cpp	Wire up prefer_aes_ctr
src/bt_peer_connection.cpp	Add AES-CTR init path, crypto negotiation
src/session_handle.cpp	Expose prefer_aes_ctr in session params
src/session_stats.cpp	Optional: metric for AES-CTR connections

~300–400 lines total. No changes to the peer wire protocol, file format, or BitTorrent spec.

[trim21]

If this approach looks good to you, I plan to bring AES-CTR support to the other major torrent clients as well — Transmission and rTorrent included

Regardless of whether obfuscation actually helps against ISP throttling, switching from RC4 to AES is worth it for the CPU savings alone.