Ticket #2004 Fix RC4/RC4Drop hex passphrase parsing and add format validation

[alleria173]

Description
Fixes #2004 reported that non-space delimiters in hex passphrase input for RC4 encoding were changing the output (should be ignored, like space) as well as non-hex characters.

Files Changed

src/core/lib/Ciphers.mjs

Added two imports at the top:

import { fromHex, toHexFast } from "./Hex.mjs";

(Utils was already imported.)

Added parseFormatString() export at the bottom of the file:

export function parseFormatString(str, formatName) {
    Utils.validateFormatInput(str, formatName);
    if (formatName === "Hex") {
        return CryptoJS.enc.Hex.parse(toHexFast(fromHex(str)));
    }
    return format[formatName].parse(str);
}

For Hex format this normalises common delimiter conventions (1f,10, 0x1f,0x10, 1f 10, 1F:10 → 1f10) before CryptoJS parses it. For all other formats it delegates to the existing format map as before.

---

src/core/operations/RC4.mjs

Updated import line:

import { format, parseFormatString } from "../lib/Ciphers.mjs";

Changed passphrase parsing in run():

// Before
passphrase = format[args[0].option].parse(args[0].string),
// After
passphrase = parseFormatString(args[0].string, args[0].option),

---

src/core/operations/RC4Drop.mjs

Identical change to RC4.mjs — same import update and same passphrase line change.

---

src/core/operations/DeriveEVPKey.mjs

Removed the global CryptoJS.enc.Hex.parse monkey-patch (lines 126–147 in the original). This patch had been added to strip whitespace from hex strings before CryptoJS parsing — a partial workaround for exactly this class of bug. It was safe to remove because:

• DeriveEVPKey.run() uses Utils.convertToByteString() → fromHex() for its own passphrase/salt, not CryptoJS.enc.Hex.parse() directly
• RC4/RC4Drop (the only operations that did use CryptoJS.enc.Hex.parse() directly for user input) now go through parseFormatString() which normalises before parsing

---

src/core/Utils.mjs

Added import:

import OperationError from "./errors/OperationError.mjs";

Added validateFormatInput() static method before convertToByteArray():

static validateFormatInput(str, type) {
    if (!str) return;
    switch (type.toLowerCase()) {
        case "hex": {
            const stripped = str.replace(/0x|\\x|%|[\s,;:\n\r]/gi, "");
            const invalid = stripped.match(/[^0-9a-fA-F]/);
            if (invalid) throw new OperationError(
                `Invalid character '${invalid[0]}' in Hex input. ` +
                `Hex accepts 0-9, a-f, A-F, and delimiters (space, comma, colon, 0x prefix).`
            );
            break;
        }
        case "base64": {
            const invalid = str.replace(/[\s]/g, "").match(/[^A-Za-z0-9+/=]/);
            if (invalid) throw new OperationError(
                `Invalid character '${invalid[0]}' in Base64 input. ` +
                `Base64 accepts A-Z, a-z, 0-9, +, /, and = (padding).`
            );
            break;
        }
        case "binary": {
            const stripped = str.replace(/[\s,;:\n\r]/g, "");
            const invalid = stripped.match(/[^01]/);
            if (invalid) throw new OperationError(
                `Invalid character '${invalid[0]}' in Binary input. Binary accepts only 0 and 1.`
            );
            break;
        }
    }
}

Added Utils.validateFormatInput(str, type) as the first line of both convertToByteArray() and convertToByteString(), covering all ~50 crypto operations that use these central conversion functions.
Existing Issue
Issue #2004

Screenshots

AI disclosure
Claude Code used to read the codebase and plan fix, as well as create tests.

Test Coverage

tests/operations/tests/RC4.mjs (new file)

Nine tests covering:

• 1f,10 (comma-delimited) produces same output as 1f10
• 0x1f,0x10 (0x-prefixed) produces same output as 1f10
• 1f 10 (space-delimited) produces same output as 1f10
• 1F:10 (colon + uppercase) produces same output as 1f10
• 1fG0 (invalid hex char) produces an error message in output
• RC4 Drop: comma-delimited and 0x-prefixed both normalise correctly
• UTF8 passphrase regression guard

Test vectors computed with a reference Python RC4 implementation against key bytes [0x1f, 0x10] and plaintext "test".

---

tests/node/tests/operations.mjs (existing file — two tests fixed)

Two pre-existing Node API tests were passing non-hex strings to Hex-format inputs. Our validation correctly caught these as bugs in the tests. Both were updated to use appropriate formats with recomputed expected values.

HMAC test (line 661):

// Before — key "idea" defaulted to Hex format (first toggleValue); "idea" is not valid hex
chef.HMAC("On Cloud Nine", {key: "idea"})
// expected: "e15c268b4ee755c9e52db094ed50add7"

// After — explicitly Latin1 format with correct expected value
chef.HMAC("On Cloud Nine", {key: {string: "idea", option: "Latin1"}})
// expected: "b128b48ec0d6b0f1a27220c396d0f3e5"

Scrypt test (line 885):

// Before — salt "salty" explicitly set to Hex format; "salty" is not valid hex
{salt: {string: "salty", option: "Hex"}}
// expected: "5446b6d86d88515894a163..."

// After — UTF8 format with correct expected value
{salt: {string: "salty", option: "UTF8"}}
// expected: "2f1417e682a7bf008c60abf8e8b8dc33..."

---

Verification

All tests pass:

• 250 / 250 Node API tests
• 2061 / 2061 operation tests (includes 9 new RC4 tests)

[alleria173]

Hi. I'd appreciate any feedback regarding my PR in case there are any adjustments I can make for it to be accepted. Thanks!