Error in ring_core_0_17_14__x25519_ge_scalarmult_base on x86_64-pc-windows-gnu, and on x86_64-pc-windows-msvc when compiling with xwin

[miruji]

Hello, I'm not an expert in networking, but there is an error that I believe I have fairly accurately debugged for Windows.

I am following the chain and have so far stopped here, as this is the last thing I found regarding this error. I have already created an issue in rustls #2943 so that they are aware that this problem affects them.

Here is a simple test example:

use ed25519_dalek::{pkcs8::EncodePrivateKey, SigningKey};
use rustls::pki_types::{PrivatePkcs8KeyDer};
use pkcs8::LineEnding;
use rustls::pki_types::pem::PemObject;
use ring::signature::Ed25519KeyPair;

fn test(signing_key: &SigningKey) -> Result<(), rustls::Error> {
    let client_private_key_pem = signing_key
      .to_pkcs8_pem(LineEnding::default())
      .expect("key is valid");

    let client_private_key = PrivatePkcs8KeyDer::from_pem_slice(client_private_key_pem.as_bytes())
      .expect("cannot open private key file");

    println!("C3");

    let a: &[u8] = client_private_key.secret_pkcs8_der();

    println!("C4");
    match Ed25519KeyPair::from_pkcs8_maybe_unchecked(a) {
        Ok(_key_pair) => {}
        Err(e) => {
            return Err(rustls::Error::General(format!("failed to parse Ed25519 private key: {:?}", e)));
        }
    }

    println!("C5");

    Ok(())
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let bytes = [42u8; 32];
    let signing_key = SigningKey::from_bytes(&bytes);

    test(&signing_key)?;
    Ok(())
}

cargo:

[package]
edition = "2024"

[dependencies]
ed25519-dalek = { version = "3.0.0-pre.6", features = ["pkcs8", "pem"] }
pkcs8 = "0.11.0-rc.11"
rustls = { version = "0.23.37", features = ["ring"] }
ring = "0.17.14"

In this code, C5 will not be triggered because there is an error in the code. Here and in rustls, this is not visible just like that; in some cases, it results in an infinite loop. That's why I am debugging the old way via console outputs + ctz suggested making a dump, and it helped to find the error.

Microsoft (R) Windows Debugger Version 10.0.17763.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [E:\Dumps\rustlsCheck.exe.4628.dmp]
User Mini Dump File with Full Memory: Only application data is available

Windows 10 Version 19045 MP (4 procs) Free x64
Product: WinNt, suite: SingleUserTS Personal
19041.1.amd64fre.vb_release.191206-1406

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1214.c78): Illegal instruction - code c000001d (first/second chance not available)

ntdll!NtWaitForMultipleObjects+0x14:
00007ffd`38f0e0f4 c3              ret

0:000> .reload /f rustlsCheck.exe
*** WARNING: Unable to verify checksum for rustlsCheck.exe
0:000> !sym noisy
noisy mode - symbol prompts on
0:000> .reload

SYMSRV:  BYINDEX: 0x9
         C:\ProgramData\dbg\sym
         ntdll.pdb
SYMSRV:  RESULT: 0x00000000
DBGHELP: ntdll - public symbols  

SYMSRV:  BYINDEX: 0xA
         C:\ProgramData\dbg\sym
         kernelbase.pdb
SYMSRV:  RESULT: 0x00000000
DBGHELP: KERNELBASE - public symbols  

************* Symbol Loading Error Summary **************
Module name            Error
SharedUserData         No error - symbol load deferred

0:000> .ecxr
SYMSRV:  BYINDEX: 0xB
         C:\ProgramData\dbg\sym
         rustlsCheck.pdb
SYMSRV:  RESULT: 0x00000000
*** WARNING: Unable to verify checksum for rustlsCheck.exe
DBGHELP: rustlsCheck - private symbols & lines 

rax=0000000000009eab rbx=0000df7181c325f7 rcx=00044fd2f9298f81
rdx=000000000000d9e8 rsi=ffffffffffffffff rdi=000493c6f58c3b85
rip=00007ff7569cec09 rsp=0000002c1b2feee0 rbp=0004799558800000
 r8=00050d6630980000  r9=0000002c1b2ff088 r10=0000f50b0b3e4cb7
r11=0005329385a44c32 r12=00023e2827f4e67c r13=00011205877aaa68
r14=00007cf9d3a33d4b r15=0002d42d0d800000
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206

rustlsCheck!ring_core_0_17_14__x25519_ge_scalarmult_base+0xc69:
00007ff7`569cec09 f2410f78da0818  insertq xmm3,xmm10,8,18h

This is through WinDbg. The error itself:

rustlsCheck!ring_core_0_17_14__x25519_ge_scalarmult_base+0xc69:
00007ff7`569cec09 f2410f78da0818  insertq xmm3,xmm10,8,18h

I am again not an expert, but it seems that it failed to process some instruction?

In ring, this is located here

This problem occurs with a GNU build via cargo build --release --target x86_64-pc-windows-gnu
Or with an MSVC xwin build via:

rustup target add x86_64-pc-windows-msvc
cargo install --locked cargo-xwin  
cargo xwin build --release --target x86_64-pc-windows-msvc

But on MSVC in debug the problem does not occur.

The system on which I tested:

OS: Windows 10 x86_64
Version: 22H2

CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz

I am also attaching the files:

• .dmp
• .pdb
• .exe

(Yes, don't mind that it's called rustlsCheck.exe, I just didn't rename the project)

ringDumpFiles.zip

updated: I tried changing my build flags to disable instructions and so on – it didn’t help, even though ctz claimed that it would solve the problem. The only thing that helps is switching to aws-lc-rs or using a debug build for ring.

[briansmith]

I saw that the Rustls issue was closed, AFAICT based on the deduction that the C compiler must be configured to be targeting the wrong CPU architecture. WDYT about that?

This problem occurs with a GNU build via cargo build --release --target x86_64-pc-windows-gnu

Which C compiler, and which version, is used in this case?

Or with an MSVC xwin build via:

It's unlikely I would make progress debugging issues the result from compiling with xwin.

It would be nice to support using clang as the C compiler when targeting -msvc, regardless. But I've not tried it yet.

I always recommend to use the MSVC C compiler.

In ring, this is located here

When investigating this issue I refactored the conditional logic regarding ADX and so your link now points to the wrong place, I think. I think you meant to point to the call to x25519_ge_scalarmult_base.

Regardless, the ADX code is guarded on:

#if defined(_WIN32)
#define OPENSSL_WINDOWS
#endif

[...]

#if !defined(OPENSSL_NO_ASM) && !defined(OPENSSL_SMALL) && \
    defined(__GNUC__) && defined(__x86_64__) && !defined(OPENSSL_WINDOWS)
#define BORINGSSL_FE25519_ADX

[...]

#if defined(BORINGSSL_FE25519_ADX)

[...]

So, I don't think the issue is that the ADX code is being used, unless there is something very wrong with the toolchain in use.

[miruji]

I saw that the Rustls issue was closed, AFAICT based on the deduction that the C compiler must be configured to be targeting the wrong CPU architecture. WDYT about that?

I think that assumption is incorrect because I have checked this further. The idea was that either I have a new CPU and Rust is compiling with such instructions, or my build configuration is set up incorrectly.

In the Rustls examples I provided, there are no extra dependencies or special settings, so that cannot be the cause.

Regarding my architecture, I specifically tried downgrading packages and rebuilding everything, and I also disabled certain instructions. However, nothing helped except using the debug flag on MSVC. On the GNU toolchain, nothing works to resolve this.

Therefore, the assumption that I did something wrong is not accurate. Moreover, I am not a Rustls user myself — it is used by the iroh library, and the same issue happens there through the dependency chain. In other words, if it breaks both in a clean example and in a larger library, it is quite incorrect to conclude that this is due to a mistake on my part.

Which C compiler, and which version, is used in this case?

As I mentioned, the GNU build fails completely with this error, while the MSVC build works, but only in debug mode without this error.

clang version 21.1.8
gcc (GCC) 15.2.1 20260209

I have the latest versions in this regard, and I haven’t encountered any issues with them—neither when building anything nor with other Rust packages.

It's unlikely I would make progress debugging issues the result from compiling with xwin.

It would be nice to support using clang as the C compiler when targeting -msvc, regardless. But I've not tried it yet.

I always recommend to use the MSVC C compiler.

xwin works perfectly everywhere; I switched to it because in debug it doesn’t have this error. I usually used GNU, but it doesn’t work here at all.

Also regarding xwin, as far as I know it handles MSVC completely correctly, just as if it were the original. I mean that of all the possible options, this is the best available when not running on actual Windows. The problem is that I’m not a Windows user, and that’s a big issue if you can’t build ring for it, since other libraries break.

I think you meant to point to the call to x25519_ge_scalarmult_base.

Yes, when I got the error, I just noticed that:

rustlsCheck!ring_core_0_17_14__x25519_ge_scalarmult_base+0xc69:
00007ff7`569cec09 f2410f78da0818  insertq xmm3,xmm10,8,18h

And then I immediately went and found this in the code to make it easier for you to understand what's there.

So, I don't think the issue is that the ADX code is being used, unless there is something very wrong with the toolchain in use.

Actually, I’m not really an expert in this area. As far as I understand, it’s just calling something that shouldn’t be called, right? It’s also strange that GNU doesn’t work at all, while xwin MSVC only fails in debug. Again, it’s strange that GNU doesn’t work at all, while xwin MSVC only works in debug. And again, it can’t be a toolchain error, because all my libraries have always compiled successfully. I mean, in several years, this is literally the first error on xwin or GNU-Windows.

[briansmith]

It doesn't make sense that insertq would appear in the compiled output because insertq is a SSE4a instruction that would only be supported by AMD CPUs. Definitely we're not trying to use any AMD-specific instructions and also it would be very strange for the compiler to generate them.

[briansmith]

I see the system that it is failing on is "Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz".

What system is the compilation happening on? Does it happen to be an AMD CPU?

[miruji]

What system is the compilation happening on? Does it happen to be an AMD CPU?

Yes, my main build system is an AMD Ryzen 7 5800X, it is a modern x86-64 processor. But again, there are no specific settings for it. That is, when I build with GNU or xwin MSVC and manually remove all flags for this processor and so on — nothing helps. However, debug mode does help.

I have a suspicion that ring compiles this exactly this way during compilation. That is, it is its own peculiarity. For example, if I use aws-lc-rs in rustls, then as far as I understand it is determined dynamically for the processor on which it will be run, and there are no problems either on GNU or on xwin.

So it is quite strange if we assume that this is a processor problem and again a build configuration issue.

[briansmith]

Look for something that is configuring the-march or -mcpu settings of the C compiler, especially using CFLAGS or similar environment variables.

insertq is a very unusual instruction. that we don't ever intentionally use exactly because no Intel CPUs support it.

[briansmith]

Perhaps this problem is caused by the cc-rs version in use. Can you try different versions of cc-rs? See rust-lang/cc-rs#1469 for an example of an issue filed against cc-rs that could easily result in the kind of behavior you are seeing.

[miruji]

I’ve debugged this and solved the problem. I’ll write a more detailed reply below. Your earlier hint about the flags was a good solution. Sorry if my English isn’t perfect, but I hope you’ll understand the issue in general.

[briansmith]

Thanks! I appreciate you taking the time to write up your solution! Don't worry about your English!

[miruji]

I’ve figured out the problem — it’s an issue with ring, if it can be called that.

Again, in aws-lc-rs it works because, as far as I understand, it determines this dynamically at runtime.

In ring, the problem arises due to the use of cc or something similar.

That is, when I build ring with GNU or MSVC on Windows, it uses AMD C flags, and we could see exactly the commands that shouldn’t have been there.

This is not an issue with xwin; it just downloads:

• Windows SDK
• MSVC CRT
  It does not affect CPU instructions. As I said, it works as if it were the original MSVC.

This is also not a Rust problem, because in its settings, in the minimal example, no build flags were used, or I tried disabling them. That also did not affect the build.

That is, ring uses my system C flags, which is rather unfortunate because it causes cascading library errors. In other words, this is the only one out of 500+ libraries that broke in this way, and the only one in many years of my using Rust for building on Windows.

So right now, you can only:

1. Do a debug build — this will avoid SIMD optimizations and the build will be general.
2. Manually clear the system flags: CFLAGS= CXXFLAGS= CPPFLAGS= cargo xwin build --release --target x86_64-pc-windows-msvc.

This needs to be done only for this library; there’s currently no other way to manually help the ring library. You can also use aws-lc-rs as a fallback.

In fact, it shouldn’t have used my system settings — in my opinion, this is a library issue. I mean, I know for sure that when you use something like this in your library, you could remove these flags. Or, alternatively, the library should warn users that there’s such a gap and that they need to adjust it manually. But no one, including rustls, has ever done that — and that’s exactly why it caused a chain error, all the way up to a major networking library.

Yes, I debugged it like this:

echo $CFLAGS
echo $CXXFLAGS
echo $CPPFLAGS
-fPIC -O3 -march=znver3 -mtune=znver3 -fomit-frame-pointer -funroll-loops -ffast-math
-fPIC -O3 -march=znver3 -mtune=znver3 -fomit-frame-pointer -funroll-loops -ffast-math -fno-exceptions -fno-rtti

These are my system flags — they shouldn’t be used during the build. At the very least, Rust has its own flags, or the library should warn about this.

And you can check for the instructions like this:

objdump -d your.dll | grep insertq

• In debug, it’s not present.
• With CFLAGS= CXXFLAGS= CPPFLAGS= cargo xwin build --release --target x86_64-pc-windows-msvc, it’s not present.
• In a normal release build, it is present.

Therefore, the issue has been resolved on my side. From here, it depends on how you want to handle this issue on your side:

• Either address the build flags in your library,
• Or warn users about this.

Again, I want to note that in other libraries this was either automatically handled and did not cause errors over the years in Rust, or it was explicitly mentioned. Here, this practice was absent, and everything simply failed one after another. That is, if 500+ packages built correctly everywhere else, here a single package simply does not build correctly in this regard. In my opinion, this is a library issue.

Once again, thank you for helping me debug this error; I really appreciate your help and your time.

[briansmith]

Thank you. I do agree that how CFLAGS and similar are used is error-prone. See the cc-rs documentation at https://docs.rs/cc/latest/cc/#external-configuration-via-environment-variables. In particular, I think you can keep your $CFLAGS as-is if you clear TARGET_CFLAGS= or similar.

[miruji]

Yes, thank you! I have experience with cc-rs, and I hope something like this could be built into ring. Or at least notify users, because I was essentially using iroh and it cascaded: rustls -> ring. It would be great if this could be handled here, or at least so that other libraries could manage it with such flags.