Zydis Formats Invalid MASM With INTEL_MASM

[widberg]

Zydis formats a few things in a way that MASM does not accept with INTEL_MASM:

1. The int3 instruction should be formatted as int 3 for CC or MASM will complain:

It will also complain about int1 but int 1 gets encoded as CD 01 instead of F1. The recommended solution is to use DB 0F1h.

test.asm(6) : error A2008: syntax error : int1
test.asm(7) : error A2008: syntax error : int3

2. The x87 floating-point registers st? should be formatted as st(?):

test.asm(6) : error A2006: undefined symbol : st1

3. The fucomp instruction should be formatted without any operands the implicit st(0) operand:

test.asm(6) : error A2152: coprocessor register cannot be first operand

[mappzor]

#⁠3 is related to #189. Error message is actually misleading e.g. fucomp st(1) is totally fine. MASM doesn't like fucomp st(0), st(N) syntax because it considers st(0) implicit and Zydis treats both as visible operands.

I've listed all instructions with implicit st(0) as first operand and ran them through ml64.exe shipped with VS2022:

.code

fadd st(0), st(1)
fcmovb st(0), st(1)
fcmovbe st(0), st(1)
fcmove st(0), st(1)
fcmovnb st(0), st(1)
fcmovnbe st(0), st(1)
fcmovne st(0), st(1)
fcmovnu st(0), st(1)
fcmovu st(0), st(1)
fcomi st(0), st(1)
fcomip st(0), st(1)
fdiv st(0), st(1)
fdivr st(0), st(1)
fmul st(0), st(1)
fsub st(0), st(1)
fsubr st(0), st(1)
fucomi st(0), st(1)
fucomip st(0), st(1)
fucomp st(0), st(1)

end

Seems like fucomp is the only one MASM dislikes.