Skip to content

x86: preserve ZF across BT-family instructions - #88

Open
DORA-B wants to merge 1 commit into
angr:masterfrom
DORA-B:fix/x86_bt_preserve_zf
Open

x86: preserve ZF across BT-family instructions#88
DORA-B wants to merge 1 commit into
angr:masterfrom
DORA-B:fix/x86_bt_preserve_zf

Conversation

@DORA-B

@DORA-B DORA-B commented Jul 28, 2026

Copy link
Copy Markdown

Summary

This completes the ZF-preservation fix started by #79 for BT, BTS, BTR, and BTC. Intel defines CF as the selected bit and ZF as unaffected for all four instructions.

PR #79 changed only the amd64 register-index decoder. Three other decoder paths still replace the flag state with CF alone, and the changed path reads raw CC_DEP1, which is not packed RFLAGS when the preceding flags are lazy.

This is also a follow-up to the remaining BTC portion of angr/angr#6067. That grouped issue is closed because its separate MUL fix used Fixes #6067; the closure does not cover these remaining VEX paths.

Missing cases after #79

Architecture Bit-index encoding Current behavior
amd64 register (0F A3/AB/B3/BB) #79 preserves ZF only if flags are already materialized; a lazy predecessor such as cmp is mishandled
amd64 immediate (0F BA /4-/7) replaces the flag state with CF alone
x86 register (0F A3/AB/B3/BB) replaces the flag state with CF alone
x86 immediate (0F BA /4-/7) replaces the flag state with CF alone

Each decoder handles both register and memory destinations, and the issue applies to BT/BTS/BTR/BTC because they share the same flag-writing code.

Root cause

VEX represents arithmetic flags lazily. After cmp, the state is conceptually:

CC_OP   = SUB
CC_DEP1 = left operand
CC_DEP2 = right operand

PR #79 reads CC_DEP1 directly and treats it as packed RFLAGS before switching CC_OP to COPY. Operand bit 6 is therefore copied as ZF. The untouched paths instead switch to COPY with CC_DEP1 containing only the new CF, which always clears ZF.

The fix materializes the previous flags with mk_amd64g_calculate_rflags_all() or mk_x86g_calculate_eflags_all(), replaces only CF with the selected bit, and stores the result in COPY form.

Reproducer

With pyvex built from current VEX master:

import angr

cases = [
    ("amd64-reg", "amd64", "450fbbcd", {"r13": 0xdeadbeef, "r9": 0x12345678}),
    ("amd64-imm", "amd64", "410fbafd05", {"r13": 0xdeadbeef}),
    ("x86-reg", "x86", "0fbbca", {"edx": 0xdeadbeef, "ecx": 0x12345678}),
    ("x86-imm", "x86", "0fbafa05", {"edx": 0xdeadbeef}),
]

for name, arch, encoding, registers in cases:
    # cmp eax, 1 sets ZF=1 lazily when EAX is 1.
    project = angr.load_shellcode(
        bytes.fromhex("83f801" + encoding), arch, load_address=0x1000
    )
    state = project.factory.blank_state(addr=0x1000)
    state.regs.eax = 1
    for register, value in registers.items():
        setattr(state.regs, register, value)
    out = project.factory.successors(state, num_inst=2).flat_successors[0]
    flags = out.solver.eval(out.regs.eflags)
    print(name, "ZF=", (flags >> 6) & 1)

Current master:

amd64-reg ZF= 0
amd64-imm ZF= 0
x86-reg ZF= 0
x86-imm ZF= 0

This branch:

amd64-reg ZF= 1
amd64-imm ZF= 1
x86-reg ZF= 1
x86-imm ZF= 1

Validation

Built pyvex 9.3.0 with this exact VEX branch and checked the cross-product of:

  • x86 and amd64;
  • register and immediate bit indices;
  • BT, BTS, BTR, and BTC;
  • register and memory destinations;
  • prior lazy ZF values 0 and 1.

Current master passes 32/64 and fails every prior-ZF=1 case. This branch passes all 64/64 cases with the destination value, CF, and ZF matching the reference.

Intel reference: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html (Volume 2A, BT/BTC/BTR/BTS Flags Affected).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant