Skip to content

s390x: handle POPCNT total-count mode - #86

Open
DORA-B wants to merge 1 commit into
angr:masterfrom
DORA-B:fix/s390x-popcnt-m3
Open

s390x: handle POPCNT total-count mode#86
DORA-B wants to merge 1 commit into
angr:masterfrom
DORA-B:fix/s390x-popcnt-m3

Conversation

@DORA-B

@DORA-B DORA-B commented Jul 27, 2026

Copy link
Copy Markdown

Summary

s390x POPCNT (0xB9E1) is an RRF-c instruction. VEX currently decodes it as RRE, discarding the M3 field, and always returns per-byte population counts. This makes the M3=8 total-count form return an incorrect result.

For example, with source 0xffffffffffffffff:

  • b9e100cc (M3=0): 0x0808080808080808
  • b9e180cc (M3=8): should return 64, but currently also returns 0x0808080808080808

IBM z/Architecture Principles of Operation SA22-7832-14, page 7-424 defines M3 bit 0 (encoded as numeric M3=8) as counting all set bits into a scalar result.

Fix

  • Decode POPCNT with the existing RRF-c-compatible formatter and extract M3.
  • Preserve the current per-byte reduction for M3=0.
  • Horizontally sum the eight byte counts when (m3 & 8) is set.
  • Keep the existing zero/nonzero CC thunk; zero is equivalent in both result forms.

Reproduction

import angr, archinfo, claripy
A = archinfo.ArchS390X()
def popcnt(code_hex, val):
    st = angr.load_shellcode(bytes.fromhex(code_hex), A).factory.blank_state(addr=0)
    st.regs.r12 = claripy.BVV(val, 64)
    s = st.step(num_inst=1).successors[0]
    return s.solver.eval(s.regs.r12) & ((1 << 64) - 1)

# b9e1 80cc = popcnt %r12,%r12,8   (M3=8, total count)
print(hex(popcnt('b9e180cc', (1<<64)-1)))   # angr 0x0808080808080808 ; want 0x40 (64)
# b9e1 00cc = popcnt %r12,%r12,0   (M3=0, per-byte) -- correct
print(hex(popcnt('b9e100cc', (1<<64)-1)))   # 0x0808080808080808 (correct)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant