An idea to solve common jpype footguns with PyGhidra

[smpeela1]

I've got an idea to help solve some a well known but silent and annoying problem with JPype (with PyGhidra).

Certain Ghidra APIs have an "output parameter" that accept things like byte[] arrays. It is tempting to call these functions with native Python constructs like bytearray. For example:

  addr = toAddr(0x401000)
  mem = currentProgram.getMemory()

  buf = bytearray(16)

  # Wrong / unreliable under JPype:
  # JPype does not treat this as a mutable Java byte[] out-param.
  n = mem.getBytes(addr, buf)

  # buf will be unchanged!
  print(buf.hex())

This is a well documented JPype phenomenon. The reason is that on the call to mem.getBytes, "buf" will be implicitly converted to a JArray so that the call can happen. This is a temporary object. When the call finishes, the temporary will indeed have been mutated -- but jpype will not sync the result back into buf. It will be silently discarded. If you aren't familiar with this phenomenon, it can be very surprising.

There are a few issues upstream about this, for example: jpype-project/jpype#314

What sucks is that the behaviour is totally silent, and there's no warning. I can only speculate why: if jpype did do the sync for you, it would have a lot of overhead on the read-only path, especially with large arrays, and jpype advertises that they support numpy, so I can only assume this is a very common case. I suppose they could have introduced a flag to do this automatically.

For humans authoring scripts, this isn't so bad. You encounter it, you spend an hour wondering what's going on, you learn, and you move on. But when using generated code from LLMs, it's especially problematic, because LLMs are terrible at following instructions and will ignore your prompt to use JArray.

To solve this, keeping it convenient for both humans and LLMs, what I did was I identified all of the Ghidra APIs that use output parameters. I think I have a pretty comprehensive list:

_TARGETS: dict[str, tuple[str, ...]] = {
    "ghidra.program.model.mem.Memory": ("getBytes",),
    "ghidra.program.model.mem.MemoryBlock": ("getBytes",),
    "ghidra.program.model.mem.MemBuffer": ("getBytes",),
    "ghidra.program.model.listing.CodeUnit": ("getBytesInCodeUnit",),
    "ghidra.program.database.mem.FileBytes": ("getOriginalBytes", "getModifiedBytes"),
    "ghidra.features.base.memsearch.bytesource.AddressableByteSource": ("getBytes",),
    "ghidra.util.DataConverter": ("getBytes", "putInt", "putLong", "putShort", "putValue", "putBigInteger"),
    "ghidra.app.util.bin.FileByteProvider": ("readBytes",),
    "ghidra.app.util.bin.RangeMappedByteProvider": ("readBytes",),
    "ghidra.trace.util.MemoryAdapter": ("getBytes",),
}

What I do is I monkey-patch the execution environment to intercept these calls and then, for each argument that is a byte array parameter, the harness will then do the sync after the call is finished. It works well enough for me. I haven't tried it with other output parameter types, as it is the byte[] ones that were the ones that affected my own workflow.

The code I'm using to install these handler is approximately 250 lines long, and supports memoryview, bytearray, lists, numpy arrays, (immutable types like bytes aren't supported). However, it is a maintenance burden, as you have to keep this list in sync with the Java code, so I am not proposing it for a patch. There are some other solutions that may work better, like changing the APIs to use ByteBuffers.

What are others doing to manage this problem?