Migrate Python dependency management from distributePyDep to standard pyproject.toml dependencies

[nblog]

Summary

I'd like to propose transitioning the Python dependency management approach from the current distributePyDep pattern in Gradle build files to leveraging the standard pyproject.toml dependencies specification, combined with a pip download workflow for offline distribution.

Current Situation

Currently, Python dependencies are managed in two separate places:

1. Gradle build.gradle files

For example, in Ghidra/Features/PyGhidra/build.gradle, dependencies are declared using distributePyDep:

distributePyDep("jpype1-1.5.2-cp313-cp313-win_amd64.whl")
distributePyDep("jpype1-1.5.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl")
// ... 20+ more lines for different platform/Python version combinations
distributePyDep("packaging-23.2-py3-none-any.whl")
distributePyDep("setuptools-68.0.0-py3-none-any.whl")
distributePyDep("wheel-0.37.1-py2.py3-none-any.whl")

2. pyproject.toml files

The same dependencies are already properly declared in Ghidra/Features/PyGhidra/src/main/py/pyproject.toml:

dependencies = [
    "Jpype1==1.5.2",
    "packaging"
]

The Problem

This dual-management approach creates confusion, especially for contributors who are familiar with Python but not Java/Gradle:

1. Discoverability: When reading Debugger-agent-dbgeng/src/main/py/pyproject.toml, one would assume only ghidratrace and pybag are dependencies. The additional dependencies in build.gradle (capstone, comtypes, pywin32) are hidden from Python developers.

2. Barrier to contribution: Python-focused contributors may struggle to understand or modify dependencies without Gradle knowledge.

3. Maintenance overhead: Dependencies need to be updated in multiple places.

Proposed Solution

Replace distributePyDep calls with a standard Python workflow:

$env:workdir="Ghidra/Features/PyGhidra/src/main/py"
pip download $env:workdir --dest $env:workdir/dist

This could be integrated into the Gradle build process, reading dependencies directly from pyproject.toml rather than duplicating them in Gradle files.

Benefits

• Single source of truth: Dependencies defined only in pyproject.toml
• Familiar workflow: Standard Python tooling that any Python developer understands
• Easier maintenance: Update dependencies in one place
• Better discoverability: All dependencies visible in standard Python project files

Questions

1. Is there a specific reason for the current distributePyDep approach that I might be missing? (e.g., offline builds, specific version pinning for wheels)
2. Would the team be open to exploring this migration?
3. Are there any technical constraints that make this approach difficult?

Context

I'm primarily a Python developer looking to contribute to Ghidra's Python components. The current setup creates a barrier for understanding the full dependency picture without diving into Gradle files, which may discourage Python-focused contributors.

Thank you for considering this proposal!

[ryanmkurtz]

Is there a specific reason for the current distributePyDep approach that I might be missing? (e.g., offline builds, specific version pinning for wheels)

It is mainly to support offline builds. We cannot assume an internet connection is available. We also have the issue of declaring their IP/license (see the Module.manifest files).

Would the team be open to exploring this migration?

The main thing I am interested in exploring is using a pip download to replace what is happening in the fetchDependencies.gradle script, but I haven't had the time to look into it yet. Even fixing that wouldn't address your main motivations though...it would just address my dislike of having to update that file.

Are there any technical constraints that make this approach difficult?

Because of the offline build requirement, we can't just look at the toml files because they don't show the transitive dependencies.