-
Notifications
You must be signed in to change notification settings - Fork 3
Description
Summary
The current shfmt-py package requires an active internet connection during installation. The setup.py script is designed to download the appropriate pre-compiled shfmt binary from the official GitHub releases for the target platform at the moment a user runs pip install.
This behavior makes it impossible to install shfmt-py in secure, airgapped, or firewalled environments. It also introduces an install-time dependency on the availability of GitHub, which can lead to non-reproducible or failed builds in CI/CD pipelines.
Expected Behavior
The shfmt-py package should be installable from a local artifact (a source distribution or a wheel) without requiring any network access during the installation process itself.
Proposed Solution
The packaging process should be updated to separate the fetching of binaries from the installation logic. I think there are many different ways to accomplish this but here are a couple that I see:
Option 1: Bundle Binaries in Distribution Artifacts
One way to achieve this could be to adjust the packaging strategy to bundle the shfmt binaries within the distribution files uploaded to PyPI. Modern packaging workflows often use tools like cibuildwheel in a CI/CD pipeline to automate this.
A possible implementation might involve these concepts:
-
A Pre-Build Step in CI: A custom command could be created (e.g., in
setup.py) that downloads all necessaryshfmtbinaries. This command would only be run by the CI/CD pipeline at release time, inside a temporary environment. The binaries would not be committed to the Git repository. -
Automated Wheel Building: After the pre-build step, a tool like
cibuildwheelcould then build the platform-specific wheels. It would be configured to find the correct, pre-downloaded binary and bundle it. A benefit of this tool is that it also handles complexities like creatingmanylinuxwheels for broad Linux compatibility. -
Automated Publishing: Finally, the CI pipeline could publish the newly created, self-contained wheels (and sdist) directly to PyPI.
This approach would automate the release process while keeping the Git repository itself free of binary files.
Option 2: Platform-Specific Packages
Create separate PyPI packages for each platform (e.g., shfmt-py-linux-x64, shfmt-py-darwin-arm64). The main shfmt-py package would become a meta-package that depends on the appropriate platform-specific package. This approach keeps individual package sizes small but increases maintenance complexity.
End-User Impact
When a user runs pip install shfmt-py:
pipwill download the most appropriate file from PyPI.- If a pre-built wheel exists for their platform, it will be used. The
shfmtbinary is already inside, and the installation will complete offline. - If they build from source,
pipwill use the sdist. Since all binaries are included in the sdist, the installation can proceed without any further network calls.
Benefits of this Approach
- Enables Airgapped Installation: Unlocks usage in secure and offline environments.
- Improves Build Reliability: Removes the external network dependency at install time, making user builds deterministic.
- Follows Best Practices: Aligns with the standard methodology for distributing Python wrappers around third-party executables.