Skip to content

Added some entry points required for bootstrapping from source. #2601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Added some entry points required for bootstrapping from source. #2601

wants to merge 1 commit into from

Conversation

KOLANICH
Copy link
Contributor

@KOLANICH KOLANICH commented Mar 8, 2021

Summary of changes

Excerpt from a DOCKERFILE to bootstrap python packaging
	git clone --depth=50 https://github.com/pypa/setuptools.git;\
	git clone --depth=50 https://github.com/pypa/setuptools_scm.git;\
	git clone --depth=50 https://github.com/pypa/pip.git;\
	git clone --depth=50 https://github.com/pypa/wheel.git;\
	export SUDO=;\
	cd ./wheel;\
	python3 -c "import re;from pathlib import Path;f=Path('setup.cfg');f.write_text(re.subn('^setup_requires.+$', '', f.read_text(), flags=re.MULTILINE)[0])";\
	PYTHONPATH=../setuptools:./src $SUDO python3 ./setup.py install;\
	cd ../setuptools;\
	python3 ./setup.py install;\
	python3 ./setup.py bdist_wheel;\
	PYTHONPATH=../pip/src $SUDO python3 -m pip install --upgrade --force-reinstall ./dist/*.whl;\
	cd ../wheel;\
	PYTHONPATH=../setuptools python3 ./setup.py bdist_wheel;\
	PYTHONPATH=../pip/src $SUDO python3 -m pip install --upgrade --force-reinstall ./dist/*.whl;\
	cd ../pip;\
	python3 ./setup.py bdist_wheel;\
	PYTHONPATH=./src $SUDO python3 -m pip install --upgrade --force-reinstall ./dist/*.whl;\
	cd ../setuptools_scm;\
	python3 ./setup.py bdist_wheel;\
	$SUDO pip3 install --upgrade ./dist/*.whl;\
	cd ..;\
	rm -rf ./setuptools ./pip ./wheel;\
	git clone --depth=50 https://github.com/uiri/toml.git;\
	cd ./toml;\
	python3 ./setup.py bdist_wheel;\
	$SUDO pip3 install --upgrade ./dist/*.whl;\
	cd ..;\
	echo todo: $SUDO pip3 install --upgrade --force-reinstall ./setuptools_scm/dist/*.whl[toml];\
	rm -rf ./toml ./setuptools_scm;\
	\
	git clone --depth=50 https://github.com/pyparsing/pyparsing.git;\
	cd ./pyparsing;\
	python3 ./setup.py bdist_wheel;\
	$SUDO pip3 install --upgrade ./dist/*.whl;\
	cd ..;\
	rm -rf ./pyparsing;\
	\
	git clone --depth=50 https://github.com/takluyver/flit.git;\
	cd flit/flit_core;\
	python3 -c "from flit_core import build_thyself;build_thyself.build_wheel('./')";\
	$SUDO pip3 install --upgrade ./*.whl;\
	cd ../..;\
	rm -rf ./flit;\
	\
	git clone --depth=50 https://github.com/pypa/packaging.git;\
	git clone --depth=50 https://github.com/pypa/pep517.git;\
	git clone --depth=50 https://github.com/pypa/build.git;\
	\
	cd pep517;\
	PYTHONPATH=../build/src:../packaging python3 -m build -xnw .;\
	$SUDO pip3 install --upgrade ./dist/*.whl;\
	\
	cd ../packaging;\
	PYTHONPATH=../build/src python3 -m build -xnw .;\
	$SUDO pip3 install --upgrade ./dist/*.whl;\
	cd ..;\
	rm -rf ./packaging;\
	\
	cd ./build;\
	PYTHONPATH=./src python3 -m build -xnw .;\
	$SUDO pip3 install --upgrade ./dist/*.whl;\
	cd ..;\
	rm -rf ./pep517 ./build ./packaging;\
Log
  1. First tried bdist_wheel for setuptools. Damn, it tries to install wheel with pip because it is a build dep. Trying to make use of non-installed wheel is unsuccesful.

  2. OK, tried to build wheel with non-installed setuptools + non-installed wheel. No success. wheel needs installed wheel. OK, wheel uses setup.cfg, it's fairly easy to remove install_requires. python3 -c import re;from pathlib import Path;f=Path('setup.cfg');f.write_text(re.subn('^setup_requires.+$', '', f.read_text(), flags=re.MULTILINE)[0]) Then python ./setup.py install succeeds (how? why?). bdist_wheel though doesn't IDK why, seems like just install doesn't register entry points.

  3. OK, added an entry point for bdist_wheel. Damn, error: cannot copy tree 'build/scripts-3.9': not a directory. Trying to solve by installing setuptools.

  4. trying to python3 ./setup.py install in setuptools dir. error: invalid command 'easy_install'. Why??? No entry point. Created a patch with them. Then install succeeds.

  5. then we can build a wheel for setuptools and install it

  6. then we can go to wheel dir, set PYTHONPATH to setuptools dir to get a get the ehtry points for bdist_wheel, build a wheel for it and install.

Closes #2550

Pull Request Checklist

@KOLANICH
Copy link
Contributor Author

KOLANICH commented Mar 18, 2021

@jaraco, is there anything that prevents it from being merged?

@benoit-pierre
Copy link
Member

Tests?

@benoit-pierre
Copy link
Member

And why do you need that? Why not run python setup.py egg_info in the setuptools directory first?

@KOLANICH
Copy link
Contributor Author

KOLANICH commented Mar 18, 2021

Will try that, though it likely won't work. We need bdist_wheel workable to install setuptools properly (setup.py install doesn't set needed entry points), so I ended-up with the following dockerfile: https://gitlab.com/KOLANICH-subgroups/docker-images/fixed_python/raw/master/Dockerfile

Tests?

I feel like the only proper way to test it is to run the part of the mentioned Dockerfile in a clean environment.

@webknjaz webknjaz closed this Mar 27, 2021
@webknjaz webknjaz reopened this Mar 27, 2021
@webknjaz
Copy link
Member

(closed/reopened to trigger GH events/CI stuff)

@jaraco
Copy link
Member

jaraco commented Aug 26, 2021

Will try that, though it likely won't work.

Any updates?

@KOLANICH
Copy link
Contributor Author

In fact no, I haven't added any tests.

@jaraco
Copy link
Member

jaraco commented Oct 3, 2021

I'm closing this (without prejudice). Feel free to comment and we can re-open. Or re-submit when ready. Thanks.

@jaraco jaraco closed this Oct 3, 2021
@KOLANICH
Copy link
Contributor Author

KOLANICH commented Oct 3, 2021

There will be no tests and likely no reopening, since it seems it is not needed by the project. If it was, it'd have been already implemented and this PR wouldn't have been existing at all.
It feels like everyone except me is OK with depending on prebuilt packages from pip. For me it is simpler to modify my CI pipelines so the sources of setuptools are patched, than to implement tests making no sense (that stuff should be "tested" every time setuptools package is built) in order to meet formal requirements for accepting of the patch into upstream. So be it.

@jaraco
Copy link
Member

jaraco commented Oct 4, 2021

The question I'd like to see answered is this one:

And why do you need that? Why not run python setup.py egg_info in the setuptools directory first?

If you add that between steps 3 and 4, does that work?

@KOLANICH
Copy link
Contributor Author

KOLANICH commented Oct 4, 2021

@benoit-pierre, @jaraco, thank you. It works for GitLab CI pipeline: https://gitlab.com/KOLANICH-subgroups/docker-images/fixed_python/-/jobs/1648667066

But not for GitHub Actions one: https://github.com/KOLANICH-libs/RichConsole.py/runs/3797498484

AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'

echo "##[group] Bootstrapping packaging";
	git clone --depth=50 https://github.com/pypa/setuptools.git;
	git clone --depth=50 https://github.com/pypa/pip.git;
	git clone --depth=50 https://github.com/pypa/wheel.git;
	cd ./setuptools;
	python3 ./setup.py egg_info;

	cd ../wheel;
	python3 -c "import re;from pathlib import Path;f=Path('setup.cfg');f.write_text(re.subn('^setup_requires.+$', '', f.read_text(), flags=re.MULTILINE)[0])";
	PYTHONPATH=../setuptools:./src sudo python3 ./setup.py install;
	cd ../;

	cd ./setuptools;
	python3 ./setup.py bdist_wheel;
	PYTHONPATH=../pip/src sudo python3 -m pip install --upgrade --pre ./dist/*.whl;

(I hope that I have correctly guessed what you mean under between steps 3 and 4)

@jaraco
Copy link
Member

jaraco commented Nov 27, 2021

The traceback in that run is thus:

Traceback (most recent call last):
    File "./setup.py", line 87, in <module>
      dist = setuptools.setup(**setup_params)
    File "/home/runner/work/RichConsole.py/RichConsole.py/setuptools/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 108, in setup
      _setup_distribution = dist = klass(attrs)
    File "/home/runner/work/RichConsole.py/RichConsole.py/setuptools/setuptools/dist.py", line 453, in __init__
      _Distribution.__init__(
    File "/usr/lib/python3.8/distutils/dist.py", line 292, in __init__
      self.finalize_options()
    File "/home/runner/work/RichConsole.py/RichConsole.py/setuptools/setuptools/dist.py", line 830, in finalize_options
      for ep in sorted(loaded, key=by_order):
    File "/home/runner/work/RichConsole.py/RichConsole.py/setuptools/setuptools/dist.py", line 829, in <lambda>
      loaded = map(lambda e: e.load(), filtered)
    File "/home/runner/work/RichConsole.py/RichConsole.py/setuptools/pkg_resources/__init__.py", line 2450, in load
      return self.resolve()
    File "/home/runner/work/RichConsole.py/RichConsole.py/setuptools/pkg_resources/__init__.py", line 2460, in resolve
      raise ImportError(str(exc)) from exc
  ImportError: type object 'Distribution' has no attribute '_finalize_feature_opts'

You'll notice that _finalize_feature_opts is an old entry point, probably present due to an existing installation of setuptools. Se here's what I think is happening:

  1. An old version of Setuptools with that entrypoint is still present.
  2. When setuptools.dist.Distribution.finalize_options is called, it enumerates all of the setuptools.finalize_distribution_options entry points and calls each.
  3. Because the old setuptools declares a setuptools.dist.Distribution._finalize_feature_opts as one of those entry points, Setuptools attempts to load it.
  4. The local copy of setuptools is newer and doesn't have that entry point.
  5. Because the local setuptools Python package is ahead on the Python path, an error occurs trying to load that entry point.

You have a few options here. You could manually uninstall setuptools from the system site packages. Or you could try running with -S to avoid the implied import site to exclude system site packages. Or you could get a CI image that includes a newer version of Setuptools without that metadata.

Regardless, I don't think adding entry points to the bootstrapping metadata would help here.

I recommend to file a new issue to track the missed expectation if there still is one.

@KOLANICH
Copy link
Contributor Author

KOLANICH commented Nov 28, 2021

Yeah, I have already solved this issue by

sudo rm -rf /usr/lib/python3/dist-packages/setuptools;
sudo rm -rf /usr/lib/python3/dist-packages/setuptools-*.egg-info;

Thanks anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Clarity on installation from source
4 participants