Remove /pkgs

[orlitzky]

Remove the /pkgs directory and handle the fallout. This is a first step towards removing the sagemath-objects, sagemath-categories (etc.) packages from the repository.

There is still a lot to do:

• Removing the corresponding build/pkgs
• Updating the documentation
• Removing the features and the thousands of "needs sagemath-foo" tags

And so on. Here we do just enough that it doesn't break the CI:

• Remove a few documentation cross-references to sagemath-foo
• Move the sage_docbuild and sage_setup "source" trees to build/pkgs to keep them working
• Remove the pkgs directory
• Update or remove miscellaneous pkgs/ references

[github-actions]

Documentation preview for this PR (built with commit 9050427; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[orlitzky]

Aside from the sagemath_doc issues, I think this actually works now. The CI failures are for the usual reasons (our of space, can't download a tarball, etc.)

[tobiasdiez]

Would it work to keep the docbuild folders for now (and remove everything else)?

[orlitzky]

I'm not sure (I was guessing at all of this), but there's not a lot stopping us from removing them. We just need to replace make doc-html and make doc-pdf with two commands that build the docs from within sage-the-distro. Something like $PYTHON_FOR_VENV src/build-docs.py with the appropriate flags/arguments. I only gave up because I don't have access to sage-the-distro to test and the CI doesn't run those commands so I have no way to know if they work.

[orlitzky]

I added a commit on to the end of this that brings back the doc targets as

doc: doc-html

doc-html:
       $(PYTHON_FOR_VENV) -m sage_docbuild all html </dev/null

doc-pdf:
       $(PYTHON_FOR_VENV) -m sage_docbuild all pdf </dev/null

That probably doesn't work but it should make it easier to experiment.

[orlitzky]

I think make doc, make doc-html, and make doc-pdf now all work and go through meson. The missing piece was using meson from the venv iff there's one installed in the venv:

doc-html:
	if [ -x "$(SAGE_VENV)/bin/meson" ]; then \
		"$(SAGE_VENV)/bin/meson" compile -C "$(SAGE_ROOT)/build/sage-distro" doc-html; \
	else \
		meson compile -C "$(SAGE_ROOT)/build/sage-distro" doc-html; \
	fi

I have actually tested it this time.

[tobiasdiez]

LGTM. Since I haven't tested this with sage-the-distro, I let someone else change it to "positive review"

[r-mb]

So this does not work for me, but maybe I did not do what I had to.
In my already installed SageMath directory, I pulled this branch, switched to it, and ran make doc-html. This fails with the following log:
meson-log.txt
I am running Debian stable. My guess is that with this change, there are now more system packages to be installed? But currently, documentation says this is "Not yet supported.".

[orlitzky]

So this does not work for me, but maybe I did not do what I had to. In my already installed SageMath directory, I pulled this branch, switched to it, and ran make doc-html. This fails with the following log: meson-log.txt I am running Debian stable. My guess is that with this change, there are now more system packages to be installed? But currently, documentation says this is "Not yet supported.".

Does the latest develop branch build? Your error has something to do with pari (fatal error: pari/pari.h: No such file or directory), which doesn't mean that I didn't inadvertently cause it, but it would be a weird coincidence.

[r-mb]

You are right, I thought develop builded because I'm working with it since weeks without issues, but it does not. Maybe something bad happened when I did all the tests in the previous PR. Let me rebuild everything and tell you again, sorry for the noise.

[orlitzky]

No problem, I really appreciate the testing.

[r-mb]

So the PARI error was actually triggered by the changes here. After checking to develop, running a clean and successful make and make doc-html, I checked out again to this branch only to get the PARI error back.

Then, I tried ./configure and make in this new branch, which were successful, and running make doc-html gave me the new error below.
I tried make doc-clean before make doc-html just in case, with no luck.

Error building the documentation.
Traceback (most recent call last):
  File "REDACTED/sage/src/build-docs.py", line 11, in <module>
    main()
    ~~~~^^
  File "REDACTED/sage/src/sage_docbuild/__main__.py", line 541, in main
    build()
    ~~~~~^^
  File "REDACTED/sage/src/sage_docbuild/builders.py", line 670, in _wrapper
    getattr(DocBuilder, build_type)(self, *args, **kwds)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/sage/src/sage_docbuild/builders.py", line 143, in f
    runsphinx()
    ~~~~~~~~~^^
  File "REDACTED/sage/src/sage_docbuild/sphinxbuild.py", line 324, in runsphinx
    sys.stderr.raise_errors()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "REDACTED/sage/src/sage_docbuild/sphinxbuild.py", line 255, in raise_errors
    raise OSError(self._error)
OSError: WARNING: lors de l'initialisation de l'extension conf.py : directive 'jupyter-execute' is already registered and will not be overridden [app.add_directive]
ninja: build stopped: subcommand failed.
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: REDACTED/.local/bin/ninja -C REDACTED/sage/build/sage-distro doc-html
make[1]: *** [Makefile:2999: doc-html] Error 1
make[1] : on quitte le répertoire « REDACTED/sage/build/make »
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make doc-html'):

It is safe to delete any log files and build directories, but they
contain information that is helpful for debugging build problems.
WARNING: If you now run 'make' again, the build directory of the
same version of the package will, by default, be deleted. Set the
environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

real 0m40,917s user 6m45,858s sys 1m28,588s
make: *** [Makefile:40: doc-html] Error 1

I need to go, and I should not be able to test again until Wednesday.

[r-mb]

It turns out I had a little bit of time tonight. So the PARI error is now gone, yeepee! But this is new:

[3/192] Generating src/doc/doc-inventory-reference-references with a custom command
FAILED: [code=1] src/doc/inventoryreferences 
REDACTED/sage/local/var/lib/sage/venv-python3.13/bin/python3 REDACTED/sage/src/build-docs.py --no-prune-empty-dirs --no-pdf-links reference/references inventory -o src/doc --source REDACTED/sage/build/sage-distro/src/doc
[reference] Sphinx v8.2.3 en cours d'exécution
[reference] chargement des traductions [en]... fait
[reference] WARNING: lors de l'initialisation de l'extension conf.py : directive 'jupyter-execute' is already registered and will not be overridden [app.add_directive]
[reference] WARNING: lors de l'initialisation de l'extension conf.py : directive 'jupyter-kernel' is already registered and will not be overridden [app.add_directive]
[reference] WARNING: lors de l'initialisation de l'extension conf.py : directive 'jupyter-input' is already registered and will not be overridden [app.add_directive]
[reference] WARNING: lors de l'initialisation de l'extension conf.py : directive 'jupyter-output' is already registered and will not be overridden [app.add_directive]
[reference] WARNING: lors de l'initialisation de l'extension conf.py : directive 'thebe-button' is already registered and will not be overridden [app.add_directive]
[reference] WARNING: L’entrée 'static' de html_static_path n’existe pas
[reference] WARNING: Le fichier de favicon 'favicon.ico' n’existe pas
[reference] chargement de l'environnement pickled... La configuration a changé (2 options : 'html_static_path', 'linkcode_resolve')
[reference] fait
[reference] construction en cours [mo] : cibles périmées pour les fichiers po 0
[reference] écriture...
[reference] construction [inventory] : cibles périmées pour les fichiers sources 0
[reference] mise à jour de l'environnement : 0 ajouté(s), 0 modifié(s), 0 supprimé(s)
[reference] lecture des sources...
[reference] recherche des fichiers périmés... aucun résultat trouvé
[reference] aucune cible n'est périmée.
[reference] documents en préparation... fait
[reference] copie des ressources...
[reference] copie des ressources: fait
[reference] La compilation a réussi, 7 avertissements.
[reference] The inventory file is in src/doc/inventory/en/reference/references.

Sorry for my French. If you need I can perhaps change my locales settings. It then fails with the same error as in my above comment.
This happens when I do make doc-html just after ckecking out to this branch after a clean make and make doc-clean on develop.

If I further do make clean, make doc-clean, make configure, ./configure and make before (all successful), then the error before is different. First, I get tons of lines like:

[reference] REDACTED/sage/build/sage-distro/src/doc/en/reference/references/index.rst:7229: WARNING: La citation [ZZ2005] n'est pas référencée [ref.citation]

Then:

[reference] vérification de la cohérence... fait
[reference] documents en préparation... fait
[reference] copie des ressources...
[reference] copie des ressources: fait
[reference] écriture... [100%] index
[reference] La compilation a réussi, 1723 avertissements.
[reference] The inventory file is in src/doc/inventory/en/reference/references.
Error building the documentation.

Then the above error (from my previous comment) again. If I run make doc-html again, then it is the situation of the beginning of this comment.

[orlitzky]

I don't understand why it's failing, and it's not the French that's the problem! Aside from the FAILED, there are no errors?

[r-mb]

Yes, it is not very clear why it failed, and it's not even (or so I believe) giving a path to a log file which could help us...
Doing grep -r 'FAILED: \[code=1\] src/doc/inventoryreferences' only gives me logs/install.log which contains exactly the above input. I went to logs/pkgs to do grep -ri FAILED but saw nothing relevant...

I actually don't know what's going on. What is weird is the line saying La compilation a réussi, 7 avertissements., basically it went fine? Maybe some warnings can be interpreted as failures? I don't know, it's just a blind guess.

[tobiasdiez]

Is this ready for review?

[orlitzky]

Is this ready for review?

It works for me, but only in a clinical setting. I don't use the sage distro any more, and can't build it on my riscv machine, so I'm doing this on a boring amd64 box at work where the git checkout is clean, and sage was never installed, etc.

@r-mb was nice enough to try it but was running into docbuild issues that I haven't been able to reproduce. Maybe some dependency issue that snuck in after replacing the pseudo-package with meson?

[r-mb]

Sorry, these are extremely busy weeks for me so I had no time not test your last updates. I just did, and, good news! The bug is gone, and make doc-html now works fine, even after a make distclean. I don't know if there is another thing that I should test.

For the rest, I really am not knowledgable at all on this to give a "s: positive review". So this is a "s: does not break my workflow review". All I can say is that it seems to simplify the build process, which might help new people (just like me) to more easily learn/debug/help with building.

[tobiasdiez]

@orlitzky the pdf doc build CI still seems to have a problem. Otherwise this looks good to me as well.

[orlitzky]

The latest force-push drops about half of the commits, hoping to narrow down the problem. I think the CI failure may actually be one of the random ones, I'll try re-pushing.

[orlitzky]

The docbuild is OK this time around, and the one new failure is from ECL (unrelated).

I think it's ready for review and testing. It's a little ugly, but this one is safer. The rest can be removed a little bit at a time.

[tobiasdiez]

Okay!