Update docs build and packaging setup #8
+17,779
−5,321
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rn` local namespace.
…ic situations where casampi is not available.
…pipeline processing job.
…te building_the_pipeline.md .
…l requirements*.txt files.
…od with the new pure Python-based approach.
…line packaging/installation. * major packaging specifications have been migrated into pyproject.toml * update requirements_dev.txt & requirements_docs.txt * consolidate the duplicated code of the Git-based version string parser into the infrastructure/version.py module, which now includes a CLI interface and can be used as a standalone tool. * refine the version string detection method to handle the case where the import source code is from repo:build/lib directory.
Merge in PIPE/pipeline from sd-reference-manual-update to update-docs-build-and-packaging-setup * commit '10bcdc124ab6461db1eed693e927c9fcb9112dff': update description on freqeuncy reference frame of output images update for formatting
* remove more references to `h_export_calstate` and `h_import_calstate` from CLI functions. * corrections in the docstring of `h_init`, `h_resume`, `h_save`, and `h_weblog`. * improve the docstring formatting of task argument `parallel`. * remove `vis` as task argument of `hif_analyzealpha()` * improve the docstring of `hif_editimlist`.
…bitbucket.nrao.edu/scm/pipe/pipeline into update-docs-build-and-packaging-setup
* Use "MeasurementSet" instead of "Measurement Set". * Correct some formatting issues in the docstring of `hifa_spwphaseup`.
…pos` and `hif_gaincal` from the CLI of `pipeline.hif`.
…s not initialized yet.
…ethod of calculating Python object sizes; add implicit Pipeline runtime dependencies built-in monolithic CASA in `requirements.txt` for documentation purposes.
…zation refactoring.
… the latest `Pympler/asizeof` version (ver 1.1).
…sub-directory structure for initialization of a pipeline PPR execution session.
…sion comparisons into the regression test `working/` directory.
…r task identification; instead, use a `ContextVar` global variable - `current_task_name` - to help domain objects and imaging heuristics to identify the top-level CLI task being executed.
…uration management. - Changed the condition for Dask client initialization to prevent accidental recursive process creation. - Added new functions to sanitize the environment for child processes in MPI contexts. - Improved logging and configuration handling for Dask clusters. - Introduced new test configuration files (in YAML) for LocalCluster, SLurmCluster, and HTCondorCluster use cases. - Restrict the use of `pyvirtualdisplay` to non-xvfb-run instances (either worker or client process). - Improve the default n_worker heuristics. Previously, starting a Dask LocalCluster from within an active MPI process would cause a fatal `MPI_Init_thread` error. The new Dask worker processes would inherit the parent's MPI environment, leading to a nested initialization crash. This change decouples the Dask cluster, starting it as a process with MPI env variables sanitized. Both the MPI server process and the Dask work process now connect to this client process smoothly and can be utilized for operations enabled in different parallelization models in one session (e.g., tclean/parallel=true and tier0future in a single session).
…ment of the daskcluster over a processing session life-cycle. PIPE-2375: improve the Dask worker setup plugin; improve the logic for default login of job name and worker number. PIPE-2884: allow optional automatic creation of dask cluster upon pipeline package import PIPE-2885: allow the creation of Dask htcondorcluster from an execute node of the htcondor cluster network. PIPE-2886: attach xvfb setup to worker process on-demand to address the plotms execution issue on the Dask workers in jobqueue Dask cluster types. PIPE-2887: Allow optional automatic creation of dask cluster from pipeline main process (serial session) or mpiclient process (mpicasa session) By removing the MPI env variables confusing MPI band initialization before creation of dak custer. The newly spawned dask worker process will not confuse the MPI communication and registration in a local cluster scenario.
…tring using `pipeline.__version__` as it's dynamically generated from Git history rather than a literal specification.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.