♻️ Integrate aiida-shell into core (aiidateam#7600)

Fold the `aiida-shell` package into core. Entry-point names are carried
over verbatim, and node type strings are derived from those names rather
than from module paths, so nodes written by `aiida-shell` keep loading
unchanged.

Source:

  calculations/shell.py  ->  aiida/calculations/shell.py
  parsers/shell.py       ->  aiida/parsers/plugins/shell.py
  data/pickled.py        ->  aiida/orm/nodes/data/pickled.py
  data/entry_point.py    ->  aiida/orm/nodes/data/entry_point.py
  data/code.py           ->  aiida/orm/nodes/data/code/shell.py
  launch.py              ->  aiida/engine/launch.py
  LICENSE.txt            ->  open_source_licenses.txt

Tests, mirroring where the code landed:

  calculations/test_shell.py -> tests/calculations/test_shell.py
  parsers/test_shell.py      -> tests/parsers/test_shell.py
  data/test_pickled.py       -> tests/orm/data/test_pickled.py
  data/test_entry_point.py   -> tests/orm/data/test_entry_point.py
  data/test_code.py          -> tests/orm/data/code/test_shell.py
  test_launch.py             -> tests/engine/test_launch_shell_job.py

The package markers, `py.typed` and `__version__` are dropped, as are
`test_version.py`, the documentation, CI workflows and repository
configuration.

The package's `conftest.py` is dissolved onto core's fixtures. Its
`aiida_profile` override goes: core's own fixture already provides a
broker and honours `--db-backend` and `--broker-backend`, both of which
the override ignored, so the CI job running on ZeroMQ without a
RabbitMQ service would have failed. `generate_computer` and
`generate_code` become `aiida_computer_local`, `aiida_localhost` and
`aiida_code_installed`. What is genuinely specific to these tests stays
as `generate_shell_code` and `generate_shell_calc_job` in
`tests/calculations/conftest.py`, and the parser helpers in
`tests/parsers/conftest.py`. `generate_shell_calc_job` keeps a name of
its own because core's `generate_calc_job` has a different signature
and is used by four other modules in the same directory.

Six edits go beyond a verbatim copy of the source, each forced by the
destination:

- `launch_shell_job` imports `ShellJob` inside the function body.
  `aiida.calculations.shell` imports `aiida.engine`, so importing it at
  module level from `aiida/engine/launch.py` is a cycle.
- The module-level `submit` is aliased as `_submit_process`, because
  `launch_shell_job` takes a boolean `submit` argument shadowing it.
- `ShellJob` and `ShellParser` no longer declare `__all__`, as no other
  core calculation job or parser does; both are reached through their
  entry points.
- The `nodes` argument of `process_arguments_and_nodes` and
  `prepare_filenames` is annotated `Data`, not `SinglefileData`, which
  is what the port declares. The narrower annotation left the
  `FolderData` and `RemoteData` branches unreachable.
- `prepare_filenames` returns `dict[str, str | None]`, which is what it
  always built and what `write_folder_data` already accepted.
- Imports of sibling data modules are absolute, `t.Union` becomes `|`,
  and one unused unpacked variable is prefixed, for rules core enables
  and `aiida-shell` did not.

- Exception messages are assigned to `msg` before being raised, as core
  does and `aiida-shell` did not. Three defensive guards keep the inline
  form: mypy proves them unreachable and tolerates a bare `raise` there,
  but not the assignment in front of it.

`dill` becomes a runtime dependency, exempt from import checking since
it ships no type information, and floored at 0.3.6. `PickledData`
pickles with `recurse=True`, which arrived in 0.3.0, but only 0.3.6
round-trips what it writes; the requirement came over unpinned, so the
minimum-requirements job resolved `dill==0.2`.

Landing the data plugins in core also brings them into sweeps that walk
every registered node type. `test_all_node_fields` keeps a regression
file per type, and `test_data_exporters` needs a way to build a dummy
instance of anything carrying `_prepare_*` methods, which `ShellCode`
inherits from `InstalledCode`.

Two smaller consequences. The ported `launch_shell_job` tests delete the
computer they create in a `finally` and assert on the log record's
level rather than its position, since the computer is labelled
`localhost` and leaking it breaks every later test in the worker that
wants `aiida_localhost`. And `verdi devel` no longer describes the
localhost it creates as made by `aiida.engine.launch_shell_job`, a
string copied out of `aiida-shell` years ago that named nothing until
this commit, and now names a function that did not create it.

Author: GeigerJ2

environment.yml

@@ -12,6 +12,7 @@ dependencies:
 - circus~=0.19.0
 - click-spinner~=0.1.8
 - click<8.3,>=8.1.0
+- dill<1,>=0.3.6
 - disk-objectstore~=1.5.0
 - docstring_parser
 - python-graphviz~=0.19

open_source_licenses.txt

@@ -3,6 +3,12 @@ projects:
 
 Python:
   * aiida/transports/transport.py
+  * aiida/calculations/shell.py
+  * aiida/engine/launch.py (the `launch_shell_job` function and its helpers)
+  * aiida/orm/nodes/data/code/shell.py
+  * aiida/orm/nodes/data/entry_point.py
+  * aiida/orm/nodes/data/pickled.py
+  * aiida/parsers/plugins/shell.py
 
 The respective copyright notices are reproduced below.
 
@@ -286,3 +292,34 @@ FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
 OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+
+==========================================================================
+
+aiida-shell license:
+
+The modules listed above were integrated from the `aiida-shell` package
+(https://github.com/aiidateam/aiida-shell), which was distributed under the
+MIT license reproduced below.
+
+MIT License
+
+Copyright (c) 2022 Sebastiaan P. Huber
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pyproject.toml

@@ -35,6 +35,7 @@ dependencies = [
   'circus~=0.19.0',
   'click-spinner~=0.1.8',
   'click>=8.1.0,<8.3',
+  'dill>=0.3.6,<1',
   'disk-objectstore~=1.5.0',
   'docstring-parser',
   'graphviz~=0.19',
@@ -75,6 +76,7 @@ requires-python = '>=3.10'
 
 [project.entry-points.'aiida.calculations']
 'core.arithmetic.add' = 'aiida.calculations.arithmetic.add:ArithmeticAddCalculation'
+'core.shell' = 'aiida.calculations.shell:ShellJob'
 'core.stash' = 'aiida.calculations.stash:StashCalculation'
 'core.templatereplacer' = 'aiida.calculations.templatereplacer:TemplatereplacerCalculation'
 'core.transfer' = 'aiida.calculations.transfer:TransferCalculation'
@@ -117,8 +119,10 @@ requires-python = '>=3.10'
 'core.code.abstract' = 'aiida.orm.nodes.data.code.abstract:AbstractCode'
 'core.code.containerized' = 'aiida.orm.nodes.data.code.containerized:ContainerizedCode'
 'core.code.installed' = 'aiida.orm.nodes.data.code.installed:InstalledCode'
+'core.code.installed.shell' = 'aiida.orm.nodes.data.code.shell:ShellCode'
 'core.code.portable' = 'aiida.orm.nodes.data.code.portable:PortableCode'
 'core.dict' = 'aiida.orm.nodes.data.dict:Dict'
+'core.entry_point' = 'aiida.orm.nodes.data.entry_point:EntryPointData'
 'core.enum' = 'aiida.orm.nodes.data.enum:EnumData'
 'core.float' = 'aiida.orm.nodes.data.float:Float'
 'core.folder' = 'aiida.orm.nodes.data.folder:FolderData'
@@ -127,6 +131,7 @@ requires-python = '>=3.10'
 'core.list' = 'aiida.orm.nodes.data.list:List'
 'core.numeric' = 'aiida.orm.nodes.data.numeric:NumericType'
 'core.orbital' = 'aiida.orm.nodes.data.orbital:OrbitalData'
+'core.pickled' = 'aiida.orm.nodes.data.pickled:PickledData'
 'core.remote' = 'aiida.orm.nodes.data.remote.base:RemoteData'
 'core.remote.stash' = 'aiida.orm.nodes.data.remote.stash.base:RemoteStashData'
 'core.remote.stash.compress' = 'aiida.orm.nodes.data.remote.stash.compress:RemoteStashCompressedData'
@@ -166,6 +171,7 @@ requires-python = '>=3.10'
 
 [project.entry-points.'aiida.parsers']
 'core.arithmetic.add' = 'aiida.parsers.plugins.arithmetic.add:ArithmeticAddParser'
+'core.shell' = 'aiida.parsers.plugins.shell:ShellParser'
 'core.templatereplacer' = 'aiida.parsers.plugins.templatereplacer.parser:TemplatereplacerParser'
 
 [project.entry-points.'aiida.schedulers']
@@ -401,6 +407,22 @@ module = [
 ]
 warn_return_any = true
 
+[[tool.mypy.overrides]]
+# Modules integrated from `aiida-shell`, kept at the strictness that package enforced. `disallow_untyped_calls` stays
+# off because they call into core's own as-yet-untyped API, such as `List`, `Dict` and `PluginVersionProvider`.
+disallow_any_generics = true
+disallow_incomplete_defs = true
+disallow_subclassing_any = true
+disallow_untyped_defs = true
+module = [
+  'aiida.calculations.shell',
+  'aiida.orm.nodes.data.code.shell',
+  'aiida.orm.nodes.data.entry_point',
+  'aiida.orm.nodes.data.pickled',
+  'aiida.parsers.plugins.shell'
+]
+warn_return_any = true
+
 [[tool.mypy.overrides]]
 check_untyped_defs = false
 module = 'tests.*'
@@ -411,6 +433,7 @@ module = [
   'bpython.*',
   'bs4.*',
   'circus.*',
+  'dill.*',
   'flask_restful.*',
   'graphviz.*',
   'kiwipy.*',