✨ process dump: JSON Data and workflow outputs

[elinscott]

TL;DR

A MultiplyAddWorkChain, dumped as it is today:

$ verdi process dump 5 --include-outputs --path dump-plain
Report: Using specified output path: `<my-pwd>/dump-plain`
Success: Raw files for process `5` dumped into folder `dump-plain`.

$ tree -a dump-plain
dump-plain
├── 01-multiply-6
│   ├── .aiida_dump_safeguard
│   ├── aiida_node_metadata.yaml
│   └── inputs
│       └── source_file
├── 02-ArithmeticAddCalculation-8
│   ├── .aiida_dump_safeguard
│   ├── aiida_node_metadata.yaml
│   ├── inputs
│   │   ├── .aiida
│   │   │   ├── calcinfo.json
│   │   │   └── job_tmpl.json
│   │   ├── aiida.in
│   │   └── _aiidasubmit.sh
│   └── outputs
│       ├── aiida.out
│       ├── _scheduler-stderr.txt
│       └── _scheduler-stdout.txt
├── aiida_dump_log.json
├── .aiida_dump_safeguard
├── aiida_node_metadata.yaml
└── README.md

Crucially, the number the workflow computed is nowhere in that tree.

With this PR, it is dumped when using the new flags:

$ verdi process dump 5 --include-outputs --include-workflow-outputs --include-data-json --path dump-with-flags
Report: Using specified output path: `<my-pwd>/dump-with-flags`
Success: Raw files for process `5` dumped into folder `dump-with-flags`.

$ tree -a dump-with-flags
dump-with-flags
├── 01-multiply-6
│   ├── .aiida_dump_safeguard
│   ├── aiida_node_metadata.yaml
│   ├── inputs
│   │   └── source_file
│   ├── node_inputs
│   │   ├── x.json
│   │   └── y.json
│   └── node_outputs
│       └── result.json
├── 02-ArithmeticAddCalculation-8
│   ├── .aiida_dump_safeguard
│   ├── aiida_node_metadata.yaml
│   ├── inputs
│   │   ├── .aiida
│   │   │   ├── calcinfo.json
│   │   │   └── job_tmpl.json
│   │   ├── aiida.in
│   │   └── _aiidasubmit.sh
│   ├── node_inputs
│   │   ├── code.json
│   │   ├── x.json
│   │   └── y.json
│   ├── node_outputs
│   │   ├── remote_folder.json
│   │   └── sum.json
│   └── outputs
│       ├── aiida.out
│       ├── _scheduler-stderr.txt
│       └── _scheduler-stdout.txt
├── aiida_dump_log.json
├── .aiida_dump_safeguard
├── aiida_node_metadata.yaml
├── node_outputs
│   └── result.json
└── README.md

$ cat dump-with-flags/node_outputs/result.json
5

Problem

Discussed extensively in #7588: a dump does not contain Data nodes nor the returned outputs of WorkflowNodes.

Changes

Two new flags

• off by default
• exposed on verdi process dump, verdi group dump and verdi profile dump
• added as keywords on ProcessNode.dump(), Group.dump() and Profile.dump().

--include-data-json writes every linked Data node that has no repository content as <link-label>.json, wherever the dump already places that node's repository-backed siblings. A Dict writes its dictionary, a List its list, an Int/Float/Str/Bool its bare value, and anything else its attributes (see e.g. remote_folder.json above, which holds the calculation's remote_path)

--include-workflow-outputs gives each WorkflowNode a node_outputs directory of the nodes it returned, with repository-backed returns copied out exactly as they are for a calculation. These files are not dumped with --include-outputs: that flag governs a calculation's CREATE outputs, and a workflow's RETURN outputs are something separate. (We could possibly reconsider the flag names though...)

Note that...

• Labels of one namespace share a file: alphas__filled and alphas__empty become a single alphas.json rather than a file per port.
• A label that would have to be written inside or on top of another label's value is dropped with a warning instead
• A node that carries repository content keeps its directory — so a namespace holding both kinds is split between alphas/ and alphas.json.

Pre-existing collisions

copy_tree has pre-existing issues with file collisions that appear under --flat. This PR mitigates the risk (JSON files are written after the repository copies, so on a name collision the JSON is skipped with a warning); the collision issue should be dealt with in a separate PR.

Notes

Addresses #7588. The issue's secondary question — whether a single-file repository should flatten to <label>.<ext> — is left for a follow-up.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.71%. Comparing base (4019b32) to head (715972d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7589      +/-   ##
==========================================
+ Coverage   80.69%   80.71%   +0.03%     
==========================================
  Files         581      581              
  Lines       47139    47147       +8     
==========================================
+ Hits        38035    38051      +16     
+ Misses       9104     9096       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[GeigerJ2]

Thanks for opening the PR, @elinscott. Some top-level design questions. I wrote most of the dumping code quite a while ago and would do plenty differently now, but, I'd try to keep it backwards compatible here, and we can discuss bigger reworks within the framing of v3.

• --include-outputs currently is a no-op on a WorkflowNode, so it could absorb --include-workflow-outputs, to keep the CLI surface smaller?

• --include-data-json names a format where its neighbours name content, so maybe --include-data-values? Process nodes already get their attributes and extras into aiida_node_metadata.yaml while Data nodes (currently) get nothing, and the flag is really closing that asymmetry.

• alphas__empty merging into alphas.json while repo-backed alphas__filled becomes alphas/filled/ is new here: before, a link label always mapped straight onto a path. Always nesting (alphas/empty.json) keeps that rule and drops the clash handling in _nest_by_link_label.

• The elif writes a node's database content only when it has no repository files. Should that be an if? Additive, and the flag would mean the same thing for every node.

  aiida-core/src/aiida/tools/_dumping/executors/process.py

  Lines 647 to 651 in 715972d

  	if node.base.repository.list_object_names():
  	linked_node_path.parent.mkdir(parents=True, exist_ok=True)
  	node.base.repository.copy_tree(linked_node_path)
  	elif self.config.include_data_json:
  	json_values[link_label] = _serialize_data_node(node)

• _serialize_data_node reads like the pydantic Entity.serialize()/from_serialized() pair but means something quite different, a bare value with no identity and no way back, so maybe _json_content to match the local json_values/_dump_io_json vocabulary. Worth saying in a docstring why a fourth view exists at all, given _prepare_<format>, serialize() and the REST translators: none of them produce the bare 2 that makes cat node_outputs/sum.json worth having. Its isinstance chain is also closed, though complete for aiida-core today, and duck-typing on .value isn't the fix since that would hand json.dumps an enum member. singledispatch is behaviour-identical and reads better, but the win I'd actually care about, a plugin type opting in, needs a public dispatcher, and this one is a private name in the private aiida.tools._dumping, nothing re-exported. Turning it into a supported extension point is a v3 follow-up, flagging it now so we design towards it rather than discover it later; landing the shape here is cheap and optional.

  aiida-core/src/aiida/tools/_dumping/executors/process.py

  Lines 46 to 61 in 715972d

  	def _serialize_data_node(node: orm.Node) -> Any:
  	"""Return the JSON-native content of a ``Data`` node.
  	``Dict``, ``List`` and the ``BaseType`` scalars answer their ``value``, any other node its attributes
  	(``EnumData`` deliberately falls in the second group: its ``value`` is the enum member, not JSON). Attributes
  	pass through ``clean_value`` on storage, which refuses anything that is not JSON-serializable, so the result can
  	always be handed to ``json.dumps``.
  	:param node: The node to serialize
  	:return: A JSON-native object
  	"""
  	if isinstance(node, (orm.Dict, orm.List, orm.BaseType)):
  	return node.value
  	return node.base.attributes.all

  what that would look like

  # aiida/tools/_dumping/executors/process.py
  from functools import singledispatch
  
  
  @singledispatch
  def _json_content(node: orm.Data) -> Any:
      """Return the JSON-native content of a ``Data`` node: its attributes, unless its type says otherwise."""
      return node.base.attributes.all
  
  
  @_json_content.register
  def _(node: orm.Dict | orm.List | orm.BaseType) -> Any:
      """``Dict``, ``List``, and the ``BaseType`` scalars (``Int``, ``Float``, ``Bool``, ``Str``) answer ``value``."""
      return node.value

  A plugin could then opt in without an aiida-core change, once the dispatcher is public:

  @_json_content.register
  def _(node: MyPluginData) -> Any:
      return node.value

  Output-identical on every aiida.data type in core, EnumData included, and a plugin only registers if it wants something different: dispatch is on the MRO, so a subclass of Dict or Int inherits value and anything else gets the attributes default, with no plugin code at all.

• One file per node, or one node_outputs.json with an entry per link label? Per-file can do everything the questions above ask for: nest always, write arraydata.json beside arraydata/, carry uuid and pk. It just has to answer each of them separately, and carrying identity turns sum.json from 2 into an object, which costs it the thing it is best at. One file settles all three at once, since an entry is an object with identity, content, and a pointer when there are files, at the price of cat node_outputs/sum.json. Per-file is what's in the PR and fine to ship, I'd just rather we chose it than defaulted into it. What do you think?

  where it would live, and one entry per node

  02-ArithmeticAddCalculation-8/
  ├── aiida_node_metadata.yaml     # unchanged, the process node itself
  ├── inputs/, outputs/            # unchanged
  ├── node_inputs.json             # every INPUT_CALC link
  ├── node_outputs.json            # every CREATE/RETURN link, nested by namespace
  └── node_outputs/                # only when a linked node has repository content
      └── alphas/
          └── filled/
              └── data.txt
  

  {
    "sum": {"uuid": "...", "pk": 12, "node_type": "data.core.int.Int.", "value": 2},
    "alphas": {
      "empty":  {"uuid": "...", "node_type": "data.core.int.Int.", "value": 2},
      "filled": {"uuid": "...", "node_type": "data.core.folder.FolderData.",
                 "repository": "node_outputs/alphas/filled/"}
    }
  }

---

Zooming out, and this probably wants its own issue: the reason all of the above are open questions is that the dump was designed around processes, and Data nodes never got the same treatment. They enter only as "copy the linked node's repository into node_inputs/<label>/", so as bytes with no identity. A Dict has no bytes and therefore reaches no file, which is #7588; a repository-backed node's database side vanishes, which is the elif; code.json carries no label because identity was never modelled; and namespaces had to be solved twice, once as directories and once inside a JSON document. v3 could close all four at once by giving Data nodes what process nodes already have, identity plus content.

v3 sketch

The internal vocabulary is already right, only the directory names it maps onto are misleading:

# _generate_calculation_io_mapping
aiida_entities = ['repository', 'retrieved', 'inputs', 'outputs']
default_dirs   = ['inputs',     'outputs',   'node_inputs', 'node_outputs']

So internal repository lands in a directory called inputs/, and internal retrieved lands in one called outputs/. Using the internal names on disk gives one meaning per word:

concept	today	proposed
the process node's own repository	inputs/	repository/
what came back from the scheduler	outputs/	retrieved/
repository content of input-linked Data	node_inputs/	inputs/
repository content of output-linked Data	node_outputs/	outputs/
database side of every linked Data	nothing	aiida_node_links.json

repository works for a CalcFunctionNode too, whose repository holds source_file rather than a submission script.

MultiplyAddWorkChain-5/
├── aiida_node_metadata.yaml        # the workflow node itself
├── aiida_node_links.json           # RETURN: result
├── 01-multiply-6/
│   ├── aiida_node_metadata.yaml
│   ├── aiida_node_links.json       # INPUT_CALC: x, y | CREATE: result
│   └── repository/
│       └── source_file
└── 02-ArithmeticAddCalculation-8/
    ├── aiida_node_metadata.yaml
    ├── aiida_node_links.json       # INPUT_CALC: x, y, code | CREATE: sum, remote_folder
    ├── repository/
    │   ├── .aiida/{calcinfo,job_tmpl}.json
    │   ├── _aiidasubmit.sh
    │   └── aiida.in
    └── retrieved/
        ├── _scheduler-std{err,out}.txt
        └── aiida.out

One entry per link, always identity and the database side, plus a repository pointer when the node has files. That is the elif gone, and alphas__filled nests identically on disk and in the document, so _nest_by_link_label goes too.

A repo/ + db/ top-level split is the other obvious shape, and I'd argue against it: it organises the dump by AiiDA's storage implementation rather than by what someone is looking for, so a single calculation's inputs would split across db/inputs/x.json and repo/inputs/structure/, and you would need to know where AiiDA stores each type before knowing where to look.