Node contraction for node deletion while keeping provenance

[agoscinski]

Design

To delete intermediate nodes without loosing provenance information, we offer users to contract the graph at the points where the nodes are delete. An example

[D_in] --> (C_1) --> [D_mid] --> (C_2) --> [D_out]

The contraction of nodes C_1, C_2 and D_mid results in in the provenance of

[D_in] --> (C_contract) --> [D_out]

So it is lost how exactly the result has been calculated but not that these nodes were connected. Note that these contracted new links as a consequence can connect two ProcessNodes. In case we delete D_mid we get

[D_in] --> (C_1) --> (C_2) --> [D_out]

One use case would be to delete all the restarts or retries of a Workflow without deleting the connection.

User API

An integration into delete_nodes did not make sense because contraction delete_nodes has the travers rules as argument and contraction is not compliant with deletion by traversal rules. So this features gets its own function with almost the same interface as delete_nodes

def contract_nodes(
    pks: Iterable[int],
    dry_run: bool | Callable[[set[int]], bool] = True,
    backend: StorageBackend | None = None,
) -> tuple[set[int], bool]:

Similar I think we will offer in verdi the endpoint verdi node contract

Examples

Example 1

---

Example 2

---

Example 3

Considerations on v3

No API break is needed. Even no database migration is needed because it introduces only a new LinkType which is stored as string in the database so no migration needed.

Considerations on collab

@khsrali
There shouldn't be much problems with the collaboration feature, because this replacement deletion works in 2 steps, it deletes existing nodes/links and adds new nodes/links, there is no mutation of existing nodes happening. The collaboration feature can already handle deletions so this shouldn't be an issue to support this replacement deletion.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: eac2664d-e926-486d-aef6-1fce756d1fb0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 19.93958% with 265 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.18%. Comparing base (e56a906) to head (f5aeab4).

Files with missing lines	Patch %	Lines
src/aiida/tools/graph/deletions.py	19.01%	179 Missing ⚠️
src/aiida/storage/psql_dos/backend.py	3.71%	26 Missing ⚠️
src/aiida/tools/archive/imports.py	0.00%	21 Missing ⚠️
src/aiida/cmdline/commands/cmd_node.py	26.09%	17 Missing ⚠️
...torage/sqlite_zip/migrations/versions/main_0002.py	0.00%	7 Missing ⚠️
src/aiida/orm/nodes/contracted.py	78.95%	4 Missing ⚠️
src/aiida/storage/sqlite_zip/migrator.py	0.00%	4 Missing ⚠️
src/aiida/orm/utils/links.py	0.00%	3 Missing ⚠️
src/aiida/orm/utils/node.py	0.00%	2 Missing ⚠️
src/aiida/orm/implementation/storage_backend.py	50.00%	1 Missing ⚠️
... and 1 more

❗ There is a different number of reports uploaded between BASE (e56a906) and HEAD (f5aeab4). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (e56a906)	HEAD (f5aeab4)
	3	1

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #7569       +/-   ##
===========================================
- Coverage   80.67%   30.18%   -50.48%     
===========================================
  Files         581      583        +2     
  Lines       47068    47394      +326     
===========================================
- Hits        37967    14302    -23665     
- Misses       9101    33092    +23991     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[agoscinski]

ping @giovannipizzi

[giovannipizzi]

Thanks! This is a sort of "git rebase with squash" in a sense? Interesting, but I would like to discuss if we really need to break the rule on not having Calc to Calc nodes just for this (requires also adapting QueryBuilder and a number of other places, eg how to navigate from node to node), and see in practice how we would use it.

Also, the deletion issue is slightly more general. Eg I might want to delete just on output node with raw data, keeping the actual nodes there (or even a single file), ideally keeping track of the fact I deleted stuff from a node. Maybe thinking first to a global solution would help (contraction might not be needed if I keep all nodes but remove the actual content? This also should help in not having to rewrite traversal queries, since with the compaction proposed here, the graph topology changes. This feature might still be useful, but I would suggest to first identify the actual use cases we want to cover