Error for export when cascading vertically related leaves

[samuelbray32]

Describe the bug

Situation:

• Take a set of tables: Session -> Table1 -> Table2
• Say a user has logged restrictions for extport in ExportSelection.Table
  • {table_name':"Table1", "restriction": restr1}
  • {table_name':"Table2", "restriction": restr2}
• Assume these restrictions define at least some non-overlapping entries in Table1 when cascaded up

Issue:

• ExportSelection.get_restr_graph() makes a RestrGraph with these as leaves and cascades up
• Cascade iteratively cascades up each leaf, adding tables touched along the way to visited
• Because the cascade of each leaf is sequential, this means that parents of the second leaf cascaded will have already been marked as visited, preventing addition of elements that should be logged by the restriction of the next leaf

To Reproduce

from spyglass.common import Nwbfile
import datajoint as dj
from datajoint.condition import make_condition
from spyglass.utils.dj_graph import RestrGraph
from spyglass.common.common_usage import ExportSelection

from spyglass.decoding.v1.clusterless import ClusterlessDecodingV1
from spyglass.position.v1 import TrodesPosV1

seed_table = ClusterlessDecodingV1()
seed_restr = "nwb_file_name LIKE 'Winnie20220714%'"
seed_table & seed_restr

seed_table_2 = TrodesPosV1()
seed_restr_2 = "nwb_file_name LIKE 'Wallie20220914%'"

leaves = [
    {
        "table_name": seed_table.full_table_name,
        "restriction": seed_restr,
    },
    {
        "table_name": seed_table_2.full_table_name,
        "restriction": seed_restr_2,
    },
]

restr_graph = RestrGraph(
    seed_table=ExportSelection(),
    leaves=leaves,
    verbose=True,
    cascade=False,
    include_files=True,
)
restr_graph.cascade()
restr_graph.enforce_restr_strings()

query = TrodesPosV1 & restr_graph._get_node(TrodesPosV1)["restr"]
set(query.fetch("nwb_file_name")) # ONLY CAPTURES ONE OF THE LEAVES' SESSIONS

Proposed Soln
Make an optional parameter for_export in RestrGraph. When True for a graph:

1. Reset the visited flag before cascading each leaf
2. When revisiting a leaf, make the restriction the union of entries from the existing restriction and the new one being cascaded

[samuelbray32]

Looking into impementation:

RestrGraph._set_restr already has the option to merge restrictions in a node if replace=False

[CBroz1]

This hadn't occurred to me, you're right that mutually exclusive leaf restrictions are not merged. I think I might focus the fix on resetting 'visited' after cascading each leaf in the main cascade loop

spyglass/src/spyglass/utils/dj_graph.py

Lines 865 to 877 in 33f30f2

	to_visit = self.leaves - self.visited
	for table in tqdm(
	to_visit,
	desc="RestrGraph: cascading restrictions",
	total=len(to_visit),
	disable=not (show_progress or self.verbose),
	):
	restr = self._get_restr(table)
	self._log_truncate(
	f"Start {direction:<4}: {self._camel(table)}, {restr}"
	)
	self.cascade1(table, restr, direction=direction)

A more thoughtful approach would break 'visited' into 'visited' and 'banned'. Right now, it's playing double duty