Description
Hi,
A quick suggestion...
The method…
sciluigi.dependencies.DependencyHelpers._parse_outputitem()
(see below)
…is used to verify an 'out target' is either a 'TargetIfo' object a 'list' or a 'dict'.
But, since the target exists() method is used by (sci)luigi to decide if it will run the command (again) – AND exists() ONLY checks if the output directory 'exists' (not that it has useful data) I've found I often need to subclass the Target or LocalTarget class.
For example I subclassed the LUIGI LocalTarget class with 'class ToppasLocalTarget(LocalTarget)' which decides whether the out_target subdirectory structure matches the expected output of the TOPPAS file and – if it does not – returns False for exists().
Point being, the _parse_outputitem() does not recognize my subclass, and errors out the run with:
raise Exception('Input item is neither callable, TargetInfo, nor list: %s' % val)
Exception: Input item is neither callable, TargetInfo, nor list: <Bioproximity.common.luigi_extensions.local_target.ToppasLocalTarget object at 0x10d9d1190>
It might be useful to have it instead check the root parent class, instead of the class itself – otherwise my child class fails the sciluigi run because it doesn't recognize it as a valid target :)
Thanks :)
def _parse_outputitem(self, val, targets):
'''
Recursively loop through lists of TargetInfos, or
callables returning TargetInfos, or lists of ...
(repeat recursively) ... and return all targets.
'''
if callable(val):
val = val()
if isinstance(val, TargetInfo):
targets.append(val.target)
elif isinstance(val, list):
for valitem in val:
targets = self._parse_outputitem(valitem, targets)
elif isinstance(val, dict):
for _, valitem in iteritems(val):
targets = self._parse_outputitem(valitem, targets)
else:
raise Exception('Input item is neither callable, TargetInfo, nor list: %s' % val)
return targets
Activity