Allow failed Transmogrifier containers

Why these changes are being introduced:

It was suggested in a previous PR that we raise an exception if ANY
Transmogrifier containers fail.  This appeared to make sense from a data
integrity POV, ensuring that all input files were properly transformed
by Transmogrifier for analysis.  In retrospect, this fell short in two ways:

1. For very large runs, a very small number of containers failing may be
admissible given the run will still contain valuable data and should
be allowed to continue.

2. More subtle, a failed Transmogrifier container may not be an intermittent
bug, but potentially an indication that a code change in Transmog is
problematic.  The responsibility here should be for this application to
surface this in a meaningful way during analysis (out of scope here),
not halt the run completely.

How this addresses that need:
* Creates new env var ALLOW_FAILED_TRANSMOGRIFIER_CONTAINERS, that defaults
to 'true' in the Config class, to allow failed containers
* Continue with run even if failed containers are present

Side effects of this change:
* Runs with failed Transmog containers will continue.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-383

Author: ghukill

README.md

@@ -138,6 +138,7 @@ TRANSMOGRIFIER_MAX_WORKERS=# max number of Transmogrifier containers to run in p
 TRANSMOGRIFIER_TIMEOUT=# timeout for a single Transmogrifier container; default is 5 hours
 TIMDEX_BUCKET=# when using CLI command 'timdex-sources-csv', this is required to know what TIMDEX bucket to use
 PRESERVE_ARTIFACTS=# if 'true', intermediate artifacts like transformed files, collated records, etc., will not be automatically removed
+ALLOW_FAILED_TRANSMOGRIFIER_CONTAINERS=# if 'true' (default), the run will continue even if some Transmogrifier containers failed to complete successfully
 ```
 
 ## CLI commands

abdiff/config.py

@@ -22,6 +22,7 @@ class Config:
         "TRANSMOGRIFIER_TIMEOUT",
         "TIMDEX_BUCKET",
         "PRESERVE_ARTIFACTS",
+        "ALLOW_FAILED_TRANSMOGRIFIER_CONTAINERS",
     )
 
     def __getattr__(self, name: str) -> Any:  # noqa: ANN401
@@ -88,6 +89,13 @@ def preserve_artifacts(self) -> bool:
             self.PRESERVE_ARTIFACTS and self.PRESERVE_ARTIFACTS.strip().lower() == "true"
         )
 
+    @property
+    def allow_failed_transmogrifier_containers(self) -> bool:
+        return bool(
+            self.ALLOW_FAILED_TRANSMOGRIFIER_CONTAINERS
+            and self.ALLOW_FAILED_TRANSMOGRIFIER_CONTAINERS.strip().lower() == "true"
+        )
+
 
 def configure_logger(logger: logging.Logger, *, verbose: bool) -> str:
     if verbose:

abdiff/core/run_ab_transforms.py

@@ -112,7 +112,7 @@ def run_ab_transforms(
     # process results
     log_file = aggregate_logs(run_directory, containers)
     logger.info(f"Log file created: {log_file}")
-    if exceptions:
+    if not CONFIG.allow_failed_transmogrifier_containers and exceptions:
         raise RuntimeError(  # noqa: TRY003
             f"{len(exceptions)} / {len(containers)} containers failed "
             "to complete successfully."