feat: use multiprocessing workers for enhanced performance

Author: angstwad

.github/workflows/test-app.yaml

@@ -23,7 +23,7 @@ jobs:
         uses: astral-sh/setup-uv@v6
 
       - name: install dependencies
-        run: uv sync --verbose
+        run: uv sync --all-extras --verbose
 
       - name: pylint
         run: uv run pylint --fail-on=E src/

README.md

@@ -266,18 +266,17 @@ constraints against themselves.
 
 ### Increase Rendering Speed with Concurrency
 
-Hydrator has added support for asynchronous hydration of cluster and package manifests using event-based concurrency. It
-is disabled by default. To use, pass the `--workers` flag during invocation. For example:
+Hydrator is designed for high-performance, parallel execution of CPU-bound hydration tasks. It automatically uses a process-based concurrency model to take full advantage of multi-core systems, bypassing Python's Global Interpreter Lock (GIL).
+
+By default, Hydrator will spawn a number of worker processes equal to the number of CPU cores on the system. You can override this behavior by passing the `--workers` flag:
 
 ```shell
 hydrate -v --workers=32 cluster sot.csv --gatekeeper-validation --split-output
 ```
 
-The recommended number of workers to use is 2 for every CPU core available. If you are still not seeing the performance
-improvements you would like, we recommend allocating more CPUs to hydrator and to increase the number of workers
-accordingly.  For example, if you have allocated 16 (v)CPUS, use 32 workers.
+The optimal number of workers can vary depending on the workload and system resources. A good starting point is one worker per available CPU core, but experimentation is encouraged. More available CPUs will always result in better performance.
 
-> **Note:** Increasing the number of concurrent workers can dramatically reduce the usefulness of hydrator output, as the logs from concurrent tasks will be interleaved in the console. If you are troubleshooting a local developer workflow, it is suggested to disable hydration concurrency by omitting the `--workers` flag to ensure clear, sequential output.
+> **Note:** When multiple workers are active, their log outputs will be interleaved in the console. For troubleshooting a specific hydration, it is recommended to limit the run to a single worker (`--workers=1`) to ensure clear, sequential output.
 
 ### Filtering
 
@@ -377,7 +376,7 @@ Note that functionality is split across subcommands:
 
 ```
 $ hydrate --help
-usage: hydrate [-h] [-v | -q] [--workers WORKERS] {cluster,package} ...
+usage: hydrate [-h] [-v | -q] [--workers WORKERS] [--version] {cluster,package} ...
 
 positional arguments:
   {cluster,package}
@@ -388,7 +387,8 @@ options:
   -h, --help         show this help message and exit
   -v, --verbose      increase output verbosity; -vv for max verbosity
   -q, --quiet        output errors only
-  --workers WORKERS
+  --workers WORKERS  Overrides the number of worker processes to use for hydration. By default the number of worker processes is the number of processors on the machine.
+  --version          show program's version number and exit
 ```
 
 ## Development
@@ -432,7 +432,7 @@ of this writing, asyncronous hydration using asyncio completes approximately 85%
 
 There is an additional set of tests assets at `tests/assets/platform_valid_async_perf_testing` which contain ~1000
 clusters for evaluation and test of performance enhancements/improvements. Hydration of these resources completes in
-approximately ~149s.
+approximately ~44s.
 
 ### Building Docker Container
 

src/hydrator/__main__.py

@@ -14,7 +14,6 @@
 # limitations under the License.
 #
 ###############################################################################
-import asyncio
 import pprint
 import sys
 
@@ -101,7 +100,7 @@ def main() -> int:
         logger.error(e)
         return 1
 
-    return asyncio.run(cli.run())
+    return cli.run()
 
 
 if __name__ == '__main__':