CorrDiff: Adding NVTX markers and removing redundant regression model inference (#306)

* Refactor saving results to improve perf and overlap with inferencing

* Update blossom-ci.yml (#295)

* Change pip install commands with the correct PyPI package name (#298)

* Make number of writer workers parameterized

* add comment for writer workers

* revert readme and blossom-ci

* 2nd revert of readme

* Adding NVTX markers and removing redundant regression model inference

* Code formatting

* Adding nvtx to pyproject.toml

* Updating CHANGELOG.md

---------

Co-authored-by: David Pruitt <dpruitt@login-eos01.eos.clusters.nvidia.com>
Co-authored-by: Kaustubh Tangsali <71059996+ktangsali@users.noreply.github.com>
Co-authored-by: Abdullah <37012364+Saydemr@users.noreply.github.com>
Co-authored-by: David Pruitt <dpruitt@nvidia.com>
Co-authored-by: David Pruitt <daviddpruitt@gmail.com>
Co-authored-by: Akshay Subramaniam <asubramaniam@login-eos01.eos.clusters.nvidia.com>

Author: 6 people

CHANGELOG.md

@@ -34,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Molecular Dynamics example.
 - Improved usage of GraphPartition, added more flexible ways of defining a partitioned graph.
 - Physics-Informed Stokes Flow example.
+- Profiling markers, benchmarking and performance optimizations for CorrDiff inference.
 
 ### Changed
 
@@ -53,7 +54,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Dependencies
 
 - Removed experimental SFNO dependencies
-- Added CorrDiff dependencies (cftime, einops, pyspng)
+- Added CorrDiff dependencies (cftime, einops, pyspng, nvtx)
 - Made tqdm a required dependency
 
 ## [0.4.0] - 2023-11-20