Merge pull request #1960 from AdeelH/backport

[BACKPORT] Backport changes to the 0.21 branch for v0.21.3 release

Author: AdeelH

.github/workflows/continuous_integration.yml

@@ -18,6 +18,10 @@ jobs:
 
       - run: df -hT
 
+      - run: rm -rf /opt/hostedtoolcache
+
+      - run: df -hT
+
       - run: ./scripts/cibuild
 
       - run: df -hT

.github/workflows/release.yml

@@ -23,6 +23,8 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      - run: rm -rf /opt/hostedtoolcache
+
       - run: ./scripts/cibuild
 
       - run: docker system prune -f

docs/release.rst

@@ -60,6 +60,13 @@ Minor or Major Version Release
         docker push quay.io/azavea/raster-vision:pytorch-<version>
 
 #.  Make a GitHub `tag <https://github.com/azavea/raster-vision/tags>`_ and `release <https://github.com/azavea/raster-vision/releases>`_ using the previous release as a template.
+#. Remove artifacts from previous builds. From the repo root:
+
+    .. code-block:: console
+
+        rm -rf build/ dist/ *.egg-info
+        rm -rf rastervision_*/build rastervision_*/dist rastervision_*/*.egg-info
+
 #.  Publish all packages to PyPI. This step requires `twine <https://twine.readthedocs.io/en/stable/>`__ which you can install with
 
     .. code-block:: console
@@ -105,12 +112,20 @@ Minor or Major Version Release
 #.  Announce the new release in our `forum <https://github.com/azavea/raster-vision/discussions>`_, and with a blog post if it's a big release.
 #.  Make a PR to the master branch that updates the version number to the next development version, ``X.Y.Z-dev``. For example, if the last release was ``0.20.1``, update the version to ``0.20.2-dev``.
 
-Bug Fix Release
+Patch Release
 -----------------
 
-This describes how to create a new bug fix release, using incrementing from 0.8.0 to 0.8.1 as an example. This assumes that there is already a branch for a minor release called ``0.8``.
+This describes how to create a new patch release (AKA a bug-fix release), using an increment from 0.8.0 to 0.8.1 as an example. This assumes that there is already a branch for a minor release called ``0.8``.
+
+#. Backport changes to the ``0.8`` branch. To create a patch release (version 0.8.1), we need to backport all the commits on the ``master`` branch that have been added since the last patch release onto the ``0.8`` branch. To do this:
+
+   #. Create a new branch from the ``0.8`` branch. Let's call it ``backport``.
+   #. Cherry-pick each commit that we want to include from the ``master`` branch onto the ``backport`` branch.
+   #. Make a PR against the ``0.8`` branch from the ``backport`` branch. The title of the PR should start with ``[BACKPORT]``.
+#. Update changelog and version on the ``0.8`` branch. Make and merge a PR against ``0.8`` (but not ``master``) that adds a changelog for the new release and increments the version to ``0.8.1`` throughout the repo. Wait for the ``0.8`` branch to be built by GitHub Actions and the ``0.8`` Docker images to be published to Quay. If that is successful, we can proceed to the next steps of actually publishing a release.
+#. Publish the new version to PyPI. Follow the same instructions for PyPI as listed above for minor/major version releases.
+#. Using the GitHub UI, make a new release. Use ``v0.8.1`` as the tag, and the ``0.8`` branch as the target.
+#. Update changelog and version on the ``master`` branch. Make and merge a PR against ``master`` that 
 
-#.  To create a bug fix release (version 0.8.1), we need to backport all the bug fix commits on the ``master`` branch that have been added since the last bug fix release onto the ``0.8`` branch. For each bug fix PR on ``master``, we need to create a PR against the ``0.8`` branch based on a branch of ``0.8`` that has cherry-picked the commits from the original PR. The title of the PR should start with [BACKPORT].
-#.  Make and merge a PR against ``0.8`` (but not ``master``) that increments the version in each ``setup.py`` file to ``0.8.1``. Then wait for the ``0.8`` branch to be built by GitHub Actions and the ``0.8`` Docker images to be published to Quay. If that is successful, we can proceed to the next steps of actually publishing a release.
-#.  Using the GitHub UI, make a new release. Use ``0.8.1`` as the tag, and the ``0.8`` branch as the target.
-#.  Publish the new version to PyPI. Follow the same instructions for PyPI that are listed above for minor/major version releases.
+   * includes the cherry-picked commit that updates the changelog for ``0.8.1`` and 
+   * increments the version to ``0.8.2-dev`` throughout the repo.

rastervision_core/rastervision/core/data/label/chip_classification_labels.py

@@ -155,20 +155,26 @@ def extend(self, labels: 'ChipClassificationLabels') -> None:
         for cell in labels.get_cells():
             self.set_cell(cell, *labels[cell])
 
-    def save(self, uri: str, class_config: 'ClassConfig',
-             crs_transformer: 'CRSTransformer') -> None:
+    def save(self,
+             uri: str,
+             class_config: 'ClassConfig',
+             crs_transformer: 'CRSTransformer',
+             bbox: Optional[Box] = None) -> None:
         """Save labels as a GeoJSON file.
 
         Args:
             uri (str): URI of the output file.
             class_config (ClassConfig): ClassConfig to map class IDs to names.
             crs_transformer (CRSTransformer): CRSTransformer to convert from
                 pixel-coords to map-coords before saving.
+            bbox (Optional[Box]): User-specified crop of the extent. Must be
+                provided if the corresponding RasterSource has bbox != extent.
         """
         from rastervision.core.data import ChipClassificationGeoJSONStore
 
         label_store = ChipClassificationGeoJSONStore(
             uri=uri,
             class_config=class_config,
-            crs_transformer=crs_transformer)
+            crs_transformer=crs_transformer,
+            bbox=bbox)
         label_store.save(self)

rastervision_core/rastervision/core/data/label/object_detection_labels.py

@@ -294,20 +294,26 @@ def prune_duplicates(
             score_threshold=score_thresh)
         return ObjectDetectionLabels.from_boxlist(pruned_boxlist)
 
-    def save(self, uri: str, class_config: 'ClassConfig',
-             crs_transformer: 'CRSTransformer') -> None:
+    def save(self,
+             uri: str,
+             class_config: 'ClassConfig',
+             crs_transformer: 'CRSTransformer',
+             bbox: Optional[Box] = None) -> None:
         """Save labels as a GeoJSON file.
 
         Args:
             uri (str): URI of the output file.
             class_config (ClassConfig): ClassConfig to map class IDs to names.
             crs_transformer (CRSTransformer): CRSTransformer to convert from
                 pixel-coords to map-coords before saving.
+            bbox (Optional[Box]): User-specified crop of the extent. Must be
+                provided if the corresponding RasterSource has bbox != extent.
         """
         from rastervision.core.data import ObjectDetectionGeoJSONStore
 
         label_store = ObjectDetectionGeoJSONStore(
             uri=uri,
             class_config=class_config,
-            crs_transformer=crs_transformer)
+            crs_transformer=crs_transformer,
+            bbox=bbox)
         label_store.save(self)

rastervision_core/rastervision/core/data/label/semantic_segmentation_labels.py

@@ -357,6 +357,7 @@ def save(self,
              uri: str,
              crs_transformer: 'CRSTransformer',
              class_config: 'ClassConfig',
+             bbox: Optional[Box] = None,
              tmp_dir: Optional[str] = None,
              save_as_rgb: bool = False,
              raster_output: bool = True,
@@ -373,6 +374,8 @@ def save(self,
             crs_transformer (CRSTransformer): CRSTransformer to configure CRS
                 and affine transform of the output GeoTiff.
             class_config (ClassConfig): The ClassConfig.
+            bbox (Optional[Box]): User-specified crop of the extent. Must be
+                provided if the corresponding RasterSource has bbox != extent.
             tmp_dir (Optional[str], optional): Temporary directory to use. If
                 None, will be auto-generated. Defaults to None.
             save_as_rgb (bool, optional): If True, Saves labels as an RGB
@@ -397,6 +400,7 @@ def save(self,
             uri=uri,
             crs_transformer=crs_transformer,
             class_config=class_config,
+            bbox=bbox,
             tmp_dir=tmp_dir,
             save_as_rgb=save_as_rgb,
             discrete_output=raster_output,
@@ -529,6 +533,7 @@ def save(self,
              uri: str,
              crs_transformer: 'CRSTransformer',
              class_config: 'ClassConfig',
+             bbox: Optional[Box] = None,
              tmp_dir: Optional[str] = None,
              save_as_rgb: bool = False,
              discrete_output: bool = True,
@@ -547,6 +552,8 @@ def save(self,
             crs_transformer (CRSTransformer): CRSTransformer to configure CRS
                 and affine transform of the output GeoTiff(s).
             class_config (ClassConfig): The ClassConfig.
+            bbox (Optional[Box]): User-specified crop of the extent. Must be
+                provided if the corresponding RasterSource has bbox != extent.
             tmp_dir (Optional[str], optional): Temporary directory to use. If
                 None, will be auto-generated. Defaults to None.
             save_as_rgb (bool, optional): If True, saves labels as an RGB
@@ -577,6 +584,7 @@ def save(self,
             uri=uri,
             crs_transformer=crs_transformer,
             class_config=class_config,
+            bbox=bbox,
             tmp_dir=tmp_dir,
             save_as_rgb=save_as_rgb,
             discrete_output=discrete_output,

rastervision_core/rastervision/core/data/label_store/chip_classification_geojson_store.py

@@ -30,7 +30,8 @@ def __init__(self,
                 in GeoJSON file to pixel coords.
             bbox (Optional[Box], optional): User-specified crop of the extent.
                 If provided, only labels falling inside it are returned by
-                :meth:`.ChipClassificationGeoJSONStore.get_labels`.
+                :meth:`.ChipClassificationGeoJSONStore.get_labels`. Must be
+                provided if the corresponding RasterSource has bbox != extent.
         """
         self.uri = uri
         self.class_config = class_config
@@ -51,7 +52,8 @@ def save(self, labels: ChipClassificationLabels) -> None:
             class_ids,
             self.crs_transformer,
             self.class_config,
-            scores=scores)
+            scores=scores,
+            bbox=self.bbox)
         json_to_file(geojson, self.uri)
 
     def get_labels(self) -> ChipClassificationLabels:

rastervision_core/rastervision/core/data/label_store/object_detection_geojson_store.py

@@ -32,7 +32,8 @@ def __init__(self,
                 in GeoJSON file to pixel coords.
             bbox (Optional[Box], optional): User-specified crop of the extent.
                 If provided, only labels falling inside it are returned by
-                :meth:`.ObjectDetectionGeoJSONStore.get_labels`.
+                :meth:`.ObjectDetectionGeoJSONStore.get_labels`. Must be
+                provided if the corresponding RasterSource has bbox != extent.
         """
         self.uri = uri
         self.class_config = class_config

rastervision_core/rastervision/core/data/label_store/semantic_segmentation_label_store.py

@@ -4,6 +4,7 @@
 
 import numpy as np
 import rasterio as rio
+import rasterio.windows as rio_windows
 from tqdm.auto import tqdm
 
 from rastervision.pipeline.file_system import (
@@ -58,7 +59,8 @@ def __init__(
             class_config (ClassConfig): Class config.
             bbox (Optional[Box], optional): User-specified crop of the extent.
                 If provided, only labels falling inside it are returned by
-                :meth:`.SemanticSegmentationLabelStore.get_labels`.
+                :meth:`.SemanticSegmentationLabelStore.get_labels`. Must be
+                provided if the corresponding RasterSource has bbox != extent.
             tmp_dir (Optional[str], optional): Temporary directory to use. If
                 None, will be auto-generated. Defaults to None.
             vector_outputs (Optional[Sequence[VectorOutputConfig]], optional):
@@ -207,11 +209,17 @@ def save(self,
         make_dir(local_root)
 
         height, width = labels.extent.size
+        if self.bbox is not None:
+            bbox_rio_window = self.bbox.rasterio_format()
+            transform = rio_windows.transform(bbox_rio_window,
+                                              self.crs_transformer.transform)
+        else:
+            transform = self.crs_transformer.transform
         out_profile = dict(
             driver='GTiff',
             height=height,
             width=width,
-            transform=self.crs_transformer.transform,
+            transform=transform,
             crs=self.crs_transformer.image_crs,
             blockxsize=min(self.rasterio_block_size, width),
             blockysize=min(self.rasterio_block_size, height))
@@ -257,6 +265,7 @@ def write_smooth_raster_output(
         out_profile.update(dict(count=num_bands, dtype=dtype))
 
         extent = labels.extent
+
         with rio.open(scores_path, 'w', **out_profile) as ds:
             windows = [Box.from_rasterio(w) for _, w in ds.block_windows(1)]
             with tqdm(windows, desc='Saving pixel scores') as bar:
@@ -310,7 +319,10 @@ def write_vector_outputs(self, labels: SemanticSegmentationLabels,
                 bar.set_postfix(vo.dict())
                 class_mask = (label_arr == vo.class_id).astype(np.uint8)
                 polys = vo.vectorize(class_mask)
-                polys = [self.crs_transformer.pixel_to_map(p) for p in polys]
+                polys = [
+                    self.crs_transformer.pixel_to_map(p, bbox=self.bbox)
+                    for p in polys
+                ]
                 geojson = geoms_to_geojson(polys)
                 out_uri = vo.get_uri(vector_output_dir, self.class_config)
                 json_to_file(geojson, out_uri)

rastervision_core/rastervision/core/data/label_store/utils.py

@@ -16,8 +16,8 @@ def boxes_to_geojson(
         class_ids: Sequence[int],
         crs_transformer: 'CRSTransformer',
         class_config: 'ClassConfig',
-        scores: Optional[Sequence[Union[float, Sequence[float]]]] = None
-) -> dict:
+        scores: Optional[Sequence[Union[float, Sequence[float]]]] = None,
+        bbox: Optional['Box'] = None) -> dict:
     """Convert boxes and associated data into a GeoJSON dict.
 
     Args:
@@ -30,6 +30,8 @@ def boxes_to_geojson(
             Optional list of score or scores. If floats (one for each box),
             property name will be "score". If lists of floats, property name
             will be "scores". Defaults to None.
+        bbox (Optional[Box]): User-specified crop of the extent. Must be
+            provided if the corresponding RasterSource has bbox != extent.
 
     Returns:
         dict: Serialized GeoJSON.
@@ -46,7 +48,10 @@ def boxes_to_geojson(
             boxes,
             desc='Transforming boxes to map coords',
             delay=PROGRESSBAR_DELAY_SEC) as bar:
-        geoms = [crs_transformer.pixel_to_map(box.to_shapely()) for box in bar]
+        geoms = [
+            crs_transformer.pixel_to_map(box.to_shapely(), bbox=bbox)
+            for box in bar
+        ]
 
     # add box properties (ID and name of predicted class)
     with tqdm(

rastervision_core/rastervision/core/data/raster_transformer/stats_transformer.py

@@ -97,7 +97,8 @@ def transform(self,
     def from_raster_sources(cls,
                             raster_sources: List['RasterSource'],
                             sample_prob: Optional[float] = 0.1,
-                            max_stds: float = 3.) -> 'StatsTransformer':
+                            max_stds: float = 3.,
+                            chip_sz: int = 300) -> 'StatsTransformer':
         """Build with stats from the given raster sources.
 
         Args:
@@ -113,7 +114,10 @@ def from_raster_sources(cls,
             StatsTransformer: A StatsTransformer.
         """
         stats = RasterStats()
-        stats.compute(raster_sources=raster_sources, sample_prob=sample_prob)
+        stats.compute(
+            raster_sources=raster_sources,
+            sample_prob=sample_prob,
+            chip_sz=chip_sz)
         stats_transformer = StatsTransformer.from_raster_stats(
             stats, max_stds=max_stds)
         return stats_transformer
@@ -162,4 +166,4 @@ def stats(self) -> RasterStats:
 
     def __repr__(self) -> str:
         return repr_with_args(
-            self, means=self.means, std=self.stds, max_stds=self.max_stds)
+            self, means=self.means, stds=self.stds, max_stds=self.max_stds)

rastervision_core/rastervision/core/rv_pipeline/rv_pipeline.py

@@ -17,7 +17,7 @@
 from rastervision.core.rv_pipeline import TRAIN, VALIDATION
 from rastervision.pipeline.file_system.utils import (
     download_if_needed, zipdir, get_local_path, upload_or_copy, make_dir,
-    sync_to_dir, file_exists)
+    sync_from_dir, file_exists)
 
 log = logging.getLogger(__name__)
 
@@ -252,8 +252,8 @@ def bundle(self):
                 shutil.copy(path, join(bundle_dir, fn))
 
             if file_exists(self.config.analyze_uri, include_dir=True):
-                sync_to_dir(self.config.analyze_uri, join(
-                    bundle_dir, 'analyze'))
+                analyze_dst = join(bundle_dir, 'analyze')
+                sync_from_dir(self.config.analyze_uri, analyze_dst)
 
             path = download_if_needed(self.config.get_config_uri(), tmp_dir)
             shutil.copy(path, join(bundle_dir, 'pipeline-config.json'))

rastervision_core/requirements.txt

@@ -2,7 +2,7 @@ rastervision_pipeline==0.21.2
 shapely==2.0.1
 geopandas==0.13.2
 numpy==1.25.0
-pillow==9.3.0
+pillow==10.0.1
 pyproj==3.4.0
 rasterio==1.3.7
 pystac==1.6.1

rastervision_pytorch_backend/rastervision/pytorch_backend/examples/semantic_segmentation/spacenet_vegas.py

@@ -98,7 +98,9 @@ def build_scene(spacenet_cfg: SpacenetConfig,
     label_uri = spacenet_cfg.get_geojson_uri(id)
 
     raster_source = RasterioSourceConfig(
-        uris=[image_uri], channel_order=channel_order)
+        uris=[image_uri],
+        channel_order=channel_order,
+        transformers=[StatsTransformerConfig()])
 
     # Set a line buffer to convert line strings to polygons.
     vector_source = GeoJSONVectorSourceConfig(

rastervision_pytorch_backend/rastervision/pytorch_backend/examples/test.py

@@ -11,13 +11,13 @@
     file_to_json, sync_from_dir, upload_or_copy, download_or_copy, file_exists,
     sync_to_dir, NotReadableError, download_if_needed)
 
-NEW_VERSION = '0.21'
+NEW_VERSION_FULL = '0.21.2'
 NEW_VERSION_MAJOR_MINOR = '0.21'
 
 EXAMPLES_MODULE_ROOT = 'rastervision.pytorch_backend.examples'
 EXAMPLES_PATH_ROOT = '/opt/src/rastervision_pytorch_backend/rastervision/pytorch_backend/examples'  # noqa
-REMOTE_PROCESSED_ROOT = f's3://raster-vision/examples/{NEW_VERSION}/processed-data'
-REMOTE_OUTPUT_ROOT = f's3://raster-vision/examples/{NEW_VERSION}/output'
+REMOTE_PROCESSED_ROOT = f's3://raster-vision/examples/{NEW_VERSION_FULL}/processed-data'
+REMOTE_OUTPUT_ROOT = f's3://raster-vision/examples/{NEW_VERSION_FULL}/output'
 LOCAL_RAW_ROOT = '/opt/data/raw-data'
 LOCAL_PROCESSED_ROOT = '/opt/data/examples/processed-data'
 LOCAL_OUTPUT_ROOT = '/opt/data/examples/output'

rastervision_pytorch_learner/rastervision/pytorch_learner/dataset/transform.py

@@ -227,7 +227,8 @@ def semantic_segmentation_transformer(
             y = np.array(y)
             out = apply_transform(transform, image=x, mask=y)
             x, y = out['image'], out['mask']
-            y = y.astype(int)
+    if y is not None:
+        y = y.astype(int)
     return x, y