Skip to content

Commit b54ea6c

Browse files
AdeelHAdeelH
authored
Merge pull request #1857 from AdeelH/rel
Pre-release fixes and improvements
2 parents ee3fbef + 5121058 commit b54ea6c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+990
-437
lines changed

Diff for: ‎.coveragerc

+1
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,4 @@ exclude_lines =
2424
@(?:abc\.)?abstractmethod
2525
@(?:abc\.)?abstractproperty
2626
@overload
27+
pass

Diff for: ‎Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
ARG UBUNTU_VERSION=22.04
21
ARG CUDA_VERSION
2+
ARG UBUNTU_VERSION
33
FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-runtime-ubuntu${UBUNTU_VERSION}
44

55
# wget: needed below to install conda

Diff for: ‎docker/build

+22-7
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ function usage() {
1313
Build Docker images.
1414
1515
Options:
16-
--arm64 will build image for arm64 architecture
16+
--arm64 Build image for arm64 architecture.
1717
"
1818
}
1919

@@ -27,13 +27,28 @@ then
2727

2828
PLATFORM="amd64"
2929
IMAGE_EXT=""
30-
if [ "${1:-}" = "--arm64" ]
31-
then
32-
PLATFORM="arm64"
33-
IMAGE_EXT="-arm64"
34-
fi
30+
CUDA_VERSION="12.1.1"
31+
UBUNTU_VERSION="22.04"
32+
33+
while [[ $# -gt 0 ]]
34+
do
35+
case "$1" in
36+
--arm64)
37+
PLATFORM="arm64"
38+
IMAGE_EXT="-arm64"
39+
shift
40+
;;
41+
*)
42+
echo "Unknown option: $1"
43+
usage
44+
exit 1
45+
;;
46+
esac
47+
done
3548

3649
DOCKER_BUILDKIT=1 docker build \
37-
--platform linux/${PLATFORM} --build-arg CUDA_VERSION="11.7.1" \
50+
--platform linux/${PLATFORM} \
51+
--build-arg CUDA_VERSION="${CUDA_VERSION}" \
52+
--build-arg UBUNTU_VERSION="${UBUNTU_VERSION}" \
3853
-t raster-vision-pytorch${IMAGE_EXT} -f Dockerfile .
3954
fi

Diff for: ‎docker/run

+1-1
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ do
107107
;;
108108
--jupyter-lab)
109109
find_free_port_in_range 8888 9999
110-
JUPYTER="-v ${RASTER_VISION_NOTEBOOK_DIR}:/opt/notebooks -v ${HOME}/.jupyter:/root/.jupyter:ro -p $FREE_PORT:$FREE_PORT"
110+
JUPYTER="-v ${RASTER_VISION_NOTEBOOK_DIR}:/opt/notebooks -v ${HOME}/.jupyter:/root/.jupyter -p $FREE_PORT:$FREE_PORT"
111111
# run jupyter lab in the background
112112
CMD=(/bin/bash -c "jupyter lab --ip 0.0.0.0 --port $FREE_PORT --no-browser --allow-root --notebook-dir=/opt/notebooks & bash")
113113
echo "Starting Jupyter Lab server at 0.0.0.0:$FREE_PORT. This may take a few seconds."

Diff for: ‎docs/release.rst

+32-20
Original file line numberDiff line numberDiff line change
@@ -9,32 +9,44 @@ Minor or Major Version Release
99
------------------------------
1010

1111
#. It's a good idea to update any major dependencies before the release.
12-
#. Update the docs if needed. See the `docs README <{{ repo }}/docs/README.md>`__ for instructions.
13-
#. Checkout the ``master`` branch, re-build the docker image (``docker/build``), and push it to ECR (``docker/ecr_publish``).
14-
#. Execute all `tutorial notebooks <{{ repo }}/docs/usage/tutorials/>`__ and make sure they work correctly. Do not commit output changes unless code behavior has changed.
15-
#. Run all :ref:`rv examples` and check that evaluation metrics are close to the scores from the last release. (For each example, there should be a link to a JSON file with the evaluation metrics from the last release.) This stage often uncovers bugs, and is the most time consuming part of the release process. There is a `script <{{ repo_examples }}/test.py>`__ to help run the examples and collect their outputs. See the associated `README <{{ repo_examples }}/README.md>`__ for details.
16-
#. Collect all model bundles, and check that they work with the ``predict`` command and sanity check output in QGIS.
17-
#. Update the :ref:`model zoo` by uploading model bundles and sample images to the right place on S3. If you use the ``collect`` command (`described here <{{ repo_examples }}/README.md>`__), you should be able to sync the ``collect_dir`` to ``s3://azavea-research-public-data/raster-vision/examples/model-zoo-<version>``.
18-
#. Update the notebooks that use models from the model zoo so that they use the latest version and re-run.
19-
#. Update `tiny_spacenet.py <{{ repo_examples }}/tiny_spacenet.py>`__ if needed and ensure the line numbers in every ``literalinclude`` of that file are correct. Tip: you can find all instances by searching the repo using the regex: ``\.\. literalinclude:: .+tiny_spacenet\.py$``.
20-
#. Test :ref:`setup` and :ref:`quickstart` instructions and make sure they work.
21-
#. Test examples from :ref:`pipelines plugins`.
12+
#. Test examples:
2213

23-
.. code-block:: console
14+
#. Checkout the ``master`` branch, re-build the docker image (``docker/build``), and push it to ECR (``docker/ecr_publish``).
15+
#. Follow the instructions in `this README <{{ repo_examples }}/README.md>`__ to do the following:
16+
17+
#. Run all :ref:`rv examples` and check that evaluation metrics are close to the scores from the last release. (For each example, there should be a link to a JSON file with the evaluation metrics from the last release.) This stage often uncovers bugs, and is the most time consuming part of the release process.
18+
#. Collect all model bundles, and check that they work with the ``predict`` command and sanity check output in QGIS.
19+
#. Update the :ref:`model zoo` by uploading model bundles and sample images to the right place on S3. If you use the ``collect`` command (`see <{{ repo_examples }}/README.md>`__), you should be able to sync the ``collect_dir`` to ``s3://azavea-research-public-data/raster-vision/examples/model-zoo-<version>``.
20+
#. Screenshot the outputs of the ``compare`` command (for each example) and include them in the PR described below.
2421

25-
rastervision run inprocess rastervision.pipeline_example_plugin1.config1 -a root_uri /opt/data/pipeline-example/1/ --splits 2
26-
rastervision run inprocess rastervision.pipeline_example_plugin1.config2 -a root_uri /opt/data/pipeline-example/2/ --splits 2
27-
rastervision run inprocess rastervision.pipeline_example_plugin2.config3 -a root_uri /opt/data/pipeline-example/3/ --splits 2
22+
#. Test notebooks:
2823

29-
#. Test examples from :ref:`bootstrap`.
24+
#. Update the `tutorial notebooks <{{ repo }}/docs/usage/tutorials/>`__ that use models from the model zoo so that they use the latest version.
25+
#. Execute all `tutorial notebooks <{{ repo }}/docs/usage/tutorials/>`__ and make sure they work correctly. Do not commit output changes unless code behavior has changed.
3026

31-
.. code-block:: console
27+
#. Test/update docs:
28+
29+
#. Update the docs if needed. See the `docs README <{{ repo }}/docs/README.md>`__ for instructions.
30+
#. Update `tiny_spacenet.py <{{ repo_examples }}/tiny_spacenet.py>`__ if needed and ensure the line numbers in every ``literalinclude`` of that file are correct. Tip: you can find all instances by searching the repo using the regex: ``\.\. literalinclude:: .+tiny_spacenet\.py$``.
31+
#. Test :ref:`setup` and :ref:`quickstart` instructions and make sure they work.
32+
#. Test examples from :ref:`pipelines plugins`.
33+
34+
.. code-block:: console
35+
36+
rastervision run inprocess rastervision.pipeline_example_plugin1.config1 -a root_uri /opt/data/pipeline-example/1/ --splits 2
37+
rastervision run inprocess rastervision.pipeline_example_plugin1.config2 -a root_uri /opt/data/pipeline-example/2/ --splits 2
38+
rastervision run inprocess rastervision.pipeline_example_plugin2.config3 -a root_uri /opt/data/pipeline-example/3/ --splits 2
39+
40+
#. Test examples from :ref:`bootstrap`.
41+
42+
.. code-block:: console
43+
44+
cookiecutter /opt/src/cookiecutter_template
3245
33-
cookiecutter /opt/src/cookiecutter_template
46+
#. Update the `the changelog <{{ repo }}/docs/changelog.rst>`__, and point out API changes.
47+
#. Fix any broken badges on the GitHub repo readme.
3448

35-
#. Update the `the changelog <{{ repo }}/docs/changelog.rst>`__, and point out API changes.
36-
#. Fix any broken badges on the GitHub repo readme.
37-
#. Update the version number. This occurs in several places, so it's best to do this with a find and replace over the entire repo.
49+
#. Update the version number. This occurs in several places, so it's best to do this with a find-and-replace over the entire repo.
3850
#. Make a PR to the ``master`` branch with the preceding updates. In the PR, there should be a link to preview the docs. Check that they are building and look correct.
3951
#. Make a git branch with the version as the name, and push to GitHub.
4052
#. Ensure that the docs are building correctly for the new version branch on `readthedocs <https://readthedocs.org/projects/raster-vision/>`_. You will need to have admin access on your RTD account. Once the branch is building successfully, Under *Versions -> Activate a Version*, you can activate the version to add it to the sidebar of the docs for the latest version. (This might require manually triggering a rebuild of the docs.) Then, under *Admin -> Advanced Settings*, change the default version to the new version.

Diff for: ‎docs/usage/tutorials/pred_and_eval_ss.ipynb

+1-1
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@
4343
},
4444
"source": [
4545
"Load a :class:`Learner` with a trained model from bundle -- :meth:`.Learner.from_model_bundle`\n",
46-
"---------------------------------------------------------------------------------------------"
46+
"----------------------------------------------------------------------------------------------"
4747
]
4848
},
4949
{

Diff for: ‎rastervision_core/rastervision/core/box.py

+21-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
1-
from typing import TYPE_CHECKING, Callable, Dict, Union, Tuple, Optional, List
1+
from typing import (TYPE_CHECKING, Callable, Dict, List, Optional, Sequence,
2+
Tuple, Union)
23
from typing_extensions import Literal
34
from pydantic import PositiveInt as PosInt, conint
45
import math
@@ -484,3 +485,22 @@ def within_aoi(window: 'Box', aoi_polygons: List[Polygon]) -> bool:
484485
if w.within(polygon):
485486
return True
486487
return False
488+
489+
def __contains__(self, query: Union['Box', Sequence]) -> bool:
490+
"""Check if box or point is contained within this box.
491+
492+
Args:
493+
query: Box or single point (x, y).
494+
495+
Raises:
496+
NotImplementedError: if query is not a Box or tuple/list.
497+
"""
498+
if isinstance(query, Box):
499+
ymin, xmin, ymax, xmax = query
500+
return (ymin >= self.ymin and xmin >= self.xmin
501+
and ymax <= self.ymax and xmax <= self.xmax)
502+
elif isinstance(query, (tuple, list)):
503+
x, y = query
504+
return self.xmin <= x <= self.xmax and self.ymin <= y <= self.ymax
505+
else:
506+
raise NotImplementedError()

Diff for: ‎rastervision_core/rastervision/core/data/label/chip_classification_labels.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ def get_cell_scores(self, cell: Box) -> Optional[Sequence[float]]:
118118
"""
119119
result = self.cell_to_label.get(cell)
120120
if result is not None:
121-
return result.score
121+
return result.scores
122122
else:
123123
return None
124124

Diff for: ‎rastervision_core/rastervision/core/data/label/object_detection_labels.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,7 @@ def get_class_ids(self) -> np.ndarray:
163163
def __len__(self) -> int:
164164
return self.boxlist.get().shape[0]
165165

166-
def __str__(self) -> str:
166+
def __str__(self) -> str: # prama: no cover
167167
return str(self.boxlist.get())
168168

169169
def to_boxlist(self) -> NpBoxList:

Diff for: ‎rastervision_core/rastervision/core/data/label/semantic_segmentation_labels.py

+17-24
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
1-
from typing import (TYPE_CHECKING, Any, Iterable, List, Optional, Sequence,
2-
Union)
1+
from typing import (TYPE_CHECKING, Any, Iterable, List, Optional, Sequence)
32
from abc import abstractmethod
43

54
import numpy as np
@@ -23,8 +22,8 @@ def __init__(self, extent: Box, num_classes: int, dtype: np.dtype):
2322
"""Constructor.
2423
2524
Args:
26-
extent (Box): The extent of the region to which
27-
the labels belong, in global coordinates.
25+
extent (Box): The extent of the region to which the labels belong,
26+
in global coordinates.
2827
num_classes (int): Number of classes.
2928
"""
3029
self.extent = extent
@@ -143,9 +142,8 @@ def transform_shape(x, y, z=None):
143142
del self[window]
144143

145144
@classmethod
146-
def make_empty(cls, extent: Box, num_classes: int, smooth: bool = False
147-
) -> Union['SemanticSegmentationDiscreteLabels',
148-
'SemanticSegmentationSmoothLabels']:
145+
def make_empty(cls, extent: Box, num_classes: int,
146+
smooth: bool = False) -> 'SemanticSegmentationLabels':
149147
"""Instantiate an empty instance.
150148
151149
Args:
@@ -157,8 +155,7 @@ def make_empty(cls, extent: Box, num_classes: int, smooth: bool = False
157155
SemanticSegmentationDiscreteLabels object. Defaults to False.
158156
159157
Returns:
160-
Union[SemanticSegmentationDiscreteLabels,
161-
SemanticSegmentationSmoothLabels]: If smooth=True, returns a
158+
SemanticSegmentationLabels: If smooth=True, returns a
162159
SemanticSegmentationSmoothLabels. Otherwise, a
163160
SemanticSegmentationDiscreteLabels.
164161
@@ -174,15 +171,14 @@ def make_empty(cls, extent: Box, num_classes: int, smooth: bool = False
174171
extent=extent, num_classes=num_classes)
175172

176173
@classmethod
177-
def from_predictions(cls,
178-
windows: Iterable['Box'],
179-
predictions: Iterable[Any],
180-
extent: Box,
181-
num_classes: int,
182-
smooth: bool = False,
183-
crop_sz: Optional[int] = None
184-
) -> Union['SemanticSegmentationDiscreteLabels',
185-
'SemanticSegmentationSmoothLabels']:
174+
def from_predictions(
175+
cls,
176+
windows: Iterable['Box'],
177+
predictions: Iterable[Any],
178+
extent: Box,
179+
num_classes: int,
180+
smooth: bool = False,
181+
crop_sz: Optional[int] = None) -> 'SemanticSegmentationLabels':
186182
"""Instantiate from windows and their corresponding predictions.
187183
188184
Args:
@@ -202,8 +198,7 @@ def from_predictions(cls,
202198
windows. Defaults to None.
203199
204200
Returns:
205-
Union[SemanticSegmentationDiscreteLabels,
206-
SemanticSegmentationSmoothLabels]: If smooth=True, returns a
201+
SemanticSegmentationLabels: If smooth=True, returns a
207202
SemanticSegmentationSmoothLabels. Otherwise, a
208203
SemanticSegmentationDiscreteLabels.
209204
"""
@@ -349,8 +344,7 @@ def from_predictions(cls,
349344
extent: Box,
350345
num_classes: int,
351346
crop_sz: Optional[int] = None
352-
) -> Union['SemanticSegmentationDiscreteLabels',
353-
'SemanticSegmentationSmoothLabels']:
347+
) -> 'SemanticSegmentationDiscreteLabels':
354348
labels = cls.make_empty(extent, num_classes)
355349
labels.add_predictions(windows, predictions, crop_sz=crop_sz)
356350
return labels
@@ -522,8 +516,7 @@ def from_predictions(cls,
522516
extent: Box,
523517
num_classes: int,
524518
crop_sz: Optional[int] = None
525-
) -> Union['SemanticSegmentationDiscreteLabels',
526-
'SemanticSegmentationSmoothLabels']:
519+
) -> 'SemanticSegmentationSmoothLabels':
527520
labels = cls.make_empty(extent, num_classes)
528521
labels.add_predictions(windows, predictions, crop_sz=crop_sz)
529522
return labels

Diff for: ‎rastervision_core/rastervision/core/data/label_source/semantic_segmentation_label_source.py

+8-17
Original file line numberDiff line numberDiff line change
@@ -7,24 +7,12 @@
77
from rastervision.core.data.label import SemanticSegmentationLabels
88
from rastervision.core.data.label_source.label_source import LabelSource
99
from rastervision.core.data.raster_source import RasterSource
10+
from rastervision.core.data.utils import pad_to_window_size
1011

1112
if TYPE_CHECKING:
1213
from rastervision.core.data import CRSTransformer
1314

1415

15-
def fill_edge(label_arr: np.ndarray, window: Box, extent: Box,
16-
fill_value: int) -> np.ndarray:
17-
"""If window goes over the edge of the extent, buffer with fill_value."""
18-
if window.ymax <= extent.ymax and window.xmax <= extent.xmax:
19-
return label_arr
20-
21-
x = np.full(window.size, fill_value)
22-
ylim = extent.ymax - window.ymin
23-
xlim = extent.xmax - window.xmin
24-
x[0:ylim, 0:xlim] = label_arr[0:ylim, 0:xlim]
25-
return x
26-
27-
2816
class SemanticSegmentationLabelSource(LabelSource):
2917
"""A read-only label source for semantic segmentation."""
3018

@@ -101,8 +89,9 @@ def get_label_arr(self, window: Optional[Box] = None) -> np.ndarray:
10189
of the null class as defined by the class_config.
10290
10391
Args:
104-
window (Optional[Box], optional): Window to get labels for. If
105-
None, returns a label array covering the full extent of the scene.
92+
window (Optional[Box], optional): Window (in pixel coords) to get
93+
labels for. If None, returns a label array covering the full
94+
extent of the scene.
10695
10796
Returns:
10897
np.ndarray: Label array.
@@ -113,8 +102,10 @@ def get_label_arr(self, window: Optional[Box] = None) -> np.ndarray:
113102
label_arr = self.raster_source.get_chip(window)
114103
if label_arr.ndim == 3:
115104
label_arr = np.squeeze(label_arr, axis=2)
116-
label_arr = fill_edge(label_arr, window, self.extent,
117-
self.class_config.null_class_id)
105+
h, w = label_arr.shape
106+
if h < window.height or w < window.width:
107+
label_arr = pad_to_window_size(label_arr, window, self.extent,
108+
self.class_config.null_class_id)
118109
return label_arr
119110

120111
@property

Diff for: ‎rastervision_core/rastervision/core/data/label_store/semantic_segmentation_label_store.py

+4-15
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
from rastervision.core.data.label_source import SemanticSegmentationLabelSource
1818
from rastervision.core.data.raster_transformer import RGBClassTransformer
1919
from rastervision.core.data.raster_source import RasterioSource
20+
from rastervision.core.data.utils import write_window
2021

2122
if TYPE_CHECKING:
2223
from rastervision.core.data import (VectorOutputConfig,
@@ -266,7 +267,8 @@ def write_smooth_raster_output(
266267
score_arr = self._scores_to_uint8(score_arr)
267268
else:
268269
score_arr = score_arr.astype(dtype)
269-
self._write_array(ds, window, score_arr)
270+
score_arr = score_arr.transpose(1, 2, 0)
271+
write_window(ds, score_arr, window)
270272

271273
# save pixel hits too
272274
np.save(hits_path, labels.pixel_hits)
@@ -291,8 +293,7 @@ def write_discrete_raster_output(
291293
if self.class_transformer is not None:
292294
label_arr = self.class_transformer.class_to_rgb(
293295
label_arr)
294-
label_arr = label_arr.transpose(2, 0, 1)
295-
self._write_array(ds, window, label_arr)
296+
write_window(ds, label_arr, window)
296297

297298
def write_vector_outputs(self, labels: SemanticSegmentationLabels,
298299
vector_output_dir: str) -> None:
@@ -314,18 +315,6 @@ def write_vector_outputs(self, labels: SemanticSegmentationLabels,
314315
out_uri = vo.get_uri(vector_output_dir, self.class_config)
315316
json_to_file(geojson, out_uri)
316317

317-
def _write_array(self, dataset: rio.DatasetReader, window: Box,
318-
arr: np.ndarray) -> None:
319-
"""Write array out to a rasterio dataset. Array must be of shape
320-
(C, H, W).
321-
"""
322-
rio_window = window.rasterio_format()
323-
if len(arr.shape) == 2:
324-
dataset.write_band(1, arr, window=rio_window)
325-
else:
326-
for i, band in enumerate(arr, start=1):
327-
dataset.write_band(i, band, window=rio_window)
328-
329318
def _clip_to_extent(self,
330319
extent: Box,
331320
window: Box,

0 commit comments

Comments
 (0)