SoilGrids exception will kill materialize

[robmarkcole]

The error arises in the soilgrids client library (gantian127/soilgrids#13) but surfaces during materialize and will kill the job

format=2026-01-20 11:14:37,502 loglevel=INFO   logger=rslearn.dataset.manage materialize_window() L436  Materializing 11 item groups in layer sentinel2 via data source
format=2026-01-20 11:14:41,071 loglevel=ERROR  logger=rslearn.main __call__() L756  Error materializing windows: WCS sever error 
<?xml version='1.0' encoding="UTF-8" ?>
<ServiceExceptionReport version="1.2.0"
xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wcs/1.0.0/OGC-exception.xsd">
  <ServiceException>msImageCreate(): Image handling error. Attempt to allocate raw image failed, out of memory.
  </ServiceException>
</ServiceExceptionReport>

 16%|██████████████████████████▎                                                                                                                                      | 35/214 [11:25<58:28, 19.60s/it]
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/main.py", line 748, in __call__
    return materialize_dataset_windows(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/manage.py", line 494, in materialize_dataset_windows
    window_summary = materialize_window(
                     ^^^^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/manage.py", line 439, in materialize_window
    retry(
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/manage.py", line 77, in retry
    return fn()
           ^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/manage.py", line 440, in <lambda>
    fn=lambda: data_source.materialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/data_sources/soilgrids.py", line 325, in materialize
    RasterMaterializer().materialize(
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/materialize.py", line 517, in materialize
    composite = build_composite(
                ^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/materialize.py", line 469, in build_composite
    return compositing_methods[compositing_method](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/materialize.py", line 211, in build_first_valid_composite
    read_raster_window_from_tiles(
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/dataset/materialize.py", line 93, in read_raster_window_from_tiles
    src = tile_store.read_raster(
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/tile_stores/tile_store.py", line 258, in read_raster
    return self.tile_store.read_raster(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/data_sources/soilgrids.py", line 266, in read_raster
    self._download_geotiff(
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/data_sources/soilgrids.py", line 210, in _download_geotiff
    client.get_coverage_data(**kwargs)
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/soilgrids/soilgrids.py", line 186, in get_coverage_data
    raise Exception('WCS sever error \n{}'.format(error_info))
Exception: WCS sever error 
<?xml version='1.0' encoding="UTF-8" ?>
<ServiceExceptionReport version="1.2.0"
xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wcs/1.0.0/OGC-exception.xsd">
  <ServiceException>msImageCreate(): Image handling error. Attempt to allocate raw image failed, out of memory.
  </ServiceException>
</ServiceExceptionReport>

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/rce/olmoearth-crop-yield/.venv/bin/rslearn", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/main.py", line 866, in main
    handler()
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/main.py", line 804, in dataset_materialize
    apply_on_windows_args(fn, args)
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/main.py", line 386, in apply_on_windows_args
    apply_on_windows(
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/rslearn/main.py", line 378, in apply_on_windows
    for _ in tqdm.tqdm(outputs, total=num_batches):
  File "/home/rce/olmoearth-crop-yield/.venv/lib/python3.12/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 873, in next
    raise value
Exception: WCS sever error 
<?xml version='1.0' encoding="UTF-8" ?>
<ServiceExceptionReport version="1.2.0"
xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wcs/1.0.0/OGC-exception.xsd">
  <ServiceException>msImageCreate(): Image handling error. Attempt to allocate raw image failed, out of memory.
  </ServiceException>
</ServiceExceptionReport>

This is only impacting a small number of locations but manages to terminate the entire materialize job. How to handle?

[robmarkcole]

created gantian127/soilgrids#14 so there will be a dedicated exception to catch

[favyen2]

If it is a transient error, this can be handled with --retry-max-attempts 10 --retry-backoff-seconds 5 to retry a few times.

If it is a permanent error, it can be handled with --ignore-errors. That will continue processing other windows even if some of them fail to materialize.

I do think it would be best to always run all steps with ignoring errors and then show summary at the end with any errors that popped up, but currently it doesn't do that.