DTED regression for voids with 1.3.59

[kjurka]

This PR changes the handling of voids in DTED:
https://github.com/ngageoint/sarpy/pull/535/files#diff-7279fd3655c7d057eae1bb04161914b0bc14ae1decdbc06e706b048021ed9f69L373

I have a DTED dataset that has voids wherever there is water. I can't share that here, but I was able to get non void filled DTED tiles from USGS. https://earthexplorer.usgs.gov/ These have voids not over water, but possibly places where they had processing problems. I pulled n33_w119_3arc_v1.dt1 I set up a correctly laid out dted folder and geoid and then ran the below code:

import numpy as np
from sarpy.io.DEM.DTED import DTEDInterpolator, DTEDList

dted_paths = '/share/home/kjurka@emagsys.com/tmp/void_dem_tiles'
ll = np.array([33.938923,-118.388885])

dted_list = DTEDList(dted_paths)
dem_interpolator = DTEDInterpolator.from_reference_point(ll, dted_list, pad_value=0.05)

print(dem_interpolator.get_elevation_hae(ll[0], ll[1]))

With current 1.3.59, I get an elevation of -32803.00222977538 while with 1.3.58, I get an elevation of -37.00222977537916.

The old code looked bogus to me, replacing voids with -1, but the new code doesn't handle them at all. I'd need to study my real DTED dataset more carefully to see if there are any voids other than for water, but if it was just water, replacing with 0 would be reasonable. Your thoughts or other ways to handle this?

[kjurka]

I misspoke about what the voids in my DTED dataset represent. They are not just water/ocean and appear in large/strange shapes/sizes. So replacing with 0 universally is not a good solution (but, I guess better than -32767 m). In the absence of any other data, I'm not sure what could be done. Perhaps if get_elevation_hae returned NaN or some other marker, I could replace with the SCP height.

[pressler-vsc]

Thanks for pointing this out @kjurka. I agree that fixing the reading of DTED nulls in #535 has some strange interactions with DTEDInterpolator and really uncovers the fact that sarpy.io.DEM.DTED is not currently set up to accommodate them. It's possible null-handling in DTED interpolation will be added as a feature in the future.
In the interim, I'd recommend accessing the underlying DTED values (e.g. raw_values = DTEDReader('/path/to/dted')[...]) such that you can then control the interpolation, null-handling, bounds-checking, etc. required for your application and DEM source. Does something like that make sense?

[kjurka]

What we've done for now is eliminate DTED tiles from our dataset that have any voids as it was a relatively small percentage in places that we don't care as much about.

Maybe the missing_error flag should be passed further down to the actual DTED reading. If missing_error is True, fail if there are voids. If missing_error is False, then replaces voids with 0, the same as if the tile wasn't there at all. That would be a little more consistent/sane than the current very negative elevations given.

[petersontex]

@dhaverporth I recommend we investigate this enhancement request and add it to our future release planning.

[MkRead]

We encountered this issue as well, and were thinking along the same lines as kjurka (12-Nov '24). If SARpy could test for bad DTED postings and either return NaN for those postings or use numpy masked arrays to flag them, that could help users accommodate DTED tiles with a variety of issues. Once bad postings are interpolated together with good ones, they can no longer be reliably detected with a threshold test. The existing interface could be maintained for backward compatibility by adding an optional keyword argument (e.g. return_mask) to get_elevation_hae. If it would be helpful to submit a code prototype / pull request, just let me know.

[jonlorax]

Investigating DTED and this issue, believe a partial solution would be in io/DEM/DTED.py::_repair_values().
Now investigating how best handle the void error and bubble something intelligent to the user.
So yes @MkRead we would like and then look into a prototype/PR is you have a solution idea. --Thanks

[jonlorax]

@MkRead can you submit your code prototype/Pull Request into this branch: https://github.com/ngageoint/sarpy/tree/DTED_fixes_569.
Thanks,
JohnO