Time-dependent problem at de Gerlache crater

[sampotter]

@steo85it

What has been done with this so far? I would like to push this along

[sampotter]

Added start of example to ./examples/gerlache in a156047.

This will download Mike's DEM from PGDA (here) and convert it to a .npy file which can be easily memory-mapped.

Next to do: make a Delaunay mesh of de Gerlache.

[steo85it]

I'm running these tests on my side, making meshes at different resolutions, etc ... here is a first plot at 500m/px for max values over 1 month (it seems that eps is not optimal for this resolution, and I had colorbars optimized for Mercury ^^). Else it seems to work (also including occultations from the outer topography).

I'm using https://github.com/steo85it/raster2mesh to generate the meshes, I can check if that can be easily called from your example.
Else, it's great that you set up a more structured example/test: we should run the example at different resolutions and compare results (which ones? fluxes? T?) to some ground-truth (non-compressed FF). Also, I should update raster2mesh to get (xmin,xmax,ymin,ymax) as input instead of selecting them manually on a plot (it's useful, but not great to repeat tests consistently).

[sampotter]

Gotcha. I like the look of these plots, they are interesting. How big are these meshes? And where are the scripts you're currently using? It will be helpful if you can share the code you use to call SPICE to generate sun positions.

[steo85it]

I run it with this script from the "applications repo" I shared some time ago.
I have to figure out how to unpack this (and especially the large proc_objfil.py which it calls) and include it in a structure such as the one you set up in the ./examples/gerlache (which is quite useful to have as a basis to do it).

[steo85it]

About the code for SPICE, these lines should be useful. These could be useful, too. (Btw, I made a few adjustments to paths, etc... today. I'll push them if useful, but it doesn't change much.) Also, if you have trouble understanding what I'm doing in that mess, feel free to ask (even though it would maybe be easier for me to unpack it).

[steo85it]

How big are these meshes?

This one was a small test. Barely 2500 triangles with a resolution of 500 mpp. Here is another one at 250 mpp (87000 triangles), but I should start to save some statistics (also, there are some imperfections, not sure if related to eps, embree vs CGAL, or what). Let me check if it's easy to integrate things in your cleaner structure.

Also, I noticed that I get in trouble loading point-clouds from Mike's 5mpp DEM (even before meshing my selection, at least on my laptop) ... it's probably best using a downsampled version of it to 20 or 50 mpp (anyway, we won't be able to use meshes down to 5mpp for a whole crater). I'll make one with gdal and load it.

[steo85it]

For a start, I'd replace the content of convert_geotiff_to_npy.py and make_mesh.py by these lines (in one or 2 files, whatever is clearer).
I'll do it after dinner, if that's ok for you (we can simplify by skipping the outer/inner mesh, if we don't plan to focus on occultations from far away topography - not sure that's relevant for the paper - and also set the coordinates to cut as input parameters). Then I'll be happy to have some help cleaning it up a bit. ;)

[steo85it]

See updates in gerlache branch in github/python-flux for the steps you mentioned (generate mesh - resolution is argument - and convert to cartesian).
Next steps (besides cleaning): generate FF and illuminate using tools from other examples.

[sampotter]

I see the imperfections. Since there are speckles in the first plot, I guess it has to do with using Embree?

So that I understand: to make your mesh, you load Mike's DEM as a point cloud and then use a tool to convert a point cloud into a 3D mesh?

I was planning on using meshpy with a user refinement function which checks the area of each 3D triangle. This way it should be possible to use one threshold inside an ROI containing the crater, and another outside for coarse-grained occlusion.

I have to run some errands until early afternoon today (going to get ultrasound #2 of our baby :o) but will be back later.

[steo85it]

So that I understand: to make your mesh, you load Mike's DEM as a point cloud and then use a tool to convert a point cloud into a 3D mesh?

Almost! :) I load a downsampled (to 50mpp, with gdal_translate ldem_87s_5mpp.tif -tr 50 50 ldem_87s_50mpp.tif) version of Mike's DEM, cut out the part that I need (interactively or not), indeed convert it to a point-cloud, then use Open3d (with Poisson's method) to get the mesh at whatever resolution (>=50mpp, in this case).

I was planning on using meshpy with a user refinement function which checks the area of each 3D triangle. This way it should be possible to use one threshold inside an ROI containing the crater, and another outside for coarse-grained occlusion.

I got a bit annoyed with meshpy, dmesh etc since they are really slow at high-resolution. Open3d is very fast: 73 seconds for 6924777 faces of a 60mpp mesh of the lsp, 9 seconds for 394033 faces of the 250mpp mesh (I haven't checked in detail how good/regular/water-tight the mesh is, but the result usually shows no artifacts nor imperfections).
Also, you might get in trouble if you do what you plan to for the outer mesh, because of "holes" at the transitions between outer and inner mesh. I solved it (after a while of trying different solutions, including the one you mentioned) by meshing the whole DEM at the target resolution (as it's fast), then cutting out the inner DEM (it's in the code I linked yesterday, I didn't copy it to the example because I don't think these occlusions are really relevant for the paper).

I have to run some errands until early afternoon today (going to get ultrasound #2 of our baby :o) but will be back later.

That's super cool, enjoy and congrats! (great that they let you in, too, in these covid times) :)

[steo85it]

In any case, I would rather import the GTiff/raster to an xarray, e.g.,

import xarray as xr
ds=xr.open_dataset(raster_path, engine="rasterio")

instead of several simple .npy, to preserve useful information such as reference frames, etc... which one later needs for coordinate transformations.

[nschorgh]

To avoid any interpolation from occurring I simply do this to generate a mesh:

    points_mesh = a[:,:2]   # mesh based on (x,y)  
    triangulation = scipy.spatial.Delaunay(points_mesh)  
    V, F = triangulation.points, triangulation.simplices
    V = np.row_stack([V.T, a[:,2]]).T   # add z to vertex coordinates

where I might create (x,y) from a raster. It's surprisingly fast (used it for millions of points), avoids loss of accuracy due to interpolation, and works for all quad-tree situations. We want to triangulate an existing point cloud, not create any other type of mesh.
gdal_translate is presumably good at downsampling a raster.

[steo85it]

Yes, that was the other option: using gdal libraries to read and downsample the mesh within the code

from osgeo import gdal
ds = gdal.Open('input.tif')
ds = gdal.Translate('output.tif', ds, xRes, yRes)

or

import xarray as xr
rds = xr.open_dataset(raster_path, engine="rasterio")
downsampled = rds.rio.reproject(rds.rio.crs, resolution=(500.,500.),resampling=Resampling.cubic_spline)

and then do what @nschorgh suggests. For my applications, I still think my approach is more flexible, but maybe here we can cut part of Mike's GTiff and process it in this way w/o importing many libraries.
I can re-implement it in this way, if you prefer, I'm anyway doing something similar for another project.

[sampotter]

I am not convinced by either of your approaches to meshing. Sorry... :-(

Please see the script make_mesh_sfp.py in the gerlache branch. It creates a 40K triangle mesh of the region [-25, 25] x [-25, 25] (stereo x/y) in 42 seconds on my computer. This uses Shewchuk's Triangle via MeshPy with a refinement function should_refine (max area in km^2 of the 3D triangle).

This seems like a reasonable amount of time to wait for a triangle mesh since we will spend much more time on the backend actually solving the problem. Maybe it would make sense to compare the amount of time it takes to construct the mesh with the compressed FF matrix assembly time.

Some details:

1. this loads the original 5 mpp DEM as a memory-mapped file so the whole DEM doesn't need to be loaded to memory
2. it uses linearly interpolation to look up subpixel values from the original DEM
3. it builds the Delaunay mesh in the stereographic x/y plane
4. for refinement, it maps each stereographic triangle to Cartesian coordinates and computes the area

Here is a screenshot of the resulting mesh:

That's super cool, enjoy and congrats! (great that they let you in, too, in these covid times) :)

Thanks. We are very excited. :-)