Investigate combining datasets for dssterrapixel.plate and DSSPngL5to12_*

[twsouthwick]

Worth keeping in mind here that the DSS dataset is splintered into these sub-plate-files only because of the previous 4 GB file size limit. On the blob storage, the handling here could be simplified to pull data from one large pyramid, rather than splitting between the one file for levels 0-8 (dssterrapixel.plate) and the collection of plates for levels 5-12 (DSSpngL5to12_x{1}_y{2}.plate).

Relevant here is that I'm pretty sure that our wwtfiles blob container already contains the loose PNG files as a single 0-12 tile pyramid in the dsstp container.

Originally posted by @pkgw in #87 (comment)

[pkgw]

Quick update from reviewing my notes from inventorying the storage containers:

• wwtfiles/dsstp is a subset: TOAST levels 8-12, ~21 million blobs, 727 GiB
• wwtfiles/dss appears to be the whole shebang: TOAST levels 0-12, 22.3 million blobs, 800 GiB

(Note that level N+1 is 4× as big as level N, so even though levels 0 to 7 are more than half of the levels, their total size is much smaller than level 12.)

The file name format in dss is DSSTerraPixelL{L}X{X}Y{Y}.png

[pkgw]

Now that we can work with plate files in blob storage, I think we can close this issue for the time being.

[twsouthwick]

@pkgw Is the 4gb file size limit due to the fileformat itself? It would be great to combine them all into a single platefile

[pkgw]

I'm not sure. I thought it was due to filesystem issues, but from some quick Wikipedia-ing it looks like NTFS has supported files larger than 4 GiB for a long time? And I assume that would be the relevant filesystem.

[twsouthwick]

Hmm that could be interesting. Azure blobs support huge blobs as well (terabytes) so it could simplify plate files. That would really make it simple to have a data-driven endpoint.

I'm going to reopen this to track if it's possible - probably longer term, so feel free to close again.