RSContextNEO Data Duplication of 3DEP

[MattReimer]

I've noticed that in some cases the 3DEP fetcher is fetching a lot more data than it needs:

I've made the DEMs transparent red so you can see the overlap. In this case we have two different projections represented: 11N and 12N but that means a full duplication of data (nearly 10Gb for a HUC12!!)

NOTE: Even though this is the same (ish) mechanism as dem_builder.py I am expicitly referring to /riverscapes-tools/packages/rs_context_neo/rscontextneo/src/fetch_dem.py on the rscontext-neo git branch for now. Eventually we'll want to move this into rscommons once it works

The HUC in question is 170402180402

And the files we download are.

USGS_1M_11_x72y487_ID_SouthernGaps_D23.tif
USGS_1M_11_x72y487_ID_SouthernID_2018_D19.tif
USGS_1M_11_x72y488_ID_SouthernGaps_D23.tif
USGS_1M_11_x72y488_ID_SouthernID_2018_D19.tif
USGS_1M_11_x72y489_ID_SouthernGaps_D23.tif
USGS_1M_11_x72y489_ID_SouthernID_2018_D19.tif
USGS_1M_11_x73y487_ID_SouthernGaps_D23.tif
USGS_1M_11_x73y487_ID_SouthernID_2018_D19.tif
USGS_1M_11_x73y488_ID_SouthernGaps_D23.tif
USGS_1M_11_x73y488_ID_SouthernID_2018_D19.tif
USGS_1M_11_x73y489_ID_SouthernID_2018_D19.tif
USGS_1M_11_x74y487_ID_SouthernGaps_D23.tif
USGS_1M_11_x74y487_ID_SouthernID_2018_D19.tif
USGS_1M_11_x74y488_ID_SouthernGaps_D23.tif
USGS_1M_11_x74y488_ID_SouthernID_2018_D19.tif
USGS_1M_11_x74y489_ID_SouthernID_2018_D19.tif
USGS_1M_12_x24y487_ID_SouthernGaps_D23.tif
USGS_1M_12_x24y487_ID_SouthernID_2018_D19.tif
USGS_1M_12_x24y488_ID_SouthernGaps_D23.tif
USGS_1M_12_x24y488_ID_SouthernID_2018_D19.tif
USGS_1M_12_x24y489_ID_SouthernGaps_D23.tif
USGS_1M_12_x24y489_ID_SouthernID_2018_D19.tif
USGS_1M_12_x25y487_ID_SouthernGaps_D23.tif
USGS_1M_12_x25y487_ID_SouthernID_2018_D19.tif
USGS_1M_12_x25y488_ID_SouthernGaps_D23.tif
USGS_1M_12_x25y488_ID_SouthernID_2018_D19.tif
USGS_1M_12_x25y489_ID_SouthernID_2018_D19.tif
USGS_1M_12_x26y487_ID_SouthernGaps_D23.tif
USGS_1M_12_x26y487_ID_SouthernID_2018_D19.tif
USGS_1M_12_x26y488_ID_SouthernGaps_D23.tif
USGS_1M_12_x26y488_ID_SouthernID_2018_D19.tif
USGS_1M_12_x26y489_ID_SouthernID_2018_D19.tif

Questions:

The query we're making is:

    params = {
        "polygon": ",".join([f"{lat} {long}" for lat, long in polygon_coords]),
        "datasets": "Digital Elevation Model (DEM) 1 meter",
        "prodFormats": "GeoTIFF",
    }

and this comes from code in rscommons
https://github.com/Riverscapes/riverscapes-tools/blob/master/lib/commons/rscommons/national_map.py#L120

1. Could we be making a different query to filter out one of these duplicate datasets?
2. Could we filter it afterwards using metadata we get back from the TNM library?

[philipbaileynar]

Three thoughts

1 - @weblorin I wonder if there's a master tile index layer for the US similar to what you found for New Zealand? We could make this available to the tool so that it could determine which tiles to download.

2 - Another option is to use a simple UTM zone layer to determine which UTM zone to use and skip those rasters that are not in the desired zone? Note in Matt's example above he's on the boundary between UTM 11N and 12N. The search is returning duplicate rasters, one in each projection.

3 - Do we need to consider "missions" or data quality? Or does the National Map only possess one raster per tile?

[MattReimer]

In my script now I consented different projections and I favour the one with the most coverage so we don't need to reproject. That doesn't stop us from downloading the rasters though