Skip to content

Commit 4513272

Browse files
NewGraphEnvironmentNewGraphEnvironment
committed
README: substantive rewrite — coverage, pipeline pointer, rstac example, sister collections, roadmap
The README opened with "live-fRee" / "play-hArd" badges + a generic two-paragraph framing, then a working rstac example tightly coupled to NewGraphEnvironment internal helpers (rfp::rfp_bcd_get_data, fpr::fpr_photo_resize_convert). Replaced with: - Status / coverage / api badges — concrete information rather than vibes (status: functional; files: ~58k; api: images.a11s.one). - Lead paragraph naming the actual scope (LidarBC DEM collection, ~58k GeoTIFFs hosted on the provincial objectstore) and the standard the catalog conforms to (STAC), with explicit API endpoint. - "Pipeline" pointer to scripts/README.md which already documents the 6-step flow (fetch → validate-access → collection-create → item-create → item-validate → s3-sync), key concepts (COG, STAC, pgstac, date extraction, validation caching), and quick-start commands. README links there rather than duplicating. - rstac quick-start rewritten to use bcdata::bcdc_query_geodata() directly instead of going through the private rfp wrapper. Anyone with bcdata installed can run this example. - QGIS Data Source Manager section retained (with the screenshots) — the v3.42+ native STAC support is a real entry point for non-R users. - "Sister collections on the same endpoint" section: stac_uav_bc + stac_airphoto_bc are also public and serve via the same images.a11s.one endpoint. - "Roadmap" section: uv migration (#16), structured logging + benchmarking (#6), true footprint geometry vs bounding-box (#2), validation-failure triage (#11). Surfaces in-flight R&D direction. Public-clean: no private-repo references (rfp, fpr, soul, sred, etc.). The rstac example uses only public packages (bcdata, sf, rstac, jsonlite, geojsonsf, tibble, purrr). README.md regenerated by hand (mirrors Rmd) since the source Rmd needs the staticimports + functions context to render and that's not relevant to the markdown emitted on GitHub. No functional change.
1 parent 1a52e46 commit 4513272

2 files changed

Lines changed: 133 additions & 106 deletions

File tree

‎README.Rmd‎

Lines changed: 51 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ output:
44
html_document:
55
code_folding: hide
66
self_contained: true
7-
params:
7+
params:
88
rmd_on: true
99
update_query: false
1010

@@ -24,122 +24,107 @@ source('scripts/functions.R')
2424

2525

2626
```{r build, echo= FALSE, eval = FALSE}
27-
# we need to chabge the rmd_on param to FALSE to render the README.md for the github landing page
27+
# we need to change the rmd_on param to FALSE to render the README.md for the github landing page
2828
rmarkdown::render(
29-
"README.Rmd",
30-
output_format = "github_document",
29+
"README.Rmd",
30+
output_format = "github_document",
3131
params = list(rmd_on = FALSE, update_query = FALSE)
3232
)
3333
3434
rmarkdown::render(
35-
"README.Rmd",
36-
output_format = "html_document",
37-
output_file = "index.html",
35+
"README.Rmd",
36+
output_format = "html_document",
37+
output_file = "index.html",
3838
params = list(rmd_on = TRUE, update_query = FALSE)
3939
)
4040
```
4141

4242
<!-- README.md is generated from README.Rmd. Please edit that file -->
4343

4444

45-
![groovy](https://img.shields.io/badge/live-fRee-green)
46-
![fUn](https://img.shields.io/badge/play-hArd-red)
45+
![status](https://img.shields.io/badge/status-functional-green)
46+
![files](https://img.shields.io/badge/files-~58k-blue)
47+
![api](https://img.shields.io/badge/api-images.a11s.one-orange)
4748

48-
The goal of [`stac_dem_bc`](https://github.com/NewGraphEnvironment/stac_dem_bc) is to serve the British Columbia lidarBC digital elevation model collection. It can be queried by location and/or time using our API via the lovely [`rstac` R package](https://brazil-data-cube.github.io/rstac/)
49-
and [QGIS (v3.42+)](https://qgis.org/). Still a work in progress but currently functioning at [https://images.a11s.one](https://images.a11s.one).
50-
See below how to add STAC items to QGIS.
49+
The goal of [`stac_dem_bc`](https://github.com/NewGraphEnvironment/stac_dem_bc) is to serve British Columbia's [LidarBC](https://lidar.gov.bc.ca/) digital elevation model collection (~58,000 GeoTIFFs hosted on the provincial objectstore) as a [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org/), queryable by location and time via the [`rstac` R package](https://brazil-data-cube.github.io/rstac/), QGIS (v3.42+), or any STAC-compliant client. The API endpoint is <https://images.a11s.one>.
5150

5251
<br>
5352

54-
This work leverages work in a sister repo [`stac_uav_bc`](http://www.newgraphenvironment.com/stac_uav_bc) with the registration of the stac done with scripts [here](https://github.com/NewGraphEnvironment/stac_uav_bc/blob/main/scripts/web.R) and [here](https://github.com/NewGraphEnvironment/stac_uav_bc/tree/main/scripts/config).
53+
The build pipeline lives under [`scripts/`](scripts/README.md) — a six-step flow (fetch → validate-access → collection-create → item-create → item-validate → s3-sync) that turns the raw provincial GeoTIFF URLs into a searchable pgstac catalog. See the [scripts README](scripts/README.md) for the pipeline walkthrough; key concepts (COG, STAC, pgstac, date extraction from filenames, validation caching for incremental rebuilds) are explained there in plain language.
5554

5655
<br>
5756

5857
```{r fig, echo=FALSE, out.width="100%", fig.align="center"}
59-
60-
fpr::fpr_photo_resize_convert("fig/cover.png", path = "fig")
6158
knitr::include_graphics("fig/cover.JPG")
6259
```
6360

6461
<br>
6562

66-
Here we will use [`bcdata`](https://github.com/bcgov/bcdata) (indirectly through our Reproducable Field Projects [`rfp`](https://www.newgraphenvironment.com/rfp/) package) to give us the Bulkley River watershed group - then query the `stac-dem-bc` collection for all the DEMs that land in that watershed group for the timeframe between 2018 and 2020.
63+
## Query the collection
64+
65+
Use [`bcdata`](https://github.com/bcgov/bcdata) to define an area of interest, then query the `stac-dem-bc` collection for DEM tiles intersecting it. Below: all DEMs covering the Bulkley River watershed group between 2018 and 2020.
66+
6767
```{r api1, eval=params$update_query}
6868
69-
aoi <- rfp::rfp_bcd_get_data(
70-
"whse_basemapping.fwa_watershed_groups_poly",
71-
col_filter = "watershed_group_name",
72-
col_filter_value = "Bulkley River"
73-
) |>
69+
aoi <- bcdata::bcdc_query_geodata("freshwater-atlas-watershed-groups") |>
70+
bcdata::filter(WATERSHED_GROUP_NAME == "Bulkley River") |>
71+
bcdata::collect() |>
7472
sf::st_transform(crs = 4326)
7573
76-
# we don't use this but could feed to the `bbox` param
77-
bbox <- aoi |>
78-
sf::st_bbox() |>
79-
as.numeric()
80-
8174
date_start <- "2018-01-01T00:00:00Z"
8275
date_end <- "2020-12-31T00:00:00Z"
8376
8477
# use rstac to query the collection
8578
q <- rstac::stac("https://images.a11s.one/") |>
8679
rstac::stac_search(
87-
# collections = "imagery-uav-bc-dev",
8880
collections = "stac-dem-bc",
89-
# bbox = bbox,
9081
intersects = jsonlite::fromJSON(
9182
geojsonsf::sf_geojson(
9283
aoi, atomise = TRUE, simplify = FALSE
9384
),
9485
simplifyVector = FALSE
95-
# ugly but just grabs the geometry list item using the new pipe
9686
) |> (\(x) x$geometry)(),
9787
datetime = paste0(date_start, "/", date_end)
9888
) |>
9989
rstac::post_request()
10090
101-
# get deets of the items
91+
# get details of the items
10292
r <- q |>
10393
rstac::items_fetch()
10494
10595
# burn the results locally so we can serve it instantly on index.html builds
106-
# Save to RDS file
10796
saveRDS(r, "data/stac_result.rds")
10897
```
10998

11099

111100
```{r}
112101
r <- readRDS("data/stac_result.rds")
113102
# build the table to display the info
114-
tab <- tibble::tibble(url_download = purrr::map_chr(r$features, ~ purrr::pluck(.x, "assets", "image", "href"))) |>
115-
dplyr::mutate(
116-
link_download = ngr::ngr_str_link_url(url_base = url_download, anchor_text = url_download)
117-
) |>
118-
dplyr::select(link_download)
103+
tab <- tibble::tibble(
104+
url_download = purrr::map_chr(r$features, ~ purrr::pluck(.x, "assets", "image", "href"))
105+
)
119106
```
120107

121108
<br>
122109

123-
`r if (params$rmd_on == FALSE) "Please see http://www.newgraphenvironment.com/stac_dem_bc for published table of collection links/details."`
124-
110+
`r if (params$rmd_on == FALSE) "Please see http://www.newgraphenvironment.com/stac_dem_bc for the published table of collection links."`
125111

126112

127-
```{r tab-uav-imagery-cap, echo=FALSE, results="asis", eval=identical(params$rmd_on, TRUE)}
113+
```{r tab-cap, echo=FALSE, results="asis", eval=identical(params$rmd_on, TRUE)}
128114
my_caption <- "DEM download links."
129-
130115
my_tab_caption_rmd()
131116
```
132117

133118

134119
```{r tab-uav-imagery, echo=FALSE, eval=identical(params$rmd_on, TRUE)}
135-
tab |>
120+
tab |>
136121
my_dt_table(cols_freeze_left = 2, escape = FALSE)
137122
```
138123

139124

140-
As of QGIS 3.42 - ONE can also access stac items (orthoimagery, Digital Surface Models and Digital Terrain Models in our case) directly via
141-
the Data Source Manager. See a blog with details [here](https://www.lutraconsulting.co.uk/blogs/stac-in-qgis). It looks like this in the
142-
th `Layer / Data Source Manager` toolbar in QGIS:
125+
## QGIS Data Source Manager (v3.42+)
126+
127+
QGIS 3.42 added native STAC support — connect directly to the catalog and filter by the current map view. See [Lutra Consulting's STAC-in-QGIS blog post](https://www.lutraconsulting.co.uk/blogs/stac-in-qgis) for a walk-through.
143128

144129
```{r echo=FALSE, fig.cap="Connecting to https://images.a11s.one"}
145130
knitr::include_graphics("fig/a11sone01.png")
@@ -149,3 +134,25 @@ knitr::include_graphics("fig/a11sone01.png")
149134
knitr::include_graphics("fig/a11sone02.png")
150135
```
151136

137+
138+
## Sister collections on the same endpoint
139+
140+
The same `images.a11s.one` STAC API serves several complementary BC collections:
141+
142+
- [`stac_uav_bc`](https://github.com/NewGraphEnvironment/stac_uav_bc) — UAV imagery, organized by watershed
143+
- [`stac_airphoto_bc`](https://github.com/NewGraphEnvironment/stac_airphoto_bc) — historic airphoto thumbnails (1963–2019)
144+
145+
146+
## Roadmap
147+
148+
- **uv-based Python dependency management** ([#16](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/16)) — migrate from conda to uv for faster, more reproducible Python environments.
149+
- **Structured logging + performance benchmarking** ([#6](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/6)) — instrument the pipeline so build performance is quantifiable across runs.
150+
- **True footprint geometry** ([#2](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/2)) — recalculate per-item footprints to exclude no-data pixels rather than using bounding boxes; gives accurate spatial-overlap queries.
151+
- **Validation-failure triage** ([#11](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/11)) — STACError on specific item JSONs; underlying root cause + automated retry.
152+
153+
Browse [open issues](https://github.com/NewGraphEnvironment/stac_dem_bc/issues) for the full backlog.
154+
155+
156+
## License
157+
158+
[MIT](LICENSE).

‎README.md‎

Lines changed: 82 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -3,110 +3,93 @@ stac_dem_bc
33

44
<!-- README.md is generated from README.Rmd. Please edit that file -->
55

6-
![groovy](https://img.shields.io/badge/live-fRee-green)
7-
![fUn](https://img.shields.io/badge/play-hArd-red)
6+
![status](https://img.shields.io/badge/status-functional-green)
7+
![files](https://img.shields.io/badge/files-~58k-blue)
8+
![api](https://img.shields.io/badge/api-images.a11s.one-orange)
89

910
The goal of
1011
[`stac_dem_bc`](https://github.com/NewGraphEnvironment/stac_dem_bc) is
11-
to serve the British Columbia lidarBC digital elevation model
12-
collection. It can be queried by location and/or time using our API via
13-
the lovely [`rstac` R
14-
package](https://brazil-data-cube.github.io/rstac/) and [QGIS
15-
(v3.42+)](https://qgis.org/). Still a work in progress but currently
16-
functioning at <https://images.a11s.one>. See below how to add STAC
17-
items to QGIS.
12+
to serve British Columbia's [LidarBC](https://lidar.gov.bc.ca/) digital
13+
elevation model collection (~58,000 GeoTIFFs hosted on the provincial
14+
objectstore) as a [SpatioTemporal Asset Catalog
15+
(STAC)](https://stacspec.org/), queryable by location and time via the
16+
[`rstac` R package](https://brazil-data-cube.github.io/rstac/), QGIS
17+
(v3.42+), or any STAC-compliant client. The API endpoint is
18+
<https://images.a11s.one>.
1819

1920
<br>
2021

21-
This work leverages work in a sister repo
22-
[`stac_uav_bc`](http://www.newgraphenvironment.com/stac_uav_bc) with the
23-
registration of the stac done with scripts
24-
[here](https://github.com/NewGraphEnvironment/stac_uav_bc/blob/main/scripts/web.R)
25-
and
26-
[here](https://github.com/NewGraphEnvironment/stac_uav_bc/tree/main/scripts/config).
22+
The build pipeline lives under [`scripts/`](scripts/README.md) — a
23+
six-step flow (fetch → validate-access → collection-create →
24+
item-create → item-validate → s3-sync) that turns the raw provincial
25+
GeoTIFF URLs into a searchable pgstac catalog. See the
26+
[scripts README](scripts/README.md) for the pipeline walkthrough; key
27+
concepts (COG, STAC, pgstac, date extraction from filenames, validation
28+
caching for incremental rebuilds) are explained there in plain language.
2729

2830
<br>
2931

3032
<img src="fig/cover.JPG" width="100%" style="display: block; margin: auto;" />
3133

3234
<br>
3335

34-
Here we will use [`bcdata`](https://github.com/bcgov/bcdata) (indirectly
35-
through our Reproducable Field Projects
36-
[`rfp`](https://www.newgraphenvironment.com/rfp/) package) to give us
37-
the Bulkley River watershed group - then query the `stac-dem-bc`
38-
collection for all the DEMs that land in that watershed group for the
39-
timeframe between 2018 and 2020.
36+
## Query the collection
37+
38+
Use [`bcdata`](https://github.com/bcgov/bcdata) to define an area of
39+
interest, then query the `stac-dem-bc` collection for DEM tiles
40+
intersecting it. Below: all DEMs covering the Bulkley River watershed
41+
group between 2018 and 2020.
4042

4143
``` r
42-
aoi <- rfp::rfp_bcd_get_data(
43-
"whse_basemapping.fwa_watershed_groups_poly",
44-
col_filter = "watershed_group_name",
45-
col_filter_value = "Bulkley River"
46-
) |>
44+
aoi <- bcdata::bcdc_query_geodata("freshwater-atlas-watershed-groups") |>
45+
bcdata::filter(WATERSHED_GROUP_NAME == "Bulkley River") |>
46+
bcdata::collect() |>
4747
sf::st_transform(crs = 4326)
4848

49-
# we don't use this but could feed to the `bbox` param
50-
bbox <- aoi |>
51-
sf::st_bbox() |>
52-
as.numeric()
53-
5449
date_start <- "2018-01-01T00:00:00Z"
55-
date_end <- "2020-12-31T00:00:00Z"
50+
date_end <- "2020-12-31T00:00:00Z"
5651

5752
# use rstac to query the collection
5853
q <- rstac::stac("https://images.a11s.one/") |>
5954
rstac::stac_search(
60-
# collections = "imagery-uav-bc-dev",
6155
collections = "stac-dem-bc",
62-
# bbox = bbox,
6356
intersects = jsonlite::fromJSON(
64-
geojsonsf::sf_geojson(
65-
aoi, atomise = TRUE, simplify = FALSE
66-
),
57+
geojsonsf::sf_geojson(aoi, atomise = TRUE, simplify = FALSE),
6758
simplifyVector = FALSE
68-
# ugly but just grabs the geometry list item using the new pipe
6959
) |> (\(x) x$geometry)(),
7060
datetime = paste0(date_start, "/", date_end)
7161
) |>
7262
rstac::post_request()
7363

74-
# get deets of the items
75-
r <- q |>
76-
rstac::items_fetch()
64+
# fetch all matching items
65+
r <- q |> rstac::items_fetch()
7766

78-
# burn the results locally so we can serve it instantly on index.html builds
79-
# Save to RDS file
80-
saveRDS(r, "data/stac_result.rds")
67+
# extract download URLs
68+
tab <- tibble::tibble(
69+
url_download = purrr::map_chr(
70+
r$features,
71+
~ purrr::pluck(.x, "assets", "image", "href")
72+
)
73+
)
8174
```
8275

83-
``` r
84-
r <- readRDS("data/stac_result.rds")
85-
# build the table to display the info
86-
tab <- tibble::tibble(url_download = purrr::map_chr(r$features, ~ purrr::pluck(.x, "assets", "image", "href"))) |>
87-
dplyr::mutate(
88-
link_download = ngr::ngr_str_link_url(url_base = url_download, anchor_text = url_download)
89-
) |>
90-
dplyr::select(link_download)
91-
```
76+
Please see <http://www.newgraphenvironment.com/stac_dem_bc> for the
77+
published table of collection links.
9278

93-
<br>
79+
## QGIS Data Source Manager (v3.42+)
9480

95-
Please see <http://www.newgraphenvironment.com/stac_dem_bc> for
96-
published table of collection links/details.
97-
98-
As of QGIS 3.42 - ONE can also access stac items (orthoimagery, Digital
99-
Surface Models and Digital Terrain Models in our case) directly via the
100-
Data Source Manager. See a blog with details
101-
[here](https://www.lutraconsulting.co.uk/blogs/stac-in-qgis). It looks
102-
like this in the th `Layer / Data Source Manager` toolbar in QGIS:
81+
QGIS 3.42 added native STAC support — connect directly to the catalog
82+
and filter by the current map view. See [Lutra Consulting's STAC-in-QGIS
83+
blog post](https://www.lutraconsulting.co.uk/blogs/stac-in-qgis) for a
84+
walk-through.
10385

10486
<div class="figure">
10587

10688
<img src="fig/a11sone01.png" alt="Connecting to https://images.a11s.one" width="100%" />
10789
<p class="caption">
10890

10991
Connecting to <https://images.a11s.one>
92+
11093
</p>
11194

11295
</div>
@@ -117,6 +100,43 @@ Connecting to <https://images.a11s.one>
117100
<p class="caption">
118101

119102
Using the field of view in QGIS to filter results
103+
120104
</p>
121105

122106
</div>
107+
108+
## Sister collections on the same endpoint
109+
110+
The same `images.a11s.one` STAC API serves several complementary BC
111+
collections:
112+
113+
- [`stac_uav_bc`](https://github.com/NewGraphEnvironment/stac_uav_bc) —
114+
UAV imagery, organized by watershed
115+
- [`stac_airphoto_bc`](https://github.com/NewGraphEnvironment/stac_airphoto_bc) —
116+
historic airphoto thumbnails (1963–2019)
117+
118+
## Roadmap
119+
120+
- **uv-based Python dependency management**
121+
([#16](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/16))
122+
— migrate from conda to uv for faster, more reproducible Python
123+
environments.
124+
- **Structured logging + performance benchmarking**
125+
([#6](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/6)) —
126+
instrument the pipeline so build performance is quantifiable across
127+
runs.
128+
- **True footprint geometry**
129+
([#2](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/2)) —
130+
recalculate per-item footprints to exclude no-data pixels rather than
131+
using bounding boxes; gives accurate spatial-overlap queries.
132+
- **Validation-failure triage**
133+
([#11](https://github.com/NewGraphEnvironment/stac_dem_bc/issues/11))
134+
— STACError on specific item JSONs; underlying root cause + automated
135+
retry.
136+
137+
Browse [open issues](https://github.com/NewGraphEnvironment/stac_dem_bc/issues)
138+
for the full backlog.
139+
140+
## License
141+
142+
[MIT](LICENSE).

0 commit comments

Comments
 (0)