Skip to content

Commit 924733e

Browse files
micahwiesner67natemcintoshpre-commit-ci[bot]github-actions[bot]zsusswein
authored
Add API v2 (#385)
* start of reading either API * add news entry * start of new tests * add new api test file * needed to make sure every reference date existed The initial version of this test file had only some of the reference dates of the whole time span * first working version of reader for api v2 More to come * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set up proportion active argument. Need to do a bit more testing I think * Air formatter Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix test issues * attempting fixes suggested on the internet * love messing up my less than, equal, greater than * changes to allow use of the new field These changes should only be merged into main at the same time as the config generator library gets these changse merged * forgot to add facility active field to metadata * roxygenize + tagging cmdstanr release version * removing rogue comma in list * adding facility active proportion to test json * removing big exclusions test as this test parquet does not exist * Update R/config.R Co-authored-by: Nate McIntosh <NMcIntosh@cdc.gov> * exposing API input container and DATA_API argument for simplicity toggling apis * adding input data api container to make test target * Update NEWS.md * Update R/config.R Co-authored-by: Nate McIntosh <NMcIntosh@cdc.gov> * adding logging to read_data to explicitly state API used * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * checking value for numeric * configuring with air * updating roxygen documentation * pre commit and updating glue in DESCRIPTION file * updating roxygen * adding readme to tests/testthat/data * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updating documentation in config * Apply suggestions from code review Co-authored-by: Zachary Susswein <46581799+zsusswein@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updating documentation in read_data function * adding tests for facility active proportion * roxygenizeing * making DATA lowercase --------- Co-authored-by: Nate McIntosh <natemcintosh@gmail.com> Co-authored-by: Nate McIntosh <NMcIntosh@cdc.gov> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Zachary Susswein <46581799+zsusswein@users.noreply.github.com>
1 parent b86c447 commit 924733e

File tree

14 files changed

+354
-103
lines changed

14 files changed

+354
-103
lines changed

‎DESCRIPTION‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ License: Apache License (>= 2)
2020
Encoding: UTF-8
2121
Remotes:
2222
github::epiforecasts/EpiNow2@9b8cd4fcceca41ac34545a38989f6f295ddeeaf7,
23-
github::stan-dev/cmdstanr
23+
github::stan-dev/cmdstanr@da99e2ba954658bdad63bffb738c4444c33a4e0e
2424
Roxygen: list(markdown = TRUE)
2525
RoxygenNote: 7.3.2
2626
Suggests:
@@ -51,7 +51,9 @@ Imports:
5151
tidyr,
5252
tidybayes,
5353
optparse,
54-
Microsoft365R
54+
Microsoft365R,
55+
stringr,
56+
glue
5557
Additional_repositories:
5658
https://stan-dev.r-universe.dev
5759
URL: https://cdcgov.github.io/cfa-epinow2-pipeline/

‎Makefile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,21 @@ IMAGE_NAME=cfa-epinow2-pipeline
33
BRANCH=$(shell git branch --show-current)
44
CONFIG_CONTAINER=rt-epinow2-config
55
CNTR_MGR=docker
6+
DATA_API=v1
67
ifeq ($(BRANCH), main)
78
TAG=latest
89
else
910
TAG=$(BRANCH)
1011
endif
1112

13+
ifeq ($(DATA_API),v1)
14+
API_CONTAINER := nssp-etl
15+
else ifeq ($(DATA_API),v2)
16+
API_CONTAINER := nssp-etl-api-v2
17+
else
18+
$(error Unknown DATA_API '$(DATA_API)'. Expected v1 or v2)
19+
endif
20+
1221
CONFIG=test.json
1322
POOL="cfa-epinow2-$(TAG)"
1423
TIMESTAMP:=$(shell date -u +"%Y%m%d_%H%M%S")
@@ -38,12 +47,14 @@ config: ## Generates a configuration file for running the model
3847
--disease="COVID-19,Influenza,RSV" \
3948
--state=all \
4049
--output-container=nssp-rt-v2 \
50+
--input-container=$(API_CONTAINER) \
4151
--job-id=$(JOB) \
4252
--report-date-str=$(REPORT_DATE)
4353

4454
rerun-config: ## Generate a configuration file to rerun a previous model
4555
uv run azure/generate_rerun_configs.py \
4656
--output-container=nssp-rt-v2 \
57+
--input-container=$(API_CONTAINER) \
4758
--job-id=$(JOB) \
4859
--report-date-str=$(REPORT_DATE)
4960

@@ -84,6 +95,7 @@ test-batch: ## Run GitHub Actions workflow and then job.py for testing on Azure
8495
--disease="COVID-19,Influenza,RSV" \
8596
--state=NY \
8697
--output-container=nssp-rt-testing \
98+
--input-container=$(API_CONTAINER) \
8799
--job-id=$(JOB) \
88100
--report-date-str=$(REPORT_DATE)
89101
uv run --env-file .env \

‎NEWS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# CFAEpiNow2Pipeline v0.2.0
22

33
## Features
4+
5+
* Add the ability to read data from API v2 as well as v1
46
* dependabot updating package docker/login-action from version 3 to version 4 new
57
* Adding image tag validation when dependabot PRs are opened and automatic update to NEWs md
68
* Add a utility script for reading, preparing, and uploading Rt review decisions

‎R/config.R‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,12 @@ Data <- S7::new_class(
158158
#' "YYYY-MM-DD".
159159
#' @param output_container An optional string specifying the output blob storage
160160
#' container.
161+
#' @param facility_active_proportion A numeric value between 0 and 1 specifying
162+
#' the proportion of days during the modeling period that facilities must have
163+
#' reported at least one informative discharge diagnosis (DDI) to be included in
164+
#' the analysis. Default is 0.94 (require
165+
#' active reporting for >=53 of 56 days in the training period).
166+
#' Lower values allow inclusion of facilities with fewer active days.
161167
#' @family config
162168
#' @export
163169
Config <- S7::new_class(
@@ -190,7 +196,26 @@ Config <- S7::new_class(
190196
# Would add default values, but Roxygen isn't happy about them yet.
191197
sampler_opts = S7::class_list,
192198
exclusions = S7::S7_class(Exclusions()),
193-
output_container = character_or_null
199+
output_container = character_or_null,
200+
facility_active_proportion = S7::new_property(
201+
S7::class_double,
202+
default = quote(0.94),
203+
validator = \(value) {
204+
if (
205+
rlang::is_bare_numeric(value) &&
206+
length(value) == 1 &&
207+
value >= 0 &&
208+
value <= 1
209+
) {
210+
NULL
211+
} else {
212+
paste0(
213+
"Invalid value for facility_active_proportion. ",
214+
"It must be a single numeric value between 0 and 1."
215+
)
216+
}
217+
}
218+
)
194219
)
195220
)
196221

‎R/pipeline.R‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ orchestrate_pipeline <- function(
8585
)
8686
read_json_into_config(
8787
config_path,
88-
c("exclusions", "output_container")
88+
c("exclusions", "output_container", "facility_active_proportion")
8989
)
9090
},
9191
error = function(con) {
@@ -94,6 +94,7 @@ orchestrate_pipeline <- function(
9494
}
9595
)
9696
if (typeof(config) == "logical") {
97+
cli::cli_warn("Failed to read config", class = "Bad_config")
9798
return(invisible(FALSE))
9899
}
99100

@@ -113,15 +114,19 @@ orchestrate_pipeline <- function(
113114
logfile_connection <- file(file.path(logfile_path, "logs.txt"), open = "wt")
114115
sink(
115116
logfile_connection,
117+
# "output" means general (non-error) output
116118
type = "output",
117119
append = TRUE,
118120
# Send output to logs and to console
119121
split = TRUE
120122
)
121123
sink(
122124
logfile_connection,
125+
# "message" means error messages
123126
type = "message",
124127
append = TRUE
128+
# Unfortunately, we can't split messages to both console and log file.
129+
# It will error out if we try
125130
)
126131
on.exit(sink(file = NULL))
127132
cli::cli_alert_info("Starting run at {Sys.time()}")
@@ -189,7 +194,8 @@ execute_model_logic <- function(config, input_dir, output_dir) {
189194
geo_value = config@geo_value,
190195
report_date = config@report_date,
191196
max_reference_date = config@max_reference_date,
192-
min_reference_date = config@min_reference_date
197+
min_reference_date = config@min_reference_date,
198+
facility_active_proportion = config@facility_active_proportion
193199
)
194200

195201
# rlang::is_empty() checks for empty and NULL values
@@ -289,7 +295,8 @@ execute_model_logic <- function(config, input_dir, output_dir) {
289295
config@exclusions@blob_storage_container
290296
),
291297
# Add the config container here when refactoring out to outer func
292-
run_at = format(Sys.time(), "%Y-%m-%dT%H:%M:%S%z")
298+
run_at = format(Sys.time(), "%Y-%m-%dT%H:%M:%S%z"),
299+
facility_active_proportion = config@facility_active_proportion
293300
)
294301

295302
write_model_outputs(

0 commit comments

Comments
 (0)