Add parameter to build and run model res workflow to run ingest on batch (#380)

This currently errors out on dev branches. Is it worth it to provide
permissions for all dev branches to ensure new features can be added
without them going into our main datalake?


A test was run by removing prox_nearest_metra_route_dist_ft as an input
variable in the ingest script. As desired, it returned null values in
assessment_card
```

SELECT
meta_pin, prox_nearest_metra_route_dist_ft
from assessment_card
where run_id = '2025-05-29-agitated-sam'
limit 10
```
Final run with up to date changes
https://github.com/ccao-data/model-res-avm/actions/runs/15330192847

---------

Co-authored-by: Jean Cochrane <[email protected]>
Co-authored-by: Jean Cochrane <[email protected]>

Author: Damonamajor

.dockerignore

@@ -3,6 +3,7 @@ README.md
 renv/profile
 docs/
 input/
+!input/.gitkeep
 renv/library/
 renv/sandbox/
 renv/staging/

.github/workflows/build-and-run-model.yaml

@@ -51,11 +51,31 @@ on:
         description: Calculate SHAP values
         default: false
         required: true
+      repro_ingest:
+        type: boolean
+        description: Run ingest stage before running model
+        default: false
+        required: true
   push:
     branches: [master, "*-assessment-year"]
 
 jobs:
+  parse-command:
+    runs-on: ubuntu-latest
+    outputs:
+      command: ${{ steps.set.outputs.command }}
+    steps:
+      - name: Determine DVC command
+        id: set
+        shell: bash
+        run: |
+          if [[ "${{ inputs.repro_ingest }}" == "true" ]]; then
+            echo "command=dvc unfreeze ingest && dvc repro" >> $GITHUB_OUTPUT
+          else
+            echo "command=" >> $GITHUB_OUTPUT
+          fi
   build-and-run-model:
+    needs: parse-command
     permissions:
       # contents:read and id-token:write permissions are needed to interact
       # with GitHub's OIDC Token endpoint so that we can authenticate with AWS
@@ -65,8 +85,10 @@ jobs:
       # required in order to allow the reusable called workflow to push to
       # GitHub Container Registry
       packages: write
-    uses: ccao-data/actions/.github/workflows/build-and-run-batch-job.yaml@main
+    uses: ccao-data/actions/.github/workflows/build-and-run-batch-job.yaml@Add-parameter-to-build-and-run-model-res-workflow-to-run-ingest-on-Batch
     with:
+      ref: Add-parameter-to-build-and-run-model-res-workflow-to-run-ingest-on-Batch
+      command: ${{ needs.parse-command.outputs.command }}
       backend: "ec2"
       vcpu: "40"
       memory: "158000"

Dockerfile

@@ -16,7 +16,8 @@ RUN apt-get update && \
         libcurl4-openssl-dev libssl-dev libxml2-dev libgit2-dev git \
         libudunits2-dev python3-dev python3-pip python3-venv libgdal-dev \
         libgeos-dev libproj-dev libfontconfig1-dev libharfbuzz-dev \
-        libfribidi-dev pandoc curl gdebi-core && \
+        libfribidi-dev pandoc curl gdebi-core \
+        libglpk-dev libglpk40 && \
     rm -rf /var/lib/apt/lists/*
 
 # Install Quarto
@@ -30,12 +31,14 @@ RUN pip install --no-cache-dir dvc[s3]
 # Copy R bootstrap files into the image
 COPY renv.lock .Rprofile DESCRIPTION requirements.txt ./
 COPY renv/profiles/reporting/renv.lock reporting-renv.lock
+COPY renv/profiles/dev/renv.lock dev-renv.lock
 COPY renv/ renv/
 
 # Install R dependencies. Restoring renv first ensures that it's
 # using the same version as recorded in the lockfile
 RUN Rscript -e 'renv::restore(packages = "renv"); renv::restore()'
 RUN Rscript -e 'renv::restore(lockfile = "reporting-renv.lock")'
+RUN Rscript -e 'renv::restore(lockfile = "dev-renv.lock")'
 
 # Set the working directory to the model directory
 WORKDIR /model-res-avm/

pipeline/00-ingest.R

@@ -25,6 +25,8 @@ noctua_options(unload = TRUE)
 # Establish Athena connection
 AWS_ATHENA_CONN_NOCTUA <- dbConnect(
   noctua::athena(),
+  s3_staging_dir   = "s3://ccao-athena-results-us-east-1/",
+  region_name      = "us-east-1",
   rstudio_conn_tab = FALSE
 )