Skip to content

Commit 12ef43e

Browse files
authored
Add documentation about copying pacta data for use with webapp (#244)
1 parent eb78ff5 commit 12ef43e

File tree

1 file changed

+64
-0
lines changed

1 file changed

+64
-0
lines changed

README.md

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,3 +266,67 @@ See the [`workflow.scenario.preparation` README](https://github.com/RMI-PACTA/wo
266266
The required files are:
267267

268268
- dependent on what sceanrios are meant to be included
269+
270+
## Promotion of Datasets to PROD
271+
272+
### `workflow.transiton.monitor`
273+
274+
Data sets to prepare images from [`workflow.transition.monitor`](https://github.com/RMI-PACTA/workflow.transition.monitor/) are stored in the `pactadatadev` Storage Account (`RMI-SP-PACTA-DEV` Resource Group), in the file share `workflow-data-preparation-outputs`.
275+
The dataset used is defined by the directory name in the build config for each image (`build/config/rmi_pacta_YYYYqX_ZZZZ.json`), in the `data_share_path` key.
276+
277+
### `workflow.pacta.webapp` and `workflow.pacta.dashboard`
278+
279+
For the [workflow.pacta.webapp](https://github.com/RMI-PACTA/workflow.pacta.webapp/) and [workflow.pacta.dashboard](https://github.com/RMI-PACTA/workflow.pacta.dashboard/) images, the PACTA data is expected as a bind mount to the docker image (rather than "baked in", as with `workflow.transition.monitor`).
280+
For Azure Container Instances running on our tenant, the expected file share to mount is `pacta_data`, in the `rmipactawebappdata` Storage Account (in the `RMI-SP-PACTA-WEU-PAT-DEV` Resource Group).
281+
The top level directories in that File Share correspond to the directories in the `pactadatadev/workflow-data-preparation-outputs` file share, and should be passed as environment variables to the docker image (see workflow repos for more detail).
282+
283+
### Transferring from `pactadatadev` to `rmipactawebappdata`
284+
285+
Prepared datasets can be copied from `pactadatadev` to `rmipactawebappdata` with the following commands:
286+
287+
```sh
288+
289+
DIRNAME="2023Q4_20240718T150252Z" # Change as needed.
290+
TOKEN_START=$(date -u -j '+%Y-%m-%dT%H:%MZ')
291+
TOKEN_EXPIRY=$(date -u -j -v "+20M" '+%Y-%m-%dT%H:%MZ')
292+
293+
DESTINATION_ACCOUNT_NAME="rmipactawebappdata"
294+
DESTINATION_SHARE="pacta-data"
295+
DESTINATION_SAS="$(
296+
az storage share generate-sas \
297+
--account-name $DESTINATION_ACCOUNT_NAME \
298+
--expiry $TOKEN_EXPIRY \
299+
--permissions rcw \
300+
--name $DESTINATION_SHARE \
301+
--start $TOKEN_START \
302+
--output tsv
303+
)"
304+
305+
# note permissions are different. rcl allows listing contents, rcw above is to write
306+
SOURCE_ACCOUNT_NAME="pactadatadev"
307+
SOURCE_SHARE="workflow-data-preparation-outputs"
308+
SOURCE_SAS="$(
309+
az storage share generate-sas \
310+
--account-name $SOURCE_ACCOUNT_NAME \
311+
--expiry $TOKEN_EXPIRY \
312+
--permissions rcl \
313+
--name $SOURCE_SHARE \
314+
--start $TOKEN_START \
315+
--output tsv
316+
)"
317+
318+
COPY_SOURCE="https://$SOURCE_ACCOUNT_NAME.file.core.windows.net/$SOURCE_SHARE/$DIRNAME"?$SOURCE_SAS
319+
COPY_DESTINATION="https://$DESTINATION_ACCOUNT_NAME.file.core.windows.net/$DESTINATION_SHARE/$DIRNAME?$DESTINATION_SAS"
320+
echo "$COPY_SOURCE"
321+
echo "$COPY_DESTINATION"
322+
323+
azcopy copy \
324+
"$COPY_SOURCE" \
325+
"$COPY_DESTINATION" \
326+
--as-subdir=false \
327+
--recursive
328+
329+
330+
```
331+
332+

0 commit comments

Comments
 (0)