BYOC_ID: 46056 Refactor District Nursing by LucyEmma22 · Pull Request #1327 · Public-Health-Scotland/source-linkage-files

LucyEmma22 · 2026-05-13T09:22:16Z

Data Required:

extract_district_nursing --> get_boxi_extract_path(year = year, type = "dn")
dn_costs --> get_dn_raw_costs_path()
dn_contacts --> fs::path(get_slf_dir(), "Costs", "DN-Contacts-Numbers-for-Costs.csv")
population_lookup --> get_pop_path(type = "hscp")

To Do/Check:

read_extract_district_nursing:

Check table name for extract_district_nursing.
Check whether data should be filtered by year. If so, check year column name.
Check whether to keep Patient Data Zone 2011 (Contact).

get_dn_costs_path:

Based on refactoring of get_ch_costs_path.

process_costs_dn:

New function.
Check if file paths should be included as arguments. If so, these file path functions will need to be refactored.
Check it is okay to add dn_costs to log_slf_event.
Check table name for dn_costs, dn_contacts and population_lookup.
Check column names for dn_costs, dn_contacts and population_lookup.
Check whether all cost columns are numeric in Denodo view. If so converting costs to numeric can be removed.

log_slf_event:

Added dn_cost to type list.

…e-linkage-files into refactor-dn

LucyEmma22 · 2026-05-13T10:59:24Z

+      names_to = "year",
+      names_pattern = "(\\d{4})_cost",
+      values_to = "cost"
+    )


This should be removed if we are using pivoted costs.

Agreed! costs are now pivoted so we can remove this code. Will need testing again once the data is available

Jennit07 · 2026-05-20T12:39:04Z

+                             # dn_raw_costs_path = get_dn_raw_costs_path(), # TODO: Check if needed. If it is function will need to be refactored to include the BYOC_MODE argument.
+                             # dn_raw_contacts_path = fs::path(get_slf_dir(), "Costs", "DN-Contacts-Numbers-for-Costs.csv"), # TODO: Check if needed. If it is function will need to be refactored to include the BYOC_MODE argument.
+                             # pop_path = get_pop_path(type = "hscp"), # TODO: Check if needed. If it is function will need to be refactored to include the BYOC_MODE argument.


Yes - these will be required and include the BYOC_MODE argument. We agreed that going forward any dependencies should be displayed in the function arguments to make them more visible

Jennit07 · 2026-05-20T12:39:27Z

+                             BYOC_MODE = FALSE,
+                             run_id = NA,
+                             run_date_time = NA) {
+  log_slf_event(stage = "process", status = "start", type = "dn_costs", year = year) # TODO: Check this is necessary.


Good spot! - yes i think we need this!

Jennit07 · 2026-05-20T12:41:18Z

+      BYOC_MODE = BYOC_MODE
+    )
+
+  log_slf_event(stage = "process", status = "complete", type = "dn_costs", year = "all") # TODO: Check this is necessary.


Yes - good spot! I added a comment to CH costs as a reminder to do this also!

Jennit07 · 2026-05-20T12:44:12Z

+    denodo_connect,
+    dbplyr::in_schema("sdl", "sdl_district_nursing_source")
+  ) %>% # TODO: Check table name.
+    # TODO: Check whether to filter by year.


Yes - i think filter by year is missing here

Do you know what the year column is/will be?

Oh this one is interesting actually - This works similar to SPARRA how the extracts are saved in hscdiip separately, the main difference is that DN is an archived dataset but this will also be a file ingest by NSS. Currently, each extract is split into FY with the name, however once this is in Denodo, i think NSS will create a FY column for us and the data would be stacked as one view.

For now, i think you could call the column year and add placeholder code with a TODO so that we remember to check this when we get the data to test

Jennit07 · 2026-05-20T12:45:18Z

+    process_costs_dn(
+      denodo_connect = get_denodo_connection(BYOC_MODE = BYOC_MODE),
+      BYOC_MODE = FALSE,
+      run_id = NA,
+      run_date_time = NA
+    ),


Ive answered the TODO comments above, they would also need added to the targets as parameters here too

Yes good point!

Jennit07 · 2026-05-20T12:46:07Z

+    process_costs_dn(
+      denodo_connect = get_denodo_connection(BYOC_MODE = BYOC_MODE),
+      BYOC_MODE = FALSE,
+      run_id = NA,
+      run_date_time = NA


See above: also need to add extra parameters here for addtitional files

Jennit07

Hi @LucyEmma22 Thanks for doing this PR - great work. Everything looks good, ive added some comments answering some of your code questions. A few changes required here. But overall looks good and i can test again when the data is ready! Thanks

OluwatobiOni · 2026-05-22T14:19:03Z

+  if (isTRUE(BYOC_MODE)) {
+    dn_costs_path <- file.path(
+      denodo_output_path(),
+      stringr::str_glue("Cost_DN_Lookup.parquet")


we are now using cost_ch_lookup.parquet, changing everything to small letters for consistency. NSS's copy_to_s3 will take cost_dn_lookup.parquet as it has been effected on their end as small letters.

OluwatobiOni · 2026-05-22T14:19:19Z

+    dn_costs_path <- get_file_path(
+      directory = fs::path(get_slf_dir(), "Costs"),
+      file_name = stringr::str_glue(
+        "Cost_DN_Lookup.parquet"


same comment as above

OluwatobiOni · 2026-05-22T14:20:54Z

+                             BYOC_MODE = FALSE,
+                             run_id = NA,
+                             run_date_time = NA) {
+  log_slf_event(stage = "process", status = "start", type = "dn_costs", year = year) # TODO: Check this is necessary.


You are correct, the line of code needs to be added. type should be dn_cost_lookup for consistency.

OluwatobiOni · 2026-05-22T14:22:00Z

+      BYOC_MODE = BYOC_MODE
+    )
+
+  log_slf_event(stage = "process", status = "complete", type = "dn_costs", year = "all") # TODO: Check this is necessary.


Type should be dn_cost_lookup for consistency.

LucyEmma22 and others added 11 commits May 12, 2026 17:05

Refactor read_extract_district_nursing

41a8774

Refactor process_extract_district_nursing

d959d94

Refactor get_dn_costs_path

01a97c1

Add dn_costs to log_slf_event

888f0f0

Add process_costs_dn function

eb87c01

Update documentation

9bdee0b

Add district nursing parameters to targets

1df0395

Add district nursing to dummy_targets

36d12e5

Add district nursing to run_sdl

028a92c

Merge branch 'refactor-dn' of github.com:Public-Health-Scotland/sourc…

6735382

…e-linkage-files into refactor-dn

Style code

6288367

LucyEmma22 added On hold Waiting for something / someone outside of our control BYOC dependency missing Preprod data not available labels May 13, 2026

LucyEmma22 commented May 13, 2026

View reviewed changes

Jennit07 reviewed May 20, 2026

View reviewed changes

Jennit07 mentioned this pull request May 20, 2026

BYOC_ID: 45936 - Refactor sc care homes #1318

Draft

4 tasks

Jennit07 reviewed May 20, 2026

View reviewed changes

Jennit07 requested changes May 20, 2026

View reviewed changes

Jennit07 added ready for PHS testing PR complete labels May 20, 2026

OluwatobiOni reviewed May 22, 2026

View reviewed changes

Conversation

LucyEmma22 commented May 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jennit07 May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jennit07 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jennit07 May 20, 2026 •

edited

Loading