cuci (Clean and Unify Collected Information) is an R package for standardising multi-year survey datasets. You declare your target variable set once in a YAML file - names, types, valid values, recode rules, missing codes - and cuci applies that declaration consistently to every raw file, producing clean data and a full audit trail of every decision made along the way.
Survey data collected over multiple years rarely arrives in the same
shape. A variable called IO_Kjonn in 2022 might be gender_var in
2023 and Kjonn in 2024. Coding schemes change. Missing-value codes
differ. Types are inconsistent. Handling this manually - with per-year
scripts or copy-pasted cleaning code - is error-prone and hard to audit.
cuci centralises all of these rules in a single YAML file and enforces them automatically, so your cleaning logic lives in one place and every raw file goes through exactly the same process.
# Install the development version from GitHub
# install.packages("pak")
pak::pak("folkehelsestats/cuci")Everything in cuci flows from a YAML file - the variable map - that describes what each variable in your clean output should look like.
# config/variable_map.yml
kjonn:
colnames: [Kjonn, IO_Kjonn, gender_var] # known aliases across raw files
keywords: [kjonn, sex, gender] # fallback: matched by substring
type: integer # target R type
label: "Respondentens kjønn"
value: # the complete valid value set
1: "Mann"
2: "Kvinne"
recode: # standardise old codings
0: 2
1: 1
missing: # codes that mean "no answer"
- 8
- 9
alder:
colnames: [Alder, age_yrs]
keywords: [alder, age]
type: integer
label: "Alder i år"
value: ~
recode: ~
missing:
- 99Each top-level key (kjonn, alder, …) is the canonical name - the
name the variable will have in the cleaned output regardless of what it
was called in the raw file. See vignette("cuci-introduction") for a
full explanation of every field.
library(cuci)
config <- load_config("config/variable_map.yml")
# Inspect what was parsed
names(config$var_map)
#> [1] "kjonn" "alder" "yrkesstatus" "siv" "antpers" "tob1"
config$type_map
#> kjonn alder yrkesstatus siv antpers tob1
#> "integer" "integer" "integer" "integer" "integer" "integer"
config$value_map$kjonn
#> num_value chr_value
#> 1: 1 Mann
#> 2: 2 KvinneProvide a manifest table with one row per raw file and the year it belongs to.
library(data.table)
manifest <- data.table(
path = c(
"data/raw/survey_2023.csv",
"data/raw/survey_2024.csv"
),
year_tag = c(2023L, 2024L)
)
# Returns a named list - one cleaned data.table per file
results <- run_pipeline(
file_manifest = manifest,
config_file = "config/variable_map.yml",
log_dir = "logs/matching",
output_path = "data/clean", # directory: one CSV per file
merge = FALSE
)
results[["survey_2023.csv"]]
#> year kjonn alder yrkesstatus siv antpers tob1
#> 1: 2023 2 75 2 3 1 2
#> 2: 2023 2 42 1 1 2 2
#> ...
# Or stack all years into one master dataset
master <- run_pipeline(
file_manifest = manifest,
config_file = "config/variable_map.yml",
log_dir = "logs/matching",
output_path = "data/clean/master.csv", # single file
merge = TRUE
)cuci tries to match every raw column name to a canonical variable through three layers, stopping at the first successful match:
Layer 1 - EXACT Column name is a known alias in colnames:
Confidence: HIGH | No review needed
"IO_Kjonn" -> "kjonn" ✓
Layer 2 - FUZZY Column name is close to a known alias (typo tolerance)
Confidence: MEDIUM | Flagged for review
"Alderr" -> "alder" ✓ (one edit distance away)
Layer 3 - KEYWORD Column name contains a keyword from keywords:
Confidence: LOW | Always flagged for review
"io_kjonn" -> "kjonn" ✓ (contains "kjonn")
Columns that do not match any layer are dropped from the output and
recorded in the audit log as unmatched.
Because each step of the pipeline is an exported function, you can call any of them independently. This is useful when exploring a new file, debugging a specific variable, or building a custom workflow.
library(data.table)
config <- load_config("config/variable_map.yml")
raw <- fread("data/raw/survey_2023.csv")
# names(raw): IO_Kjonn, Alder, Yrkesstatus, Siv, Antpers, Tob1
match_result <- match_columns(
raw_colnames = tolower(names(raw)),
config = config
)
# Structured log of every matching decision
match_result$match_log
#> raw_name canonical method confidence needs_review
#> 1: yrkesstatus yrkesstatus exact high FALSE
#> 2: alder alder exact high FALSE
#> 3: siv siv exact high FALSE
#> 4: antpers antpers exact high FALSE
#> 5: tob1 tob1 exact high FALSE
#> 6: io_kjonn kjonn keyword [kjonn] low TRUE
# Pretty-print to the console
print_match_report(match_result, "survey_2023.csv")
#>
#> ╔══════════════════════════════════════╗
#> Column Matching Report: survey_2023.csv
#> ╚══════════════════════════════════════╝
#>
#> ✔ EXACT MATCHES (5):
#> alder -> alder
#> yrkesstatus -> yrkesstatus
#> ...
#>
#> 🚫 KEYWORD/PREDICTED MATCHES - must verify (1):
#> io_kjonn -> kjonn matched by keyword [kjonn]out <- clean_dataset(
dt = raw,
config = config,
year_tag = 2023L,
dataset_label = "survey_2023.csv"
)
# The cleaned data.table
out$data
#> year kjonn alder yrkesstatus siv antpers tob1
#> 1: 2023 2 75 2 3 1 2
#> 2: 2023 2 42 1 1 2 2
#> ...
# Issues found during cleaning - check this before saving
out$issues
#> variable issue_type detail
#> 1: siv coerce_failure Target: integer. 1 value(s) -> NA: D. ...
#> 2: tob1 unexpected_values Values not declared in YAML: 8, 9. ...clean_dataset() never silently corrupts data. If a column cannot be
safely converted to its declared type, it is left unchanged and the
problem is recorded in out$issues. Fix the YAML or the source data and
re-run.
validate_dataset(out$data, config, "survey_2023.csv")
#>
#> ================================================
#> Validation Report: survey_2023.csv
#> ================================================
#> Rows : 20
#> Columns : year, kjonn, alder, yrkesstatus, siv, antpers, tob1
#>
#> Missingness per column:
#> year 0%
#> kjonn 0%
#> alder 0%
#> yrkesstatus 0%
#> siv 5% <-- 1 row could not be coerced
#> antpers 0%
#> tob1 0%
#>
#> Value conformance issues:
#> ⚠ tob1 unexpected: 8, 9 valid: 1, 2
#> ================================================# Useful after interactive exploration - produce the audit trail on demand
export_match_log(
match_result = match_result,
issues_dt = out$issues,
dataset_label = "survey_2023.csv",
year_tag = 2023L,
log_dir = "logs/matching"
)
#> Log directory: C:/Projects/survey/logs/matching
#> Match log saved: .../match_log_survey_2023_csv.csv
#> Issues log saved: .../issues_survey_2023_csv.csv
#> Master log updated: .../match_log_MASTER.csv (6 total decisions)summarise_master_log("logs/matching")
#>
#> ╔══════════════════════════════════════════════════════╗
#> MASTER AUDIT SUMMARY - All Datasets
#> ╚══════════════════════════════════════════════════════╝
#>
#> Decisions by match method (all datasets):
#> exact 268
#> unmatched 2
#> keyword [kjonn] 1
#>
#> Columns flagged for review per dataset:
#> survey_2023.csv (year 2023): 1 column(s)
#> survey_2024.csv (year 2024): 1 column(s)
#>
#> ⚠⚠ ALL PREDICTED (keyword) MATCHES - verify these:
#> [survey_2023.csv | 2023] io_kjonn -> kjonn (keyword [kjonn])Three CSV files are maintained in log_dir throughout the life of a
project:
| File | Written | Purpose |
|---|---|---|
match_log_<dataset>.csv |
Once per dataset per run | Column-level matching decisions for one file |
issues_<dataset>.csv |
When problems exist | Coercion failures and unexpected values |
match_log_MASTER.csv |
Cumulative across all runs | All decisions from all datasets in one place |
The master log is the primary audit tool. Open it in Excel or load it
with fread() to see every decision made across all years. Re-running a
dataset replaces its existing rows in the master log - no duplication.
master_log <- data.table::fread("logs/matching/match_log_MASTER.csv", quote = "")
# All predicted matches - need human verification
master_log[grepl("^keyword", method)]
# All columns that were dropped because they could not be matched
master_log[method == "unmatched", .(dataset, year, raw_name)]| Function | Purpose |
|---|---|
load_config() |
Parse variable_map.yml into R lookup structures |
match_columns() |
Map raw column names to canonical names (3 layers) |
print_match_report() |
Print a formatted matching summary to the console |
clean_dataset() |
Recode, coerce, and validate one dataset |
validate_dataset() |
Print a post-clean quality report |
export_match_log() |
Write per-dataset and master audit log CSVs |
summarise_master_log() |
Print a cross-dataset summary from the master log |
run_pipeline() |
Orchestrate all of the above for a manifest of files |
build_keyword_patterns() |
Compile regex patterns from YAML keyword lists |
Every function in this table is exported and can be called
independently. See vignette("cuci-introduction") for detailed examples
of when and why to use each one on its own.
vignette("cuci-introduction")- full walkthrough with single-function examples and four common workflows?run_pipeline,?clean_dataset,?export_match_log- function reference pages- GitHub Issues - bug reports and feature requests