Skip to content

Commit 038d82c

Browse files
committed
minor updates to data quality
1 parent 7a01903 commit 038d82c

2 files changed

Lines changed: 6 additions & 6 deletions

File tree

239 KB
Loading

materials/sections/data-quality.qmd

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,13 @@ An overall score is then calculated for each dataset, according to the following
1414
$$score = \frac{R_{pass} + O_{pass}}{R_{pass} + R_{fail} + O_{pass}}$$
1515
where R is the number of Required checks that pass or fail, and O is the number of Optional checks that pass or fail. This ensures that datasets are not penalized for failing optional checks. This system allows the Arctic Data Center to evaluate FAIR-ness of its datasets over time. The figure below shows the quality scores of the repository through time for metadata FAIRness.
1616

17-
![](images/adc-fair-timeseries.png){width="80%" fig-align="center"}
17+
![](images/adc-fair-timeseries.png){fig-align="center"}
1818
To evaluate data quality over a wide range of data types and disciplines, the Arctic Data Center first conceptualized four categories of data quality checks. These categories are: congruency, accessibility, validity, and accuracy.
1919

2020
#### Congruency {.unnumbered}
2121

2222
**Is the file what it says it is?**
2323

24-
Examples:
2524
- data format matches format listed
2625
- checksum of file matches documentation
2726
- variable names in file match documentation
@@ -30,7 +29,6 @@ Examples:
3029

3130
**Can the file be accessed and read correctly?**
3231

33-
Examples:
3432
- text formatted files valid according to subtype (eg: csv files match format)
3533
- binary files validate against declared type (eg: hdf5 is valid against HDF5 standard)
3634
- characters in text files are valid against declared encoding
@@ -83,15 +81,17 @@ Warnings are issued for checks that failed, but are optional. Within each row, o
8381
Some output is collapsed for readability:
8482

8583
![](images/adc-data-quality-warning-detail.png){fig-align="center"}
84+
The figure below shows a summary of the percentage of files that pass our data quality checks, by check:
85+
86+
![](images/results-by-check.png){fig-align="center"}
8687

87-
At the Arctic Data Center we can use these checks to gain insights about our data holdings. TODO: maybe insert figures here?
8888

8989
### Checking Data Quality in R
9090

9191
Poll:
9292

93-
- What are the most common quality issues you see? (free text)
94-
- What kind of data quality checks do you use on your own data? (free text)
93+
- What are the most common quality issues you see?
94+
- What kind of data quality checks do you use on your own data?
9595
- Do you have ideas for checks you think we should add to the ADC quality suite?
9696

9797
Although the Arctic Data Center has implemented data quality checks that are automated, the vast majority of data quality work still falls on researchers doing analysis. In this section we will discuss a few tools that researchers can use in R to do general data quality work.

0 commit comments

Comments
 (0)