Skip to content

Add col_select parameter to read_resource() #123

Description

@peterdesmet

read_delim() has a col_select parameter that allows to select columns before reading data. This can vastly improve read speed:

file <- "HG_OOSTENDE-gps-2017.csv.gz"
all_cols <- function(x) {
  data <- readr::read_csv(x, progress = FALSE, col_type = readr::cols(.default = "c"))
}
select_cols <- function(x) {
  data <- readr::read_csv(x, progress = FALSE, col_type = readr::cols(.default = "c"), col_select = c("event-id", "timestamp"))
}
microbenchmark(all_cols(file), select_cols(file), times = 3)

Unit: seconds
              expr       min        lq      mean   median       uq      max neval
    all_cols(file) 30.749202 33.715649 38.549023 36.68210 42.44893 48.21577     3
 select_cols(file)  7.999319  9.125165  9.932861 10.25101 10.89963 11.54825     3

I think read_resource() should also allow this.

Implementation

  1. Cf. read_delim() I would name the argument col_select and put it after resource_name, i.e. read_resource(package, resource_name, col_select)
  2. col_select indicates:

Columns to include in the results. You can use the same mini-language as dplyr::select() to refer to the columns by name. Use c() to use more than one selection expression. Although this usage is less common, col_select also accepts a numeric column index. See ?tidyselect::language for full details on the selection language.
Personally, I think a vector of strings should be sufficient ...

  1. Look up the columns in schema and return error if not found. Note, there is some similar code already in:
    field_names_collapse <- paste(field_names, collapse = "`, `")
    col_names <- colnames(data)
    col_names_collapse <- paste(col_names, collapse = "`, `")
    assertthat::assert_that(
    identical(field_names, col_names),
    msg = glue::glue(
    "Field names in `schema` must match column names in data:",
    "\u2139 Field names: `{field_names_collapse}`",
    "\u2139 Column names: `{col_names_collapse}`",
    .sep = "\n"
    )
  2. Reduce schema to selected columns.
  3. Pass col_select to the internal read_delim() function
  4. Return df
  5. Add tests, including one where the column names are selected and thus provided in a different order than in the file.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions