read_delim() has a col_select parameter that allows to select columns before reading data. This can vastly improve read speed:
file <- "HG_OOSTENDE-gps-2017.csv.gz"
all_cols <- function(x) {
data <- readr::read_csv(x, progress = FALSE, col_type = readr::cols(.default = "c"))
}
select_cols <- function(x) {
data <- readr::read_csv(x, progress = FALSE, col_type = readr::cols(.default = "c"), col_select = c("event-id", "timestamp"))
}
microbenchmark(all_cols(file), select_cols(file), times = 3)
Unit: seconds
expr min lq mean median uq max neval
all_cols(file) 30.749202 33.715649 38.549023 36.68210 42.44893 48.21577 3
select_cols(file) 7.999319 9.125165 9.932861 10.25101 10.89963 11.54825 3
I think read_resource() should also allow this.
Implementation
- Cf.
read_delim() I would name the argument col_select and put it after resource_name, i.e. read_resource(package, resource_name, col_select)
col_select indicates:
Columns to include in the results. You can use the same mini-language as dplyr::select() to refer to the columns by name. Use c() to use more than one selection expression. Although this usage is less common, col_select also accepts a numeric column index. See ?tidyselect::language for full details on the selection language.
Personally, I think a vector of strings should be sufficient ...
- Look up the columns in
schema and return error if not found. Note, there is some similar code already in:
|
field_names_collapse <- paste(field_names, collapse = "`, `") |
|
col_names <- colnames(data) |
|
col_names_collapse <- paste(col_names, collapse = "`, `") |
|
assertthat::assert_that( |
|
identical(field_names, col_names), |
|
msg = glue::glue( |
|
"Field names in `schema` must match column names in data:", |
|
"\u2139 Field names: `{field_names_collapse}`", |
|
"\u2139 Column names: `{col_names_collapse}`", |
|
.sep = "\n" |
|
) |
- Reduce schema to selected columns.
- Pass
col_select to the internal read_delim() function
- Return df
- Add tests, including one where the column names are selected and thus provided in a different order than in the file.
read_delim() has a
col_selectparameter that allows to select columns before reading data. This can vastly improve read speed:I think
read_resource()should also allow this.Implementation
read_delim()I would name the argumentcol_selectand put it afterresource_name, i.e.read_resource(package, resource_name, col_select)col_selectindicates:schemaand return error if not found. Note, there is some similar code already in:frictionless-r/R/check_schema.R
Lines 60 to 70 in 421c22f
col_selectto the internalread_delim()function