collapse version 2.0.1

• %in% with set_collapse(mask = "%in%") does not warn about overidentification when used with data frames.

• Fixed several typos in the documentation.

collapse version 2.0.0

collapse 2.0, released in Mid-October 2023, introduces fast table joins and data reshaping capabilities alongside other convenience functions, and enhances the packages global configurability, including interactive namespace control.

Potentially breaking changes

• In a grouped setting, if .data is used inside fsummarise() and fmutate(), and .cols = NULL, .data will contain all columns except for grouping columns (in-line with the .SD syntax of data.table). Before, .data contained all columns. The selection in .cols still refers to all columns, thus it is still possible to select all columns using e.g. grouped_data %>% fsummarise(some_expression_involving(.data), .cols = seq_col(.)).

Other changes

• In qsu(), argument vlabels was renamed to labels. But vlabels will continue to work.

Bug Fixes

• Fixed a bug in the integer methods of fsum(), fmean() and fprod() that returned NA if and only if there was a single integer followed by NA's e.g fsum(c(1L, NA, NA)) erroneously gave NA. This was caused by a C-level shortcut that returned NA when the first element of the vector had been reached (moving from back to front) without encountering any non-NA-values. The bug consisted in the content of the first element not being evaluated in this case. Note that this bug did not occur with real numbers, and also not in grouped execution. Thanks @blset for reporting (#432).

Additions

• Added join(): class-agnostic, vectorized, and (default) verbose joins for R, modeled after the polars API. Two different join algorithms are implemented: a hash-join (default, if sort = FALSE) and a sort-merge-join (if sort = TRUE).

• Added pivot(): fast and easy data reshaping! It supports longer, wider and recast pivoting, including handling of variable labels, through a uniform and parsimonious API. It does not perform data aggregation, and by default does not check if the data is uniquely identified by the supplied ids. Underidentification for 'wide' and 'recast' pivots results in the last value being taken within each group. Users can toggle a duplicates check by setting check.dups = TRUE.

• Added rowbind(): a fast class-agnostic alternative to rbind.data.frame() and data.table::rbindlist().

• Added fmatch(): a fast match() function for vectors and data frames/lists. It is the workhorse function of join(), and also benefits ckmatch(), %!in%, and new operators %iin% and %!iin% (see below). It is also possible to set_collapse(mask = "%in%") to replace base::"%in%" using fmatch(). Thanks to fmatch(), these operators also all support data frames/lists of vectors, which are compared row-wise.

• Added operators %iin% and %!iin%: these directly return indices, i.e. %[!]iin% is equivalent to which(x %[!]in% table). This is useful especially for subsetting where directly supplying indices is more efficient e.g. x[x %[!]iin% table] is faster than x[x %[!]in% table]. Similarly fsubset(wlddev, iso3c %iin% c("DEU", "ITA", "FRA")) is very fast.

• Added vec(): efficiently turn matrices or data frames / lists into a single atomic vector. I am aware of multiple implementations in other packages, which are mostly inefficient. With atomic objects, vec() simply removes the attributes without copying the object, and with lists it directly calls C_pivot_longer.

Improvements

• set_collapse() now supports options 'mask' and 'remove', giving collapse a flexible namespace in the broadest sense that can be changed at any point within the active session:

  • 'mask' supports base R or dplyr functions that can be masked into the faster collapse versions. E.g. library(collapse); set_collapse(mask = "unique") (or, equivalently, set_collapse(mask = "funique")) will create unique <- funique in the collapse namespace, export unique() from the namespace, and detach and attach the namespace again so R can find it. The re-attaching also ensures that collapse comes right after the global environment, implying that all it's functions will take priority over other libraries. Users can use fastverse::fastverse_conflicts() to check which functions are masked after using set_collapse(mask = ...). The option can be changed at any time. Using set_collapse(mask = NULL) removes all masked functions from the namespace, and can also be called simply to ensure collapse is at the top of the search path.

  • 'remove' allows removing arbitrary functions from the collapse namespace. E.g. set_collapse(remove = "D") will remove the difference operator D(), which also exists in stats to calculate symbolic and algorithmic derivatives (this is a convenient example but not necessary since collapse::D is S3 generic and will call stats::D() on R calls, expressions or names). This is safe to do as it only modifies which objects are exported from the namespace (it does not truly remove objects from the namespace). This option can also be changed at any time. set_collapse(remove = NULL) will restore the exported namespace.

  For both options there exist a number of convenient keywords to bulk-mask / remove functions. For example set_collapse(mask = "manip", remove = "shorthand") will mask all data manipulation functions such as mutate <- fmutate and remove all function shorthands such as mtt (i.e. abbreviations for frequently used functions that collapse supplies for faster coding / prototyping).

• set_collapse() also supports options 'digits', 'verbose' and 'stable.algo', enhancing the global configurability of collapse.

• qM() now also has a row.names.col argument in the second position allowing generation of rownames when converting data frame-like objects to matrix e.g. qM(iris, "Species") or qM(GGDC10S, 1:5) (interaction of id's).

• as_factor_GRP() and finteraction() now have an argument sep = "." denoting the separator used for compound factor labels.

• alloc() now has an additional argument simplify = TRUE. FALSE always returns list output.

• frename() supports both new = old (pandas, used to far) and old = new (dplyr) style renaming conventions.

• across() supports negative indices, also in grouped settings: these will select all variables apart from grouping variables.

• TRA() allows shorthands "NA" for "replace_NA" and "fill" for "replace_fill".

• group() experienced a minor speedup with >= 2 vectors as the first two vectors are now hashed jointly.

• fquantile() with names = TRUE adds up to 1 digit after the comma in the percent-names, e.g. fquantile(airmiles, probs = 0.001) generates appropriate names (not 0% as in the previous version).

collapse version 1.9.6

• New vignette on collapse's handling of R objects.

• print.descr() with groups and option perc = TRUE (the default) also shows percentages of the group frequencies for each variable.

• funique(mtcars[NULL, ], sort = TRUE) gave an error (for data frame with zero rows). Thanks @NicChr (#406).

• Added SIMD vectorization for fsubset().

• vlengths() now also works for strings, and is hence a much faster version of both lengths() and nchar(). Also for atomic vectors the behavior is like lengths(), e.g. vlengths(rnorm(10)) gives rep(1L, 10).

• In collap[v/g](), the ... argument is now placed after the custom argument instead of after the last argument, in order to better guard against unwanted partial argument matching. In particular, previously the n argument passed to fnth was partially matched to na.last. Thanks @ummel for alerting me of this (#421).

collapse version 1.9.5

• Using DATAPTR_RO to point to R lists because of the use of ALTLISTS on R-devel.

• Replacing != loop controls for SIMD loops with < to ensure compatibility on all platforms. Thanks @albertus82 (#399).

collapse version 1.9.4

• Improvements in get_elem()/has_elem(): Option invert = TRUE is implemented more robustly, and a function passed to get_elem()/has_elem() is now applied to all elements in the list, including elements that are themselves list-like. This enables the use of inherits to find list-like objects inside a broader list structure e.g. get_elem(l, inherits, what = "lm") fetches all linear model objects inside l.

• Fixed a small bug in descr() introduced in v1.9.0, producing an error if a data frame contained no numeric columns - because an internal function was not defined in that case. Also, POSIXct columns are handled better in print - preserving the time zone (thanks @cdignam-chwy #392).

• fmean() and fsum() with g = NULL, as well as TRA(), setop(), and related operators %r+%, %+=% etc., setv() and fdist() now utilize Single Instruction Multiple Data (SIMD) vectorization by default (if OpenMP is enabled), enabling potentially very fast computing speeds. Whether these instructions are utilized during compilation depends on your system. In general, if you want to max out collapse on your system, consider compiling from source with CFLAGS += -O3 -march=native -fopenmp and CXXFLAGS += -O3 -march=native in your .R/Makevars.

collapse version 1.9.3

• Added functions fduplicated() and any_duplicated(), for vectors and lists / data frames. Thanks @NicChr (#373)

• sort option added to set_collapse() to be able to set unordered grouping as a default. E.g. setting set_collapse(sort = FALSE) will affect collap(), BY(), GRP(), fgroup_by(), qF(), qG(), finteraction(), qtab() and internal use of these functions for ad-hoc grouping in fast statistical functions. Other uses of sort, for example in funique() where the default is sort = FALSE, are not affected by the global default setting.

• Fixed a small bug in group() / funique() resulting in an unnecessary memory allocation error in rare cases. Thanks @NicChr (#381).

collapse version 1.9.2

• Further fix to an Address Sanitizer issue as required by CRAN (eliminating an unused out of bounds access at the end of a loop).

• qsu() finally has a grouped_df method.

• Added options option("collapse_nthreads") and option("collapse_na.rm"), which allow you to load collapse with different defaults e.g. through an .Rprofile or .fastverse configuration file. Once collapse is loaded, these options take no effect, and users need to use set_collapse() to change .op[["nthreads"]] and .op[["na.rm"]] interactively.

• Exported method plot.psmat() (can be useful to plot time series matrices).

collapse version 1.9.1

• Fixed minor C/C++ issues flagged by CRAN's detailed checks.

• Added functions set_collapse() and get_collapse(), allowing you to globally set defaults for the nthreads and na.rm arguments to all functions in the package. E.g. set_collapse(nthreads = 4, na.rm = FALSE) could be a suitable setting for larger data without missing values. This is implemented using an internal environment by the name of .op, such that these defaults are received using e.g. .op[["nthreads"]], at the computational cost of a few nanoseconds (8-10x faster than getOption("nthreads") which would take about 1 microsecond). .op is not accessible by the user, so function get_collapse() can be used to retrieve settings. Exempt from this are functions .quantile, and a new function .range (alias of frange), which go directly to C for maximum performance in repeated executions, and are not affected by these global settings. Function descr(), which internally calls a bunch of statistical functions, is also not affected by these settings.

• Further improvements in thread safety for fsum() and fmean() in grouped computations across data frame columns. All OpenMP enabled functions in collapse can now be considered thread safe i.e. they pass the full battery of tests in multithreaded mode.

collapse version 1.9.0

collapse 1.9.0 released mid of January 2023, provides improvements in performance and versatility in many areas, as well as greater statistical capabilities, most notably efficient (grouped, weighted) estimation of sample quantiles.

Changes to functionality

• All functions renamed in collapse 1.6.0 are now depreciated, to be removed end of 2023. These functions had already been giving messages since v1.6.0. See help("collapse-renamed").

• The lead operator F() is not exported anymore from the package namespace, to avoid clashes with base::F flagged by multiple people. The operator is still part of the package and can be accessed using collapse:::F. I have also added an option "collapse_export_F", such that setting options(collapse_export_F = TRUE) before loading the package exports the operator as before. Thanks @matthewross07 (#100), @edrubin (#194), and @arthurgailes (#347).

• Function fnth() has a new default ties = "q7", which gives the same result as quantile(..., type = 7) (R's default). More details below.

Bug Fixes

• fmode() gave wrong results for singleton groups (groups of size 1) on unsorted data. I had optimized fmode() for singleton groups to directly return the corresponding element, but it did not access the element through the (internal) ordering vector, so the first element/row of the entire vector/data was taken. The same mistake occurred for fndistinct if singleton groups were NA, which were counted as 1 instead of 0 under the na.rm = TRUE default (provided the first element of the vector/data was not NA). The mistake did not occur with data sorted by the groups, because here the data pointer already pointed to the first element of the group. (My apologies for this bug, it took me more than half a year to discover it, using collapse on a daily basis, and it escaped 700 unit tests as well).

• Function groupid(x, na.skip = TRUE) returned uninitialized first elements if the first values in x where NA. Thanks for reporting @Henrik-P (#335).

• Fixed a bug in the .names argument to across(). Passing a naming function such as .names = function(c, f) paste0(c, "-", f) now works as intended i.e. the function is applied to all combinations of columns (c) and functions (f) using outer(). Previously this was just internally evaluated as .names(cols, funs), which did not work if there were multiple cols and multiple funs. There is also now a possibility to set .names = "flip", which names columns f_c instead of c_f.

• fnrow() was rewritten in C and also supports data frames with 0 columns. Similarly for seq_row(). Thanks @NicChr (#344).

Additions

• Added functions fcount() and fcountv(): a versatile and blazing fast alternative to dplyr::count. It also works with vectors, matrices, as well as grouped and indexed data.

• Added function fquantile(): Fast (weighted) continuous quantile estimation (methods 5-9 following Hyndman and Fan (1996)), implemented fully in C based on quickselect and radixsort algorithms, and also supports an ordering vector as optional input to speed up the process. It is up to 2x faster than stats::quantile on larger vectors, but also especially fast on smaller data, where the R overhead of stats::quantile becomes burdensome. For maximum performance during repeated executions, a programmers version .quantile() with different defaults is also provided.

• Added function fdist(): A fast and versatile replacement for stats::dist. It computes a full euclidian distance matrix around 4x faster than stats::dist in serial mode, with additional gains possible through multithreading along the distance matrix columns (decreasing thread loads as the matrix is lower triangular). It also supports computing the distance of a matrix with a single row-vector, or simply between two vectors. E.g. fdist(mat, mat[1, ]) is the same as sqrt(colSums((t(mat) - mat[1, ])^2))), but about 20x faster in serial mode, and fdist(x, y) is the same as sqrt(sum((x-y)^2)), about 3x faster in serial mode. In both cases (sub-column level) multithreading is available. Note that fdist does not skip missing values i.e. NA's will result in NA distances. There is also no internal implementation for integers or data frames. Such inputs will be coerced to numeric matrices.

• Added function GRPid() to easily fetch the group id from a grouping object, especially inside grouped fmutate() calls. This addition was warranted especially by the new improved fnth.default() method which allows orderings to be supplied for performance improvements. See commends on fnth() and the example provided below.

• fsummarize() was added as a synonym to fsummarise. Thanks @arthurgailes for the PR.

• C API: collapse exports around 40 C functions that provide functionality that is either convenient or rather complicated to implement from scratch. The exported functions can be found at the bottom of src/ExportSymbols.c. The API does not include the Fast Statistical Functions, which I thought are too closely related to how collapse works internally to be of much use to a C programmer (e.g. they expect grouping objects or certain kinds of integer vectors). But you are free to request the export of additional functions, including C++ functions.

Improvements

• fnth() and fmedian() were rewritten in C, with significant gains in performance and versatility. Notably, fnth() now supports (grouped, weighted) continuous quantile estimation like fquantile() (fmedian(), which is a wrapper around fnth(), can also estimate various quantile based weighted medians). The new default for fnth() is ties = "q7", which gives the same result as (f)quantile(..., type = 7) (R's default). OpenMP multithreading across groups is also much more effective in both the weighted and unweighted case. Finally, fnth.default gained an additional argument o to pass an ordering vector, which can dramatically speed up repeated invocations of the function on the dame data:

  # Estimating multiple weighted-grouped quantiles on mpg: pre-computing an ordering provides extra speed. 
  mtcars %>% fgroup_by(cyl, vs, am) %>%
      fmutate(o = radixorder(GRPid(), mpg)) %>% # On grouped data, need to account for GRPid()
      fsummarise(mpg_Q1 = fnth(mpg, 0.25, o = o, w = wt),
                 mpg_median = fmedian(mpg, o = o, w = wt),
                 mpg_Q3 = fnth(mpg, 0.75, o = o, w = wt))
  # Note that without weights this is not always faster. Quickselect can be very efficient, so it depends 
  # on the data, the number of groups, whether they are sorted (which speeds up radixorder), etc...

• BY now supports data-length arguments to be passed e.g. BY(mtcars, mtcars$cyl, fquantile, w = mtcars$wt), making it effectively a generic grouped mapply function as well. Furthermore, the grouped_df method now also expands grouping columns for output length > 1.

• collap(), which internally uses BY with non-Fast Statistical Functions, now also supports arbitrary further arguments passed down to functions to be split by groups. Thus users can also apply custom weighted functions with collap(). Furthermore, the parsing of the FUN, catFUN and wFUN arguments was improved and brought in-line with the parsing of .fns in across(). The main benefit of this is that Fast Statistical Functions are now also detected and optimizations carried out when passed in a list providing a new name e.g. collap(data, ~ id, list(mean = fmean)) is now optimized! Thanks @ttrodrigz (#358) for requesting this.

• descr(), by virtue of fquantile and the improvements to BY, supports full-blown grouped and weighted descriptions of data. This is implemented through additional by and w arguments. The function has also been turned into an S3 generic, with a default and a 'grouped_df' method. The 'descr' methods as.data.frame and print also feature various improvements, and a new compact argument to print.descr, allowing a more compact printout. Users will also notice improved performance, mainly due to fquantile: on the M1 descr(wlddev) is now 2x faster than summary(wlddev), and 41x faster than Hmisc::describe(wlddev). Thanks @statzhero for the request (#355).

• radixorder is about 25% faster on characters and doubles. This also benefits grouping performance. Note that group() may still be substantially faster on unsorted data, so if performance is critical try the sort = FALSE argument to functions like fgroup_by and compare.

• Most list processing functions are noticeably faster, as checking the data types of elements in a list is now also done in C, and I have made some improvements to collapse's version of rbindlist() (used in unlist2d(), and various other places).

• fsummarise and fmutate gained an ability to evaluate arbitrary expressions that result in lists / data frames without the need to use across(). For example: mtcars |> fgroup_by(cyl, vs, am) |> fsummarise(mctl(cor(cbind(mpg, wt, carb)), names = TRUE)) or mtcars |> fgroup_by(cyl) |> fsummarise(mctl(lmtest::coeftest(lm(mpg ~ wt + carb)), names = TRUE)). There is also the possibility to compute expressions using .data e.g. mtcars |> fgroup_by(cyl) |> fsummarise(mctl(lmtest::coeftest(lm(mpg ~ wt + carb, .data)), names = TRUE)) yields the same thing, but is less efficient because the whole dataset (including 'cyl') is split by groups. For greater efficiency and convenience, you can pre-select columns using a global .cols argument, e.g. mtcars |> fgroup_by(cyl, vs, am) |> fsummarise(mctl(cor(.data), names = TRUE), .cols = .c(mpg, wt, carb)) gives the same as above. Three Notes about this:

  • No grouped vectorizations for fast statistical functions i.e. the entire expression is evaluated for each group. (Let m...

collapse version 1.8.9

• Fixed some warnings on rchk and newer C compilers (LLVM clang 10+).

• .pseries / .indexed_series methods also change the implicit class of the vector (attached after "pseries"), if the data type changed. e.g. calling a function like fgrowth on an integer pseries changed the data type to double, but the "integer" class was still attached after "pseries".

• Fixed bad testing for SE inputs in fgroup_by() and findex_by(). See #320.

• Added rsplit.matrix method.

• descr() now by default also reports 10% and 90% quantiles for numeric variables (in line with STATA's detailed summary statistics), and can also be applied to 'pseries' / 'indexed_series'. Furthermore, descr() itself now has an argument stepwise such that descr(big_data, stepwise = TRUE) yields computation of summary statistics on a variable-by-variable basis (and the finished 'descr' object is returned invisibly). The printed result is thus identical to print(descr(big_data), stepwise = TRUE), with the difference that the latter first does the entire computation whereas the former computes statistics on demand.

• Function ss() has a new argument check = TRUE. Setting check = FALSE allows subsetting data frames / lists with positive integers without checking whether integers are positive or in-range. For programmers.

• Function get_vars() has a new argument rename allowing select-renaming of columns in standard evaluation programming, e.g. get_vars(mtcars, c(newname = "cyl", "vs", "am"), rename = TRUE). The default is rename = FALSE, to warrant full backwards compatibility. See #327.

• Added helper function setattrib(), to set a new attribute list for an object by reference + invisible return. This is different from the existing function setAttrib() (note the capital A), which takes a shallow copy of list-like objects and returns the result.