Categorical Data documentation is missing data frame usage examples

[nathanrboyer]

The DataFrames.jl documentation on Categorical Data never actually shows how to use these types in a data frame. I assume the intent is to replace a data frame column with a CategoricalVector or PooledVector, but I don't see how to do that illustrated in the documentation.

[bkamins]

Can you please suggest what example would be useful? I can add it.

[nathanrboyer]

Having never used CategoricalArrays.jl or PooledArrays.jl, I can't make a good suggestion on what to add. To me, that entire page looks like it belongs in the CategoricalArrays.jl documentation since it never mentions a data frame df. It only shows to how to work with the categorical vector cv independently. I assume since the documentation page is there, CategoricalArrays can be used within DataFrames, but I don't know how.

[bkamins]

There are tutorials showing how to do it that are linked in the main page. Here is a specific file https://raw.githubusercontent.com/bkamins/Julia-DataFrames-Tutorial/refs/heads/master/07_factors.ipynb.
In general there is nothing special about these types (except for the fact to know that they exist- and I guess this was the intent of the material you linked).

[nathanrboyer]

I assume the point is that one should use cdf over df for this type of categorical data to improve memory usage and speed of the data frame, so I guess I would prove that with data frame examples and benchmarks. I didn't know types other than Vector were allowed as DataFrame columns, and that usage is not shown on the current documentation page.

using DataFrames, CategoricalArrays
v = ["Group A", "Group B", "Group A", "Group A", "Group B", "Group A"]
df = DataFrame(value = 1:6, group = v)
cv = categorical(v)
cdf = DataFrame(value = 1:6, group = cv)

[bkamins]

For speed and memory PooledVector is used CategoricalVector is used when you indeed need to model categorical data. Performace is also discussed in the tutorials linked to the documentation, specifically here https://github.com/bkamins/Julia-DataFrames-Tutorial/blob/master/11_performance.ipynb.

[nathanrboyer]

I've still not seen anything demonstrating why these types would be useful to use within a data frame.

The second paragraph of the documentation says this:

... we can represent the data more efficiently by replacing the strings with indices into a small pool of levels. There are two benefits of doing this. The first is that such vectors will tend to use less memory. The second is that they can be efficiently grouped using the groupby function.

There are two common types that allow to perform level pooling:

• PooledVector from PooledArrays.jl;
• CategoricalVector from CategoricalArrays.jl.

It suggests that memory footprint and efficiency is the main benefit for both packages.

My testing shows a modest benefit to the categorical data frame in these metrics, except when sorting:

using CategoricalArrays, DataFrames, Chairmarks

data = rand(10000)                                # vector of length 10,000 with randomized numerical elements between 0 and 1
v = rand(('A', 'B'), 10000)                       # vector of length 10,000 with randomized 'A' or 'B' elements
cv = categorical(v, ordered=true, compress=true)  # categorical vector version of `v` (~72 μs)

df_standard = DataFrame(category = v, data = data)      # data frame using `v` (~5 μs)
df_categorical = DataFrame(category = cv, data = data)  # data frame using `cv` (~4 μs)

@b unique(df_standard.category)     # unique values in data frame category column (~25 μs)
@b levels(df_categorical.category)  # unique values in data frame category column (~0.031 μs)
# However, the creation of `cv` with the `categorical` function takes 3x longer than `unique`.

@b groupby(df_standard, :category)     # create grouped data frame by category (~19 μs)
@b groupby(df_categorical, :category)  # create grouped data frame by category (~15 μs)

@b sort(df_standard, :category)     # sort the original dataframe (~125 μs)
@b sort(df_categorical, :category)  # sort the original dataframe (~375 μs)

Base.summarysize(df_standard)     # Memory size of data frame (121,000 bytes)
Base.summarysize(df_categorical)  # Memory size of data frame (91,000 bytes)

I don't see any non-performance benefits demonstrated that I can't already do easily with a regular vector or data frame.
What data frame manipulations become easier when data is stored as a CategoricalVector?

[nalimilan]

Char columns isn't the best test as it's a relatively small and efficient type. Try with String columns.

[nathanrboyer]

My point stands either way.
If the desired objective of using CategoricalArrays is performance improvement of data frame operations, then those operations and speedups should be shown in the documentation.
If the desired objective of using CategoricalArrays is new functionality or convenience enabled for data frame operations, then those novel operations should be shown off in the documentation.

Again, there are no data frames used at all on that page of the documentation, so I don't get what it is trying to tell me about how I should be working with data frames containing categorical data.

[bkamins]

The original point of this manual entry is to inform the user about this functionality. It is unrelated to DataFrames.jl, but it is often wanted by users (especially those coming from R), so we thought it is useful to inform them that such functionality in the ecosystem exists. The point of existence of this package is described here: https://categoricalarrays.juliadata.org/stable/.

In particular, there are no functions in DataFrames.jl that allow to work with categorical data and this package is not in [deps] of DataFrames.jl in https://github.com/JuliaData/DataFrames.jl/blob/main/Project.toml. This is a different case from missing that is a next entry in the manual (for which there are special functions in DataFrames.jl).

Having said that we can add an example how to put a CategoricalVector into a data frame, like e.g. df = DataFrame(cv=cv) using a cv vector from the example, but this is all to it. You just put a CategoricalVector column into a data frame, if you want. Would this addition be useful?

[nalimilan]

Maybe it would make sense to show how to convert all String columns to categorical using transform(df, names(df, String) .=> categorical, renamecols=false)?

[nathanrboyer]

Even if you include how to put CategoricalVectors into a DataFrame, which would be a good addition, I am still missing examples of why I would want to do that.

https://categoricalarrays.juliadata.org/stable/ says:

"The main feature of CategoricalArray is that it maintains a pool of the levels which can ... [#1 be] used for pretty printing (e.g. in cross tables or plots) ... [or #2 for] comparing values using the < and > operators."

The documentation shows #2: how to add, order, and compare the levels themselves. However, I don't see any examples of how to #1: use that level data information to do something useful, especially as it relates to grouping/printing/ploting/etc. data stored in the columns next to the CategoricalVector columns.

[bkamins]

In general I think that all these would be good additions but to CategoricalArrays.jl package (and maybe we should link from DataFrames.jl to that part of the documentation). The reason is that you ask: "Why do we need CategoricalArrays.jl?". And I think this question should be answered in a package that provides this functionality.