`All nested columns must have the same number of elements` error only when using `regex = T`

[damonbayer]

When manually specifying variable names, I am able to gather_draws successfully, but, if I specify the same columns via a regular expression, I receive an error:

library(tidybayes)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)

my_tidy_draws <- 
  expand_grid(.chain = 1:2, .iteration = 1:5) |> 
  mutate(.draw = tidybayes:::draw_from_chain_and_iteration_(.chain, .iteration)) |> 
  mutate(`aa[1]` = rnorm(n()), `aa[2]` = rnorm(n()), `ab[1]` = rnorm(n())) |> 
  tidy_draws()


my_tidy_draws |> gather_draws(aa[i], ab[i])
#> # A tibble: 30 × 6
#> # Groups:   i, .variable [3]
#>        i .chain .iteration .draw .variable  .value
#>    <int>  <int>      <int> <int> <chr>       <dbl>
#>  1     1      1          1     1 aa         0.0500
#>  2     1      1          2     2 aa        -0.187 
#>  3     1      1          3     3 aa         1.35  
#>  4     1      1          4     4 aa         0.637 
#>  5     1      1          5     5 aa         0.856 
#>  6     1      2          1     6 aa        -0.0731
#>  7     1      2          2     7 aa        -1.37  
#>  8     1      2          3     8 aa         1.49  
#>  9     1      2          4     9 aa        -0.765 
#> 10     1      2          5    10 aa        -1.39  
#> # ℹ 20 more rows
my_tidy_draws |> gather_draws(`a.*`[i], regex = T)
#> Error in `unnest_legacy()`:
#> ! All nested columns must have the same number of elements.

^{Created on 2025-01-07 with reprex v2.1.1}

[damonbayer]

Having looked at the source code a bit, it's not really surprising that this happens -- the regex = T case is handled quite differently from regex = F. I wonder if the code base could be simplified by processing the regex ahead of time to create a list of non-regex expressions.

[Lionel-Re]

Hello,
I am new to this package and, together with @FANTA-TCHAKOUNTE, under the supervision of @katossky and @JMeyer31, we are working on solving this issue as part of an assignment at ENSAI (School of Statistical Engineering, France).
We have read the contributing guidelines and will try to be as autonomous as possible, but we may need to ask for guidance from time to time.
We have a solid background in statistics and a reasonable one in computer science. If any maintainer feels that this issue might be too ambitious for beginners, please let us know. If you have time, we would also appreciate it if you could guide us towards a project or issue where our help might be more useful.

Best regards,
Lionel

[damonbayer]

@Lionel-Re I opened up a PR to show the beginning of a potential solution: #332. I am not a maintainer of the project, but I'm happy to help you, if you are interested in solving the issue.

[Lionel-Re]

@damonbayer thanks a lot for your proposed solution! I've tested your modification (replacing unnest_legacy() with unnest()):
long_draws = unnest(nested_draws, -!dimension_names)
It works perfectly on my side, and I no longer encounter the issue of extra rows filled with NA in the .value column.
From your perspective, is the problem fully solved by this modification, or do you think there's anything else we should verify or improve?

Thanks again for your great help!

[FANTA-TCHAKOUNTE]

@damonbayer thanks a lot for your proposed solution! I've tested your modification (replacing unnest_legacy() with unnest()):

long_draws <- nested_draws %>%
  unnest(-!!dimension_names) %>%
  drop_na()

This works perfectly on my side, and it fully resolves the issue by removing the extra rows filled with NA values.
I think adding the drop_na() step right after the unnest() ensures that no unwanted NA rows appear in the final result.
Would you agree with integrating this solution into the PR? Or do you think there's anything else we should consider before finalizing?
Thanks again for your great help!

[damonbayer]

@Lionel-Re @FANTA-TCHAKOUNTE

1. For clarity, I think you should probably drop_na only on the .value column.
2. Using drop_na will drop values that are intentionally NA, but this may be okay. Maybe @mjskay can comment.
3. You should add a test to demonstrate that the issue is fixed and prevent developers from accidentally re-introducing the bug.

You can make a PR against my branch to add the drop_na and the test: https://github.com/damonbayer/tidybayes/tree/fix_330

[FANTA-TCHAKOUNTE]

Hello @damonbayer,
Thanks again for your input and your suggestion to drop the NA values from the .value column. After further testing and consideration, we decided to take a different approach.
Instead of using drop_na() right after the unnest(), we opted to handle the issue within the data processing pipeline by filtering out the NA values during the process of gathering and unnesting the variables. This way, we maintain more control over which specific rows and columns are affected, ensuring that we only drop the unwanted NA values and preserve valid data.
We’ve tested this new approach, and it works as expected, keeping the data clean and ready for further analysis. We appreciate your insight and will make sure to integrate this approach into the next steps.
Thanks for your help!
Best regards,
@Lionel-Re & @FANTA-TCHAKOUNTE