Open
Description
Currently the roundtrip tests (
) only use a simple data generation logic, which doesn't include any missing values, as far as I see:dataframe-interchange-tests/tests/strategies.py
Lines 75 to 87 in 8c1fc08
Currently the roundtrip tests (
) only use a simple data generation logic, which doesn't include any missing values, as far as I see:dataframe-interchange-tests/tests/strategies.py
Lines 75 to 87 in 8c1fc08
Activity
honno commentedon Jan 10, 2023
Yeah this would be neat. Quite possible but a bit tricky to implement, given we'd need to figure out what protocol adopters allow missing values in which scenarios (i.e. dtypes) for an initial "missing values mask" (not necessarily an actual mask, just what we use on our end to figure out where missing values exist for a given test example), and a translation of said mask for every protocol adopter in
wrappers.py
.jorisvandenbossche commentedon Jan 10, 2023
Another option could be to not create numpy arrays (like in the snippet above), but plain python lists of scalar values for a column, and then let each library convert that to their native representation in wrappers.py
mock_to_toplevel
.For example, then we could say that missing values are represented as
None
(eg["a", "b", None]
for strings). That avoids needing a sentinel or mask approach to make it work with numpy arrays.honno commentedon Jan 10, 2023
Ah I wasn't concerned about masking the original numpy arrays, as I assume it's easier to create an initial test example (an adopter's native column object) and then create/manipulate a new test example from that with "randomly" placed missing values (either/both sentinel or mask where appropriate).
Just to note I initially was going to generate plain Python lists but found all the adopters consumed numpy arrays, so thought I could utilise Hypothesis'
nps.arrays()
tool to very easily generated complex test examples for all the dtypes.