TST(string dtype): Resolve xfails in pytables #60795

rhshadrach · 2025-01-26T13:23:32Z

closes #xxxx (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Looks like using where that results in empty will still give object dtype. xfailing those tests here and plan to tackle in a followup.

pandas/io/pytables.py

…xfail_pytable_misc

rhshadrach · 2025-01-29T02:36:24Z

pandas/tests/io/pytables/test_append.py

+            if using_infer_string:
+                # TODO: Test is incorrect when not using_infer_string.
+                #       Should take the last 4 rows uncondiationally.
+                expected = expected[16:]


Would like to make sure this is correct.

So the string type is just truncating the last 4 rows? Is it an invalid unicode sequence?

The DataFrame is all NaN in rows 0-15 inclusive (L250 in this PR), and it is being appended to the store with dropna=True.

So this is currently a bug then? (that indirectly gets fixed somehow by the presence of a string dtype? do you know if this also happens if there is no string column?)

@jorisvandenbossche - the cases in this test above show that when dropna=True, the NA rows are dropped for e.g. floats.

I missed L239:

# nan some entire rows (string are still written!)

This goes back to #3013. I am wondering if at one point the line df.ix[0:15,:] = np.nan did not put the NaNs in string columns, in which case it makes sense to not drop these rows?

In any case, the behavior for string dtype is now working equivalently to other dtypes.

…xfail_pytable_misc

WillAyd · 2025-02-03T14:17:07Z

pandas/tests/io/pytables/test_append.py

+            if using_infer_string:
+                # TODO: Test is incorrect when not using_infer_string.
+                #       Should take the last 4 rows uncondiationally.
+                expected = expected[16:]


So the string type is just truncating the last 4 rows? Is it an invalid unicode sequence?

WillAyd · 2025-02-03T14:18:48Z

pandas/tests/io/pytables/test_append.py

@@ -822,10 +826,11 @@ def test_append_raise(setup_path):
        df["foo"] = Timestamp("20130101")
        store.append("df", df)
        df["foo"] = "bar"
+        shape = "(30,)" if using_infer_string else "(1, 30)"


Isn't this still a bug? Not sure why we would expect a different shape with the string data types?

I assumed it was due to array-backed vs 1d ndarray backed data. But I haven't checked too deeply.

This error message got changed in #60829; we no longer report on the shape.

WillAyd · 2025-02-03T14:19:51Z

pandas/tests/io/pytables/test_errors.py

+                    "Cannot serialize the column [datetime1]\nbecause its data "
+                    "contents are not [string] but [date] object dtype"
+                ),
+                re.escape("[date] is not implemented as a table column"),


I think the original error message here is much clearer - is there no way to catch and raise that for the string types?

Not sure what you mean, this error is not raised for string types. It's being raised for date types.

When infer_string=False, this function is passed a single object block with a mix of strings and dates. In this case, the data of the block is inferred as mixed, and then checked column-by-column. This is where the top message (which I think is confusing) is raised. When infer_string=True, each string array is fed into this function individually and does not raise. Then the object block is fed in containing only dates. This is inferred as dates, and the corresponding error message is raised.

rhshadrach · 2025-02-08T19:01:27Z

@WillAyd - friendly ping

jorisvandenbossche · 2025-02-08T19:54:13Z

pandas/tests/io/pytables/test_store.py

+@pytest.mark.xfail(
+    using_string_dtype() and has_pyarrow,
+    reason="TODO(infer_string): can't encode '\ud800': surrogates not allowed",
+)


Do you know if the error happens when writing or reading? (I suppose reading because even if it was object dtype we try to read it as strings?)

This is probably not something we can fix? Wondering (not for this PR) if we should fall back to object dtype in such a case (or at least give that option), so the user can still load the data and inspect it.

I hadn't looked into this one - currently it raises on data creation. Fixing that by specifying object dtype, you're right it's the reading that raises here. The error happens in Series.str.decode(encoding, errors=errors). I am wondering if we should add a dtype argument to decode so that you can force object dtype - that seems generally useful to alleviate this pain-point, and then we could use it to provide a fallback in pytables.

PyArrow strings not supporting surrogates is another interesting case in regards to PDEP-13. #58455

I am wondering if we should add a dtype argument to decode so that you can force object dtype - that seems generally useful to alleviate this pain-point, and then we could use it to provide a fallback in pytables.

+1

I listed this issue in #59328 as one of the breaking changes when users upgrade to the string dtype (in this case at least when using pyarrow).

But so we should also list this as one of the differences between the pyarrow and python engines.

I do wonder if we should start nudging users down the path of referring to our strings as utf8. For other encodings, the binary type is the best bet for now, and probably for a while (I'm not aware of any non-utf8 Arrow array work, but maybe its out there)

WillAyd · 2025-02-10T14:24:01Z

Great work thanks @rhshadrach

lumberbot-app · 2025-02-10T14:24:23Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout 2.3.x
git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 4511251ccf409f2ba71cab0283bdf751697ee539

You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #60795: TST(string dtype): Resolve xfails in pytables'

Push to a named branch:

git push YOURFORK 2.3.x:auto-backport-of-pr-60795-on-2.3.x

Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #60795 on branch 2.3.x (TST(string dtype): Resolve xfails in pytables)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

rhshadrach · 2025-02-11T22:19:32Z

@meeseeksdev backport to 2.3.x

lumberbot-app · 2025-02-11T22:20:02Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout 2.3.x
git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 4511251ccf409f2ba71cab0283bdf751697ee539

You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #60795: TST(string dtype): Resolve xfails in pytables'

Push to a named branch:

git push YOURFORK 2.3.x:auto-backport-of-pr-60795-on-2.3.x

Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #60795 on branch 2.3.x (TST(string dtype): Resolve xfails in pytables)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

(cherry picked from commit 4511251)

rhshadrach · 2025-02-11T22:27:19Z

Backport PR: #60916

…0916) * ENH: Improved error message and raise new error for small-string NaN edge case in HDFStore.append (#60829) * Add clearer error messages for datatype mismatch in HDFStore.append. Raise ValueError when nan_rep too large for pytable column. Add and modify applicable test code. * Fix missed tests and correct mistake in error message. * Remove excess comments. Reverse error type change to avoid api changes. Move nan_rep tests into separate function. (cherry picked from commit 57340ec) * TST(string dtype): Resolve xfails in pytables (#60795) (cherry picked from commit 4511251) * Adjust test --------- Co-authored-by: Jake Thomas Trevallion <[email protected]>

…ading with condition (#60967) * ENH: Improved error message and raise new error for small-string NaN edge case in HDFStore.append (#60829) * Add clearer error messages for datatype mismatch in HDFStore.append. Raise ValueError when nan_rep too large for pytable column. Add and modify applicable test code. * Fix missed tests and correct mistake in error message. * Remove excess comments. Reverse error type change to avoid api changes. Move nan_rep tests into separate function. (cherry picked from commit 57340ec) * TST(string dtype): Resolve xfails in pytables (#60795) (cherry picked from commit 4511251) * BUG(string dtype): Resolve pytables xfail when reading with condition (#60943) (cherry picked from commit 0ec5f26) --------- Co-authored-by: Jake Thomas Trevallion <[email protected]>

* ENH: Improved error message and raise new error for small-string NaN edge case in HDFStore.append (#60829) * Add clearer error messages for datatype mismatch in HDFStore.append. Raise ValueError when nan_rep too large for pytable column. Add and modify applicable test code. * Fix missed tests and correct mistake in error message. * Remove excess comments. Reverse error type change to avoid api changes. Move nan_rep tests into separate function. (cherry picked from commit 57340ec) * TST(string dtype): Resolve xfails in pytables (#60795) (cherry picked from commit 4511251) * BUG(string dtype): Resolve pytables xfail when reading with condition (#60943) (cherry picked from commit 0ec5f26) * Backport PR #60940: ENH: Add dtype argument to str.decode --------- Co-authored-by: Jake Thomas Trevallion <[email protected]>

…cked strings (#60984) * ENH: Improved error message and raise new error for small-string NaN edge case in HDFStore.append (#60829) * Add clearer error messages for datatype mismatch in HDFStore.append. Raise ValueError when nan_rep too large for pytable column. Add and modify applicable test code. * Fix missed tests and correct mistake in error message. * Remove excess comments. Reverse error type change to avoid api changes. Move nan_rep tests into separate function. (cherry picked from commit 57340ec) * TST(string dtype): Resolve xfails in pytables (#60795) (cherry picked from commit 4511251) * BUG(string dtype): Resolve pytables xfail when reading with condition (#60943) (cherry picked from commit 0ec5f26) * Backport PR #60940: ENH: Add dtype argument to str.decode * Backport PR #60938: ENH(string dtype): Implement cumsum for Python-backed strings --------- Co-authored-by: Jake Thomas Trevallion <[email protected]>

TST(string dtype): Resolve xfails in pytables

120b0bd

rhshadrach added Testing pandas testing functions or related to the test suite IO HDF5 read_hdf, HDFStore Strings String extension data type and string data labels Jan 26, 2025

rhshadrach added this to the 2.3 milestone Jan 26, 2025

Cleanup

adb0349

rhshadrach commented Jan 26, 2025

View reviewed changes

pandas/io/pytables.py Outdated Show resolved Hide resolved

rhshadrach marked this pull request as draft January 26, 2025 13:47

rhshadrach mentioned this pull request Jan 26, 2025

BUG: is_*_array returns true on empty object dtype #60796

Merged

5 tasks

rhshadrach added 4 commits January 28, 2025 20:02

Merge branch 'main' of https://github.com/pandas-dev/pandas into str_…

1413d27

…xfail_pytable_misc

Revert code change

70563ce

type-ignore

d12f66d

xfails and fixes

bc5b697

rhshadrach commented Jan 29, 2025

View reviewed changes

rhshadrach added 2 commits January 30, 2025 16:22

Merge branch 'main' of https://github.com/pandas-dev/pandas into str_…

55ea98b

…xfail_pytable_misc

More strict xfail

cb56243

rhshadrach requested review from jorisvandenbossche and WillAyd February 2, 2025 21:19

rhshadrach marked this pull request as ready for review February 2, 2025 21:19

WillAyd reviewed Feb 3, 2025

View reviewed changes

rhshadrach added 2 commits February 5, 2025 18:09

Merge branch 'main' into str_xfail_pytable_misc

733b123

Merge branch 'main' into str_xfail_pytable_misc

dae84aa

jorisvandenbossche reviewed Feb 8, 2025

View reviewed changes

jorisvandenbossche approved these changes Feb 8, 2025

View reviewed changes

[16:] -> [-4:]

4df00dc

rhshadrach mentioned this pull request Feb 9, 2025

ENH: Improved error message and raise new error for small-string NaN edge case in HDFStore.append #60829

Merged

3 tasks

WillAyd approved these changes Feb 10, 2025

View reviewed changes

WillAyd merged commit 4511251 into pandas-dev:main Feb 10, 2025
42 checks passed

lumberbot-app bot added the Still Needs Manual Backport label Feb 10, 2025

rhshadrach added a commit to rhshadrach/pandas that referenced this pull request Feb 11, 2025

TST(string dtype): Resolve xfails in pytables (pandas-dev#60795)

413563e

(cherry picked from commit 4511251)

rhshadrach mentioned this pull request Feb 16, 2025

ENH: Add dtype argument to str.decode #60940

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST(string dtype): Resolve xfails in pytables #60795

TST(string dtype): Resolve xfails in pytables #60795

rhshadrach commented Jan 26, 2025 •

edited

Loading

rhshadrach Jan 29, 2025

WillAyd Feb 3, 2025

rhshadrach Feb 4, 2025

jorisvandenbossche Feb 8, 2025

rhshadrach Feb 9, 2025

WillAyd Feb 3, 2025

WillAyd Feb 3, 2025

rhshadrach Feb 4, 2025

rhshadrach Feb 5, 2025

WillAyd Feb 3, 2025

rhshadrach Feb 4, 2025 •

edited

Loading

rhshadrach commented Feb 8, 2025

jorisvandenbossche Feb 8, 2025

rhshadrach Feb 9, 2025

rhshadrach Feb 9, 2025

jorisvandenbossche Feb 9, 2025

jorisvandenbossche Feb 9, 2025

WillAyd Feb 10, 2025

WillAyd commented Feb 10, 2025

lumberbot-app bot commented Feb 10, 2025

rhshadrach commented Feb 11, 2025

lumberbot-app bot commented Feb 11, 2025

rhshadrach commented Feb 11, 2025

TST(string dtype): Resolve xfails in pytables #60795

TST(string dtype): Resolve xfails in pytables #60795

Conversation

rhshadrach commented Jan 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhshadrach Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

rhshadrach commented Feb 8, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Feb 10, 2025

lumberbot-app bot commented Feb 10, 2025

rhshadrach commented Feb 11, 2025

lumberbot-app bot commented Feb 11, 2025

rhshadrach commented Feb 11, 2025

rhshadrach commented Jan 26, 2025 •

edited

Loading

rhshadrach Feb 4, 2025 •

edited

Loading