https://pandas.pydata.org/community/blog/pandas-3.0.html
Dedicated string data type by default: string columns are now inferred as the new str dtype instead of object, providing better performance and type safety
When I run command I get error:
df.to_parquet("test.parquet", engine="fastparquet")
df.to_parquet("test.parquet", engine="fastparquet")
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/h/aa/lib/python3.13/site-packages/pandas/core/frame.py", line 3135, in to_parquet
return to_parquet(
self,
...<7 lines>...
**kwargs,
)
File "/home/h/aa/lib/python3.13/site-packages/pandas/io/parquet.py", line 490, in to_parquet
impl.write(
~~~~~~~~~~^
df,
^^^
...<6 lines>...
**kwargs,
^^^^^^^^^
)
^
File "/home/juris/postpro/lib/python3.13/site-packages/pandas/io/parquet.py", line 342, in write
self.api.write(
~~~~~~~~~~~~~~^
path,
^^^^^
...<4 lines>...
**kwargs,
^^^^^^^^^
)
^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 1340, in write
write_simple(filename, data, fmd,
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
row_group_offsets=row_group_offsets,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
compression=compression, open_with=open_with,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
has_nulls=None, append=False, stats=stats)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 1001, in write_simple
write_to_file(f)
~~~~~~~~~~~~~^^^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 985, in write_to_file
rg = make_row_group(f, row_group, fmd.schema,
compression=compression, stats=stats)
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 802, in make_row_group
chunk = write_column(f, coldata, column,
compression=comp, stats=st)
File "/home/juris/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 635, in write_column
repetition_data, definition_data, encode[encoding](data, selement), 8 * b'\x00'
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 388, in encode_plain
out = convert(data, se)
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 298, in convert
raise ValueError('Error converting column "%s" to bytes using '
'encoding %s. Original error: '
'%s' % (data.name, ct, e))
ValueError: Error converting column "NAME" to bytes using encoding UTF8. Original error: Unable to avoid copy while creating an array as requested.
This error can solve if I column NAME change to object.
df["NAME"] = df["NAME"].astype("object")
I think there are some problems with pandas 3.0 type "str" conversion to bytes.
https://pandas.pydata.org/community/blog/pandas-3.0.html
Dedicated string data type by default: string columns are now inferred as the new str dtype instead of object, providing better performance and type safety
When I run command I get error:
df.to_parquet("test.parquet", engine="fastparquet")
File "/home/h/aa/lib/python3.13/site-packages/pandas/core/frame.py", line 3135, in to_parquet
return to_parquet(
self,
...<7 lines>...
**kwargs,
)
File "/home/h/aa/lib/python3.13/site-packages/pandas/io/parquet.py", line 490, in to_parquet
impl.write(
~~~~~~~~~~^
df,
^^^
...<6 lines>...
**kwargs,
^^^^^^^^^
)
^
File "/home/juris/postpro/lib/python3.13/site-packages/pandas/io/parquet.py", line 342, in write
self.api.write(
~~~~~~~~~~~~~~^
path,
^^^^^
...<4 lines>...
**kwargs,
^^^^^^^^^
)
^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 1340, in write
write_simple(filename, data, fmd,
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
row_group_offsets=row_group_offsets,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
compression=compression, open_with=open_with,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
has_nulls=None, append=False, stats=stats)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 1001, in write_simple
write_to_file(f)
~~~~~~~~~~~~~^^^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 985, in write_to_file
rg = make_row_group(f, row_group, fmd.schema,
compression=compression, stats=stats)
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 802, in make_row_group
chunk = write_column(f, coldata, column,
compression=comp, stats=st)
File "/home/juris/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 635, in write_column
repetition_data, definition_data, encode[encoding](data, selement), 8 * b'\x00'
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 388, in encode_plain
out = convert(data, se)
File "/home/h/aa/lib/python3.13/site-packages/fastparquet/writer.py", line 298, in convert
raise ValueError('Error converting column "%s" to bytes using '
'encoding %s. Original error: '
'%s' % (data.name, ct, e))
ValueError: Error converting column "NAME" to bytes using encoding UTF8. Original error: Unable to avoid copy while creating an array as requested.
This error can solve if I column NAME change to object.
df["NAME"] = df["NAME"].astype("object")
I think there are some problems with pandas 3.0 type "str" conversion to bytes.