AttributeError: 'pyarrow.lib.DataType' object has no attribute 'tz'

[zzl221000]

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/deadpool.py", line 871, in raw_runner2
    results = fn(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^
  File "/app/src/xx_data_tool/pipeline/base.py", line 191, in start_from_toml
    pipeline.run()
  File "/app/src/xx_data_tool/pipeline/base.py", line 146, in run
    self.sink.write(processed)
  File "/app/src/xx_data_tool/pipeline/connects/odps_connect.py", line 48, in write
    self.write(item)
  File "/app/src/xx_data_tool/pipeline/connects/odps_connect.py", line 45, in write
    writer.write(pa.Table.from_batches([batch]).to_pandas())
  File "/app/.venv/lib/python3.12/site-packages/odps/models/tableio.py", line 546, in write
    self._write_contents(writer, *args)
  File "/app/.venv/lib/python3.12/site-packages/odps/models/tableio.py", line 683, in _write_contents
    writer.write(arg)
  File "/app/.venv/lib/python3.12/site-packages/odps/tunnel/io/writer.py", line 1066, in write
    super(BufferedArrowWriter, self).write(data)
  File "/app/.venv/lib/python3.12/site-packages/odps/tunnel/io/writer.py", line 893, in write
    col = self._localize_timezone(column_dict[lower_name])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/odps/tunnel/io/writer.py", line 753, in _localize_timezone
    if col.type.tz is not None:
       ^^^^^^^^^^^
AttributeError: 'pyarrow.lib.DataType' object has no attribute 'tz'

[wjsi]

Could you please give an example of data you are writing?

[zzl221000]

@wjsi
Data cannot be provided, but the writing process can be explained:
Use DuckDB to read the NDJSON file,
then convert it to pandas for writing.

[wjsi]

@wjsi Data cannot be provided, but the writing process can be explained: Use DuckDB to read the NDJSON file, then convert it to pandas for writing.

We need at least types of data or piece of mock data to spot what actually happens. This description is too general. Also you need to provide your PyODPS version as well as version of pyarrow.

[zzl221000]

@wjsi
versions

pyarrow==20.0.0
pydops==0.12.5

odps type is datetime
It seems related to the exception below

  File "/app/.venv/lib/python3.12/site-packages/odps/models/tableio.py", line 552, in write
    self._write_contents(writer, *args)
    │    │               │        └ (                   rid  entity_id fin_no first_annt_dt info_update_dt  \
    │    │               │          0              7029333  189686395  44093    2025-09...
    │    │               └ <odps.tunnel.io.writer.BufferedArrowWriter object at 0x7f909cad5940>
    │    └ <function TableArrowWriter._write_contents at 0x7f912bceed40>
    └ <odps.models.tableio.TableArrowWriter object at 0x7f9128c97b30>
  File "/app/.venv/lib/python3.12/site-packages/odps/models/tableio.py", line 693, in _write_contents
    writer.write(arg)
    │      │     └                    rid  entity_id fin_no first_annt_dt info_update_dt  \
    │      │       0              7029333  189686395  44093    2025-09-...
    │      └ <function BufferedArrowWriter.write at 0x7f912bb654e0>
    └ <odps.tunnel.io.writer.BufferedArrowWriter object at 0x7f909cad5940>
  File "/app/.venv/lib/python3.12/site-packages/odps/tunnel/io/writer.py", line 1087, in write
    super(BufferedArrowWriter, self).write(data)
          │                    │           └                    rid  
entity_id fin_no first_annt_dt info_update_dt  \
          │                    │             0              7029333  189686395  44093    2025-09-...
          │                    └ <odps.tunnel.io.writer.BufferedArrowWriter object at 0x7f909cad5940>
          └ <class 'odps.tunnel.io.writer.BufferedArrowWriter'>
  File "/app/.venv/lib/python3.12/site-packages/odps/tunnel/io/writer.py", line 914, in write
    col = self._localize_timezone(column_dict[lower_name])
          │    │                  │           └ 'send_end_datetime'
          │    │                  └ {'rid': <pyarrow.lib.ChunkedArray object at 0x7f911c769000>
          │    │                    [
          │    │                      [
          │    │                        7029333,
          │    │                        2305843009220693287,
          │    │                        34587645138274802...
          │    └ <classmethod(<function BaseArrowWriter._localize_timezone at 0x7f912bb64a40>)>
          └ <odps.tunnel.io.writer.BufferedArrowWriter object at 0x7f909cad5940>
  File "/app/.venv/lib/python3.12/site-packages/odps/tunnel/io/writer.py", line 774, in _localize_timezone
    if col.type.tz is not None:
       │   └ <attribute 'type' of 'pyarrow.lib.ChunkedArray' objects>
       └ <pyarrow.lib.ChunkedArray object at 0x7f909ca220e0>
         [
         6 nulls
         ]
AttributeError: 'pyarrow.lib.DataType' object has no attribute 'tz'

[wjsi]

Seems caused by columns with all nulls. We will try to fix it in the next release. Thanks for report. Currently you may try converting the column which is assumed to be datetime as datetime type in pandas before writing to circumvent this issue.