Skip to content

Flaky test test_clickhouse_errors.py::TestOOMError::test_oom, Unexpected EOF while reading bytes #579

@koletzilla

Description

@koletzilla

Describe the bug

Test fails with

FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
 +  where 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes' = RunResult(status=<RunStatus.Error: 'error'>, timing=[TimingInfo(name='compile', started_at=datetime.datetime(2025, 12,...ey=[], time_spine=None, previous_batch_results=None, batch=None, _has_this=None), agate_table=None, batch_results=None).message

I can see them happening with the Native/TCP (port 9440) connection. I have not been able to reproduce it with the regular HTTP (port 8443) connection

Looks like this may be related to some of the existing EOF situations that are also found in other languaje clients, but I have not dig enough on it.

Examples

Steps to reproduce

Use the Native protocol (DBT_CH_TEST_PORT=9440) and run

 pytest tests/integration/adapter/clickhouse/test_clickhouse_errors.py --flake-finder --flake-runs=1000 -n=3

In the terminal you can see different errors:

FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[121] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[129] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[89] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[201] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[137] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[65] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[104] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[176] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[74] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n  ClickHouse exception:  Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[34] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[114] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[75] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[202] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[226] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[105] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[35] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Searching for bad queries, I see the ones that the test creates:

select *
from system.query_log
WHERE
event_date = today() 
and event_time > now() - INTERVAL 1 hour
and type > 2
and current_database like '%dbt_clickhouse%'

That returns as expected

Code: 241. DB::Exception: (total) memory limit exceeded: would use 32.99 GiB (attempt to allocate chunk of 32.00 GiB bytes), current RSS: 1011.44 MiB, maximum: 14.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: while executing 'FUNCTION repeat(toString(__table1.number) :: 0, 100000_UInt32 :: 1) -> repeat(toString(__table1.number), 100000_UInt32) String : 2'. (MEMORY_LIMIT_EXCEEDED) (version 25.8.1.8702 (official build))

ClickHouse server

  • ClickHouse Server version: 25.8.1.8702

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions