An error occurred in a middle of a select stream processing should be thrown

[slvrtrn]

Min repro (note: wait_end_of_query setting is not used):

import { createClient } from '@clickhouse/client'

void (async () => {
  const client = createClient()
  let rowsCount = 0
  try {
    const resultSet = await client.query({
      query: `
        SELECT
          throwIf(number = 2, 'There was an error in the stream!') AS e,
          randomPrintableASCII(1) AS s
        FROM system.numbers
        LIMIT 2000
      `,
      format: 'JSONEachRow',
      clickhouse_settings: {
        max_block_size: '1',
      },
    })
    console.log(await resultSet.json())
  } catch (err) {
    console.error(`Got ${rowsCount} rows before an error:`, err)
  }
  await client.close()
})()

Prints:

[
  { e: 0, s: '}' },
  {
    exception: "Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 2_UInt8) :: 3, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 2_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.8.3.59 (official build))"
  }
]

However, the client should handle and throw during the stream consumption instead of providing the error as a row.

Caveat: the stream error representation depends on the format.

For example:

• JSON, JSONCompact, JSONStrings, JSONCompactStrings - the exception resides in the exception field and can be extracted from there

  {
      "meta":
      [
    	  {
    		  "name": "throwIf(equals(number, 2), 'There was an error in the stream!')",
    		  "type": "UInt8"
    	  },
    	  {
    		  "name": "randomPrintableASCII(2000)",
    		  "type": "String"
    	  }
      ],
  
      "data":
      [
  
      ],
  
      "rows": 0,
  
      "exception": "Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 2_UInt8) :: 3, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 2_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.8.3.59 (official build))"
  }
  

• JSONObjectEachRow behaves similarly to JSON

  curl "http://localhost:8123" --data-binary "select throwIf(number = 10, 'There was an error in the stream!') AS e, randomPrintableASCII(2) AS s from system.numbers limit 3000 SETTINGS max_block_size=1 FORMAT JSONObjectEachRow"
  {
      "row_1": {"e":0,"s":"hK"},
      "row_2": {"e":0,"s":"RT"},
      "row_3": {"e":0,"s":"dT"},
      "row_4": {"e":0,"s":"JU"},
      "row_5": {"e":0,"s":"W="},
      "row_6": {"e":0,"s":"a&"},
      "exception": "Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 10_UInt8) :: 3, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 10_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.8.3.59 (official build))"
  }
  

• CSV, TSV - those formats contain the error in the last line

  curl "http://localhost:8123" --data-binary "select throwIf(number = 2, 'There was an error in the stream!'), randomPrintableASCII(2) from system.numbers limit 3 SETTINGS max_block_size=1 FORMAT TSV" 
  0	NK
  Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 2_UInt8) :: 4, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 2_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.8.3.59 (official build))
  

• JSONEachRow

  curl "http://localhost:8123" --data-binary "select throwIf(number = 2, 'There was an error in the stream!') AS e, randomPrintableASCII(2) AS s from system.numbers limit 3 SETTINGS max_block_size=1 FORMAT JSONEachRow"
  {"e":0,"s":"4S"}
  {"exception": "Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 2_UInt8) :: 4, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 2_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.8.3.59 (official build))"}
  

• JSONCompact*EachRow

  curl "http://localhost:8123" --data-binary "select throwIf(number = 5, 'There was an error in the stream!') AS e, randomPrintableASCII(2) AS s from system.numbers limit 3000 SETTINGS max_block_size=1 FORMAT JSONCompactEachRow"
  [0, "$,"]
  ["Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 5_UInt8) :: 3, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 5_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.8.3.59 (official build))"]
  

• Worst case: JSONColumnsWithMetadata

  curl "http://localhost:8123" --data-binary "select throwIf(number = 200, 'There was an error in the stream!'), randomPrintableASCII(2000) from system.numbers limit 300 SETTINGS max_block_size=1 FORMAT JSONColumnsWithMetadata"
  {
      "meta":
      [
    	  {
    		  "name": "throwIf(equals(number, 200), 'There was an error in the stream!')",
    		  "type": "UInt8"
    	  },
    	  {
    		  "name": "randomPrintableASCII(2000)",
    		  "type": "String"
    	  }
      ]Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 200_UInt8) :: 3, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 200_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.8.3.59 (official build))
  

• Parquet: probably won't be able to catch it, unless we add parquet-wasm as a package dependency to parse the last row

[slvrtrn]

Related: ClickHouse/ClickHouse#46426

[mshustov]

ClickHouse seems to attach X-ClickHouse-Exception-Code header, which can be used to detect the problem. see ClickHouse/clickhouse-connect#413 and ClickHouse/clickhouse-java#1879

[mshustov]

@slvrtrn we can test the ClickHouse behaviour after ClickHouse/ClickHouse#68800 (available in 24.11)

[mshustov]

depends on ClickHouse/ClickHouse#75175