JSON: null-valued typed path is dropped from the returned JsonObject instead of being read as null

[polyglotAI-bot]

Describe the bug

When reading a JSON column in JsonReadMode.Binary (the default), a typed path whose value is SQL NULL is dropped from the returned JsonObject entirely. The key is absent rather than present with a JSON null, so a caller cannot distinguish "path not present in this row" from "path present but null". This is a lossy read for Nullable(...) typed paths.

The server itself reports the path with an explicit null, so the driver diverges from the server's own JSON rendering.

For a nested typed path (JSON(a.b Nullable(Int64))), the entire parent subtree disappears too — the document comes back as {}.

Scope of what I measured:

• Affects JsonReadMode.Binary (default) and, by code path, JsonReadMode.None. JsonReadMode.String is not affected (the raw server string is returned verbatim).
• Not specific to backtick-quoted paths — reproduces with a plain unquoted path.
• Affects hinted Nullable(...) typed paths only. Nulls nested inside containers (Array(Nullable(Int32)), Map(String, Nullable(Int64))) are preserved correctly, and unhinted dynamic paths are correctly absent (see "Expected behaviour").

Steps to reproduce

1. Create a table with a typed Nullable JSON path and insert a null for it:

   CREATE OR REPLACE TABLE t (data JSON(x Nullable(Int64))) ENGINE = Memory;
   INSERT INTO t VALUES ('{"x": null}');

2. Read the column with the driver in the default JsonReadMode.Binary.
3. Observe that the returned JsonObject is {} and ContainsKey("x") is false.

Expected behaviour

The path should be present with a JSON null value — {"x":null} — matching what the server reports.

Server ground truth (same server the driver was talking to, ClickHouse 26.5.1.882):

$ curl -s --data-binary "SELECT CAST('{\"x\": null}' AS JSON(x Nullable(Int64))) AS d FORMAT JSONEachRow"
{"d":{"x":null}}

$ curl -s --data-binary "SELECT CAST('{\"a\":{\"b\":null}}' AS JSON(a.b Nullable(Int64))) AS d FORMAT JSONEachRow"
{"d":{"a":{"b":null}}}

$ curl -s --data-binary "SELECT CAST('{\"s\":null}' AS JSON(s Nullable(String))) AS d FORMAT JSONEachRow"
{"d":{"s":null}}

Contrast case that must keep its current behaviour — for an unhinted / dynamic path the server omits the key itself, so the driver dropping it is correct:

$ curl -s --data-binary "SELECT CAST('{\"x\":null}' AS JSON) AS d FORMAT JSONEachRow"
{"d":{}}

So the fix should apply to hinted paths, not become a blanket "always materialize a null key".

Code example

Reproduction run as NUnit tests in ClickHouse.Driver.Tests (inserting via SQL so only the read path is exercised):

private async Task<JsonObject> ReadJson(string colType, string json, JsonReadMode mode = JsonReadMode.Binary)
{
    using var c = TestUtilities.GetTestClickHouseClient(jsonReadMode: mode);
    var t = CreateTableName();
    await c.ExecuteNonQueryAsync($"CREATE OR REPLACE TABLE {t} (data {colType}) ENGINE = Memory");
    await c.ExecuteNonQueryAsync($"INSERT INTO {t} VALUES ('{json}')");
    using var reader = await c.ExecuteReaderAsync($"SELECT data FROM {t}");
    Assert.That(reader.Read(), Is.True);
    return (JsonObject)reader.GetValue(0);
}

[Test]
public async Task UnquotedTypedNullablePath_Null()
{
    var r = await ReadJson("JSON(x Nullable(Int64))", "{\"x\": null}");
    Assert.That(r.ContainsKey("x"), Is.True);   // fails: false, document is {}
}

[Test]
public async Task NestedTypedNullablePath_Null()
{
    var r = await ReadJson("JSON(a.b Nullable(Int64))", "{\"a\":{\"b\":null}}");
    Assert.That(r.ContainsKey("a"), Is.True);   // fails: false, whole subtree gone
}

Error log

Observed results (3 failing, 5 passing out of the 8 shapes I swept):

Failed R1_UnquotedTypedNullablePath_Null
  R1 json={} containsKey=False
  Assert.That(r.ContainsKey("x"), Is.True) -> Expected: True  But was: False

Failed R2_NullableString_Null
  R2 json={} containsKey=False

Failed R3_NestedTypedNullablePath_Null
  R3 json={}
  whole subtree vanished

Passed R4_StringReadMode_Null                  (JsonReadMode.String unaffected)
Passed C1_NullableWithValue_Present            (Nullable path with value 42)
Passed C2_ArrayNullsPreserved                  (Array(Nullable(Int32)) [1,null,3])
Passed C3_MapNullValue                         (Map(String, Nullable(Int64)) null value)
Passed C4_UnhintedDynamicNull_AbsentIsCorrect  (plain JSON, absence is correct)

Root cause

ClickHouse.Driver/Types/JsonType.cs:74, in ReadAsJsonObject:

HintedTypes.TryGetValue(name, out var hintedType);
if (ReadJsonNode(reader, hintedType) is not { } jsonNode)
{
    continue;
}

ReadJsonNode → ReadJsonValue maps a SQL NULL to a CLR null (JsonType.cs:388, the null => null arm). The is not { } pattern is a null test, so a null value takes the continue branch and the path is never added to the JsonObject — the value has already been consumed from the reader, it is simply discarded. For a dotted path the continue also happens before the parent-object walk (JsonType.cs:79-92), which is why the whole subtree is missing rather than just the leaf.

Worth noting the write path is deliberately asymmetric with this: WritePocoFields (JsonType.cs:238-248) goes out of its way to write nulls for hinted Nullable types only. So the driver writes such a null faithfully and then cannot read it back.

The existing test Write_WithNullableHintedProperty_ShouldWriteNull (JsonTypeTests.cs:899) does not catch this because it asserts result["Value"] Is.Null — the JsonObject indexer returns null both for an absent key and for a present JSON null, so the assertion passes either way.

Suggested fix

In ReadAsJsonObject, distinguish "no value" from "null value". Since the value is already fully consumed from the reader before the check, the null can simply be materialized: walk the parent path and assign JsonValue-null (i.e. a JSON null) instead of continue-ing — but only when the path is a hinted Nullable one, so that the unhinted/dynamic contrast case above keeps omitting the key.

Write_WithNullableHintedProperty_ShouldWriteNull should also be strengthened to assert ContainsKey, not just an indexer-null, so the round trip is actually pinned.

Note this is independent of #503 (backtick-quoted typed paths) — it reproduces with a plain unquoted path on current main.

Configuration

Environment

• Client version: current main (1912030)
• Language version: C# / .NET 10
• .NET version: net10.0
• OS: Ubuntu 24.04 (x64, Docker)

ClickHouse server

• ClickHouse Server version: 26.5.1.882
• ClickHouse Server non-default settings, if any: none
• CREATE TABLE statements for tables involved:

  CREATE OR REPLACE TABLE t (data JSON(x Nullable(Int64))) ENGINE = Memory;

• Sample data: INSERT INTO t VALUES ('{"x": null}');

---

_{Found by automated analysis of the client while working on #502 / #503 (JSON quoted typed paths), where the round-trip case was switched to a non-null value rather than depending on this behaviour. Verified against a live ClickHouse server rather than by code inspection.}