[BUG]: Row group reader failed on string list

[zhouchanghai]

Library Version

6.0.0

.NET Runtime

.net 8

OS and Architecture

Windows (64-bit)

How to reproduce?

When a column is List, the row group reader will fail with error like " System.ArgumentException : Definition levels buffer is too small for field 'Tags/list/element' (length 2 < numValues 2)"

Failing test

using System.Collections.Generic;
    using System.IO;
    using System.Threading.Tasks;
    using Parquet;
    using Parquet.Schema;
    using Parquet.Serialization;
    using Xunit;

public class ParquetNetListColumnReadReproTests
{
    [Fact]
    public async Task ReadStringListColumn_WithLowLevelRowGroupReader_ShouldRoundTrip()
    {
        // Arrange - write a parquet with a single List<string> column using the high-level serializer.
        var records = new List<Record>
        {
            new Record { Tags = new List<string> { "a", "b", "c" } },
            new Record { Tags = new List<string> { "d" } },
        };

        using var stream = new MemoryStream();
        await ParquetSerializer.SerializeAsync(records, stream);
        stream.Position = 0;

        // Act - read the leaf column (Tags/list/element) back with the low-level row group reader.
        await using var reader = await ParquetReader.CreateAsync(stream);
        DataField field = reader.Schema.GetDataFields()[0];

        using var rowGroupReader = reader.OpenRowGroupReader(0);
        var numValues = (int)rowGroupReader.GetMetadata(field)!.MetaData!.NumValues;

        var values = new string[numValues];
        var repetitionLevels = new int[numValues];

        // BUG: this throws "Definition levels buffer is too small for field 'Tags/list/element'".
        await rowGroupReader.ReadAsync(field, values, repetitionLevels);

        // Assert - the four physical string values should have round-tripped.
        Assert.Equal(new[] { "a", "b", "c", "d" }, values);
    }

    private sealed class Record
    {
        public List<string>? Tags { get; set; }
    }
}

[RobSwDev]

Hit the same sort of issue trying to read data from http://ftp.ebi.ac.uk/pub/databases/opentargets/platform/23.12/output/etl/parquet/targets/
The code below throws System.ArgumentException: 'Definition levels buffer is too small for field 'go/list/element/id' (length 326 < numValues 326)'

Is there a way to work around this in 6.x? I am able to read the file in 5.6.

public static class Reader
{
    const string dir = @"MyDownloadsDirectory";
    const string file = @"part-00000-70041866-e16c-42ce-b724-442f59bf453a-c000.snappy.parquet";

    public static async Task Main()
    {
        await using ParquetReader reader = await ParquetReader.CreateAsync(Path.Combine(dir, file));
        DataField field = reader.Schema.DataFields.Single(f => f.Path.ToString() == "go/list/element/id");

        using ParquetRowGroupReader rowGroup = reader.OpenRowGroupReader(0);
        int numValues = (int)rowGroup.GetMetadata(field)!.MetaData!.NumValues;   // 1887

        string[] values = new string[numValues];
        int[] repetitionLevels = new int[numValues];

        // Throws: System.ArgumentException: 'Definition levels buffer is too small for field 'go/list/element/id' (length 326 < numValues 326)'
        await rowGroup.ReadAsync(field, values, repetitionLevels);
    }
}

[RobSwDev]

@aloneguid
I cloned the repo and had a quick play, and this may well be caused by a simple typo on line 220:

parquet-dotnet/src/Parquet/ParquetRowGroupReader.cs

Lines 220 to 221 in 0b86ddb

	using IMemoryOwner<int>? definitionlevels = MemoryOwner<int>.Allocate(field.MaxDefinitionLevel > 0 ? (int)RowCount : 0);
	using IMemoryOwner<ReadOnlyMemory<char>> rawValues = MemoryOwner<ReadOnlyMemory<char>>.Allocate(values.Length);

definitionlevels should be allocated with length values.Length not RowCount. (the same as rawValues is on the line below)

At least, that makes my naive test above pass.

Caveat: I can't claim to know the first thing about the parquet format.