PipeReader + SequenceReader file offset

[bencyoung-Fignum]

I'm trying to write a file processor that captures the file offset of the start of each line in a very large file. I'm currently using PipeReader and SequenceReader for performance, and I can implement the line reading, but I'm finding it very hard to reverse engineer where I actually am in the file... Does anyone have any advice or examples of how to do this? It's somewhat tempting to go back to Stream and do it manually at this point

[huoyaoyuan]

2 ideas:

• Try to decode how many bytes have been consumed through SequencePosition. I don't know if this achievable when buffer pooling etc are involved.
• Implement a custom FilePipeReader, instead of using PipeReader.Create(FileStream). This is not hard since FileStream is just a wrapper of RandomAccess and tracks the position by itself. You still need to copy the buffer management logic from StreamPipeReader.

[jmelkins]

It's hard to know what might be going wrong without seeing some code, but it is possible you have been hit by the issue #75866

In the pattern where you find the start of the next line, increment a stored long by the number of bytes between that position and the start position, then slice the ReadOnlySequence and loop back to start the search again, if you use SequencePosition.GetOffset to obtain the number of bytes between the current line start and the previous one then you will sometimes get a value that is too large, since GetOffset is dubiously implemented as the offset from the start of the memory buffer, not from the start of the sequence.

The workaround is that issue: #75866 (comment)