Structured logs for each read request

[TomAugspurger]

Kvikio currently emits some logs related to reads. I'm working on optimizing cudf-polars IO, and these logs had a couple of shortcomings:

1. The logs aren't machine readable. The lines are formatted like FileHandle::pread(buf=..., size=..., ...). It didn't have the path being read or the start/end times.
2. Logical reads are logged, rather than physical reads (I think). kvikio automatically breaks large multi-MB reads into smaller chunks. Both are interesting, but I was especially interested in the physical reads.

Have we considered emitting structured logs (in JSON format, say, but this might be configurable)? https://github.com/rapidsai/kvikio/compare/main...TomAugspurger:kvikio:tom/read-log-traces?expand=1 has a POC for that. The lines include

• source (path / URL)
• start, end (int counters)
• offset, size
• threadId
• bytesRead (the bytes actually read, which might differ from size)
• backend (local or remote)
• status
• isDeviceBuffer (maybe this should be a "destination" enum?)
• requestId: correlate with the logical request that induced this physical read (a logical read might be split into multiple physical reads)

It also includes a (private) tool for converting the structured logs to chrome://tracing format, which can be loaded into tools like https://ui.perfetto.dev/. https://gist.github.com/TomAugspurger/135c38917a3a4cbe0cfc6eb80b503bff is an example file, shown here:

If this is helpful, I can clean things up and submit a pR.

[TomAugspurger]

Discussed a bit today and one potential way to improve the POC: we should integrate these new logs into the existing logging framework:

• use the TRACE log level to control whether these physical reads are logged. This pairs nicely with the other settings: =DEBUG is logical and =TRACE is physical (and logical)
• Separate the data logged from the output format with a new parameter. Then we can have all the logs emitted as JSON or the custom strings we've used up till now