metrics: Object Store

[peterbarnett03]

Object Store Metrics

Right now we serve no information on the object store performance, so this ticket looks to add some basic metrics to track.

Update the /metrics endpoint to serve the following metrics:

New Object Store Retrieval Metrics

• object_store_retrieval_duration_seconds_bucket: Time to complete object store retrievals, bucketed into 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 10, inf seconds, with types
• object_store_retrieval_duration_seconds_sum: Total time spent retrieving objects from object store
• object_store_retrieval_duration_seconds_count: Total number of object store retrieval requests

These metrics also add a bucket param, which while always the same, will make visualization easier when grouping across multiple sets.

E.g.

...
object_store_retrieval_duration_seconds_bucket{bucket="influxdb_data",le="0.05"} 2
object_store_retrieval_duration_seconds_bucket{bucket="influxdb_data",le="0.1"} 3
object_store_retrieval_duration_seconds_bucket{bucket="influxdb_data",le="0.25"} 5
...
object_store_retrieval_duration_seconds_sum{bucket="influxdb_data"} 0.68
object_store_retrieval_duration_seconds_count{bucket="influxdb_data"} 5

New Object Store Persistence Metrics

• object_store_write_duration_seconds_bucket: Time to write to object store, bucketed into 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 10, inf seconds, with types
• object_store_write_duration_seconds_sum: Total time spent writing objects to object store
• object_store_write_duration_seconds_count: Total number of object store write requests

These metrics also add a bucket param, which while always the same, will make visualization easier when grouping across multiple sets. Additionally, they add a type, which can be either parquet, wal, or none.

E.g.

...
object_store_write_duration_seconds_bucket{bucket="influxdb_data",type="parquet",le="0.05"} 2
object_store_write_duration_seconds_bucket{bucket="influxdb_data",type="parquet",le="0.1"} 3
object_store_write_duration_seconds_bucket{bucket="influxdb_data",type="parquet",le="0.25"} 5
...
object_store_write_duration_seconds_sum{bucket="influxdb_data",type="parquet"} 0.68
object_store_write_duration_seconds_count{bucket="influxdb_data",type="parquet"} 5

[hiltontj]

@peterbarnett03 couple things:

1. Re: bucket label. This is only applicable to cloud stores. For file- or memory-based object stores, I think the label can be omitted or none.
2. We might also be able to track the number of bytes transferred to the store, e.g., object_store_write_bytes_total and object_store_retrieve_bytes_total.
3. I will prefix the metric names with influxdb3_ as for other metrics added to monolith.

[hiltontj]

So, I am working on this now and looked at what is implemented in IOx. There is an entire suite of metrics implemented in IOx for the object store, which we have access to today in InfluxDB Core/Enterprise, in the object_store_metrics crate.

These provide metrics similar to what is described in the original issue, with the following differences:

• there is no bucket label
• metrics do not distinguish file type
• metrics are provided for each kind of object store request: put, get, get_ranges, etc., instead of just read vs. write
• there are different categories of metrics: total duration histograms (used by all requests), byte counters (used by get and put type requests), time-to-first-byte duration histograms (used by get requests), object counters (used by list requests)

I would like to propose that we use the suite of metrics from IOx for the following reasons:

• it can be implemented with only a few lines of code
• it will make the metrics reported in this case consistent with IOx
• there are more, finer grained metrics reported than what are described in the original issue

The drawback is that we will not report bucket or file type. I think both bucket and type could be added upstream to IOx, as described in the issue. bucket being known when initializing the object store, and type being extractable from the file path written (which would then include other file types than parquet and wal).

[hiltontj]

#26365 contains the changes needed to leverage the IOx object store metrics.

[peterbarnett03]

Sounds good, thanks Trevor. And I feel like we could still get the bucket metrics in some way, just by pulling from all nodes pushing to the same bucket, though that'd require some additional work figuring out which clusters are writing to the same bucket. But nothing concerning right now. Just thinking out loud.

Appreciate it.

[hiltontj]

@peterbarnett03 I have an upstream PR to get bucket added to the metrics in IOx: https://github.com/influxdata/influxdb_iox/pull/14283

So that will be coming to a Core and Enterprise near you soon.

The type on the other hand may be more tricky. I will follow up on that in this upstream issue on IOx: https://github.com/influxdata/influxdb_iox/issues/14278