Flow Runs Graph - should display more than 20k run results

[MattMacGillivray]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar request and didn't find it.
• I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

The flow runs graph appears limited to the last 20k flow runs that match my criteria.

I should be able to see flow run statistics across a greater set of data.
Grafana handles graphing large datasets like this well by aggregating by time period, allowing you to visualize large sets of data, yet also zoom in to see specific time ranges.

Prefect should allow the same.
Currently, I can only see 1-2hrs of data at a time, which isn't particularly helpful if I am trying to ensure all my runs have a similar duration.

Describe the proposed behavior

Grafana auto aggregates based on the time period you choose to view.
Grafana allows you to display multiple lines on the same graph.
Grafana allows you to select a timerange on the graph itself to zoom in.

https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/

Example Use

In my case, displaying more than 1-2hrs of data if i have lots of flow runs happening.

Additional context

No response

[serinamarie]

Hi @MattMacGillivray, thank you for raising this issue regarding the limitations on displaying flow run results on the flow runs graph. A member of the UI team mentioned this might require a shift in the way we currently visualize data.

We note that the current limitation might seem reasonable from an orchestration standpoint, but from an observability perspective, we recognize the value of supporting larger datasets, possibly through some form of aggregation.

While we do not have anything planned for this on the roadmap, we are very open to improvements that accommodate larger datasets. This is something we're interested in supporting in the long term.

Have a good weekend!