fix grafana dashboards dropdowns for data and train dashboard

[alanwguo]

Why are these changes needed?

Previously the dropdown for variables for data and train dashboard wasn't working for a few reasons:

• Data dashboard used the ray_data_allocated_bytes metric which doesn't seem to be guaranteed metric to be emitted when ray data is used
• Both data and train dashboard used label_values which only shows values for live metrics. Since these variables represent entities that are expected to stop emitting metrics over time, I changed to use a query that checks for any values over the time range selected based on the approach suggested here

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[matthewdeng]

Thank you!

[matthewdeng]

Should this be ray_train_run_name instead of SessionName=~\"$SessionName\"?

[alanwguo]

no, this is right. THis is filtering the dropdown to only show train run names that are part of the selected Session.

[matthewdeng]

I realized that ray_train_report_total_blocked_time_s may not be recorded in the case that the user does not call ray.train.report. Would it be better to use a metric that will always be recorded? E.g. right now ray_train_worker_group_start_total_time_s should always be there.