Performance report compute time stats are inaccurate with lots of tasks

[gjoseph92]

If there are more tasks than the task stream length limit (distributed.scheduler.dashboard.tasks.task-stream-length), then I think the calculation of "number of tasks", "compute time", etc. would be an underestimate, because the deque of tasks would roll over. When generating the performance report, we're just summing up data from the task stream:

distributed/distributed/scheduler.py

Lines 7327 to 7335 in 51a63ea

	task_stream = self.get_task_stream(start=start)
	total_tasks = len(task_stream)
	timespent = defaultdict(int)
	for d in task_stream:
	for x in d.get("startstops", []):
	timespent[x["action"]] += x["stop"] - x["start"]
	tasks_timings = ""
	for k in sorted(timespent.keys()):
	tasks_timings += f"\n<li> {k} time: {format_time(timespent[k])} </li>"

I'd propose that in Scheduler.performance_report, if total_tasks == self.plugins[TaskStreamPlugin.name].buffer.maxlen (or something like that, but written more nicely), we just prepend add a >= to every value in the performance report ("number of tasks: >=100000", "compute time: >= 123456s", etc.).

[jrbourbeau]

Thanks for reporting @gjoseph92. That makes sense. It'd be great if we could find a way to have accurate metrics for this portion in the performance report when exceeding task-stream-length but, as you suggested, even just adding a >= would be a nice improvement.

[ypogorelova]

Hi @jrbourbeau and @gjoseph92, I'm looking at this issue and have a question. As far as I understand if I have the length of the task stream that is greater than both settings distributed.scheduler.dashboard.status.task-stream-length and distributed.scheduler.dashboard.tasks.task-stream-length, it divides it into chunks with a length that equal to max(distributed.scheduler.dashboard.status.task-stream-length, distributed.scheduler.dashboard.tasks.task-stream-length) and calculates tasks_timings for each chunk. It seems to me as accurate behavior. Am I missing something here? Could you please guys share more details of what is expected behavior here?

[ypogorelova]

Hi @jrbourbeau! Did you have a chance to take a look at my comment above?