Performance issue seen by CMS for ROOT's use of tbb::task_arena

[Dr15Jones]

Explain what you would like to see improved and how.

ROOT's use of their own tbb::task_arena (from #14807) can cause the main thread of a program to be left with no work to do.

The issue is the following. CMS creates its own tbb::task_arena in the main thread (to control the number of threads the application is allowed to use). Then in that arena, we do a tbb::task::wait until the application has finished all work. That means the main thread is part of CMS's arena. Now say we call a ROOT API (e.g. TTree::Fill) which behind the scenes makes use of tbb concurrency. For simplicity, let us say the application is using 2 threads. If the ROOT API happens to be called from the main thread (a 50/50 chance) then ROOT's arena temporarily takes control of the main thread and the ROOT API makes use of both the main thread and the one TBB worker thread in the job. On the other hand, if the ROOT API is called from the TBB worker thread, that worker thread becomes part of the ROOT arena but the main thread does not. This means the ROOT API only has access to 1 thread and the main thread will remain idle.

We have seen this issue in real world applications.

We understand that for the non-framework user's use the EnableImplicitMT(<number of threads>) API needs the ability to set the number of of threads to use and that does require the use of a tbb::task_arena. However, for an application that is controlling the threads directly via TBB already (like the CMS framework) we request that a different API (either passing a special value to EnableImplicitMT or a new function entirely) which will cause ROOT to use the tbb::task_arena already in use by the calling thread. This can even still work with

root/core/imt/src/RTaskArena.cxx

Line 93 in 9b8ed94

RTaskArenaWrapper::RTaskArenaWrapper(unsigned maxConcurrency) : fTBBArena(new ROpaqueTaskArena{})

by being able to pass oneapi::tbb::attach to the constructor of ROpaqueTaskArena.

E.g.

RTaskArenaWrapper::RTaskArenaWrapper(unsigned maxConcurrency) : 
  fTBBArena(maxConcurrency == RTaskArenaWrapper::Attach? new ROpaqueTaskArena{oneapi::tbb::attach} : new ROpaqueTaskArena{})

Where RTaskArenaWrapper::Attach == std::numeric_limits<unsigned>::max()

ROOT version

6.32 and beyond

Installation method

build from source

Operating system

linux

Additional context

No response

[Dr15Jones]

@makortel FYI

[hahnjo]

Hi @Dr15Jones @makortel, if I remember correctly our discussion in #14807 the request was to avoid task stealing across unrelated activities (for example unrelated CMSSW events): #14807 (comment) This (continues to) works because you call this_task_arena::isolate() so an event thread will only pick up descendant tasks, but not unrelated tasks created by other threads. Additional threads not directly employed by CMSSW can "help" by picking up any task.

The situation you are describing here can happen if the framework's task_arena is "out-of-work", but the waiting thread actually doesn't pick up tasks from ROOT's task_arena. Is that understanding correct? I see how sharing a single task_arena should solve this, but would this still guarantee that tasks are not intermingled? Would it be possible to make joining the ROOT task_arena explicit by enqueuing a "final" task that does an explicit call to ROOT?

[makortel]

The situation you are describing here can happen if the framework's task_arena is "out-of-work", but the waiting thread actually doesn't pick up tasks from ROOT's task_arena. Is that understanding correct?

Yes, this is correct.

I see how sharing a single task_arena should solve this, but would this still guarantee that tasks are not intermingled?

As long as this_task_arena::isolate() is used, even with the same task_arena, the tasks will not intermingle.

Example with isolate()

For example, in

task_group group;

group.run([&]{
  // Run in thread A
  this_task_arena::isolate([&] { m_treeOnFile1->Fill(); });
});
group.run([&]{
  // Run in thread B
  this_task_arena::isolate([&] { m_treeOnFile2->Fill(); });
});

// Run in thread C
group.wait();

the this_task_arena::isolate() guarantees that no tasks related to m_treeOnFileN->Fill() will run on the thread that is directly calling the m_treeOnFileM->Fill() when N != M.

• if A != B and A == C then C will only run tasks from m_treeOnFile1->Fill() until that Fill() finishes, and only then could pick tasks from m_treeOnFile2->Fill()
  • if A != B and B == C same as above but flip 1 and 2
• if A != B and A != C and B != C then C can run tasks from both of the m_treeOnFile1->Fill() and m_treeOnFile2->Fill() without restrictions
• in all cases where A != B
  • A will only run tasks from m_treeOnFile1->Fill() until that Fill() finishes, and only then could pick tasks from m_treeOnFile2->Fill()
  • B will only run tasks from m_treeOnFile2->Fill() until that Fill() finishes, and only then could pick tasks from m_treeOnFile1->Fill()

This is the behavior CMS wants, even given the following downside. If A == C or B == C, in the case where there are no more tasks for C to run because some other thread is running the last task from the Fill() started on C, the C thread will be temporarily idle until the last task finishes. This behavior is a consequence of the synchronous nature of the Fill() function. Even so, a given Fill() function will finish at least as fast as it would if only one thread would be used. Without the isolate(), that cannot be guaranteed, because the thread that called the Fill() could pick unrelated tasks that may finish later than the last task of the Fill().

We want the output-related tasks to finish as soon as they can in order to allow other tasks to start. For example output from another event waiting to be written to the same file, or starting to process a new event from the input using the same concurrent event slot.

Same example with explicit task_arena

task_group group;
task_arena arena;

group.run([&]{
  // Run in thread A
  arena.execute([&]{
    this_task_arena::isolate([&] { m_treeOnFile1->Fill(); });
  });
});
group.run([&]{
  // Run in thread B
  arena.execute([&]{
    this_task_arena::isolate([&] { m_treeOnFile2->Fill(); });
  });
});

// Run in thread C
group.wait();

In this case thread C will never run any tasks from either Fill() calls. The threads A and B share tasks as in the previous isolate()-only example, i.e. they execute tasks only from "their own Fill()" until their Fill() finishes.

[makortel]

Would it be possible to make joining the ROOT task_arena explicit by enqueuing a "final" task that does an explicit call to ROOT?

We didn't really understand the question. From where would the "final task" be started? What exactly would the "explicit call to ROOT" be?

[hahnjo]

Same example with explicit task_arena

Hm, to exactly reproduce ROOT's IMT setup, shouldn't arena.execute (from ROOT) be called inside of this_task_arena::isolate (from CMSSW)?

Would it be possible to make joining the ROOT task_arena explicit by enqueuing a "final" task that does an explicit call to ROOT?

We didn't really understand the question. From where would the "final task" be started? What exactly would the "explicit call to ROOT" be?

My idea was to add a task that explicitly transfers over the thread from CMSSW to ROOT:

group.run([&]{
   arena.wait();
});

But given your other reply, this is probably less useful than just sharing the same task_arena. It would also wait for all tasks in arena, not only those required to fulfill group.wait()...

[Dr15Jones]

If I may, what is the status of this?

[hahnjo]

Hi @Dr15Jones, we discussed this topic and agreed that we should have a new interface. @pcanal floated the idea of using an enum class overload, but we didn't yet have the time to implement it.

[Dr15Jones]

Is there an ETA for when this might be changed? This issue is starting to affect CMS' ability to improve our CPU performance in the case of high IO loads.

[pcanal]

It should be available later this week.