High latencies

[harald-lang]

I noticed very high latencies for kernel dispatches using AQL. Synchronous dispatches take up to 21 µs. Asynchronous (batch) dispatches help to hide latencies. However, kernel dispatching still takes 6 µs (in average), which is still far to slow for fine-grained offloading.

In my experiments I set HSA_ENABLE_INTERRUPT to 0, which greatly improves robustness of the kernel offload times. With interrupts enabled, latencies vary from 6 to 15 microseconds.

System setup:

• Kaveri APU (no dGPU)
• Kernel 4.0.0-100002-generic #201511031149 SMP
• kfd-v1.6.1 (7fb04c4 from git repo HSA-Drivers-Linux-AMD)
• HSA-Runtime 1.0.3 (fa0ef7e from git repo HSA-Runtime-AMD)
• CL offline compiler (CLOC) v0.9.8

[harald-lang]

just FYI... the same experiment with a discrete NVidia card connected via PCIe takes 2 microseconds.

[harald-lang]

Update:
Setting the iGPU's frequency from "auto" to 720 MHz reduced the dispatch time to ~ 3µs.
Installing a dGPU to connect the display doesn't have an impact at all.

[aditya4d]

Hi,
Interesting observation. Are you still facing this?

[harald-lang]

Hi,
yes, nothing changed so far.
I wonder if something is wrong with my system configuration.
It would be very interesting to know, if other HSA developers are also facing this.

[aditya4d]

Hi,
Can you try spawning a new thread (run an empty function) and time it? It
may be the CPU bottleneck.
You can try ROC drivers with AMD discrete card with disabling integrated
graphics in bios.
Thank you for profiling!! (Y)

On Thursday, February 4, 2016, Harald Lang notifications@github.com wrote:

Hi,
yes, nothing changed so far.
I wonder if something is wrong with my system configuration.
It would be very interesting to know, if other HSA developers are also
facing this.

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

Regards,

Aditya Atluri,

USA.

[gstoner]

ROC Driver only runs with Haswell CPU and FIJI based GPU. I am having the team look into this to see if it regression, but it would be help to understand which NVIDIA GPU ( need model number) are you comparing the APU too. Which APU is it and model number.

greg
On Feb 4, 2016, at 8:56 PM, Aditya Avinash Atluri <notifications@github.commailto:notifications@github.com> wrote:

Hi,
Can you try spawning a new thread (run an empty function) and time it? It
may be the CPU bottleneck.
You can try ROC drivers with AMD discrete card with disabling integrated
graphics in bios.
Thank you for profiling!! (Y)

On Thursday, February 4, 2016, Harald Lang <notifications@github.commailto:notifications@github.com> wrote:

Hi,
yes, nothing changed so far.
I wonder if something is wrong with my system configuration.
It would be very interesting to know, if other HSA developers are also
facing this.

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

Regards,

Aditya Atluri,

USA.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/18#issuecomment-180169638.

[aditya4d]

Hi Greg,

How about disabling integrated graphics on APU?

—

Reply to this email directly or view it on GitHub
#18 (comment)
.

Regards,

Aditya Atluri,

USA.

[gstoner]

What are trying to run FIJI card on APU? We only are testing FIJI card with Xeon E5 v3, Xeon E3, I7, I5 Haswell or newer since we need PCIe Gen 3 Platform atomics with the ROC driver and runtime.

greg
On Feb 4, 2016, at 9:14 PM, Aditya Avinash Atluri <notifications@github.commailto:notifications@github.com> wrote:

Hi Greg,

How about disabling integrated graphics on APU?

—

Reply to this email directly or view it on GitHub
#18 (comment)
.

Regards,

Aditya Atluri,

USA.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/18#issuecomment-180172813.

[aditya4d]

Hi Harald,
Can you try rebuilding the driver? My guess for performance hit can be new
thread spawn or ISA loader. Can you try profiling executable.cpp in loader
directory?
Thanks!

Regards,

Aditya Atluri,

USA.

[aditya4d]

Hi Greg,
How about the new A10-7890K?

[harald-lang]

Hi Gregory,
for the NVidia experiment we used a GTX 650 connected via PCIe 2.0.
It was an entirely different system (which is out of my control). If you need more information about the system, I'll contact my colleague.

Alternatively, I can plug in a GTX 970 into the APU system and re-run the measurements...

@adityaatluri I'm going to profile the system as requested. I'll post the results ASAP.

[aditya4d]

Hi Harald,
Quick question. What is your target setup? For which configuration do you
want to solve this issue? APU? Or AMD GPU? Or NVIDIA GPU?

[gstoner]

Can I get the test your running. I can do A/B test on same hardware with Fiji vs Titan x

Sent from Outlook Mobilehttps://aka.ms/qtex0l

On Fri, Feb 5, 2016 at 3:32 AM -0800, "Harald Lang" <notifications@github.commailto:notifications@github.com> wrote:

Hi Gregory,
for the NVidia experiment we used a GTX 650 connected via PCIe 2.0.
It was an entirely different system (which is out of my control). If you need more information about the system, I'll contact my colleague.

Alternatively, I can plug in a GTX 970 into the APU system and re-run the measurements...

@adityaatlurihttps://github.com/adityaatluri I'm going to profile the system as requested. I'll post the results ASAP.

Reply to this email directly or view it on GitHubhttps://github.com//issues/18#issuecomment-180308482.

[harald-lang]

Hi Aditya,
my target setup is an APU system.

[aditya4d]

Hi Harald,
Can you please provide the code which you are trying to profile? Or code
that can simulate the same behavior? Usual AQL dispatch is writing to queue
memory and ringing the doorbell. Also, just making sure there are no other
kernel dispatches going on in the system right?
Thanks!