Improving Host Performance

[EJainDev]

Since the CLBlast API interface uses StatusCode, why use exceptions in the source code? Switching to if statements would make the code a lot faster in the error case (critical for low level libraries like BLAS). I understand that it was probably easier for development, but would you accept a PR switching to if statements.

[CNugteren]

This might be a good idea indeed. Before making a huge PR, could you perhaps first show a small diff here of what this would look like for one of the (simpler) CLBlast APIs?

[EJainDev]

The CLCudaAPI throws exceptions natively which means the try…catch can’t be removed from the top or bottom (harder to implement) level. So about only half of less-likely (CLBlast parameter errors) errors will benefit from the speed up until changes are made to the wrappers.

[EJainDev]

Here is a small diff sketch as a branch of what would happen if Xcopy switched to no exceptions, obviously sub functions have changed and this won't compile until it's done all the way.

[CNugteren]

Thanks for the sketch. Yeah that's quite a big change already and indeed only covers a small part. Almost every function will need to be re-written to pass all the error codes back down to the user. I'm not sure that's worth it. Do you have any evidence of the performance loss with exceptions?

[EJainDev]

Here is the performance loss in the error case (I used 2 cases, one is n = 0 for copy and the other is x is nullptr):

Error case 1: 477710300ns
Error case 2: 188200ns
No error: 76600ns

I assume the major difference in case 1 and case 2 comes from the fact that case 2 is caught less times than case 1.

Here is the C++ for more detail:

#include <chrono>
#include <iostream>
#include <stdexcept>
#define CL_HPP_TARGET_OPENCL_VERSION 300
#define CL_TARGET_OPENCL_VERSION 300

#include <clblast.h>

#include <CL/opencl.hpp>
#include <vector>

#include "CL/cl.h"

using std::chrono::nanoseconds;
using std::chrono::steady_clock;

int main() {
  std::vector<cl::Device> devices;
  cl::Platform::get().getDevices(CL_DEVICE_TYPE_ALL, &devices);
  auto device = devices[0];
  auto context = cl::Context(devices[0]);
  cl::CommandQueue queue{context, device};

  cl::Buffer x{context, CL_MEM_READ_WRITE, sizeof(float)};
  cl::Buffer y{context, CL_MEM_READ_WRITE, sizeof(float)};

  auto cl_x = x();
  auto cl_y = y();

  auto cl_queue = queue();

  auto start = steady_clock::now();
  clblast::Copy<float>(0, cl_x, 0, 1, cl_y, 0, 1, &cl_queue);
  auto end = steady_clock::now();
  std::cout << "Error case 1: " << (end - start).count() << "ns\n";

  start = steady_clock::now();
  cl_y = nullptr;
  try {
    clblast::Copy<float>(1, cl_x, 0, 1, cl_y, 0, 1, &cl_queue);
  } catch (...) {
  }
  end = steady_clock::now();
  std::cout << "Error case 2: " << (end - start).count() << "ns\n";

  start = steady_clock::now();
  cl_queue = queue();
  clblast::Copy<float>(1, cl_x, 0, 1, cl_y, 0, 1, &cl_queue);
  end = steady_clock::now();
  std::cout << "No error: " << (end - start).count() << "ns\n";

  queue.finish();

  return 0;
}

[CNugteren]

Error case 1: 477710300ns
Error case 2: 188200ns
No error: 76600ns

I believe you are just measuring compilation time of the OpenCL kernels here... I would suggest to modify your code and put a loop of 10 runs at least between each timer and then report back the median or so.

[EJainDev]

Here are some updated times, as it would seem, exceptions aren't too bad but it still takes half the time:

Error case 1: 45609ns
Error case 2: 54627ns
No error: 81267ns

I used average 100 iterations with 1 unrecorded call first to avoid recompilation.

[CNugteren]

exceptions aren't too bad but it still takes half the time:

I guess the real question is: how much time does the No error case take if the CLBlast code is written without exceptions., but we might not know that unless the exception logic is removed from the code. I doubt it will have a large impact though. Because now you measured the error cases, but I think we can assume those won't be important for the end-user's total application speed, because they might only happen during development or in rare situations.

[EJainDev]

I think that the assumption might work but there is also a reason that libraries like Tensorflow and Pytorch use status codes, even if it is GPU backend... However, since this is a significantly large change, I think it can be broken down into steps. Most functions call sub-functions which my diff sketch also changed, but what if instead, it went on a level based change? For example, use StatusCode return value for the functions in src/routines/level*/*.hpp but not the functions they call internally. Then move on layer by layer to decrease diff size while slowly increasing performance.

[CNugteren]

I would still suggest to not make any changes before we have a good idea of how much speed you will gain with this (see my previous suggestion). Which I doubt will be significant. Because that's the only motivation for this issue, other than that exceptions result in much simpler code.