Provide a way to initialize connection to dynolog without triggering cupti API

[yinghai]

Context in facebookincubator/dynolog#286.

In pytorch, we have only 2 call sites of libkineto_init (https://github.com/search?q=repo%3Apytorch%2Fpytorch%20libkineto_init&type=code). - One is the to when we need to register ourselves to dynolog (KINETO_USE_DAEMON=1).

• One is for the programmable torch.profiler interface (https://github.com/pytorch/pytorch/blob/6371c25b91fb66756b5a2df67741d4e7a72a7261/torch/csrc/profiler/kineto_shim.cpp#L247).

First one is triggered from global_kineto_init, but libkineto_init actually triggers a bunch of cupti API, which defeats the purpose of lazy cupti attach. Therefore we are potentially affecting the perf of the program until cuptiFinalize is called.

I think the right design would be decoupling registration to dynolog from all the cupti initialization. At global_kineto_init, we only register to dynolog but don't do cupti calls. And then when we get signal from dynolog, we trigger ondemand profile and then attach cupti context, basically following the pattern of the second use case in pytorch above.

This requires

• Some work to separate registration to dynolog from all the cupti initialization in libkineto
• Provide API in libkineto so that global_kineto_init in pytorch only trigger the registration to dynolog part
• Change pytorch code global_kineto_init accordingly.

[yinghai]

cc @briancoutinho

[briancoutinho]

@yinghai Just wanted to share where dynolog registry happens.
It gets triggered on initBaseConfig()

https://github.com/pytorch/kineto/blob/main/libkineto/src/init.cpp#L233

This will under the hood call updateBaseConfig that tries to construct the DaemonConfigloader
https://github.com/pytorch/kineto/blob/main/libkineto/src/ConfigLoader.cpp#L228-L233

and thus dynolog registry takes place
https://github.com/pytorch/kineto/blob/main/libkineto/src/ConfigLoader.cpp#L212-L218

We primarily need to call updateBaseConfig() in init to initialize dynolog, this does not trigger CUPTI as such.

[briancoutinho]

@yinghai landing the change but I will also need to sync it on PyTorch :)

Tested out open source dyno with PyTorch in test plan, let me know if you have any questions, suggestions.
#1035

[briancoutinho]

@yinghai let's wait for pytorch nightly to catch up, and we can check if this works!