[Question] How do i enable profiling for HugeCTR using Nsight Profiler?

[ramgandikota]

I am trying to run MLCommons DLRM on 2 nodes with H100 GPUs. How do i enable profiling on this to get information on time taken in compute, communication and storage?
Mostly interested in getting a clear understanding of how much time is taken by each component of our infrastructure

[shijieliu]

hi @ramgandikota you can use nsight system to get profile result. Please refer to Nsight System Doc for more detailed use guidance. Here is an example:

nsys profile  -s none -t cuda -f true -o train.qdre  --cpuctxsw none --cuda-flush-interval 100 --capture-range-end=stop --cuda-graph-trace=node python ...