I am trying to reproduce Sparsedrive. My setup is to use similar number of GPUs, but for my HPC just wrapped in a docker. The docker image and environment is according to the repository's requirements file.
When training, I get non-deterministic results, which is explainable by atomic operations with the CUDA kernel.
But for some metrics such as Collision Rate I obtained a lot worse results. Has anyone already a solution to that? I have taken the seeds as in the repository.
In the Chinese issue #124 it is suggested to change post-processing, did anyone find settings that lead to the paper metrics?
My results so far are:
Detection and Tracking
| Model |
(Det) mAP↑ |
(Det) NDS↑ |
(Tra) AMOTA↑ |
(Det) AMOTP↓ |
(Det) Recall↑ |
| Original |
0.418 |
0.525 |
0.386 |
1.254 |
0.499 |
| mn1 |
0.412 |
0.5273 |
0.374 |
1.242 |
0.516 |
| mn2 (=1) |
$=$ |
$\approx$ |
0.371 |
1.269 |
$\approx$ |
| mn3 |
0.414 |
0.5261 |
0.380 |
1.274 |
0.491 |
| mn4 (=3) |
0.419 |
0.5281 |
0.385 |
1.267 |
$=$ |
| mn5 |
$\approx$ |
0.5254 |
0.364 |
1.254 |
0.528 |
| mn6 |
0.415 |
0.5264 |
0.376 |
1.237 |
0.525 |
Mapping
| Model |
(Map) AP_ped↑ |
(Map) AP_divider↑ |
(Map) AP_boundary↑ |
(Map) mAP↑ |
| Original |
49.9 |
57.0 |
58.4 |
55.1 |
| mn1 |
52.54 |
56.63 |
59.07 |
56.08 |
| mn2 (1) |
51.72 |
57.04 |
58.53 |
55.76 |
| mn3 |
53.51 |
56.64 |
59.53 |
56.56 |
| mn4 (3) |
53.04 |
$\approx$ |
58.45 |
55.85 |
| mn5 |
53.24 |
$\approx$ |
57.60 |
55.82 |
| mn6 |
52.24 |
57.44 |
58.40 |
56.02 |
Planning
| Model |
(Pla) L2 3s↓ |
(Pla) L2 Avg.↓ |
(Pla) ColRa 3s↓ |
(Pla) ColRa Avg↓ |
| Original |
0.96 |
0.61 |
0.18 |
0.08 |
| mn1 |
0.95 |
0.61 |
0.283 |
0.122 |
| mn2 (1) |
0.93 |
$\approx$ |
0.332 |
0.163 |
| mn3 |
1.01 |
0.65 |
0.212 |
0.087 |
| mn4 (3) |
0.95 |
0.61 |
0.287 |
0.131 |
| mn5 |
0.96 |
$\approx$ |
0.309 |
0.137 |
| mn6 |
0.97 |
$\approx$ |
0.378 |
0.194 |
I'd be thankful for experience or an author's comment on that @swc-17.
I am trying to reproduce Sparsedrive. My setup is to use similar number of GPUs, but for my HPC just wrapped in a docker. The docker image and environment is according to the repository's requirements file.
When training, I get non-deterministic results, which is explainable by atomic operations with the CUDA kernel.
But for some metrics such as Collision Rate I obtained a lot worse results. Has anyone already a solution to that? I have taken the seeds as in the repository.
In the Chinese issue #124 it is suggested to change post-processing, did anyone find settings that lead to the paper metrics?
My results so far are:
Detection and Tracking
Mapping
Planning
I'd be thankful for experience or an author's comment on that @swc-17.