Issue with trajectory prediction

[sharmasushil]

Hi,

Thank you very much for your help.

I've encountered an issue: when I run the visualization, I can only see GT, with no changes in the trajectory prediction.

I can share some results using HiVT and DenseTNT if that would be helpful.
Below is HiVT

The below one is DenseTNT:

Please let me know where I am doing mistake

Thanks a million

Your were very helpful so far

[alfredgu001324]

May I know how you run the visualization? You need to save the trajectory prediction results, and then load them in the visualization step. You can take a look at the visualization part: https://github.com/alfredgu001324/MapUncertaintyPrediction/blob/main/docs/trj.md

Can you first do a sanity check regarding the trajectory results? See if they actually contain trajectories.

[sharmasushil]

Hi,
Thank you for your reply.

I am using to save pickle file for trajectory (base and unc) using below command:
python eval.py --root ../trj_data/maptrv2 --split mini_val --method base --batch_size 32 --ckpt_path /media/xxx/Storage/MapUncertaintyPrediction/HiVT_modified/epoch=63-step=411903.ckpt

then I do visualization:

python vis_std.py --version mini --dataroot ../nuscenes --split mini_val --trj_pred HiVT --map MapTR --trj_data ../trj_data/maptrv2/mini_val/data --base_results /media/xxx/Storage/MapUncertaintyPrediction/HiVT_modified/result_base.pkl --unc_results /media/xxx/Storage/MapUncertaintyPrediction/HiVT_modified/result_unc.pkl --boxes /media/xxx/Storage/MapUncertaintyPrediction/bbox.pkl --save_path /media/xxx/Storage/MapUncertaintyPrediction/Results

this is the step, I am following

Thanks

[sharmasushil]

Just wondering if this is how you save results.pickle for unc and base:

https://github.com/alfredgu001324/MapUncertaintyPrediction/blob/main/HiVT_modified/models/hivt.py#L202

[alfredgu001324]

Yes correct, you need to uncomment this line:

MapUncertaintyPrediction/HiVT_modified/models/hivt.py

Line 138 in 08baf64

# global_y_hat_agent, pi_agent, seq_id = self.visualization(data, batch_idx)

to save the trajectories.

Sorry it is indeed an ugly solution....

[sharmasushil]

I have uncommented this line, but I still have the same issue with visualization as above I mentioned.
I don't know where I am making a mistake even following everything step by step.

I believe the model is not predicting the trajectory of HiVT, even DenseTNT has some predictions.

Please advise on this

[alfredgu001324]

Uhmm so you checked the results.pickle and it is empty?

[sharmasushil]

I have checked the results. pickle and it is not empty.
here are the things in the pickle file:

`(dict, '{4: array([[[6.28811523e+02, 1.62749915e+03, 4.10397165e-02, 4.01039235e-03],\n [6.29559692e+02, 1.62690979e+03, 6.02839701e-02, 9.66788642e-03],\n [6.30304810e+02, 1.62634119e+03, 7.80631453e-02, 1.71917211e-02],\n [6.31058716e+02, 1.62578174e+03, 1.01770058e-01, 2.77737509e-02],\n [6.31825867e+02, 1.62522888e+03, 1.20038942e-01, 3.99099700e-02],\n [6.32605957e+02, 1.62468250e+03, 1.40693560e-01, 5.47035448e-02],\n [6.33396118e+02, 1.62414807e+03, 1.61892608e-01, 7.08645731e-02],\n [6.34192444e+02, 1.62362732e+03, 1.79534808e-01, 8.91276747e-02],\n [6.34999084e+02, 1.62311890e+03, 1.97023645e-01, 1.08093277e-01],\n [6.35814270e+02, 1.62262085e+03, 2.12459818e-01, 1.32769600e-01],\n [6.36632568e+02, 1.62213855e+03, 2.27409974e-01, 1.52587786e-01],\n [6.37467468e+02, 1.62166418e+03, 2.44429246e-01, 1.73584893e-01],\n [6.38304810e+02, 1.62120483e+03, 2.61574328e-01, 1.96128502e-01],\n [6.39152771e+02, 1.6207583')

If you have a pickle file for base and unc. Is it possible to share me, Just to check if there is an issue with the pickle file or checkpoints? (Just a request, can you ignore it!)

Thanks a million!
`

[alfredgu001324]

Yes, of course! I have uploaded some sample trajectories files here: https://drive.google.com/drive/folders/161c5PH_ccZYi2wj0MnLU2KJCDzwAyCSt?usp=sharing

Since this project was a while back, I already deleted the trajectory files for this paper. The pickle files from this link are from my second project published in ECCV, but the format should be the same.

Let me know if you have further questions!

[sharmasushil]

Hi,

sample trajectory which you shared doesn't work.
and give me error:

File "vis_std.py", line 291, in main
x, y = data[int(key)]['ego_pos'][0, 0].item(), data[int(key)]['ego_pos'][0, 1].item()
KeyError: 614

for visualization:
--base_results ../PATH_to_baseline_prediction_results
--unc_results ../PATH_to_unc_prediction_results \

Please advice me on this.

Thanks a million

[alfredgu001324]

Are you loading the current splits? The sample trajectories I shared is for the entire val split, which contains 4519 scenarios. If you see a key error then maybe check L249?

[sharmasushil]

Hi,

Thank you so much for your help.
I can successfully implement the DenseTNT model for trajectory prediction, but I'm having an issue as I show your output. The prediction makes sense but my case trajectory shows:

Please let me know if it looks normal (Pink dotted in prediction and red is GT)

Thanks !

[alfredgu001324]

Hi, sorry I did not quite get your question. Can you explain again? Thank you!

[flotaas]

Hi,
I've encountered a similar issue when running the visualization. The pink predicted trajectory completely overlaps with the red GT, just like the first image in this issue. Below is the command I ran “python vis_std.py
--version trainval
--dataroot ../Nuscenes/
--split val
--trj_pred HiVT
--map MapTR
--trj_data ../trj_data/maptrv2/val/data
--base_results /MapUncertaintyPrediction/HiVT_modified/val_base_result.pkl
--unc_results /MapUncertaintyPrediction/HiVT_modified/val_unc_result.pkl
--boxes /MapUncertaintyPrediction/MapTRv2_modified/bbox.pkl
--save_path /MapUncertaintyPrediction/MapTRv2_modified/Results”, where val_base_result.pkl and val_unc_result.pkl were obtained using python eval.py.

However, when I run the command using sample trajectories files (https://drive.google.com/drive/folders/161c5PH_ccZYi2wj0MnLU2KJCDzwAyCSt?usp=sharing) shared by the author, it works normally.

It seems that there is an issue with the obtained trajectory .pkl file, but I generated it by running the provided script https://github.com/alfredgu001324/MapUncertaintyPrediction/blob/main/docs/trj.md. Why is this happening?

[alfredgu001324]

@flotaas Thanks for reaching out. Unfortunately I am not too sure either, maybe can you check whether the metrics is actually being calculated first? At least just to make sure the trajectories that are being saved are the actual predicted trajectories. There should be 6 trajectories being saved.

[alfredgu001324]

@flotaas @sharmasushil Hi, can you try the new vis function I just updated? It now saves all the vehicles trajectories for each scene and group them accordingly. It should give you the correct pkl format, and the vis script should work.