Add Grad-CAM visualization tool for DonkeyCar neural networks #1205
Conversation
|
Just curious, how is this different than the donkeycar/donkeycar/parts/salient.py Line 13 in 5483490 |
|
Great question @mgagvani ! The Grad-CAM implementation actually differs from the existing salient flag in makemovie.py in several ways:
Let me know if you've got any other questions or want me to dive deeper! |
There was a problem hiding this comment.
@rxhxm - this looks like a good addition to our current salient method. However we cannot merge the code like this, simply because it has been written completely without using any of the donkey car framework, objects and methods. Any stand-alone script will very quickly stop working, because the code evolves over time, so delivering this new functionality within a stand-alone script is not the right thing to do.
However, it will be very easy to implement your new method within the MakeMovie class that currently implements the salient feature. This approach will integrate your new approach directly into the donkey car eco system, remove a lot of unnecessary code in your PR that deals with accessing data, and creating the movie from the individual frames, etc.
It would be great if you could make that change.
| @@ -0,0 +1,109 @@ | |||
| # DonkeyCar Grad-CAM Visualization | |||
There was a problem hiding this comment.
This documentation looks good. But it should go as a PR into the donkey_docs repository, so it will show up in the official documentation at docs.donkeycar.com
| @@ -0,0 +1,8 @@ | |||
| """ | |||
There was a problem hiding this comment.
If integrated into donkey, this file will become obsolete
| @@ -0,0 +1,432 @@ | |||
| """ | |||
There was a problem hiding this comment.
While the code in this file looks fine, the implementation should be integrated within the current donkey makemovie command and not within a separate script. We can extend the argument list for makemovie to add all the arguments you specify below.
|
|
||
|
|
||
| def find_images(data_path): | ||
| """ |
There was a problem hiding this comment.
This will not be needed when going through the MakeMovie class as all the data preparation is already in place.
|
|
||
|
|
||
| def load_metadata(data_path): | ||
| """ |
There was a problem hiding this comment.
Access to tub metadata and tub records is provided through the Tub class and hence this method will become obsolete.
|
|
||
| # Process images | ||
| frames = [] | ||
| for i, img_path in enumerate(image_paths): |
There was a problem hiding this comment.
The code inside this loop is the core part of the PR. This is the bit that should be translated into MakeMovie such that we can choose between the current and your new version of displaying the salient features. At that point all the tub data loading and file loading has already happened. This will greatly simplify your PR.
| return frames | ||
|
|
||
|
|
||
| def create_video(frames, output_path, fps=20): |
There was a problem hiding this comment.
This will become obsolete
| return output_path | ||
|
|
||
|
|
||
| def save_frames(frames, output_dir): |
| return output_dir | ||
|
|
||
|
|
||
| def main(): |
| return regions | ||
|
|
||
|
|
||
| def create_video(frames, output_path, fps=30): |
There was a problem hiding this comment.
This will become obsolete.
3 participants
Grad-CAM Visualization Tool for DonkeyCar
Overview
This PR adds a Gradient-weighted Class Activation Mapping (Grad-CAM) visualization tool to help better understand their neural network models. The tool provides a visual explanation of which parts of input images most influence the model's predictions, making it easier to debug and improve model performance.
Features
Implementation
The implementation consists of:
Why This is Valuable
Unlike the existing tools such as makemovie.py that simply show the model's predictions, Grad-CAM helps users understand why the model makes those predictions by visualizing which regions of the input image are most important to the model. This leads to:
Testing
The tool has been tested on various DonkeyCar models with TensorFlow 2.15 and works with different model architectures including default DonkeyCar models.
Documentation
Documentation is included with examples in the README.md file within the module directory.