Handling Optimized Model

[angry-crab]

Recently, we have been looking into optimizing tvm models with tuning and quantization. There should be a way to handle the optimized models. I don't think we should try to tune the model with CI/CD pipelines because it can be time consuming. For example, tuning centerpoint takes around 1 hour. Similar for tvm quantization, retraining and tuning are also required. I feel like these steps could be done offline and uploaded manually for now.

[angry-crab]

@ambroise-arm Do you have any comment on this?

[ambroise-arm]

I also think this is the next step.

There are a few questions that would probably need to be looked into first. For example, how tied is the optimization to a given system? Because I am pretty sure a model has to be tuned on the hardware it will run on in order to give optimal results (that way it takes into account the CPU/GPU/RAM bottlenecks), but it may be almost the same to tune on a different system. Would an optimization done for an RTX4090 be also good for a 4080? What about a 30 series, etc.

Currently a user can already do the tuning on their own and use the "user-provided models" selection with the Autoware TVM packages. If we want to have a kind of database with tuning logs that users can choose from without doing the tuning themselves, then we probably need to have some performance figures from the questions above first. And then design a solution.

EDIT: And regarding tuning in CI, unless we have GPUs in CI I don't think it will even be possible to do tuning for a GPU target that way.

[angry-crab]

Would an optimization done for an RTX4090 be also good for a 4080? What about a 30 series, etc

I think the optimization is certainly hardware dependent, but we probably don't have to worry about these problems. Maybe what we could do is to provide a sample tuning log for each commonly used hardware backend and put them into the compressed files.

we probably need to have some performance figures from the questions above first

We can create another issue regarding this.