Instead of manually encoding it ourselves inside our prompts.
https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling
When @DonggeLiu first tried this a few months ago, he ran into some issues (@DonggeLiu do you remember what they were?)
We should try this again.