The way Wharf transact calls currently work might always require loading the ABIs.
Core doesn't require this and allows for pre-defining structs/ABIs to prevent this API call, and Wharf should too.
Investigate what will be required for a session to perform transactions and preload this data, to prevent those calls from being required.