In a kernel with OTG support built as a module (so it defaults to host mode), plugging in a USB device causes an unhandled interrupt 75. This causes the interrupt to be disabled and stops the dual-role port from working at all.
This is probably due to something (uboot?) leaving an interrupt source enabled that is normally handled by the device controller (arcotg_udc).
Demonstrated when building using imx6_hdmidongle_usb_defconfig