At the moment there's no way to influence the idealWidthForHeight or idealHeightForWidth of a view via view modifiers. frame(idealWidth:idealHeight:) currently only sets the full idealSize of the underlying view but not its individual idealBlahForBlah properties (which are used by modifiers such as fixedSize(horizontal: true, vertical: false).
I'm not exactly sure if we want to set idealBlahForBlah from the frame modifier in all cases, but SwiftUI does at least to set it in basic cases such as frame(idealWidth: 100) on a text view.