The TransferToRemote supports the use of SCP and SFTP, but there is no clear way for how a non-standard destination port can be provided.
In the case of http and https, the port is part of the the address, but this convention does not extend to SCP and SFTP, where the : is used to limit the host from the path.