I've noticed that even if only one contact is identified in the custom SOQL on the component using some other identifier, it is still possible for the wrong contact to be associated with the created Site User if that contact shares a matching Email Address.
This may be an issue with the Site.CreateExternalUser method: createExternalUser(user, accountId, password)
According to the documentation, it seems to look for any contact under the Account with a matching email.
Thought I would raise this issue for awareness, maybe there is a workaround.