Controlnet with Non-Image Condition?

[aoirusann]

Currently, depth, normal, edge, pose, etc. seem to be input into ControlNet in the form of Images as conditions.
I would like to ask is it possible to use Non-Image Conditions?

This will allow ControlNet to be extended to more CV problems.
For example, Face Recognition can make face_encoding from an image:

import face_recognition
known_image = face_recognition.load_image_file("biden.jpg")
unknown_image = face_recognition.load_image_file("unknown.jpg")

biden_encoding = face_recognition.face_encodings(known_image)[0]
unknown_encoding = face_recognition.face_encodings(unknown_image)[0]

results = face_recognition.compare_faces([biden_encoding], unknown_encoding)

If ControlNet can accept face_encoding as the input, Stable Diffusion will make it very easy to duplicate anyone's face.
(I know this may involve some legal and ethical issues, this example is just to illustrate the technical possibility.)

I am not an AI technician. I may have made some low-level mistakes, please forgive me 😸 ❤️

[geroldmeisinger]

I collect information about this here https://civitai.com/articles/2078#heading-3617 -> Q: Is it possible to use non-image inputs for control nets?

A: See [Double Control] What model is most needed? and adding more "hint" to training process
To some extent I assume yes, although ultimately it has to boil down "working with spatial blocks in the latent space" (see control net paper and how the architecture works). Also see reference-only which I assume is a just-in-time algorithmic version of a control net.