Sample results, Tooling on top of big-sleep

[enricoros]

big-sleep is GORGEOUS. We need to explore what it can do, where it shines, and what to avoid.

Adding a few pics down below, but I'm still in early experimentation - will update the thread later.

Puppies

« a colorful cartoon of a dog »

• seed=553905700049900, iteration=160, lr=.07, size=256
  
• same, iteration=490
  

« a colorful cartoon of a dog with blue eyes and a heart »

• seed=555169003382600, iteration=400, lr=.07, size=256
  

Clouds

« clouds in the shape of a donut »

• seed=581307748222100, iteration=360, lr=.07, size=256
  
• seed=583134047383400, iteration=390, lr=.07, size=256
  

This post will be edited to add new samples

[lucidrains]

Heart for a nose! Lol

[enricoros]

Haha, it's creative indeed. Wanted to show the impact of seeds and iterations, for people that are puzzled by seeing completely different results. Added some more samples. Would like to figure out a list of nouns and modifiers/adjectives that work very well with big-sleep. For instance "made of" is used in the DALL-E example and seems to work very well here too.

[lucidrains]

@enricoros at this rate, we may not need DALL-E!

[htoyryla]

Wanted to show the impact of seeds and iterations, for people that are puzzled by seeing completely different results.

Thinking of me? Don't recognise myself, having worked as a visual artist working with neural networks since 2015 mainly largely writing my own code. Now learning about these new techniques, for eventually finding ways to integrate them into my own work processes.

Yes, there is something uncanny about this. From the prompt "a cityscape in the style of Lionel Feininger" I got this. Of course someone familiar with Feininger's work would see the differences, but that would miss the similarities. Like someone influenced by Feininger...

"A cityscape in the style of Paul Klee". Again, not exactly Klee, but very much in the right direction. In fact, like someone influenced by Klee and Mark Rothko.

I was never really interested in BigGAN, but working on my own GAN code and my own image materiaIs to keep to my own style. Now I am wondering, not perplexed, about how we are suddenly getting so much more interesting results from BigGAN. Is it that these pictures were always there, but difficult to find (I experimented once a little in latent space search in BigGAN but gave up). Or is it that using different conditioning vectors on different layers (as I see that the code does) increases the variety even further, even if the BigGAN is still using the same weights as before?

[lucidrains]

@htoyryla Very nice pictures!

I think the fabulous results we are seeing is from the unique combination of the multimodal network (CLIP) and the GAN. The GAN has been trained to be able to generate textures and objects to realism, so it has the capacity to paint anything it wants. All the knobs are there. CLIP helps to guide it based on the immense amount of images and text it has seen (400 million I believe). CLIP is also composed of attention, which in my mind is more powerful than the gradients you could get from convolutions.

The other thing that makes this combination special is that BigGAN is class conditioned, so it begins training at a random class as starting point. I believe the different starting points leads to a high variety of end results, even when rerunning on the same text.

[htoyryla]

@lucidrains Thanks for the clear explanation. Feels so obvious now :) Extremely interesting.

Your code projects here are an excellent resource for learning about these developments. Compact and keeping to the essentials, still fully working.

[htoyryla]

The GAN has been trained to be able to generate textures and objects to realism, so it has the capacity to paint anything it wants. All the knobs are there.

I am still thinking of how the control from CLIP to BigGAN is implemented. Where the knobs are, so to say. It does not appear to be simply through a latent into the first layer but injection into multiple layers (which makes sense to me as different layers control different visual features).

No need for a long explanation though, I can go on to investigate on my own.

[enricoros]

Love the artistic direction of this thread. My interest (other than reading the answer to @htoyryla's question), is in the tooling on top of this beautiful technology that can allow for artistic control, save/restore/collaboration, and to not waste computing resources (and precious time!) running notebook for hours for a single image which is then discarded.

I'm summarizing my ideas for the transition in the "usability" of generative technologies in this picture:

What I'm not mentioning here is the plan for "the day after", which could use trained networks to replace the manual selection process, and weed out automatically pictures that are straight out garbage (we see many :).

This would require API changes to make the library more controllable, executable in a step-by-step fashion, make latent space restorable/saveable (instead of starting from an initial seed and crossing fingers), not to mention then going into editing of the latent space from an UI (point, interpolate, etc). Am I going too far off the deep end? :)

[htoyryla]

This would require API changes to make the library more controllable, executable in a step-by-step fashion, make latent space restorable/saveable (instead of starting from an initial seed and crossing fingers), not to mention then going into editing of the latent space from an UI (point, interpolate, etc). Am I going too far off the deep end? :)

I already tried saving the latents together with each intermediate image, and then made a separate script for generating images by interpolating between two stored latents. No problems with that, worked nicely.

[enricoros]

Please share the code! Open a pull request, or fork the repo and add. @htoyryla: what are your creation flows, using this tech?

[htoyryla]

Please share the code! Open a pull request, or fork the repo and add. @htoyryla: what are your creation flows, using this tech?

My experiment was based on an earlier version, so I will make a new fork, make the necessary changes, test and let you know. Nothing fancy, just how I did it.

My workflow in art is based on my own GAN, with lots of options, my own image sets, usually quite small and focused to limit the visual world. In addition I use other tools, such as pix2pix and the like, to modify images. Here I am simply getting familiar with these new technological options.

[htoyryla]

See here https://github.com/htoyryla/big-sleep . It will store latents in a pth file (named similar to the image) when save_progress is used.

The lines for storing are here https://github.com/htoyryla/big-sleep/blob/472699165a4d792f0837239836e7e5a1f45dcd88/big_sleep/big_sleep.py#L243-L246

bsmorph.py shows then that the latents can be loaded and that it is possible to interpolate between them.
The interpolation in bsmorph.py is very crude, feel free to use your own.

There is nothing yet for continuing training from stored latents, but it should be straightforward to initialise latents from a stored one. Use lats = torch.load(filename) to read latents from a file and then initialise the latents with lats.normu and lats.cls here

big-sleep/big_sleep/big_sleep.py

Lines 72 to 73 in a7ad18c

	self.normu = torch.nn.Parameter(torch.zeros(num_latents, 128).normal_(std = 1))
	self.cls = torch.nn.Parameter(torch.zeros(num_latents, 1000).normal_(mean = -3.9, std = .3))

[htoyryla]

Here's a morph between two latents I stored:

p2l.mp4