It’s probably Stable diffusion. I use comfyui since you can watch the sausage get made but there’s also other UIs like automatic1111. Originally for a qr pattern beautifier, there is a controlnet that takes a two tone black and white “guide” image. but you can guide it to follow any image you feed it. Such as a meme edited to be black and white, or text like “GAY SEX.”

[–] SzethFriendOfNimi@lemmy.world 5 points 1 year ago* (last edited 1 year ago)

Looks like somebody created an outline mask and then used that in img2img with a prompt for the particular scenery.

I remember seeing somebody use that technique to generate the same pose and model but different colored outfits with that technique

Maybe Canny nap like used here

https://youtu.be/8cVnooYgpDc?t=13m

[–] simplylemons@lemm.ee 4 points 1 year ago

I've used InvokeAI (stable diffusion models) with ControlNets for it too, it's a bit easier to use than comfy/a1111 but not as powerful IMHO.

[–] TheBat@lemmy.world 1 points 1 year ago

There's also SD.Next with some enhancements to automatic1111.

[–] CodeInvasion@sh.itjust.works 17 points 1 year ago

This is done by combining a Diffusion model with ControlNet interface. As long as you have a decently modern Nvidia GPU and familiarity with Python and Pytorch it's relatively simple to create your own model.

The ControlNet paper is here: https://arxiv.org/pdf/2302.05543.pdf

I implemented this paper back in March. It's as simple as it is brilliant. By using methods originally intended to adapt large pre-trained language models to a specific application, the author's created a new model architecture that can better control the output of a diffusion model.

[–] TropicalDingdong@lemmy.world 4 points 1 year ago

Check this out:

https://github.com/camenduru/controlnet-colab

I started looking after you responded. Havnt gotten it to work yet.