Image-to-Image Techniques in Action

Historically, before the advent of ControlNet, the stylization effects prevalent on various video platforms were predominantly achieved through image-to-image techniques. To grasp the fundamentals of image-to-image transfer, we can break down the process. This method involves introducing noise to the original image, governed by a parameter known as "redrawing strength." The resulting noisy image serves as the foundational latent representation for the generation of new images. Subsequently, a prompt you provide steers the style of the final output.

import requests

import torch

from PIL import Image

from io import BytesIO

from diffusers import StableDiffusionImg2ImgPipeline

device = "cuda"

pipe = StableDiffusionImg2ImgPipeline.from_pretrained("zhyemmmm/ToonYou").to(device)

response = requests.get(url)

init_image = Image.open(BytesIO(response.content)).convert("RGB").resize((512, 512))

prompt = "1girl, fashion photography"

images = []

for strength in [0.05, 0.15, 0.25, 0.35, 0.5, 0.75]:

image = pipe(prompt=prompt, image=init_image, strength=strength, guidance_scale=7.5).images[0]

images.append(image)

This segment outlines how to import the necessary libraries, initialize the model, fetch an image from the web, and generate stylized versions of that image using varying levels of noise.

ControlNet with Edge Contour Conditioning

Using the iconic Mona Lisa as a reference, we can illustrate the functionality of the SDXL model's Canny control mode in contrast to the directive-level editing capabilities of the SD1.5 model. The initial step involves loading the Mona Lisa image and extracting its contours via the Canny operator.

image = cv2.Canny(np.array(original_image), 100, 200)

image = Image.fromarray(np.concatenate([image[:, :, None]] * 3, axis=2))

Next, we can load the ControlNet model and the VAE model necessary for generating images under specific control conditions.

controlnet = ControlNetModel.from_pretrained("diffusers/controlnet-Canny-sdxl-1.0-mid", torch_dtype=torch.float16)

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

pipe = StableDiffusionXLControlNetPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet, vae=vae, torch_dtype=torch.float16)

pipe.enable_model_cpu_offload()

We can then use prompts to manipulate the style of the output images in a range of artistic formats.

The second video, "Style transfer with Leonardo AI's STYLE REFERENCE feature is incredible!" showcases how advanced style transfer features can be utilized in practical applications.

Directive-Level Editing with ControlNet

In directive editing mode, users simply need to provide commands that describe the desired transformation. For example, to create an image that appears to be on fire, you would input "add fire" as your directive. This mode is not only more intuitive but also eliminates the need for additional control inputs, simplifying the process of generating stylized images.

Model Fusion Techniques

In practice, blending different models can yield unique artistic styles. This process, known as model fusion, involves combining multiple models to create a new one. By adjusting the weight of each model, you can control their contribution to the final output.

For instance, to fuse the "Anything V5" model with the "ToonYou" model, you would apply a formula to define the weights of the new model:

New Model Weight = Model A * (1 — M) + Model B * M

Where M represents the weighting coefficient. Utilizing the WebUI's "Checkpoint Merger," you can execute this fusion to explore various creative outcomes.