[Official callbacks] SDXL Controlnet CFG Cutoff by asomoza · Pull Request #9311 · huggingface/diffusers (original) (raw)

Adds a new official callback for the SDXL Controlnet pipeline.

After testing with different alternatives, I found that just adding a new callback for this specific pipeline was the best and simpler solution.

import random from datetime import datetime

import torch

from diffusers import AutoencoderKL, ControlNetModel, StableDiffusionXLControlNetPipeline from diffusers.callbacks import SDXLControlnetCFGCutoffCallback from diffusers.utils import load_image

anyline = load_image( "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/callbacks/20240829020431.png" )

controlnet = ControlNetModel.from_pretrained("TheMistoAI/MistoLine", torch_dtype=torch.float16, variant="fp16") vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16) pipe = StableDiffusionXLControlNetPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet, vae=vae, torch_dtype=torch.float16, variant="fp16", ).to("cuda")

prompt = "high quality photo of a yellow plastic huggingface emoji toy, white background" callback = SDXLControlnetCFGCutoffCallback(cutoff_step_ratio=0.3)

image = pipe( prompt, num_inference_steps=25, image=anyline, controlnet_conditioning_scale=0.6, controlnet_guidance_end=0.9, callback_on_step_end=callback, ).images[0]

image.save("cnet_callback.png")

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.