Add K-LMS scheduler from k-diffusion by anton-l · Pull Request #185 · huggingface/diffusers (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation9 Commits11 Checks0 Files changed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
This PR adds the K-LMS sampler from k-diffusion by Katherine Crowson.
At the moment it only supports discrete beta-schedules (specifically the one for Stable Diffusion) but it will be extended to support continuous sigma-schedules in a follow-up PR.
The documentation is not available anymore as the PR was closed or merged.
Comment on lines 128 to 131
| if isinstance(self.scheduler, LmsDiscreteScheduler): |
|---|
| latents = self.scheduler.step(noise_pred, i, latents, **extra_step_kwargs)["prev_sample"] |
| else: |
| latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs)["prev_sample"] |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be made cleaner by removing the branch with t->i, but for now I'm not touching the timesteps from k-lms for ease of debugging.
Comment on lines 115 to 117
| if isinstance(self.scheduler, LmsDiscreteScheduler): |
|---|
| sigma = self.scheduler.sigmas[i] |
| latent_model_input = latent_model_input / ((sigma**2 + 1) ** 0.5) |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll probably need a mechanism to specify custom coefficients for inputs in the future, since the Karras scheduler needs it too.
[WIP] Add K-LMS scheduler from k-diffusion Add K-LMS scheduler from k-diffusion
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just wondering about the i vs. sigma API since in karras_ve we pass a sigma:
| @@ -118,7 +125,10 @@ def __call__( |
|---|
| noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond) |
| # compute the previous noisy sample x_t -> x_t-1 |
| latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs)["prev_sample"] |
| if isinstance(self.scheduler, LmsDiscreteScheduler): |
| latents = self.scheduler.step(noise_pred, i, latents, **extra_step_kwargs)["prev_sample"] |
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't we maybe pass the sigma here?
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok we need it anyway - good to leave as is for me!
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The i is needed inside the step for other calculations too, and we can't reverse the sigma into i, so we'll need it like this until a more full refactor.
prompt = ["a logo of Knicks and championship "]
height = 512 # default height of Stable Diffusion
width = 512 # default width of Stable Diffusion
num_inference_steps = 100 # Number of denoising steps
guidance_scale = 7.5 # Scale for classifier-free guidance
generator = torch.manual_seed(32) # Seed generator to create the inital latent noise
batch_size = 1
yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request
test LMS with LDM
test LMS with LDM
Interchangeable sigma and timestep. Added dummy objects
Debug
cuda generator
Fix derivatives
Update tests
Rename Lms->LMS