Add Recent Timestep Scheduling Improvements to DDIM Inverse Scheduler by clarencechen · Pull Request #3865 · huggingface/diffusers (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation16 Commits18 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

clarencechen

What does this PR do?

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@HuggingFaceDocBuilderDev

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten

patrickvonplaten

patrickvonplaten

@patrickvonplaten

Thanks for the PR @clarencechen . However, a couple of things would break current setups. Can we try to not change config names? Also it seems like you are correcting a bug here? Can you add a reproducible code snippet that shows that there is a bug?

@patrickvonplaten

Actually I think we should go forward with this PR. The DDIMScheduler also has the set_alpha_to_one argument, so it'd be good to align this here.

@sayakpaul @linoytsaban what do you think about the PR?

@patrickvonplaten

@sayakpaul

@sayakpaul @linoytsaban what do you think about the PR?

I concur with your thoughts:

Actually I think we should go forward with this PR. The DDIMScheduler also has the set_alpha_to_one argument, so it'd be good to align this here.

@clarencechen

Roll timesteps by one to reflect origin-destination semantic discrepancy

Restore set_alpha_to_one option to handle negative initial timesteps

Remove set_alpha_to_zero option not used due to previous truncation

@clarencechen

@clarencechen

Use self.image_processor.preprocess in DiffEdit pipeline functions

@clarencechen

@clarencechen

@clarencechen

@clarencechen

Invert step specification leads to negative noise variance in SDE-based algs

Add first draft for proper fast tests for DPMMultistepInverseScheduler

@clarencechen

…ffusion step

Clean up diffedit fast test

@clarencechen

@clarencechen

patrickvonplaten

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sayakpaul

@@ -1037,7 +1037,7 @@ def generate_mask(
)
# 4. Preprocess image
image = preprocess(image).repeat_interleave(num_maps_per_mask, dim=0)
image = self.image_processor.preprocess(image).repeat_interleave(num_maps_per_mask, dim=0)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great reuse!

Should we then maybe just delete preprocess() to help reduce the LOC from the pipeline source?

sayakpaul

inverted_latents = [latents.detach().clone()]
with self.progress_bar(total=num_inference_steps - 1) as progress_bar:
for i, t in enumerate(timesteps[:-1]):
inverted_latents = []

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this variable being used now?

sayakpaul

@@ -1336,7 +1336,7 @@ def __call__(
self,
prompt: Optional[Union[str, List[str]]] = None,
mask_image: Union[torch.FloatTensor, PIL.Image.Image] = None,
image_latents: torch.FloatTensor = None,
image_latents: Union[torch.FloatTensor, PIL.Image.Image] = None,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image_latents is not a good name when we're accepting PIL images too. image works for me. But probably need to keep image_latents to be backward-compatible. WDYT @patrickvonplaten?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, happy to deprecate the name via **kwargs

sayakpaul

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work!

orpatashnik pushed a commit to orpatashnik/diffusers that referenced this pull request

Aug 1, 2023

@clarencechen @orpatashnik

…huggingface#3865)

Roll timesteps by one to reflect origin-destination semantic discrepancy

Restore set_alpha_to_one option to handle negative initial timesteps

Remove set_alpha_to_zero option not used due to previous truncation

Use self.image_processor.preprocess in DiffEdit pipeline functions

Invert step specification leads to negative noise variance in SDE-based algs

Add first draft for proper fast tests for DPMMultistepInverseScheduler

Clean up diffedit fast test

orpatashnik pushed a commit to orpatashnik/diffusers that referenced this pull request

Aug 1, 2023

@clarencechen @orpatashnik

…huggingface#3865)

Roll timesteps by one to reflect origin-destination semantic discrepancy

Restore set_alpha_to_one option to handle negative initial timesteps

Remove set_alpha_to_zero option not used due to previous truncation

Use self.image_processor.preprocess in DiffEdit pipeline functions

Invert step specification leads to negative noise variance in SDE-based algs

Add first draft for proper fast tests for DPMMultistepInverseScheduler

Clean up diffedit fast test

orpatashnik pushed a commit to orpatashnik/diffusers that referenced this pull request

Aug 1, 2023

@clarencechen @orpatashnik

…huggingface#3865)

Roll timesteps by one to reflect origin-destination semantic discrepancy

Restore set_alpha_to_one option to handle negative initial timesteps

Remove set_alpha_to_zero option not used due to previous truncation

Use self.image_processor.preprocess in DiffEdit pipeline functions

Invert step specification leads to negative noise variance in SDE-based algs

Add first draft for proper fast tests for DPMMultistepInverseScheduler

Clean up diffedit fast test

yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request

Dec 25, 2023

@clarencechen

…huggingface#3865)

Roll timesteps by one to reflect origin-destination semantic discrepancy

Restore set_alpha_to_one option to handle negative initial timesteps

Remove set_alpha_to_zero option not used due to previous truncation

Use self.image_processor.preprocess in DiffEdit pipeline functions

Invert step specification leads to negative noise variance in SDE-based algs

Add first draft for proper fast tests for DPMMultistepInverseScheduler

Clean up diffedit fast test

AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request

Apr 26, 2024

@clarencechen

…huggingface#3865)

Roll timesteps by one to reflect origin-destination semantic discrepancy

Restore set_alpha_to_one option to handle negative initial timesteps

Remove set_alpha_to_zero option not used due to previous truncation

Use self.image_processor.preprocess in DiffEdit pipeline functions

Invert step specification leads to negative noise variance in SDE-based algs

Add first draft for proper fast tests for DPMMultistepInverseScheduler

Clean up diffedit fast test