fix bugs for sd3 controlnet training by xduzhangjiayu · Pull Request #9489 · huggingface/diffusers (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation9 Commits3 Checks8 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

xduzhangjiayu

What does this PR do?

This PR is for fix some bugs for sd3 controlnet training

  1. fix issue for multi-gpu training (examples\controlnet\train_controlnet_sd3.py)
  2. fix issue for enabling --gradient_checkpointing in training process (src\diffusers\models\controlnet_sd3.py)

Who can review?

@yiyixuxu @sayakpaul @asomoza

@xduzhangjiayu

@sayakpaul

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good to me.

In a future PR, I will add a few more tests related to gradient checkpointing.

@xduzhangjiayu

@yiyixuxu

yiyixuxu

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

leisuzz pushed a commit to leisuzz/diffusers that referenced this pull request

Oct 11, 2024

Co-authored-by: Sayak Paul spsayakpaul@gmail.com Co-authored-by: YiYi Xu yixu310@gmail.com

@tobiasfshr

Hi, I have a question on 1: Is having unused parameters expected behavior? The following parameters seem to be unused:

transformer_blocks.11.ff_context.net.2.bias, transformer_blocks.11.ff_context.net.2.weight, transformer_blocks.11.ff_context.net.0.proj.bias, transformer_blocks.11.ff_context.net.0.proj.weight, transformer_blocks.11.attn.to_add_out.bias, transformer_blocks.11.attn.to_add_out.weight

@Zheng-Fang-CH

sformer_blo

Hi, I meet the same error, do you solve it now?

@xduzhangjiayu

Hi, I have never met this error, could you please describe in detail, thanks!

@Zheng-Fang-CH

Hi, I have never met this error, could you please describe in detail, thanks!

error
Hi, I didn't change the code in diffusers, but the weight parameters in controlnets seem not be used to compute the final loss, but obviously, these weights have been initialized.

@xduzhangjiayu

Hi, I have never met this error, could you please describe in detail, thanks!

error Hi, I didn't change the code in diffusers, but the weight parameters in controlnets seem not be used to compute the final loss, but obviously, these weights have been initialized.

Seems maybe I have met before, but I'm not sure, could you please contact me with email: xduzhangjiayu@163.com for further discussion ?

sayakpaul added a commit that referenced this pull request

Dec 23, 2024

Co-authored-by: Sayak Paul spsayakpaul@gmail.com Co-authored-by: YiYi Xu yixu310@gmail.com