The Modular Diffusers by yiyixuxu · Pull Request #9672 · huggingface/diffusers (original) (raw)

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

TO-DOs before merge

PAG needs to be a guider on its own
One more guide on how to write a PipelineBlock; and how to assemble SequentialPipelineBlocks and AutoPipelineBlocks

TO-DOs

guider needs to be a ConfigMixin
ModularPipelineBlocks.from_pretrained needs to work with official blocks
API docs
a simple test case
Merge the concept of ModularLoader into ModularPipeline
Flux/Wan/ITX
We overrode changes in this PR https://github.com/huggingface/diffusers/pull/11652/files, need to make sure the updated code works for this case

Documentations

Documentations are live here

a-r-r-o-w, apolinario, charchit7, CesarERamosMedina, sayakpaul, juancopi81, ghunkins, Wauplin, linoytsaban, stevhliu, and 3 more reacted with hooray emoji asomoza, a-r-r-o-w, hlky, charchit7, apolinario, CesarERamosMedina, peteromallet, exdysa, sam17, ghunkins, and 7 more reacted with heart emoji a-r-r-o-w, hlky, charchit7, apolinario, CesarERamosMedina, shauray8, PaulFidika, ultranity, ghunkins, linoytsaban, and 3 more reacted with rocket emoji a-r-r-o-w, apolinario, CesarERamosMedina, ghunkins, and asomoza reacted with eyes emoji

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

… completely yet (look into later)

hi this is very interesting! I'm making a Python pipeline flow visual scripting tool, that can auto-convert functions to visual nodes for fast and modular UI blocks demo. Itself is a pip package: https://pypi.org/project/nozyio/

I wanted to integrate diffusers with my flow nodes UI project but found its not very modular. But this PR may change that! Looking forward to see how this evolves.

github: https://github.com/oozzy77/nozyio happy to connect!

@oozzy77 thanks!
do you want to join a slack channel with me? if you want to experiment building something with this PR I'm eager to hear your feedback and iterate base on that

hlky mentioned this pull request

Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yiyixuxu! The other review comments address most of the PR already. To me, I already was exposed to a full implementation early-on so it's hard for me to review as user perspective, but the docs are great and give a wide coverage of everything.

Co-authored-by: Aryan aryan@huggingface.co

[CI] Fix big GPU test marker (#11786)
update
update
First Block Cache (#11180)
update
modify flux single blocks to make compatible with cache techniques (without too much model-specific intrusion code)
remove debug logs
update
cache context for different batches of data
fix hs residual bug for single return outputs; support ltx
fix controlnet flux
support flux, ltx i2v, ltx condition
update
update
Update docs/source/en/api/cache.md
Update src/diffusers/hooks/hooks.py

Co-authored-by: Dhruv Nair dhruv.nair@gmail.com

address review comments pt. 1
address review comments pt. 2
cache context refacotr; address review pt. 3
address review comments
metadata registration with decorators instead of centralized
support cogvideox
support mochi
fix
remove unused function
remove central registry based on review
update

Co-authored-by: Dhruv Nair dhruv.nair@gmail.com

Reviewers

DN6 DN6 approved these changes

sayakpaul sayakpaul approved these changes

stevhliu Awaiting requested review from stevhliu

asomoza Awaiting requested review from asomoza

+3 more reviewers

a-r-r-o-w a-r-r-o-w approved these changes

hlky hlky left review comments

Reviewers whose approvals may not affect merge requirements

Labels

roadmap

Add to current release roadmap