Skip to content

save_pretrained for serializing config. #11603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 19, 2025
Merged

Conversation

sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented May 22, 2025

What does this PR do?

Follow-up of #11539.

Test code:

from diffusers.modular_pipelines import PipelineBlock

block = PipelineBlock.from_pretrained(
    "diffusers-internal-dev/modular-depth-block-sayak", trust_remote_code=True
)
block.save_pretrained("dummy")

Currently, it serializes the config.json. I think we agreed to manually save the block module that implements the custom block. It looks like so:

{
  "_class_name": "DepthProcessorBlock",
  "_component_names": [
    "depth_processor"
  ],
  "_diffusers_version": "0.34.0.dev0",
  "auto_map": {
    "PipelineBlock": "block.DepthProcessorBlock"
  }
}

diffusers-cli custom_blocks can also do the work. More details:
#11603 (comment)

More about the CLI stuff

Currently, for custom blocks, we need the block implementation module (say block.py), and the config.json, and an optional requirements.txt. @DN6 suggested a good idea over chat.

We let the user implement block.py (or any other module name) inside a folder. We then provide a CLI tool (diffusers-cli init-block ...) that would be run from the folder and spit out the following:

├── block.py
├── config.json
└── requirements.txt

So, the config.json will be created by the CLI command itself.

There will be some reasonable defaults to serialize a desired config.json file but we will also expose certain CLI args to let the users take fine-grained control (would come in handy in case there are multiple block implementations and scripts).

@sayakpaul sayakpaul requested review from DN6 and yiyixuxu May 22, 2025 12:13
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sayakpaul sayakpaul marked this pull request as draft May 22, 2025 13:18
@sayakpaul
Copy link
Member Author

@DN6 I pushed an initial version of the diffusers-cli command we chatted about yesterday. Could you PTAL?

Provided you have implemented a block.py in a directory called custom_blocks, run:

diffusers-cli custom_blocks

And then it should spit out:

config.json
requirements.txt
block.py (should already be present)

This is a rough proposal, happy to iterate further.

@yiyixuxu
Copy link
Collaborator

thanks @sayakpaul, ok if I merge in first? I think erializes the config.json is sufficient for now

@sayakpaul
Copy link
Member Author

@yiyixuxu let's go.

@yiyixuxu yiyixuxu marked this pull request as ready for review June 19, 2025 03:38
@yiyixuxu yiyixuxu merged commit 79be5a1 into modular-refactor Jun 19, 2025
2 checks passed
@sayakpaul sayakpaul deleted the custom-blocks-saving branch June 19, 2025 13:17
yiyixuxu added a commit that referenced this pull request Jun 20, 2025
* add componentspec and configspec

* up

* up

* move methods to blocks

* Modular Diffusers Guiders (#11311)

* cfg; slg; pag; sdxl without controlnet

* support sdxl controlnet

* support controlnet union

* update

* update

* cfg zero*

* use unwrap_module for torch compiled modules

* remove guider kwargs

* remove commented code

* remove old guider

* fix slg bug

* remove debug print

* autoguidance

* smoothed energy guidance

* add note about seg

* tangential cfg

* cfg plus plus

* support cfgpp in ddim

* apply review suggestions

* refactor

* rename enable/disable

* remove cfg++ for now

* rename do_classifier_free_guidance->prepare_unconditional_embeds

* remove unused

* [modular diffusers] introducing ModularLoader (#11462)

* cfg; slg; pag; sdxl without controlnet

---------

Co-authored-by: Aryan <[email protected]>

* make loader optional

* remove lora step and ip-adapter step -> no longer needed

* rename pipeline -> components, data -> block_state

* seperate controlnet step into input + denoise

* refactor controlnet union

* reefactor pipeline/block states so that it can dynamically accept kwargs

* remove controlnet union denoise step, refactor & reuse controlnet denoisee step to accept aditional contrlnet kwargs

* allow input_fields as input & update message

* update input formating, consider kwarggs_type inputs with no name, e/g *_controlnet_kwargs

* refactor the denoiseestep using LoopSequential! also add a new file for denoise step

* change warning to debug

* fix get_execusion blocks with loopsequential

* fix auto denoise so all tests pass

* update imports on guiders

* remove modular reelated change from pipelines folder

* made a modular_pipelines folder!

* update __init__

* add notes

* add block state will also make sure modifed intermediates_inputs will be updated

* move block mappings to its own file

* make inputs truly immutable, remove the output logic in sequential pipeline, and update so that intermediates_outputs are only new variables

* decode block, if skip decoding do not need to update latent

* fix imports

* fix import

* fix more

* remove the output step

* make generator intermediates (it is mutable)

* after_denoise -> decoders

* add a to-do for guider cconfig mixin

* refactor component spec: replace create/create_from_pretrained/create_from_config to just create and load method

* refactor modular loader: 1. load only load (pretrained components only if not specific names) 2. update acceept create spec 3. move the updte _componeent_spec logic outside register_components to each method that create/update the component: __init__/update/load

* update components manager

* up

* [WIP] Modular Diffusers support custom code/pipeline blocks (#11539)

* update

* update

* remove the duplicated components_manager file I forgot to deletee

* fix import in block mapping

* add a to-do for modular loader

* prepare_latents_img2img pipeline method -> function, maybe do the same for others?

* update input for loop blocks, do not need to include intermediate

* solve merge conflict: manually add back the remote code change to modular_pipeline

* add node_utils

* modular node!

* add

* refator based on dhruv's feedbacks

* update doc format for kwargs_type

* up

* updatee modular_pipeline.from_pretrained, modular_repo ->pretrained_model_name_or_path

* save_pretrained for serializing config. (#11603)

* save_pretrained for serializing config.

* remove pushtohub

* diffusers-cli rough

---------

Co-authored-by: YiYi Xu <[email protected]>

---------

Co-authored-by: Aryan <[email protected]>
Co-authored-by: Dhruv Nair <[email protected]>
Co-authored-by: Sayak Paul <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants