-
Notifications
You must be signed in to change notification settings - Fork 6.1k
save_pretrained for serializing config. #11603
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
@DN6 I pushed an initial version of the Provided you have implemented a diffusers-cli custom_blocks And then it should spit out: config.json
requirements.txt
block.py (should already be present) This is a rough proposal, happy to iterate further. |
thanks @sayakpaul, ok if I merge in first? I think erializes the config.json is sufficient for now |
@yiyixuxu let's go. |
* add componentspec and configspec * up * up * move methods to blocks * Modular Diffusers Guiders (#11311) * cfg; slg; pag; sdxl without controlnet * support sdxl controlnet * support controlnet union * update * update * cfg zero* * use unwrap_module for torch compiled modules * remove guider kwargs * remove commented code * remove old guider * fix slg bug * remove debug print * autoguidance * smoothed energy guidance * add note about seg * tangential cfg * cfg plus plus * support cfgpp in ddim * apply review suggestions * refactor * rename enable/disable * remove cfg++ for now * rename do_classifier_free_guidance->prepare_unconditional_embeds * remove unused * [modular diffusers] introducing ModularLoader (#11462) * cfg; slg; pag; sdxl without controlnet --------- Co-authored-by: Aryan <[email protected]> * make loader optional * remove lora step and ip-adapter step -> no longer needed * rename pipeline -> components, data -> block_state * seperate controlnet step into input + denoise * refactor controlnet union * reefactor pipeline/block states so that it can dynamically accept kwargs * remove controlnet union denoise step, refactor & reuse controlnet denoisee step to accept aditional contrlnet kwargs * allow input_fields as input & update message * update input formating, consider kwarggs_type inputs with no name, e/g *_controlnet_kwargs * refactor the denoiseestep using LoopSequential! also add a new file for denoise step * change warning to debug * fix get_execusion blocks with loopsequential * fix auto denoise so all tests pass * update imports on guiders * remove modular reelated change from pipelines folder * made a modular_pipelines folder! * update __init__ * add notes * add block state will also make sure modifed intermediates_inputs will be updated * move block mappings to its own file * make inputs truly immutable, remove the output logic in sequential pipeline, and update so that intermediates_outputs are only new variables * decode block, if skip decoding do not need to update latent * fix imports * fix import * fix more * remove the output step * make generator intermediates (it is mutable) * after_denoise -> decoders * add a to-do for guider cconfig mixin * refactor component spec: replace create/create_from_pretrained/create_from_config to just create and load method * refactor modular loader: 1. load only load (pretrained components only if not specific names) 2. update acceept create spec 3. move the updte _componeent_spec logic outside register_components to each method that create/update the component: __init__/update/load * update components manager * up * [WIP] Modular Diffusers support custom code/pipeline blocks (#11539) * update * update * remove the duplicated components_manager file I forgot to deletee * fix import in block mapping * add a to-do for modular loader * prepare_latents_img2img pipeline method -> function, maybe do the same for others? * update input for loop blocks, do not need to include intermediate * solve merge conflict: manually add back the remote code change to modular_pipeline * add node_utils * modular node! * add * refator based on dhruv's feedbacks * update doc format for kwargs_type * up * updatee modular_pipeline.from_pretrained, modular_repo ->pretrained_model_name_or_path * save_pretrained for serializing config. (#11603) * save_pretrained for serializing config. * remove pushtohub * diffusers-cli rough --------- Co-authored-by: YiYi Xu <[email protected]> --------- Co-authored-by: Aryan <[email protected]> Co-authored-by: Dhruv Nair <[email protected]> Co-authored-by: Sayak Paul <[email protected]>
What does this PR do?
Follow-up of #11539.
Test code:
Currently, it serializes the
config.json
. I think we agreed to manually save the block module that implements the custom block. It looks like so:diffusers-cli custom_blocks
can also do the work. More details:#11603 (comment)
More about the CLI stuff
Currently, for custom blocks, we need the block implementation module (say
block.py
), and theconfig.json
, and an optionalrequirements.txt
. @DN6 suggested a good idea over chat.We let the user implement
block.py
(or any other module name) inside a folder. We then provide a CLI tool (diffusers-cli init-block ...
) that would be run from the folder and spit out the following:So, the
config.json
will be created by the CLI command itself.There will be some reasonable defaults to serialize a desired
config.json
file but we will also expose certain CLI args to let the users take fine-grained control (would come in handy in case there are multiple block implementations and scripts).