A1111 <> Diffusers Scheduler mapping

[patrickvonplaten]

A1111 <> Diffusers Mapping

A1111 / K-Diffusion	Diffusers	Notes
DPM++ 2M	Multistep DPM-Solver
DPM++ 2M Karras	Multistep DPM-Solver	init with use_karras_sigmas=True
DPM++ 2M SDE	Multistep DPM-Solver	init with algorithm_type="sde-dpmsolver++"
DPM++ 2M SDE Karras	Multistep DPM-Solver	init with algorithm_type="sde-dpmsolver++" and use_karras_sigmas=True
DPM++ 2S a	N/A	Very similar to DPM++ SDE
DPM++ 2S a Karras	N/A	Very similar to DPM++ SDE Karras
DPM++ SDE	Singlestep DPM-Solver
DPM++ SDE Karras	Singlestep DPM-Solver	init with use_karras_sigmas=True
DPM2	DPM Discrete Scheduler
DPM2 Karras	DPM Discrete Scheduler	init with use_karras_sigmas=True
DPM2 a	DPM Discrete Scheduler with ancestral sampling
DPM2 a Karras	DPM Discrete Scheduler with ancestral sampling	init with use_karras_sigmas=True
DPM adaptive	N/A
DPM fast	N/A	Not really used no?
Euler	Euler scheduler
Euler a	Euler Ancestral Scheduler
Heun	Heun Scheduler
LMS	Linear Multistep
LMS Karras	Linear Multistep	Init with use_karras_sigmas=True
N/A	DEIS
N/A	UniPCMultistepScheduler

Diffusers samplers have a couple more ways to be customized as can be seen from the inits of the respective samplers.

[patrickvonplaten]

@stevhliu could we maybe put this in a doc as discussed a bit during our meeting? Maybe in the overview doc page?

[stevhliu]

Nice, I'll roll this into a PR I'm working on to update the scheduler APIs!

[rbertus2000]

@patrickvonplaten: I'm confused: Shouldn't DPM++ SDE be DPMSolverSDEScheduler? And in this issue you just said that "DPM++ 2s a" = DPMSolverSinglestepScheduler.

[patrickvonplaten]

• https://huggingface.co/docs/diffusers/v0.18.2/en/api/schedulers/dpm_sde is another implementation of DPMSolverSDE but it's not the official one (the author contributed DPMSolverSDE himself), so that the default one.
  In my opinion, we could deprecate DPMSolverSDE at some point.
• RE: DPM++ 2s a yes indeed it's very similar to DPMSolverSinglestepScheduler with init with algorithm_type="sde-dpmsolver++" and use_karras_sigmas=True

[cmdr2]

Hi @patrickvonplaten Thanks for the clarification. For DPM++ 2s a - can you please confirm whether the algorithm_type should be sde-dpmsolver++ or dpmsolver++?

Your recommendation in #3953 is dpmsolver++:

DPM++ 2s a => should be the following: DPMSolverSinglestepScheduler.from_config(..., algorithm_type="dpmsolver++")

And your most-recent message here is sde-dpmsolver++:

RE: DPM++ 2s a yes indeed it's very similar to DPMSolverSinglestepScheduler with init with algorithm_type="sde-dpmsolver++" and use_karras_sigmas=True

Thanks!

[cmdr2]

Hmm, I don't think sde-dpmsolver++ is implemented for DPMSolverSinglestepScheduler

NotImplementedError: sde-dpmsolver++ does is not implemented for <class 'diffusers.schedulers.scheduling_dpmsolver_singlestep.DPMSolverSinglestepScheduler'>

[patrickvonplaten]

Ah good catch! We need to add this algorithm type. Here: https://github.com/huggingface/diffusers/pull/3344/files was the PR to add it to the Multi-Step scheduler. I think it can be more or less copied 1-to-1 to DPMSolverSinglestepScheduler - would you like to open a PR?

[cmdr2]

@patrickvonplaten Thanks! I've opened a PR at #4251

There's a discrepancy that I'm not sure about, I've raised that in the PR.

[CapsAdmin]

I tried implementing a1111's kdiffusion samplers in diffusers along with the ability to pass user changable settings from a1111 to kdiffusion.

Do note that I don't have much experience in this field, it's just something I got into for fun the last month or two.

But my findings are:

• UniPC is available in a1111, but I was not sure how to translate skip type. A1111 has logSNR, time_uniform and time_quadratic, while diffusers has linspace, leading and trailing?
• Samplers with "brownian_noise": True will pass a custom noise generator to kdiffusion that generate brownian noise
• From settings you can change the eta of Euler a and other samplers, by default it's 1.0 and it's used here in kdiffusion and I imagine in euler a's case it would go here
• You can also change s_churn = 0.0, s_tmin = 0.0, s_tmax = 0.0, s_noise = 1.0, from settings which are passed onto the kdiffusion samplers
• A1111 overrides torch.randnlike to produce consistent results in samplers that use them. The random tensors are pushed onto some array and then popped by kdiffusion calling torch.randnlike
  • An API for this was requested by AUTOMATIC1111 and solved here [Feature request] Let user provide his own randn data for samplers in sampling.py crowsonkb/k-diffusion#25 but it was never actually used in the webui, apart from the brownian noise find.
• The creation of sigmas in a1111 to be passed to a sampler seem a little involved. You can change the "rho", "sigma_min" and "sigma_max" from the ui, and there are some backwards compatible flags

I was going for 1 to 1 kind of for fun, but I think ideally some sacrifices should be made. Like I'm not sure people intentionally use the extra sampler settings, and they're all set to default as far as I know.

My implementation can be found here. I've compared various samplers to a1111 and my implementation and it seems to work, but probably there are details I've missed.
https://github.com/CapsAdmin/diffusers-a1111/

[CapsAdmin]

I've refactored and fixed some issues, samplers reproduce the same result in a1111 except for DPM2 a Karras (which is marked as a second order sampler). I also explain a bit better in code what's going on in kdiffusion.py

I can observe that samplers like Heun, DPM2 a, DPM fast, DPM++ SDE Karras produce very similar results but has a subtle variation from a1111.

I was thinking this would be the sigma values being slightly different. For example:

diffusers: [14.6172, 11.7266,  9.3438,  7.3867,  5.7891,  4.5000,  3.4648,  2.6406,
         1.9912,  1.4834,  1.0908,  0.7910,  0.5649,  0.3965,  0.2732,  0.1842,
         0.1214,  0.0779,  0.0485,  0.0292,  0.0000]

a1111: [14.6116, 11.7413,  9.3685,  7.4192,  5.8284,  4.5394,  3.5029,  2.6763,
         2.0228,  1.5112,  1.1147,  0.8110,  0.5812,  0.4096,  0.2834,  0.1921,
         0.1273,  0.0822,  0.0515,  0.0313,  0.0000]

I used the same method from the k-diffusion pipeline in diffusers to get the min max sigma and I'm not sure how to do it otherwise.

[CapsAdmin]

I also discovered a potential bug where a1111 will pass eta=1 coming from sampler settings to DPM Fast and DPM Adaptive. The default eta in kdiffusion is 0.

When they are used with eta 0, DPM fast actually produces good results and DPM adaptive finishes a lot faster.

old man, 25 steps, DPM fast, eta 0

old man, 25 steps, DPM fast, eta 1

I think this setting was supposed to be used for ancestral samplers, but accidentally brought to these samplers.

So I guess this is the reason DPM fast is infamous for producing bad results.

[xhinker]

Wow, this is great, thanks @patrickvonplaten , I have been spending lots of nights testing out the correct mapping relationship