Inpainting height and width

[Amazingldl]

import PIL
import requests
import torch
from io import BytesIO

from diffusers import StableDiffusionInpaintPipeline


def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")


img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = download_image(img_url)
mask_image = download_image(mask_url)

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

prompt = "Face of a yellow cat, high resolution, sitting on a park bench"
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, height=1024, width=1024).images[0]

I got this error:

│ D:\App\miniconda\envs\aigc\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stabl │
│ e_diffusion_inpaint.py:871 in __call__                                                           │
│                                                                                                  │
│   868 │   │   │   │                                                                              │
│   869 │   │   │   │   # concat latents, mask, masked_image_latents in the channel dimension      │
│   870 │   │   │   │   latent_model_input = self.scheduler.scale_model_input(latent_model_input   │
│ ❱ 871 │   │   │   │   latent_model_input = torch.cat([latent_model_input, mask, masked_image_l   │
│   872 │   │   │   │                                                                              │
│   873 │   │   │   │   # predict the noise residual                                               │
│   874 │   │   │   │   noise_pred = self.unet(latent_model_input, t, encoder_hidden_states=prom   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 128 but got size 64 for tensor number 2
in the list.

If I set the height=520, width=520, I got this error:

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 65 but got size 64 for tensor number 2 in
the list.

Can I customize the height and width of the output image? If so, what are the requirements for height and width?
Thanks for your helps!

[Madot10]

Hi @Amazingldl,
Something I've noticed using stable diffusion the last couple of months is about the input image height and width must be a multiple of 64 (and the same applies to the output image)

[sayakpaul]

What happens when you do 512x512?

[Amazingldl]

yes, but set output image height=1024, width=1024 should be work?

Hi @Amazingldl, Something I've noticed using stable diffusion the last couple of months is about the input image height and width must be a multiple of 64 (and the same applies to the output image)

[Amazingldl]

Do you mean this?
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, height=512, width=512).images[0]
It works, but why can't set height=1024 and width=1024, what is wrong with this size?

What happens when you do 512x512?

[Amazingldl]

Thank you all for your reply.

[rupertmenneer]

Hey everyone! I don't think the way image resolutions are handled is super clear way in the inpainting pipeline (and perhaps others).

The current code allows you pass in an image of incorrect resolution into the inpainting pipeline and then will throw a tensor mismatch error. Additionally, there are two input parameters: height and width - however there is no check to see whether the image actually matches the passed height/width nor does the pipeline resize the image w.r.t these inputs. I propose the following changes which I am happy to put into a PR:

• If height/width left are None, and the image isn’t 512*512 then warn users they have passed in an input image that is not the expected size and automatically resize it. Perhaps reject it if isn’t square aspect ratio.
• If the height/width have been passed as an input parameter then resize the input image accordingly. I feel like there is no point having the height/width parameters if the input image isn’t modified to match them.
• Improved documentation on the model card w.r.t input resolutions.

N.b. to clarify input sizes must already be a multiple of 8 - this is expected due the U-net multiplier/AE. This check already exists in the code that the passed height/width are a mult of 8, the pipeline works just fine on any mult of 8.

However it should also be stated that the inpainting model (and presumably the base model as well) is quite sensitive to resolution/scales. I have produced some results below inpainting on the same image at 32, 64, 128, 256, 512, 1024. As you can see results below 512 degrade significantly. This makes sense given the fact the model uses a vector quantised auto encoder and has learned to produce results at a specific scale and resolution, it is quite clear in the paper/model card they trained the model on 512 images. I am not sure if there is anything we can do to really address this, but might make sense to make this clear to people passing in images above/below the recommended size(s). Perhaps we could add this to the model card or documentation?

Happy to write a PR for this if there is any agreement on the above?

[sayakpaul]

CC: @patrickvonplaten @williamberman

[rupertmenneer]

To add some additional insight, the in-painting model seems to be slightly sensitive to aspect ratio as well:

896, 472

448, 472

224, 472

112, 472

prompt = "a mountain landscape"

However, I believe this is simply just the model struggling with the different resolution (we know from my previous testing that model struggles with resolutions < 512). However, since it is sensitive to resolution (512<->1024 seems best), it inherently makes it quite sensitive to aspect ratio. Not sure if we can totally extrapolate these results to the general case but seems to be quite intuitive given the architecture.

[williamberman]

Yes, the initial issue was probably due to one of the model inputs (latents?) being created according to the passed height and width parameters which were different dimensions from one of (or both) the resized mask and/or the encoded masked image.

Once there are multiple picture inputs (in this case, latents, mask, and the original image), keeping track of the right dimensions for all of them can get a bit wonky :).

I think in this case, the solution might just be re-sizing the input image before it is masked and encoded. We similarly did this in the controlnet pipeline here

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py

Lines 672 to 696 in 5c7a35a

	def prepare_image(
	self,
	image,
	width,
	height,
	batch_size,
	num_images_per_prompt,
	device,
	dtype,
	do_classifier_free_guidance=False,
	guess_mode=False,
	):
	if not isinstance(image, torch.Tensor):
	if isinstance(image, PIL.Image.Image):
	image = [image]
	if isinstance(image[0], PIL.Image.Image):
	images = []
	for image_ in image:
	image_ = image_.convert("RGB")
	image_ = image_.resize((width, height), resample=PIL_INTERPOLATION["lanczos"])
	image_ = np.array(image_)
	image_ = image_[None, :]
	images.append(image_)

@rupertmenneer lots of interesting points on the aspect ratio side! I'm not super familiar with the affect of aspect ratio on model quality in this case but it seems outside the scope of this issue. It's probably just a small fix with resizing the masked image before its encoded. If you want to propose aspect ratio based pipeline changes might be worth a separate issue :)

[rupertmenneer]

@williamberman yes, I totally agree. I think your code addresses the problem in the simplest way. I was considering adding warnings if the height/width are below 512 but I suppose it will be self-evident from output and keeps it simpler.

I also note that your attached code only works for PIL inputs and not tensor images/numpy arrays. I am not sure if that is intentional, but any how I have added the proposed changes into a PR that resizes input images to height and width (n.b. only PIL images). #3322

Possibly worth adding this for pt tensors/np arrays too..

Also to your point I totally agree it is quite difficult to keep track of all the inputs and verify they are the correct size/shape/content etc. It would be a larger piece of work but has a general DiffuserInput class been considered? This could abstract out a lot of complexity of these pipelines

[patrickvonplaten]

BTW diffusers code only requires height and width to be a multiple of 8, not 64 :-) See: #505

[Amazingldl]

solved by #3322
thanks rupertmenneer