0.15.1 version changes the weight shapes of unet !!!

[CharlieLzy]

Describe the bug

There is a horrible bug that the weight shapes of unet are changed after the version update from 0.13.1 to 0.15.1. I found it when I tried to use controlnet pipeline, the 0.13.1 has not supported controlnet yet, so I updated the diffusers. After that I tried on controlnet pipeline, and I found the shape of injected lora weights is incompatible with the unet. Then I printed the shapes of all weights in old unet checkpoint and newly converted one, as expected, different. I suspected it was controlnet‘s problem at beginning, but confirmed the bug came from a version changing after runing the previous codes. I changed the version back to 0.13.1, and everything became fine immediately. I hope this problem could be taken seriously and solved soon, cause controlnet is important, so does lora.

Reproduction

image = pipe(prompt, negative_prompt=negative_prompt, num_inference_steps=num_inference_steps, guidance_scale=7.5).images[0]

Logs

No response

System Info

diffusers 0.13.1, 0.15.1

[yiyixuxu]

Hi:

Can you provide a code snippet that's reproducible?
And maybe provide a list of parameters you think changed between these 2 versions

Thanks

YiYi

[CharlieLzy]

Hi:

Can you provide a code snippet that's reproducible? And maybe provide a list of parameters you think changed between these 2 versions

Thanks

YiYi

I just run the code(normal in 0.13.1).

from lora_diffusion import patch_pipe

model_base = "runwayml/stable-diffusion-v1-5"
device = "cuda"

pipeline = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16).to(device)
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)

lora_weight_path = "lora.safetensors"
patch_pipe(pipeline, lora_weight_path, patch_text=True, patch_unet=True, patch_ti=True)
image = pipeline(prompt, num_inference_steps=20, guidance_scale=7.5).images[0]

######### error #############
return (
self.linear(input)
+ self.dropout(self.lora_up(self.selector(self.lora_down(input))))
* self.scale
)

RuntimeError: The size of tensor a (320) must match the size of tensor b (2560) at non-singleton dimension 2

The shapes of self.linear.weight, self.lora_up.weight,self.lora_down.weight are (320,320), (2560, r), (r,320). r is the lora rank.
The lora model is previously trained on sd 1.5.

[CharlieLzy]

I think I have probably found the cause. I checked the state dict of unet carefully, find the keys and shapes are all the same from loading checkpoint directly and unet state dict from sd pipe, but the order changed. When lora works, it find the target layers in the model, and the order decides the shapes of injected lora layers. I don't know why there is change of unet state dict order due to version update. Is there any hope of correcting this problem? Because it will cause some unnecessary troubles.

[patrickvonplaten]

Hey @CharlieLzy,

Note that we can't execute:

from lora_diffusion import patch_pipe

model_base = "runwayml/stable-diffusion-v1-5"
device = "cuda"

pipeline = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16).to(device)
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)

lora_weight_path = "lora.safetensors"
patch_pipe(pipeline, lora_weight_path, patch_text=True, patch_unet=True, patch_ti=True)
image = pipeline(prompt, num_inference_steps=20, guidance_scale=7.5).images[0]

######### error #############
return (
self.linear(input)
+ self.dropout(self.lora_up(self.selector(self.lora_down(input))))
* self.scale
)

as we don't know what lora_diffusion is

[CharlieLzy]

Hi, @patrickvonplaten
lora_diffusion is from https://github.com/cloneofsimo/lora
I found a compromise way to solve my problem that using 0.14.0, which has controlnet and works well，but I confirmed that 0.15.1 really changed the keys order of unet state dict from pipeline.

[linyu0219]

@CharlieLzy cloneofsimo/lora#237

[patrickvonplaten]

I see, we should probably then pin https://github.com/cloneofsimo/lora/ to 0.14.0 no? Also we should really try to allow loading https://github.com/cloneofsimo/lora/ weight format into diffusers (cc @sayakpaul )

[CharlieLzy]

@patrickvonplaten Thanks! As there will be new functions that can only be used in the updated version, hopefully the new version will address this issue. Actually, it's just a matter of the order of unet state dict keys.

[sayakpaul]

I would incline to pin the version since it's a much easier alternative than us adding utilities to support a new format. But if we feel strongly about it, then maybe worth working with Simo (author of the LoRA SD repo) to figure this out.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.