Problem with using lora in prompt editing · AUTOMATIC1111/stable-diffusion-webui · Discussion #10897 (original) (raw)
Im using a1111 v1.20, I have been testing prompt editing and I have found that mine only works with regular words, as soon as I put a lora in a prompt edit it starts from the first frame even if i tell it to start at the middle or end of the generation.
[<lora:LowRA:0.9>:0.5]
This prompt uses a very obvious LowRA Lora and it starts from the first frame and makes it dark when it should be starting halfway through the generation.
[wearing a hat:0.5]
But if I change the lora to just wearing a hat I can see it starts half way through. I tested it with other loras as well. I have seen new posts on reddit using the loras in the prompt edit so it seems like it should be working.
I also tried disabling additional networks extension in case that had anything to do with it but it didn't fix it.
The other extensions are pretty standard like controlnet, ultimate sd upscale, open pose, multidiffusion so I don't think they are interferrring and I wasn't using them in the test.
Any ideas on what the problem is?
-EDIT: Maybe somebody can try to reproduce this on v1.2? download LowRA and use[<lora:LowRA:0.9>:0.5] in your prompt and see if it darkens on the first frame instead of halfway through?
You must be logged in to vote
Prompt editing doesn't function with extra networks, let alone any special prompt syntax. They're applied to the model weights, and immediately removed from the prompt. Your prompt that's actually being parsed is [:0.5], which ultimately does nothing.
If someone on Reddit is claiming it works they must be doing something incorrectly or naively think it changes the result.
Prompt editing doesn't function with extra networks, let alone any special prompt syntax. They're applied to the model weights, and immediately removed from the prompt. Your prompt that's actually being parsed is [:0.5], which ultimately does nothing.
If someone on Reddit is claiming it works they must be doing something incorrectly or naively think it changes the result.
You must be logged in to vote
2 replies
This should be fixable though, shouldn't it? I mean, the [...:0.5] could tell the prompt parser to add the lora token or whatever not right at the beginning of the generation but half way through. Or is it more complicated than that? Can prompts in general be changed in the middle of a generation? If that's the case a prompt like photo of a house, green grass, [<lora:LowRA:0.9>:0.5] could mean, do photo of a house, green grass during the first half of the samples and then simply continue the other half of the generation with photo of a house, green grass, <lora:LowRA:0.9>. Why should it not work like that? It would mean that the prompt would have to be re-parsed at the boundaries set by prompt editing. I do not see the problem with that right now, but I'm no expert of the inner workings of the tokenizers and stable diffusion.
Or is it more complicated than that?
It is. And it is completely unrelated to tokenizers. Commas shouldn't even be used for the reason I already outlined above because then the prompt gets parsed as something like photo of a house, green grass, which has an extra trailing comma.
Answer selected by BHMW
Update: There is now an extension available for the upcoming 1.5.0 version that does implement this functionality. https://github.com/cheald/sd-webui-loractl
See the README for specifics, as it utilizes the new extra networks architecture and expects you to follow that syntax.
You must be logged in to vote
0 replies