Something wrong with img2img samplers

[netsvetaev]

Describe your environment

• mbp m1pro
• Branch: development

Describe the bug
I’m trying new samplers with img2img (klms & keuler), but always get strange result.
I use a classic command from https://github.com/invoke-ai/InvokeAI/blob/main/docs/features/IMG2IMG.md, just with «-A klms» for example.

strength 0.5

strength 0.3

Originals:

[lstein]

I just confirmed that img2img with the plms and k* samplers is working as expected on a CUDA system. This may be an M1-specific issue, although I'd be surprised if this was the case. Could you post the original images and the exact prompts you used?

[psychedelicious]

I'm getting the same weird washed out and blurry images at lower img2img strengths. Also on M1.

@netsvetaev 's original images are the last two images they posted btw.

Here's my init image:

And here's k_lms, k_euler, and ddim each at 32 steps and -f 0.25, 0.5, 0.75.

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.25

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.5

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.75

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_lms -f 0.25

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_lms -f 0.5

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_lms -f 0.75

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f 0.25

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f 0.5

"photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f 0.75

[ranhalprin]

I notice that the sampler only runs for one step (regardless of input), this is probably the reason for the blurry results

[psychedelicious]

I notice that the sampler only runs for one step (regardless of input), this is probably the reason for the blurry results

I'm not sure that's accurate - I get very different results when I specify 1 step vs any other number:

1 step "photograph of a tree on a hill with a river" -s 1 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.25

32 steps "photograph of a tree on a hill with a river" -s 32 -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A k_euler -f 0.25

It's possible you are using a low number of steps - with img2img, the number of actual steps is a function of strength. So you specify 32 steps at strength 0.25, you get floor((32 - 1) * 0.25) --> 7 steps. The result is that for certain combinatoins of low step count and strength, it only does one step.

[lstein]

This seems to be an MPS-related bug. I don’t have a Mac to test on, but I’ll investigate any suspicious “if MPS” statements. Maybe @mh-dm or @Any-Winter-4079 could have a look?

[mh-dm]

I don't have an MPS device to test on.

[psychedelicious]

@lstein I'm not convinced it is MPS-related. I have the exact same behavior after forcing my torch device to be cpu:

in ldm/dream/devices.py:

def choose_torch_device() -> str:
    '''Convenience routine for guessing which GPU device to run model on'''
    # if torch.cuda.is_available():
    #     return 'cuda'
    # if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
    #     return 'mps'
    return 'cpu'

in ldm/generate.py:

def fix_func(orig):
    # if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
    #     def new_func(*args, **kw):
    #         device = kw.get("device", "mps")
    #         kw["device"]="cpu"
    #         return orig(*args, **kw).to(device)
    #     return new_func
    return orig

Each step now takes about 4x as long, and there are no messages about mismatched torch devices, so I think I've forced cpu successfully...

[damian0815]

I have been writing some documentation for img2img, which involved visualizing the latent space, and i noticed that when the step count does not scale with strength. if you request 100 steps with a strength of 0.3, SD will only actually do 30 steps of inference. with a strength of 0.6 it will 60. not sure if this is intended behaviour, a bug, or an oversight in the UI design, but since it seems to be surprising i'd suggest adjusting the step count to compensate for strength when doing img2img.

[psychedelicious]

Ok, there is something wrong on a deeper level. I have reverted the force-cpu changes to my local branch and run some tests with strength 0.75 and 0.85, each running from steps 15 to ~40. Sampler DDIM.

My REPL command:

"photograph of a tree on a hill with a river" -S 12345 -W 512 -H 512 -C 7.5 -I /Users/spencer/Documents/Code/initimage.png -A ddim -f {strength} -s {steps}

Using this same init image:

Strength 0.75 images (named the zip f7.5 but its f0.75):
f7.5.zip

Strength 0.85 images (named the zip f8.5 but its f0.85):
f8.5.zip

Images are prefixed by the steps count entered in the REPL (not the actual steps).

Have a scroll through those and you will notice that there are some interesting patterns. There seem to be 2 types of result which I'll call the "common" type - very close to the init image - and the "occasional" type - which diverges more. You get 4 or 5 "common" types, then a couple "occasional" types, then 4 or 5 "common" types, then a couple "occasional" types...

It's far more obvious on the f0.85 images which are "common" and which are "occasional".

Also, it seems like the more steps, the closer the "common" images are to the init image. I would have expected the images to converge but not on something very much like the original...

Here's f0.85 s200:

And here's f0.85 s201:

That ain't right!

[damian0815]

@psychedelicious you might want to try my branch document-img2img which spits out pngs for each intermediate step. check the outputs/img_samples/intermediates folder, which may help understand what's going on (this is what i'm documenting!)

[damian0815]

this is actually a subtle thing that i haven't seen any of the SD guis communicate well. because of the way the SD algorithm works (actually diffusion algorithms more generally), doing steps=50 is not equivalent to doing steps=49 then feeding the 49th image in for "one more" step. if the step count is different then the amount of denoising that happens at each step is different, all the way back to the first step. so your difference between s200 and s201 could be emerging actually from a change that happened on step 2 or 3 - two pixels get interpreted slightly differently and then 200 iterations later your spindly tree has become a bushy tree somewhere else.

the best way to get a handle on this, i found, was to try low step counts and look at the latents at each step. if there's interest i can clean up my intermediate writer and submit it as a feature..?

[psychedelicious]

@damian0815 Thanks, I don't understand what is really going on internally - thanks for explaining that. Do I understand you correct, that the patterns I am seeing are to be expected?

I checked out your branch, but it seems to be missing from scripts.modules.preview_decoder import ApproximateDecoder.

I discovered by trial and error that, in the step_callback, calling sample_to_image(sample) doesn't give you the intermediate image. Would love to have access to your intermediate writer. Thanks.

[damian0815]

yep that's the problem, see how we've already diverged enough to see with our eyes alone after about the 3rd or 4th step. top is 20 steps (-s 30 -f 0.7), bottom is 19 (-s 29 -f 0.7).

@psychedelicious i've fixed the issue on my branch - if you fetch again you can add --write_intermediates to the end of your dream> prompt and it will spit out all the latent steps to an intermediates folder under outputs/img_samples. please let me know how you get on with it!

& i'd love to hear your comments on my updated img2img docs! :)