Interpause
Some improvements that would make things easier
Utility Buttons:
Some buttons (honestly can just be hotkeys for pros but not everyone is as experienced in Krita) that does most of the common things and I think should be visible on all the panels, much like the top section where Face Restorer and Upscaler are always visible regardless of what tab you're on.
Blur Layer- for doing the gaussian blur for Masking for Inpainting. Select>Feather Selection - for doing selections of img2img but you don't want the edges to have visible seams. I feel like the size of the blur should depend on the current canvas size. When you're doing stuff in 512 x 512 of course you just want a smaller blur size but when you upscale that and you're now above 1k pixels, the size of the blur needs to be bigger as well.
Rescale Image presets - Add a x4, x2, x0.5, x0.25, (+nearest neighbor option, maybe for pixel art?) btw in the outdated video you made, there's an option for rescaling into percentage, so you don't need to do any math.
https://user-images.githubusercontent.com/88552647/205484365-d27f985f-3361-4f23-85cc-4e7b53e303e2.mp4
Layer>Transform>Offset Layer - really useful for creating tileable stuff. The usual problem with tiles is when you upscale, you also introduce seams, this honestly should just be an Automatic1111 script rather than over here 🤔 I'll try to make a feature request there too. Offset Layer will make the edges be at the center of the canvas so the upscaler can do a proper upscale of those sections, then you can just blend the two together or inpaint to remove the seams.
Outpainting
Outpainting feels wonky? Haven't really understood why, but I understand it's also hard to predict what the user wants with outpainting. I feel like the selection part is being used in a way, at least for me personally, that doesn't make sense.
For inpainting, it's used as "hey SD, look over here, disregard everything else on the canvas" and it's the same here in outpainting, but the problem then is SD also resizes the outputs into the same selection. So you get non-uniform scaling from the outpainting result. I feel like there should be either a checkbox or this "resize into selection" function should be turned off by default when using Outpainting. But the problem then is the user might not be able to see what was generated, and need to move the resulting images around if it ever clips the actual canvas borders.
Perhaps it opens the outpainting results in a new document? Or changes the canvas size to accommodate for any possible canvas clipping? Not entirely sure, so let me know your thoughts on that.
- I'll consider utility buttons for the milestone since they make things more convenient.
- For feather selection, I think it makes more sense to be relative to the selection size?
- Interesting, Ive never done tiling before so I never knew about this offset layer thing.
- Actually, outpainting (upscale too) expands the canvas, but only when nothing is selected: https://github.com/Interpause/auto-sd-paint-ext#2022-11-15. I probably should have made it more clear somewhere (I was planning to cover it when I made the new demo videos). Maybe outpainting should just ignore the selection area.
I was doing some inpainting today, and I bumped into the issue of the canvas getting expanded. I'm having to make a conscious effort to keep it the same. Why was it deemed better to have a potential for expanding the canvas for the inpainting tool, when most of the time that would just create white borders and mess up the image size I was going for? What's the purpose? Another option for outpainting, maybe? At least make it a toggle, I'd say.
@Interpause yep I've realized it too late that when nothing is selected, it expands the canvas. I agree with the selection size, that makes more sense.
I think for now, I'll just do what you suggested where you just expand the canvas on your own and use inpaint/img2img instead. That might be a better way of going about outpainting in general actually? Since in the normal way using webui, you just say how much pixels you want the image to expand by, then probably crop after when you find a good composition. But since we're already in Krita, we already know how much space we want to expand and fill. Not really sure how Mk2 works so the inpainting/img2img workaround might give worse results than Mk2. But I feel like the img2img/inpaint works more intuitively in this context
@R4rum Yea in general I think it's just not as obvious when it does and doesn't expand. I for one like the current workflow of box select, then img2img or inpaint, as I think that's the intuitive way of going about it. Saves on VRAM/memory too since the AI doesn't need to see the entire canvas (especially when I moved on to upscaling a composition I like). So just make sure you box select the section that you want SD to see and replace. If you press CTRL+A before generating an inpaint or an img2img, it ensures you don't expand the canvas.
Yeah, the box selection for an area to focus on might have its uses at times. I like that it's at least a readily available option.
Saves on VRAM/memory too since the AI doesn't need to see the entire canvas (especially when I moved on to upscaling a composition I like)
I do even selecting compositions at a high resolution, 2.5K or so. And it's really only become feasible with the Krita plugin which offers streamlined upscaling that's quicker than the highres fix. 22 extra seconds for an upscale when an image is created in 80 (when batched) is an okay tradeoff for a slightly better idea of what the final product would look like. (I wonder if upscales would also benefit from being batched, hmm...)
The minute per image might be a lot but the base size is set to 768 due to some models having been trained with that already, and I've been testing out max size yet higher at 896. I did have to go with the lowvram option but it hardly seems to have had a performance impact compared to medvram.
When I read the source code for outpainting Mk2, it seems to do something similar to img2img alt where it tries to reconstruct the original noise pattern and extend it, which can result in a more seamless generation.
@Interpause ooohh interesting, that would be cool but not as useful in my use case at least.
@R4rum I'm honestly sure if I understood half of what you said... Haven't touched much of my settings. I'm on a RTX 2060 so mine can handle the default 512 x 512 just fine, In other times, I just need to make it 448x448 but the results I get isn't really that different. Haven't fully understood what's highres fix is for. If I'm not mistaken, it just makes it so that there's no duplication/multiple heads etc going on when you're generating an image in a stretched aspect ratio? Haven't really bothered with it since it takes so long when I tested it 😅
How's the performance impact though? I'm interested in generating above the 512 x 512 size if the change in the speed of the generation is negligible. There's also the xformers thing which apparently makes things faster in general but I don't really know enough about it. All of these things are generally hard to keep track of 😆
And not sure why are you batching inside the Krita plugin? I'm assuming you're talking about like 10-20 images per generation right? I'd rather make a big batch, curate/filter them, and only then import them into Krita since loading them all into Krita right after it gets generated seems like a waste of resources unless your PC can handle it? Or maybe you're using it for maybe like texture creation, rather than making paintings/illustrations?
The upscaler option can be found at the very top of all the tabs in the non-dev branch of Krita plugin. Try it out, I'd say. First make a new image of the final size you want, select an upscaler of your choice and the plugin does the rest for you.
Those base size and max size are in my view what gives the starting point to an upscale. The upscaler I use, SwinIR_4x is very faithful to the original image. In my opinion it excels at 2x scale, and 3x scale is comparatively blazing fast as well but the level of detail won't increase really beyond making things look smoother past that. That starting level of detail is why a decently large base image seems a good idea to have when using SwinIR. Possibly if you have a smaller image as your starting point you might opt for a more resource-intensive purpose-built for the type of art you make upscaler, but I'm just guessing too.
I've not noticed "duplication/multiple heads" with the base and max size settings I use with the models I use; those kinds of problems are maybe where I'd start drawing the line of how high I'd set those even if I had the computing resources.
if the change in the speed of the generation is negligible
Depends on the upscaler. With my current workflow it's in total roughly 1.25 to 1.33 times time spent. Not negligible, but acceptable. I've not done a lot of testing, but a final size being an integer multiple of the original generated size seems that bit over 1.25x the time and an uneven multiplier resize makes it slightly slower.
In other times, I just need to make it 448x448
Have you tried
set COMMANDLINE_ARGS="--lowvram --api"
in webui-user.bat yet? Also I'm not yet sure what's a good batch size, but the speed increase comes at the cost of an increased VRAM usage so... maybe you have it too high? Batch count you can have as high as you want.
And not sure why are you batching inside the Krita plugin?
The point of Krita plugin for me is to make the workflow easier. It's a "one-stop shop" when just making images. That integrated upscaling thing isn't available in regular webui, also.
@rexelbartolome A good idea to start using the xformers. Generation time for a batch of 6 images was cut down from 07:53 (plus 128 seconds for the upscales) to 05:18 (plus 126 seconds for the upscales).
@R4rum that's great! Haven't tested it yet, I'm trying to install it too but it's telling me that it can't install it in the python version I'm using... I'll try asking around on A1111's repo. What instructions did you follow to install it?
Have you tried
set COMMANDLINE_ARGS="--lowvram --api"
Yea, generations took too long, but --medvram seems to work fine in my case :)
That integrated upscaling thing isn't available in regular webui, also.
Yea that's pretty handy when you're already working inside an upscaled image/canvas
The xformers? My troubles are going to be unrelated to yours probably. I'm on Linux and I have Python 3.10.8. Manual installation seemed to hang (or maybe it was just doing some super expensive operation on CPU), and pip install didn't work properly by defaul (the primary error message was "RuntimeError: CUTLASS submodule not found. Did you forget to run git submodule update --init --recursive ?").
However, these commands finally worked:
source ./venv/bin/activate
FORCE_CUDA=1 pip install git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
Honestly, didn't understand half of what you said again 😆 but I was able to reinstall my entire SD and got xformers working. Apparently I had some issues with running SD on my F: drive cause it couldn't find the Python 3.10 that was installed on my C: drive. Had to install the new Python on F: so it can find it.