lllyasviel
Some ideas from an artist to consider about making an animation model
Ok, I am not the best artist in the world, but I have some studies and practice with 2D and 3D animations, so pls forgive me if my explanation kind of not make a lot of sense initially, but also my english is not native, so pls forgive me.
To me the animation issue and coherence issue in videos, stem from the model not having a representation of what the character should look like. Dunno how it would be possible, but maybe look at what representation a character lora will have about major features. Not sure how to do that though.
But to me, a LORA could be a good way to have something like an art director that checks the animator inbetween or keyframe againts the character sheet that is used in animation studios.
Having said that, It's pretty clear for a proper animation model, to follow the human process of 2D animation. So you need an art director that is like a higher level guy that checks the quality and make sure it looks nice. Maybe a control net that checks what the other control nets are rendering, and like it uses the LORA biases to be just some aditional bias into the process.
Another key area would be to ask for the user, to provide keyframes, done by hand, or already premade or prerendered indidual keyframes using 3D models, maybe keep some segmentation map from a video with color masks or a 3D keyframe with difussed rendered color mask, to keep the integrity of details, such as belts, buttons, shoelaces, etc. Again, the keyframe could then simply be checked with the lora, maybe a middle point in some vector math, maybe the middle point between the lora and the keyframe could help.
Dunno, if training a LORA where segmentation maps of the pictures and some labeling of the mask, could be used, not sure about this, if telling the LORA what each element of the picture could mean.
Having said this, It's pretty clear some keyframes are needed to keep the straight ahead animator bot in check. Then the issue later is simply have an interpolation method where the details are kept drawn by the same artist.
Pretty sure normal interpolation methods using splines (like the inkscape simplification of the vector shapes and smooth of a vector shape works in inkscape) and batch normal control net videos could be provided to create some motion tracking, like similar to blender algos to check the direction and speed of the movement.
Pretty sure first a silloethe mask shape could be interpolated in the first few batches, where a single step is done linearly copying the video and controlnets reference, and then later the smaller details of the lora and keyframes are superimpossed later on top and checked in a secondary few steps to keep as close as the bias in the lora between the target points (made by the keyframes) and the concept art LORA director. And then maybe the vast majority of the process would be refining and polishing this initial mash up of two images, using normal difussion, but animating in a straight ahead method, but with a proper bias to move towards the keyframes and lora concept art.
If anything these are my first ideas to tackle a video model or a model to make sprites for games.
