= Animation Stylization Collection :imagesdir: asset :tip-caption: 💡 :note-caption: 📝 :warning-caption: ⚠️ :caution-caption: 🔔 :important-caption: ❗ :experimental: :toc:

== Environment

[source,bash]

conda create -n torch python=3.8 conda activate torch conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch pip install pytorch-lightning==1.0.2 opencv-python matplotlib joblib scikit-image torchsummary webdataset albumentations more_itertools

== Algorithm

=== https://github.com/TachibanaYoshino/AnimeGANv2[AnimeGAN: A Novel Lightweight GAN for Photo Animation]

[cols="^.^5,<.^50,<.^50"] |===

a| Setup

2+a|

. [yellow]#If you just want to use it, just skip the following step#

. download dataset from https://github.com/TachibanaYoshino/AnimeGAN/releases/tag/dataset-1[here] and unzip

. download pretrain VGG19 from https://drive.google.com/file/d/1j0jDENjdwxCDb36meP6-u5xDBzmKBOjJ/view?usp=sharing[here] and unzip, then put it to models/vgg19.npy

|Train

2+a|

• change configs/animegan_pretrain.yaml menu:dataset[root] to your path

• change configs/animeganv2.yaml menu:dataset[root] to your path

• pre-training generator

[source,bash]

make train CODE=scripts/animegan_pretrain.py CFG=configs/animegan_pretrain.yaml

• training generator (use kbd:[Ctrl+c] can stop)

[source,bash]

make train CODE=scripts/animeganv2.py CFG=configs/animeganv2.yaml

• check progress

[source,bash]

make tensorboard LOGDIR=logs/animeganv2/

|test

2+a|

. you can download my pretrained model from https://drive.google.com/drive/folders/1Bu5yIYBPGBlO4yNzUamhWdWs5o5gT1Rx?usp=sharing[here].

. run test command

[source,bash]

make infer CODE=scripts/animeganv2.py
CKPT=logs/animeganv2/version_0/checkpoints/epoch=17.ckpt
EXTRA=image_path:asset/animegan_test2.jpg # <1>

make infer CODE=scripts/animeganv2.py
CKPT=logs/animeganv2/version_0/checkpoints/epoch=17.ckpt
EXTRA=help # <2>

<1> The EXTRA paramter can pass image path or image directory path <2> Read help.

.2+| Result

a| image::animegan_test2.jpg[animegantest,pdfwidth=50%,scaledwidth=50%]

a| image::animegan_test2_out.jpg[animegantestout,pdfwidth=50%,scaledwidth=50%]

2+^.^a| image::anime_video.png[animevideo,624,432,pdfwidth=50%,scaledwidth=50%,link="https://www.bilibili.com/video/BV1gy4y1z7o2"]

|===

[NOTE]

. If you want to change style, you have run scripts/animegan_datamean.py get dataset mean difference, then change configs/animeganv2.yaml menu:dataset[data_mean].

=== https://github.com/SystemErrorWang/White-box-Cartoonization[Learning to Cartoonize Using White-box Cartoon Representations]

[cols="^.^5,<.^50,<.^50"] |===

a| Setup

2+a|

. [yellow]#If you just want to use it, just skip the following step#

. download dataset from https://drive.google.com/file/d/10SGv_kbYhVLIC2hLlz2GBkHGAo0nec-3/view[here] and unzip

. download pretrain VGG19 from https://drive.google.com/file/d/1j0jDENjdwxCDb36meP6-u5xDBzmKBOjJ/view?usp=sharing[here] and unzip, then put it to models/vgg19.npy

|Train

2+a|

• change configs/whitebox_pretrain.yaml menu:dataset[root] to your path

• change configs/whitebox.yaml menu:dataset[root] to your path

• pre-training generator

[source,bash]

make train CODE=scripts/whiteboxgan_pretrain.py CFG=configs/whitebox_pretrain.yaml

• training generator (use kbd:[Ctrl+c] can stop)

[source,bash]

make train CODE=scripts/whiteboxgan.py CFG=configs/whitebox.yaml

• check progress

[source,bash]

make tensorboard LOGDIR=logs/whitebox

|test

2+a|

. you can download my pretrained model from https://drive.google.com/drive/folders/1Bu5yIYBPGBlO4yNzUamhWdWs5o5gT1Rx?usp=sharing[here]. you can use choice whitebox-v2.zip or whitebox.zip

. run test command

[source,bash]

make infer CODE=scripts/whiteboxgan.py
CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt
EXTRA=image_path:asset/whitebox_test.jpg # <1>

make infer CODE=scripts/whiteboxgan.py
CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt
EXTRA=image_path:tests/test.flv,device:cuda,batch_size:4 # <2>

ffmpeg -i xx.mp4 -vcodec libx265 -crf 28 xxx.mp4

make infer CODE=scripts/whiteboxgan.py
CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt
EXTRA=help # <3>

<1> The EXTRA paramter can pass image path or image directory path <2> Using GPU convert video, if GPU ran out of memory, pleas use cpu <3> Read help.

.2+| Result

a| image::whitebox_test.jpg[whiteboxtest,pdfwidth=50%,scaledwidth=50%]

a| image::whitebox_test_out.jpg[whiteboxtestout,pdfwidth=50%,scaledwidth=50%]

2+^.^a| image::whitebox_video.png[whitebox_video,pdfwidth=50%,scaledwidth=50%,link="https://www.bilibili.com/video/BV14V411a73o/"]

|===

[NOTE]

. The menu:model[superpixel_fn] has a great influence on the style. The slic,sscolor refer from offical code. defualt use sscolor. . Pytorch version and official version compare results in https://github.com/zhen8838/AnimeStylized/issues/2#issuecomment-726512435[here]

=== https://github.com/minivision-ai/photo2cartoon[Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation(minivision)]

[cols="^.^5,<.^50,<.^50"] |===

a| Setup

2+a|

. [yellow]#If you just want to use it, just skip the following step#

. download dataset from https://drive.google.com/file/d/10SGv_kbYhVLIC2hLlz2GBkHGAo0nec-3/view[here] and unzip

. download model_mobilefacenet.pth from https://drive.google.com/file/d/10SGv_kbYhVLIC2hLlz2GBkHGAo0nec-3/view[here] and put into ./models

|Train

2+a|

• change configs/uagtit.yaml menu:dataset[root] to your path

• training generator (use kbd:[Ctrl+c] can stop)

[source,bash]

make train CODE=scripts/uagtit.py CFG=configs/uagtit.yaml

• check progress

[source,bash]

make tensorboard LOGDIR=logs/uagtit

|test

2+a|

. you can download my pretrained model from https://drive.google.com/drive/folders/1Bu5yIYBPGBlO4yNzUamhWdWs5o5gT1Rx?usp=sharing[here]. (I have not enough GPU and time, so this model effect not so good)

. this model requires an input image that only contains the human head. Since I don’t have time to migrate the previous tools to PyTorch, need to rely on the previous library to process images.(you need clone https://github.com/zhen8838/playground[this repo])

[source,bash]

python tools/face_crop_and_mask.py
--data_path test/model_image
--save_path test/model_image_faces
--use_face_crop True
--use_face_algin False
--face_crop_ratio 1.3

. run test command

[source,bash]

make infer CODE=scripts/uagtit.py
CKPT=logs/uagtit/version_13/checkpoints/epoch=15.ckpt
EXTRA=image_path:asset/uagtit_test.png

.2+| Result

a| image::uagtit_test.png[uagtit_test,pdfwidth=50%,scaledwidth=50%]

a| image::uagtit_test_out.png[uagtit_testout,pdfwidth=50%,scaledwidth=50%]

|===

[NOTE]

. The menu:model[light] control the model version, the light version requires GPU memory > 8G,non-light version requires GPU memory > 22G.

. Maybe you need training more epoch to get better results.

== Repository structure

[%autowidth,cols="<.^,<.^"] |=== | Path | Description | AnimeStylized | Repository root folder | ├  asset | Folder containing readme image assets | ├  <<anchor-configs,configs>> a| Folder containing configs defining model/data paramters, training hyperparamters. | ├  <<anchor-datamodules,datamodules>> a| Folder with various dataset objects and transfroms. | ├  losses | Folder containing various loss functions for training, Only very general used loss functions are added here.

| ├ <<anchor-models,models>> a| Folder containing all the models and training objects | ├  optimizers | Folder with common used optimizers | ├  <<anchor-scripts,scripts>> a| Folder with running scripts for training and inference | ├  utils | Folder with various utility functions |===

[NOTE]

[[anchor-configs]]configs::

• Each algorithm has a corresponding config file.
• Config file uses the YAML format

[[anchor-datamodules]]datamodules::

• The dataset.py,dsfunction.py,dstransform.py contains common data module object's basic component
• Basically, each algorithm has a corresponding xxxds.py

[[anchor-models]]models::

• Basically, each algorithm has a corresponding xxxnet.py
• Now, only have the gan architecture model, in future maybe add more.

[[anchor-scripts]]scripts::

• Each algorithm has a corresponding xxx.py, implement the main training step and inference here
• Each algorithm must add run_common(xxxModule, xxxDataModule) in main function, then you can use general trainer to training or testing

====

== Participation and Contribution

. Add custom LightningDataModule object as xxxds.py in datamodules dir. . Add custom Module object model architecture as xxxnet.py in networks dir. . Add custom LightningDataModule training script as xxx.py in scripts dir . Add config file in configs dir, the paramters follow your custom LightningModule and LightningDataModule . trianing your algorithm