Toonify yourself

Toonify uses a neural network to turn your uploaded photo into a unique cartoon.

We start with the classic StyleGAN model which is trained on photos of people’s faces. This was released with the StyleGAN2 code and paper and produces pretty fantastically high-quality results.

Not everyone has multiple-GPUs and weeks of time to train a model so a shortcut lots of people is called transfer learning, where they take this “pre-trained model” and then train it on some new data. This gives good results really quickly, and even more so if the new dataset is some also of faces.

Once the model is trained just a little bit it gives outputs that look like below

The output is OK for such a small amount of training on a small dataset, it’s clearly got the big eyes thing down pretty well. The problem is that the style of images in the dataset is a bit of a mish-mash, some are CG some are hand-drawn, and lots are quite low resolution. The model tries to replicate all these things and comes off worse for it.

Blend the models

As this model was fine-tuned from the original faces model, we can perform a trick where we directly swap parts of the models around.
This gets interesting because, due to the structure of StyleGAN, different layers in the model affect the appearance in different ways.
So low-resolution layers affect the pose of the head and shape of the face, while high-resolution layers control things like lighting and texture.
To take the high-resolution layers from the original model, and the low resolution from his fine-tuned cartoon model. We end up with a hybrid that has the structure of a cartoon face, but photorealistic rendering!

If we generate images using the original faces model and the blended cartoon model, you can see how there is a clear relationship between the two, the identities appear to be the same, but the features have been shifted to give them a cartoon look.

Now Toonify yourself

These StyleGAN face models can produce a huge diversity of faces and it’s actually possible to find basically any face inside the model.
It’s actually a straight forward process to search for any image of a face in the model.

So Let's do code first -

Code

Step 1: Please ensure that you're using a GPU runtime. First, we do some setup -

We clone this repository into stylegan2

Step 2: Now upload your photos to raw/. These don't need to be aligned as we'll use a face detector to grab all the faces and transform them into the correct format. One note of caution is that you'll need a pretty high-resolution picture of a face to get a sharp result (the final face crop is resized to 1024x1024 pixels)

We'll grab an example image from the internet to work with.

Step 3: We import the pre-trained networks and use the blended model

Step 4: Then we aligned the images

Step 5: We import the necessary libraries.

Then we use the following file to convert the image and also using blended model which makes it into a Toonify image

Step 6: At last we display the image and then it gives the toonify image.

Let's see the result -

More examples -

Conclusion

All things considered, the Toonify algorithm is a success. It is capable of producing exactly the effect specified above, and a wide range of input images will yield satisfactory results.

Source Code: Toonify Colab

All Image Source: Google Images

All the Rights are preserved by ArtificialGen

Toonify yourself

Recent Posts

Comments