- Researchers develop a neural network that can produce full 3D hair geometry from 2D image.
- The network is capable of rendering up to 30,000 strands of hair in milliseconds.
- It can smoothly sample and interpolate variety of hairstyles, including wavy, curly and straight.
Rendering realistic hairs on machines is probably the toughest task when digitizing virtual humans. Compared to other objects like eyes, nose and ears, hair involves a wide range of shape variations and could be extremely complex due to the level of deformability in each single strand and its volumetric structure.
It’s not like existing system cannot generate high-quality 3D hair model, but they typically require specific hardware and software setups, which aren’t easily deployable. Some modern techniques do utilize data-driven approaches, but they are not feasible for real-time applications and systems with limited memory space.
Now, researchers at Microsoft, Pinscreen and University of Southern California have built a deep-learning model that can produce full 3D hair geometry from 2D images in real-time.
Rather than synthesizing structures in the form of point clouds or volumetric grids, it creates the strand of hair directly. This method is more appropriate for non-manifold structures, and could provide higher details and accuracy.
HairNet
The neural network, what they call HairNet, is capable of continuously producing and representing 3D hair geometry. It can smoothly sample and interpolate variety of hairstyles, including wavy, curly and straight.
The neural network pipeline consists of 3 steps: preprocessing, generating hair strand and reconstruction.
HairNet contains a convolutional encoder for extracting high level features of hair (in the form of vector) from a 2D image, and a de-convolutional decoder for generating 32*32 strand-features evenly distributed on the scalp. These strand-features are then interpolated on the scalp space, which are finally represented as series of 3D points.
Network Architecture | Courtesy of researchers
In order to push the rendered hairstyles towards a more plausible space, researchers introduced ‘reconstruction loss’ and ‘collision loss’ between a body model and hair strands. They used the input image as a weight to modulate its loss, which further improved the accuracy.
Training
Researchers trained the neural network on a large dataset containing 40,000 hairstyles and 160,000 two-dimensional pictures from random viewpoints. The network then learned to regenerate 3D hair in different styles, colors and lengths, from a single 2D photo.
Rendering hair from a 2D image using HairNet | Courtesy of researchers
The AI was trained on Nvidia Titan Xp GPUs (graphical processing units) with PyTorch framework powered by CUDA deep neural network library.
Reference: arXiv:1806.07467
It can render up to 30,000 strands of hair in milliseconds. Moreover, it can mimic video and render each strand – all interacting with each other.
Future Work
The team concluded that their technique isn’t perfect yet. It cannot handle exotic hairstyles like Afro, kinky or buzz cuts. However, training the network on massive datasets containing more variations could solve this problem.
Read: AI Can Put Anyone In Any Pose | Synthesizing Human Images In Unseen Poses
The technique also fails when the hair is slightly occluded in the given image. Therefore, they plan to improve training datasets in the future by integrating more random occlusion.
