- Kaolin is a new comprehensive PyTorch library that moves 3D models into the realm of neural networks.
- It is packed with many graphics components, including rendering, shading, and lightning.
- It cuts down the task for creating a 3D model for deep learning from 300 lines to code to just 5.
3D deep learning is receiving a lot of attention these days because of its high relevance in complex tasks such as augmented reality, self-driving vehicles, and robotics. The advent of powerful graphics cards and deep learning methods have enabled the analysis of large, previously intractable 3D data.
However, performing experiments in this field is an extremely difficult task because there is no standard system to help researchers easily load common 3D datasets, and train and evaluate machine learning architectures.
This is why researchers at NVIDIA have developed a tool — Kaolin — that moves 3D models into the realm of neural networks. It’s a PyTorch library for accelerating deep learning research.
The named Kaolin comes from Kaolinite, a form of plasticine (clay) that is sometimes used to sculpt 3D models.
A Valuable Tool For Both Beginners and Experienced Developers
By implementing this comprehensive library, users can cut down the task of creating a 3D model for deep learning from 300 lines of code to just 5.
Kaolin offers features to load and preprocesses various 3D datasets and local functions to control different representations like pointclouds, voxel grids, and meshes. It also diminishes the use of unnecessary boilerplate code.
An illustration of 5 lines of code required to train and test a PointNet++ classifier for airplane vs car
The library is packed with a variety of graphics components, ranging from rendering and shading to view warping. It supports a set of evaluation metrics and loss functions for seamless evaluations. And there’s a visualization module for rendering 3D outcomes.
Kaolin can support almost all common 3D tasks, including
- Basic graphic utilities (shading, lighting, etc.)
- Differentiable rendering
- Single-image based mesh reconstruction
- Mesh classification and segmentation
- 3D superresolution on voxel grids
Reference: arXiv:1911.05063 | GitHub
Real-World Applications
Classification: Usually, the first step in the complicated processes is to identify objects in a 3D scene.
3D part segmentation: Kaolin makes it easy to customize a 3D model or rig a character for animation, by automatically identifying different parts of the model.
Image to 3D: It can also build a 3D model from the product image detected by a trained neural network.
Read: NVIDIA Develops AI That Creates Interactive Graphics From Real World
Researchers have released source code and some pre-trained models for such tasks on standard benchmarks. They intend to foster a strong open-source community around Kaolin. Contributions from other researchers and practitioners are welcomed.
