Social Presence with Codec Avatars
DescriptionOur course is devoted to telepresence: the tasks of (a) generating realistic human representations with a “decoder”, and (b) driving (animating with your body) them with an “encoder”.
Regarding generation, we will cover the process of converting a set of high quality captures into realistic digital human bodies, from data collection and processing to model learning and rendering.
We will also cover the often overlooked part of _driving_ those models from a VR headset. This process includes capturing and processing images from headset cameras, generating synthetic data that links those incomplete images to the complete state of the person, and training a model that extracts the driving signal required to drive the decoder.
A key limiting factor to the development of telepresence experiences is a large corpus of high-quality human data to feed learning algorithms, which is particularly lacking in academia. In this regard, we will present two new datasets: Avatar-256, composed of millions of calibrated images of 256 subjects captured in a high resolution dome for which, besides, every subject has also been captured from a head-mounted display, allowing researchers in academia to build their own encoders and decoders for telepresence for the first time. The second dataset, Goliath-4, features captures of the same subjects under different conditions (e.g. casual and tight-fitted clothing) captured with different devices (relightable low volume light stage, large volume photometric scanner, smartphone, and head-mounted devices), enabling research into full-body telepresence. We will also present open implementations of baseline, yet state-of-the-art algorithms that the audience can use to start working on these challenging problems. We believe that an event in which the science and data related to realistic telepresence are released to the public will help foster a vibrant research community beyond the few companies that can work on this problem at the moment.
Regarding generation, we will cover the process of converting a set of high quality captures into realistic digital human bodies, from data collection and processing to model learning and rendering.
We will also cover the often overlooked part of _driving_ those models from a VR headset. This process includes capturing and processing images from headset cameras, generating synthetic data that links those incomplete images to the complete state of the person, and training a model that extracts the driving signal required to drive the decoder.
A key limiting factor to the development of telepresence experiences is a large corpus of high-quality human data to feed learning algorithms, which is particularly lacking in academia. In this regard, we will present two new datasets: Avatar-256, composed of millions of calibrated images of 256 subjects captured in a high resolution dome for which, besides, every subject has also been captured from a head-mounted display, allowing researchers in academia to build their own encoders and decoders for telepresence for the first time. The second dataset, Goliath-4, features captures of the same subjects under different conditions (e.g. casual and tight-fitted clothing) captured with different devices (relightable low volume light stage, large volume photometric scanner, smartphone, and head-mounted devices), enabling research into full-body telepresence. We will also present open implementations of baseline, yet state-of-the-art algorithms that the audience can use to start working on these challenging problems. We believe that an event in which the science and data related to realistic telepresence are released to the public will help foster a vibrant research community beyond the few companies that can work on this problem at the moment.
Coordinator
Lecturer
Event Type
Courses
TimeThursday, 5 December 202411:00am - 12:45pm JST
LocationG602, G Block, Level 6
