AI Motion Capture:
Accelerating the pipeline from 3D modeling to production using AI.

Background

PUSH Studio, in collaboration with DePaul’s Jarvis Innovation Center, embarked on a pioneering project aimed at democratizing technology for virtual production and motion capture. This initiative focuses on leveraging off-the-shelf hardware, software, and artificial intelligence (AI) to revolutionize how motion capture is conducted, opening this domain to interested students. By introducing an innovative approach that transforms single RGB video feeds into real-time 3D humanoid models, this project addresses the inherent limitations of traditional motion capture systems in a user-friendly manner. Traditional systems typically require elaborate setups and multiple cameras, which can be restrictive and expensive. This new method represents a significant advancement in the fields of animation and virtual reality, enabling more streamlined and cost-effective motion capture processes.

Challenge 

The main challenge tackled by the AI Motion Capture initiative was the complexity and inaccessibility of conventional motion capture technology, which often deters creative professionals in animation, game development, and virtual reality due to its high costs and technical demands. Traditional motion capture systems require sophisticated equipment setups, including numerous cameras and specialized software, which are often a major roadback for students and independent creators. To overcome these, the project utilized AI to interpret and translate human movements captured via a standard RGB camera into detailed 3D animations. This approach significantly reduces the barrier to entry for utilizing advanced motion capture technology by simplifying the equipment and processes required.

Deliverables:
High-fidelity interactive prototypes

PUSH services:
Technology Prototyping, Student Education

Industry:
Artificial Intelligence, Motion Capture, Virtual Production

PUSH team: 

  • Mit Dave

Example rigging on human body

Camera feed & project image

Approach

In addressing these challenges, the project combined a standard webcam with advanced AI technologies to capture human motions, which are then visualized in real-time using actual avatars. This integration allows for the immediate representation of human movements within virtual environments, making the process highly interactive and responsive. The pipeline capitalizes on existing AI algorithms that are adept at accurately capturing and translating movements from simple video feeds into intricate 3D animations. This innovative use of straightforward technology ensures that the motion capture process is not only more accessible but also more adaptable to various creative and educational contexts.

Solution

The result of this collaborative effort is a highly accessible AI motion capture system that simplifies the traditional complexities associated with capturing human motion. By utilizing a combination of standard webcams and AI, the system allows students and creators from diverse backgrounds to capture natural human motion effortlessly and in real-time. This technology opens up new possibilities for those involved in animation, game development, and virtual reality, providing them with a powerful tool to enhance their creative projects. Moreover, the ease of use and affordability of the system ensure that it can be integrated into educational settings, allowing students to explore and interact with cutting-edge technology firsthand, thereby fostering a deeper understanding and innovation in motion capture and virtual production.


Demo at the Jarvis Innovation Center’s XR Day 2024