Mary and The Monster
Lead Technical Artist
NYU | 2019 | Magic Leap One Headset
Mary and the Monster is a shared AR experience that brings to life “Frankenstein” through the eyes of a young Mary Shelley.
Mary and the Monster is an immersive AR/VR theater experience reimagining Frankenstein through the eyes of Mary Shelley. I was responsible for the graphics pipeline—including animation with OptiTrack and Faceware, keyframing, 3D modeling, rigging, skinning, texturing, and lighting—developed in Maya and Unity for Oculus Quest and Magic Leap. The work was showcased at SIGGRAPH 2019.
Capturing the Soul of Performance
For Mary and the Monster, we brought the cast’s performances to life through a full motion capture and facial tracking pipeline. Using OptiTrack for body movement and Faceware for facial expression capture, we recorded synchronized performances that preserved every nuance of emotion. I contributed to the setup and supervision of capture sessions, data cleanup, and retargeting the performances onto digital characters—ensuring the final animation remained authentic to the actors’ intent while technically optimized for real-time playback.
From Expression to Data
Facial animation was driven through Faceware, combining live head-mounted camera footage with real-time facial tracking. Each actor’s expressions were analyzed and converted into motion data that drove the character rigs with precision. Beyond simply recording faces, we refined the retargeting curves, smoothed transitions, and adjusted lip-sync for emotional accuracy. This process allowed the digital characters to mirror the performers’ intent—maintaining the smallest details of a glance or smile across every scene.
Building the Faces Behind the Performances
Each character began as a high-resolution digital sculpt, designed to capture the anatomical precision and emotional range needed for performance capture. Once the models were finalized, we built and retargeted an automatic facial rig compatible with Faceware, enabling seamless data transfer from captured footage to the digital face. This integration allowed facial expressions—from subtle frowns to intense emotion—to drive the rig automatically, maintaining consistency across takes while dramatically reducing manual keyframing.
Refining Performance Through Motion Data
After capturing the actors’ performances with OptiTrack, the raw motion data was imported into Maya for cleanup and retargeting. Each take was processed to correct marker noise, refine joint orientation, and ensure consistent alignment with the character rigs. Subtle keyframe adjustments were then added to enhance weight, timing, and emotional clarity before exporting the final animations to Unity, where they were integrated into the AR/VR experience.
Trailer — Mary and the Monster
This trailer captures the essence of Mary and the Monster, an immersive AR/VR performance that blends theater, motion capture, and virtual storytelling. Viewers step inside Mary Shelley’s imagination, where emotion, creation, and technology converge. The real-time characters and environments were brought to life through OptiTrack, Faceware, and Unity, delivering an experience that bridges live performance and digital presence.
