Goethe’s Puppet Theatre: AI-powered puppet plays inspired by Goethe
A mixed reality puppet theatre where users create plays featuring characters from Goethe’s works using AI-driven storytelling
A project by Aaron Höhn, Oscar Pohl and Phillip Premper Owor
Supervised by Prof. Dr Frank Gabler and Prof. Philip Hausmeier
Project completed in Semester 5 | Winter 2024/2025
About the Project
Goethe’s MR Puppet Theatre is a mixed reality installation that allows users to create unique puppet plays by selecting historical and fictional characters from the world of Johann Wolfgang von Goethe. Designed for museum visitors and developed for Meta Quest 3, the experience centres on a virtual puppet theatre that users can physically place in their environment using the Quest controller. After placing two puppets in the theatre, a unique play is generated by AI, using pre-defined prompts that shape the story elements, emotional tone and dialogue.
The project features characters such as Goethe, Charlotte and Mephisto, each with motion-captured animations and character-specific actions. Once the puppets are in place, the theatre curtains close and a short, approximately 2-minute play begins, featuring interactions generated entirely in real time using ChatGPT.
Technically, the system uses NavMesh for puppet movement and the Meta Voice SDK for real-time text-to-speech synthesis. Characters are treated as unique identities, even if duplicates are selected, which enables the AI to deliver emotionally varied interactions and layered storytelling. Over 80 motion-captured animations were integrated to bring different characters to life and enhance visual realism.
The experience was developed with the support of the Klassik Stiftung Weimar and incorporates physical measurements and references from the original puppet theatre. By combining storytelling, AI and spatial computing, the project offers audiences a novel way to explore Goethe’s legacy through interaction and entertainment.
Project Gallery
