Narratron: Shadow play with a storytelling AI

Shadow play as a transcultural way of storytelling has been practiced and inherited in generations. It not only expresses human imagination through bodily interactions, but stimulates creativity and boosts early-childhood education. Inspired by this tradition, Narratron introduces the participation of large language model in shadow play storytelling process. It is designed to push the boundary of human-AI interaction by expanding the story co-creation experience to multi-sensory immersion.

Starting with a serene startup screen, users receive instructions before exploring and shaping hand shadows freely. The camera captures intricate shapes, analyzed by image classifiers to derive animal keywords. GPT 3.5 model then processes these keywords to generate a rich story with plotlines, dialogues, and descriptive elements. Simultaneously, a custom Stable Diffusion model was trained to create a corresponding image projected onto the surface, adding a visual component to the audio experience, enhancing the story-making experience.

Players progress through the story chapters by spinning the knob, resembling vintage movie projectors that adds a nostalgic touch. This tactile engagement provides a sense of agency, allowing users to dictate the pace and flow of the storytelling journey. The fusion of traditional and modern elements in Narratron offers a captivating and personalized storytelling adventure.