image: doubleA foundation

With “From Dust”, Dutch composer and director Michel van der Aa has developed an innovative fusion of opera, virtual reality (VR), and artificial intelligence (AI). Building on his experience with the VR installation „Eight“, he takes things a step further: the audience is no longer merely a spectator, but becomes the central figure in an individual, immersive narrative.

Each visitor begins with an onboarding process lasting approximately 20 minutes. During this phase, participants are asked questions about their own personality and encouraged to visualize themselves at different stages of their lives. Their responses are fed into an AI model—which operates locally, independent of any cloud services—that helps shape the virtual environment and the unique progression of the experience. This personalization is based, in part, on the five personality dimensions known as the „Big Five“ or „OCEAN“ model.

The performance itself lasts about 25 minutes. Unlike a traditional opera, it does not follow a strictly linear plot. Instead, the audience moves through poetic, dreamlike spaces that evoke various associations and emotional states. In terms of content, the work draws inspiration from texts by female poets and explores themes related to human personality.

Musically, the experience is anchored by the German vocal ensemble Sjaella from Leipzig. The six singers were chosen for their extraordinary sonic cohesion; their voices effectively form a unified musical identity. The encounter begins within a surreal point cloud, from which the singers gradually emerge. They become guides through the virtual world, establishing a symbolic and emotional connection with each individual visitor.

The interaction between the audience and the singers

The singers are by no means merely static figures within a virtual backdrop. Their body movements were captured using motion capture technology, while their faces are based on video footage. This combination allows for greater naturalism and emotional expressiveness than fully computer-generated characters, while simultaneously avoiding the typical „uncanny valley“ effect.

image: doubleA foundation

The direction of their gazes was also carefully planned for the production. During the video shoots, van der Aa stood in the exact positions where visitors would later be located. This creates highly natural-looking eye contact between the virtual singers and the participants. The bodies are animated in three dimensions, while the faces—derived from video—are integrated into the 3D models. To achieve this, the technical team developed a specialized process for incorporating both video and depth information into the virtual environment.

Interactivity is particularly evident in the spatial design and the freedom of movement. At the very beginning, for instance, visitors must choose whether to pass through a left or right portal, which can lead to different paths and locations. At certain points, visitors converge in shared spaces, while other sections are experienced individually; for example, they can approach and enter various glass domes or move between different areas.

These choices influence not only the spatial experience but also the musical composition. According to van der Aa, visitors hear different music depending on the path they take through the virtual world. The onboarding process also significantly impacts which locations visitors reach and how they perceive the singers.

One example of this is the question of whether one perceives oneself as belonging more to an orderly or a chaotic environment. This answer determines which of the distinctively designed glass domes the visitor encounters as the narrative unfolds. Other questions relate to how one interacts with others—for instance, whether one prefers to initiate a conversation or listen first. From this, among other things, a so-called „intimacy slider“ is derived, which determines how close the singers are permitted to get to the visitor. The size of the circle within which the singers move around the participant is also adjusted accordingly.

Often, it is subtle changes that personalize the experience. Not every visitor sees the same locations, hears the same music, or experiences the same spatial relationships. Consequently, a repeat visit can very well reveal entirely new experiences.

Designing the Virtual World and Technical Implementation

image: doubleA foundation

The visual design blends real and surreal elements. Particle effects play a significant symbolic role, representing life cycles and transitions; for instance, characters leave traces of their presence that either dissolve or evolve within the virtual environment.

Spatial dimensions also change dramatically: narrow, confined spaces contrast with monumental structures and towering edifices. A striking example is a tower that was originally conceived to feature an internal elevator; during development, the team opted for an external version, which creates a more profound sense of immersion and transition.

The technical implementation is built on the Unity game engine. A key challenge lay in coordinating the numerous scenes, animations, and interactive choices. Since multiple developers were working on different scenes simultaneously, the team utilized „prefabs“—modular assets that allowed for independent development without disrupting the main scene. At the same time, these various elements had to be synchronized with the production’s complex temporal dramaturgy.

Real-time 3D visualization and spatial audio contribute significantly to the sense of presence. The singers‘ voices appear to emanate precisely from their respective positions within the virtual world, forging an intense connection between music, space, and movement.

Testing with various visitor groups revealed that freedom of navigation presents its own challenges. While inexperienced VR users sometimes struggle with orientation, seasoned users often explore the environment quite intensely, pushing its boundaries. Consequently, the design must strike a balance: providing enough freedom for individual discovery while offering subtle guidance to ensure that key elements of the production are not overlooked or bypassed.

The future: Could AI agents serve as co-performers in a personalized opera?

In discussions about potential future developments, the question arises as to whether AI agents could significantly expand the personalization and interactivity of such productions. One could imagine, for instance, a virtual world in which AI-driven characters do not merely execute pre-programmed sequences, but instead respond autonomously to visitors—all while remaining within a dramaturgical framework established by the artist.

Van der Aa finds this prospect inherently compelling. He notes that in computer games, „non-player characters“ (NPCs) are becoming increasingly intelligent and are already capable of engaging in conversations with players. For artistic endeavors, this opens up the possibility of creating even more individualized experiences.

At the same time, he identifies a key challenge: where does the artist’s influence end, and where does the arbitrary nature of a randomly acting chatbot begin? For him, the distinct strength of an artistic work lies precisely in the control over its dramaturgy, content, and impact. If too much of this control is relinquished, there is a risk that while an interaction may occur, the work loses its coherent, independent artistic statement.

The vision of a fully personalized virtual world—in which AI agents or virtual performers shape the experience for an individual visitor—is reminiscent of the concept behind „The Truman Show“: an entire environment tailored to one person, while all other characters are part of a larger, overarching production. Van der Aa finds this idea fascinating, yet he also raises the question of how such a vision can be reconciled with artistic control.

A concrete step toward greater accessibility is already underway. Through a platform called „Here“, his team is developing a digital, immersive music theater experience designed for Apple Vision Pro and PC-based VR. The platform aims to make these experiences accessible from one’s own living room, thereby reaching a wider audience. In contrast, the current site-specific installation of „From Dust“ requires a space of approximately seven by seven meters, specialized technical equipment, and three staff members to oversee the experience. The new platform could significantly reduce spatial and organizational barriers to access.

Summary

„From Dust“ illustrates how the boundaries between opera, interactive installation, and digital storytelling are shifting. By combining VR, AI, music, and individual personality traits, it transforms the audience into an active component of an artistic production. What is particularly compelling is that this personalization is not merely a product of technical effects; rather, it is deeply embedded within the spatial, musical, and dramaturgical structure of the work.

Dates for upcoming performances of „From Dust“ will be published on Michel van der Aa’s website: