Transformirror

One starting point for the project was a December 2022 conversation with David Holz of Midjourney. At the time, diffusion models such as Stable Diffusion were rapidly becoming central to high-quality text-to-image generation. Latent Diffusion Models had shown how operating in latent space could reduce the computational cost of high-resolution generation, and the public release of Stable Diffusion in August 2022 made text-to-image generation broadly accessible.

Yet at the end of 2022, diffusion models were still primarily used to generate single images. Capturing live camera input, transforming it with a diffusion model, and returning it as stable continuous video remained a major technical challenge.

The question was simple but fundamental: when would it become possible to transform camera input into another image in real time with a diffusion model? Could a system generate continuously, around 20fps, while responding to the movement of a viewer?

Holz predicted that this kind of real-time diffusion-based transformation could become feasible during 2023. By late 2023, fast inference, few-step generation, and real-time image-to-image optimization had advanced enough that the idea could be implemented as an interactive installation.

The question was not only about generation speed. A central issue was temporal consistency. Diffusion models can produce a strong single image, but when live camera frames are generated independently, contours, faces, clothing, backgrounds, textures, colors, and small details can fluctuate from frame to frame. The result can feel like an unstable sequence of still images rather than video.

In an interactive installation, that instability matters. Viewers expect their bodily movement to return through the screen. If the output flickers heavily or the body collapses, it becomes difficult to feel the connection between bodily action and AI response.

The second challenge was speed and latency. Camera capture, image analysis, diffusion generation, post-processing, and LED output all needed to run with sufficiently low latency. The goal was not to put a filter on camera video, but to build a system where the viewer's body, movement, and space are reinterpreted by a diffusion model in real time.

To position Transformirror historically, it is useful to begin with Myron Krueger. From the 1970s onward, Krueger developed the idea of the responsive environment: an environment that senses human behavior and returns audiovisual feedback.

In Videoplace, the participant's body is captured as an image and interacts inside a computer-generated environment. For Krueger, the camera was not merely a recording device. It was an input device for reading the body, a sensing organ through which the environment could respond to presence and movement.

From the late 1990s through the 2000s, works such as Text Rain, Messa di Voce, and Reface [Portrait Sequencer] transformed silhouettes, voices, and faces into live visual experiences. Before Kinect, many such systems relied on background subtraction, blob tracking, and optical flow; they often saw moving pixels rather than a structured body.

Kinect changed this trajectory in 2010 by making depth sensing and skeletal structure available as a practical creative material. Transformirror is not a Kinect-style skeleton-tracking work, but it inherits the shift toward treating the body as structured input that can be read, responded to, and returned as another visual environment.

In the mid-2010s, deep learning rapidly changed image generation and image transformation. GANs introduced a powerful framework for learning data distributions through competition between generator and discriminator. DeepDream showed that neural networks could become tools for visual expression by amplifying internal features.

Neural style transfer separated content and style as feature representations and recombined them, but early methods were not practical for high-frame-rate live camera input. Feed-forward approaches to real-time style transfer shifted the field toward faster image transformation.

pix2pix, CycleGAN, and vid2vid expanded image-to-image and video-to-video synthesis. In particular, vid2vid made temporal stability a central problem, directly anticipating the temporal consistency issues later faced by real-time diffusion systems.

Memo Akten's Learning to See is a crucial precedent. It used live camera input and neural networks to reinterpret objects on a table as other visual worlds, showing that AI does not simply see reality; it rereads it through training data, model structure, and learned associations.

The significance of Transformirror extends beyond the work itself. It was also an early demonstration that real-time diffusion models could expand into interactive image production, live performance, virtual production, broadcasting, exhibitions, education, and public-space experiences.

At the beginning, realizing such a system required a complex implementation. Camera capture, GPU inference, model optimization, parallel processing, video output, LED control, prompt management, and sound generation all needed to be integrated with low latency. AI coding assistance was beginning to appear, but building a real-time system stable enough for exhibition still required deep knowledge of low-level video processing, GPU processing, and pipeline design.

Since 2024, these techniques have become more accessible. Real-time image-to-image pipelines, visual development environments such as TouchDesigner, and optimization libraries for generative AI point toward a future where systems once limited to specialists become available to more artists and designers.

Future development raises ethical questions. When a system captures viewers' bodies or faces and transforms them with a generative model, consent, privacy, data storage, likeness, model bias, and representation need to be handled carefully.

AI does not see the world as it is. It reinterprets the world through training data, model structure, and given conditions. Transformirror is not a mirror that simply reflects the viewer, but a mirror that regenerates the viewer through the visual world learned by a model.

Transformirror belongs to a long lineage of camera-input interactive art. Krueger imagined environments responsive to bodily movement; later works transformed silhouettes, faces, voices, and bodies into live visual forms; machine-learning works then expanded image and video transformation through DeepDream, style transfer, GANs, pix2pix, CycleGAN, vid2vid, and Learning to See.

In the early 2020s, diffusion models became central to image generation. But there was still a large gap between generating a single image and transforming live camera input in real time. Closing that gap required temporal consistency, low latency, fast inference, pipeline optimization, and the design of bodily interaction.

Transformirror sits at that turning point. It is both an AI image-generating artwork and an interactive visual environment that connects camera, body, diffusion model, LED display, sound, prompts, and real-time processing.