The Agent Patch

Prototype for Multi Agent Modular Control

by sonicLAB — Daito Manabe × Sinan Bokesoy · 2026

Overview

In this software, multiple AI agents listen to a piece of music, analyze its structure and progression, and reconstruct and remix it. The agents operate the parameters of a modular synth plugin, or the CV parameters of an actual physical modular synthesizer.

The current prototype is a headless application: given a 2-track source, stems, or an Ableton Live .als file, it analyzes the material, holds discussions, generates effect patterns, and renders audio — all through offline rendering.

Beyond the audio itself, the system also generates visualizations of the agents' conversations, the modular synth patching and effects, and — as a further derivative — VJ visuals.

Virtual modular rack, full view — 62 DSP modules and patching operated by the agents

Modular system — the rack in operation

Three Projections

Every output is generated from a single source of truth: measured 60fps event data (JSON). Audio, the modular visualization, the agents' discussion, and the VJ visuals stay synchronized with frame accuracy.

01 — MODULAR

Modular Rack

The actual signal flow of the effects applied by the agents, visualized as a virtual modular rack. Patch cables, knobs, LEDs and automation lanes are driven by measured values.

02 — CONVERSATION

Agent Conversation

Six roles — ANALYST / SOUND DESIGNER / ARRANGER / HYPE / CRITIC / SPATIAL — discuss the track as they listen, visualized as a node graph.

03 — VJ VISUAL

VJ Visual

VJ visuals derived from the analysis data and events, rendered offline in perfect sync with the remix's progression and effect triggers.

Agent Control Room node graph — Agent Control Room — a node graph of six agents discussing the track

Agent conversation — the agents discussing a track

VJ visual synced to the remix — VJ visual — wireframe scenography synced to the remix

Technical Details

The pipeline is fully deterministic: the same input always reproduces the same audio and visuals. Stem separation (Demucs), musical analysis, multi-agent LLM discussion generation, gesture remixing through 62 DSP processors, spatial processing, a SuperCollider-synthesized SE layer, and the rendering of every projection all run in a single shot.

INPUT (2tr / stems / .als) │ stem separation · structure & key analysis ▼ AGENT DISCUSSION (multi-agent LLM, 6 roles) │ effect pattern generation ▼ OFFLINE RENDER — 48kHz/24bit · 60fps event JSON ├─ audio remix + SC-synth SE layer │ └─ multichannel spatial (stereo / quad / 5.0 / hexa / octa ring) ├─ modular rack visual ├─ agent conversation visual └─ VJ visual

The spatial layer also supports multichannel output. The measured azimuth and width curves recorded in the event JSON are re-projected onto arbitrary speaker rings (quad / 5.0 / hexa / octa / custom) via VBAP, while everything below 150Hz is always distributed equally to all speakers to anchor the rhythmic foundation. The same curves double as synchronization channels for camera motion and lighting.

Spatial section detail — SPATIAL SECTION — modules controlling the rotation and distortion of the sound field. Azimuth maps 1:1 to the camera in the visuals

What's Next

We will extend the system toward real-time performance and interactive production workflows, including live operation in which the agents control a physical modular synthesizer via CV.

We are also exploring the development of multiple hardware modular synthesizers, and we welcome collaborations with anyone — individuals or companies — interested in this direction.

AI Reference Surface

Related reference pages

Open the FAQ, glossary, authority, measurement, and AI index pages. Each link now states what it is for.

FAQsite questions Glossaryterms Authoritysources Measurementmetrics llms.txtAI index