arXiv 2026

MAGE: Modality-Agnostic Music Generation
and Target-Source Extraction

Muhammad Usama Saleem^1*, Ravi Tejasvi¹, Tianyu Xu¹, Rajeev Nongpiur¹, Ishan Chatterjee¹, Mayur Jagdishbhai Patel², Pu Wang²

¹Google ²University of North Carolina at Charlotte
^*Work done during a research internship at Google.

PDF arXiv BibTeX

MAGE — modality-agnostic music generation and target-source extraction

Figure 1: MAGE Conditioning Interface. MAGE supports conditional music generation and mixture-grounded target-source extraction through a shared interface over text, visual, and mixture inputs. Without a reference mixture, it generates music from text, visual input, or both; with an observed mixture and a target-identifying condition, it recovers the specified source.

MAGE system overview — architecture and pipeline

Figure 2: System Overview. High-level view of the MAGE pipeline across its two operating modes — mixture-free conditional music generation (left) and mixture-grounded target-source extraction (right) — both using the same model parameters and flow-based inference procedure.

Abstract

Recent advances in multimodal audio generation have enabled music synthesis from text, visual cues, and other high-level conditions. However, most systems are designed for a single operating mode: either generating music without a reference mixture or extracting a target source from an existing mixture. This fixed-task design limits their use when different combinations of text, visual, and mixture inputs are available.

To address this gap, we propose MAGE, a modality-agnostic framework for conditional music generation and mixture-grounded target-source extraction within a shared continuous latent space. Our approach introduces three key components: (1) a Controlled Multimodal FluxFormer that models the conditional flow from noise to a target audio latent, enabling the same backbone to operate with or without a mixture condition; (2) Audio–Visual Nexus Alignment (AVNA) that maps frame-level visual features onto the audio latent sequence, allowing visual evidence to condition the generation process at the audio-token level; and (3) a cross-gated modulation mechanism that uses the aligned visual representation to regulate intermediate audio features, while text provides separate semantic guidance.

We further train MAGE with dynamic modality masking, exposing the same model to text-only, visual-only, joint text–visual, mixture-conditioned, and unconditional configurations. Experiments on the MUSIC benchmark show that MAGE provides a shared conditioning interface across both settings, and that the proposed alignment and gating components improve interference suppression in the extraction task.

Music Generation Source Separation Multimodal Learning Flow Matching Audio-Visual Alignment Target-Source Extraction

Key Contributions

1 We propose MAGE, a modality-agnostic framework that treats conditional music generation and mixture-grounded target-source extraction as two operating modes of a shared continuous latent model.
2 We introduce the Controlled Multimodal FluxFormer, a flow-matching Transformer that predicts target audio latents under supported combinations of text, visual, and mixture conditions using the same model parameters and inference procedure.
3 We propose Audio–Visual Nexus Alignment (AVNA) and cross-gated visual modulation to map frame-level visual features onto the audio latent sequence, enabling temporally-aligned visual conditioning of the audio-token stream.
4 We train with dynamic modality masking to support text-only, visual-only, joint text–visual, mixture-conditioned, and unconditional configurations, and conduct separate evaluations for generation and extraction on MUSIC.

Method

      MAGE represents audio using a learned continuous codec and uses a Controlled Multimodal FluxFormer
      to model a conditional flow from noise to the target audio latent. When no mixture is provided, the model
      generates music from text, visual input, or both. When a mixture is available, its encoded latent is supplied
      as an acoustic condition, while text or visual input specifies the source to be recovered.
    

Figure 1: MAGE Framework. The shared conditional latent model handles both mixture-free generation and mixture-grounded target-source extraction through a unified conditioning interface.

Controlled Multimodal FluxFormer architecture

Figure 2: Controlled Multimodal FluxFormer. Flow-matching Transformer backbone that predicts target audio latents under all supported conditioning combinations.

Figure 3: Audio–Visual Nexus Alignment (AVNA). Maps frame-level visual features onto the audio latent sequence for token-level visual conditioning.

Figure 4: Cross-Gated Visual Modulation. The aligned visual representation regulates intermediate audio features via cross-gated attention, while text provides a separate semantic pathway.

Figure 5: Spectrogram Analysis. Qualitative spectrogram comparisons showing MAGE's ability to generate and separate music sources under different modality conditions.

Results

Figure 6: State-of-the-Art Comparison. MAGE evaluated on the MUSIC benchmark under separate protocols for mixture-free generation and mixture-grounded target-source extraction. Compared against specialized demixing baselines and promptable extraction systems.

Dedicated separation systems are treated as valid task-specific baselines, while MAGE evaluates how a shared conditional model performs across both mixture-free generation and mixture-grounded extraction. The AVNA and cross-gated modulation components are shown to improve interference suppression in ablations.

BibTeX