MuseGAN: Multi-Track Music Generation

• The paper introduces MuseGAN, a GAN-based framework that generates symbolic multi-track music by modeling complex intra- and inter-track dependencies.
• MuseGAN employs three architectures—Jamming, Composer, and Hybrid—to effectively capture the unique temporal structures and multiple instrument interactions in polyphonic music.
• Evaluation metrics and user studies confirm MuseGAN's ability to produce stylistically coherent compositions while maintaining privacy against membership inference attacks.

MuseGAN is a family of generative adversarial network (GAN) architectures designed for symbolic, multi-track music generation in the piano-roll domain. MuseGAN addresses the unique challenges of polyphonic music generation, including the modeling of temporal structures at multiple scales, intra- and inter-track dependencies, and the requirement to produce stylistically coherent multi-instrument compositions without explicit sequential or note-level orderings (Dong et al., 2017).

1. Motivation and Challenges in Multi-Track Music Generation

MuseGAN targets the problem domain where music is represented as symbolic, multi-track piano-rolls. Unlike image generation, symbolic music generation must contend with latent temporal structures, polyphony within tracks, and complex coordination across multiple instruments. Previous approaches that relied on hand-crafted rules or single-track generation typically failed to model the intricate joint distribution governing multi-instrument polyphonic music. Existing sequence models, such as those typical in NLP, are not naturally suited to modeling the concurrency (polyphony) inherent in music, since a strict note ordering is absent in typical multi-instrumental music scenarios (Dong et al., 2017).

2. Data Representation and Interdependency Models

MuseGAN represents a music piece as a tensor $x \in \{0,1\}^{T \times R \times S \times M}$, where %%%%1%%%% is the number of bars, $R$ the time steps per bar (96 in standard settings), $S$ the pitch classes (e.g., 84 for approximately 7.5 octaves), and $M$ the number of tracks (commonly bass, drums, guitar, piano, strings). Each bar is typically $x^{(t)} \in \{0,1\}^{R \times S \times M}$.

MuseGAN proposes three generator/discriminator architectures to capture interdependency among tracks:

Model	Generator Structure	Discriminator Structure
Jamming	$M$ independent $G_i$, one per track, each with its own $z_i$	$M$ independent $D_i$
Composer	Single joint $G : z \to$ $M$-channel piano-roll	Single $D$ over all tracks
Hybrid	$M$ track-specific $G_{bar,i}$ with shared inter-track $z$ and per-track intra-track $z_i$	Joint $D$ over all tracks

In the jamming model, each track’s generation is independent, with generators/discriminators specialized per instrument. The composer model uses a global latent vector to produce all tracks jointly, enforcing cross-track coordination. The hybrid variant combines per-track flexibility with shared semantics (Dong et al., 2017).

3. Temporal Structure and Conditional Generation

MuseGAN treats each bar as a unit and generates temporal sequences bar-by-bar using a composition of temporal and bar-level generators. For from-scratch generation,

$$G(z) = \{ G_{bar}(G_{temp}(z)^{(t)}) \}_{t=1}^{T} ,$$

where $G_{temp}$ is implemented via 1D transposed convolutions acting along bars, and $G_{bar}$ produces an $M$-track piano-roll for each bar.

MuseGAN also supports track-conditional (accompaniment) generation. Given a human-composed track, an encoder $E$ maps per-bar inputs to low-dimensional embeddings, and a conditional bar generator $G^\circ_{bar}$ produces the remaining tracks as a function of both the latent input and the encoded human track, formalized as $$G^\circ(z, y) = \{ G^\circ_{bar}(z^{(t)}_{temp}, E(y^{(t)})) \}_{t=1}^T$$ (Dong et al., 2017).

4. Network Architecture and Training Objectives

The generator typically starts by projecting a 128-dimensional latent vector via an FC layer and reshaping it into a dense tensor, followed by a sequence of transposed convolutional (upsampling) blocks. Each upsampling stage consists of ConvTranspose2D (kernel = $5 \times 5$, stride = 2, padding = 2, output padding = 1), BatchNorm2D, and ReLU activation, progressing through channel sizes (e.g., $512 \to 256 \to 128 \to 64 \to 5$ channels, with the latter corresponding to tracks) (Chow et al., 25 Dec 2025). The output is passed through a sigmoid to yield binary pianoroll activations.

The discriminator processes the $M$-track pianoroll with four downsampling blocks (Conv2d, BN, LeakyReLU(0.2)) and flattens the result to yield a scalar through a final sigmoid activation. During training, the standard GAN minimax objective is used:

$$\min_G \max_D V(D,G) = \mathbb{E}_{x \sim p_{data}} [\log D(x)] + \mathbb{E}_{z \sim p_z} [\log(1 - D(G(z)))].$$

MuseGAN architectures may also be trained using Wasserstein GAN with gradient penalty (WGAN-GP), where the critic loss is:

$$\mathcal{L}_D = \mathbb{E}_{x \sim p_d} [D(x)] - \mathbb{E}_{z \sim p_z} [D(G(z))] + \lambda \mathbb{E}_{\hat{x} \sim p_{\hat{x}}} ( \|\nabla_{\hat{x}} D(\hat{x})\|_2 - 1 )^2$$

with $\lambda = 10$. Generator and encoder updates depend on the variant (from-scratch or track-conditional) (Dong et al., 2017, Chow et al., 25 Dec 2025).

5. Evaluation Metrics and Empirical Results

MuseGAN introduces a suite of intra- and inter-track metrics:

• EB (Empty Bars %): Frequency of bars with no note events.
• UPC (Used Pitch Classes): Number of pitch classes used per bar.
• QN (Qualified Notes %): Percentage of notes lasting at least 3 time steps, to penalize artifacts due to binarization.
• DP (Drum Pattern %): Measure of conformity to canonical drum rhythms.
• TD (Tonal Distance): A cross-track harmonic measure derived from tonal centroid distances.

In a large-scale user study (144 listeners, including 44 professional musicians), the hybrid and composer models scored higher for harmony and coherence on from-scratch phrase generation. In track-conditional tasks, hybrid models were preferred by professional musicians, while jamming was sometimes preferred by non-professionals, likely for stylistic flexibility (Dong et al., 2017).

Quantitatively, the jamming model best matched EB, UPC, and QN of the training data, whereas the composer and hybrid models achieved closer tonal distances (1.0–1.6) to real data. All models produced high DP, indicating successful learning of drum patterns (Dong et al., 2017).

6. Privacy Properties and Membership Inference Attacks

Research on membership inference attacks (MIA) applied to MuseGAN revealed high resilience compared to image-based GANs. Both white-box (LOGAN) and black-box Monte Carlo (MC) attacks, when targeting the MuseGAN Composer architecture on symbolic multi-track music (Lakh Pianoroll Dataset), failed to outperform random guessing (success rates near 50%). Even under deliberate overfitting (tiny training sets and prolonged epochs), no significant leakage was observed. This suggests that in high-dimensional, sparse, and structured piano-roll spaces, traditional MIA techniques, which exploit overfitting and Euclidean closeness, are largely ineffective. Discriminators in symbolic music domains do not strongly memorize individual bars, and music’s repetitive, variable temporal structure aids in privacy preservation (Chow et al., 25 Dec 2025).

7. Limitations, Extensions, and Broader Impact

MuseGAN currently generates fixed-length (typically 4-bar) phrases and exhibits binarization-induced note fragmentation. It does not model long-range song-level dependencies. Future directions include hierarchical or recurrent modeling for extended musical form, the use of note-event decoders for improved articulation, and explicit key/chord conditioning for controllable harmonic output (Dong et al., 2017). Practical applications include human-AI collaborative composition, automated accompaniment, and style-transfer within multi-instrument symbolic domains. Privacy analysis indicates minimal per-sample leakage, although further research is needed regarding audio-domain models and the impact of advanced attacks (Chow et al., 25 Dec 2025).