Controllable Generation Approaches

• Controllable generation approaches are techniques that allow precise manipulation of generative model outputs using explicit control signals.
• They employ methods such as auxiliary-conditioned models, plug-and-play guidance, and gradient-based adjustments to meet complex constraints.
• These techniques drive applications in text, image, video, and more, balancing fidelity, diversity, and efficiency while addressing scalability challenges.

Controllable generation approaches comprise a diverse array of model architectures, algorithms, and intervention mechanisms enabling precise, multi-faceted steering of generative models’ outputs according to explicit attributes, instructions, or constraints. They are foundational for applications requiring user-specified structure, style, semantics, or extrinsic compliance—including but not limited to text, image, video, motion, game level, time series, protein, and code generation. Methodological innovation in this domain is driven by the need for fine-grained, sample-efficient, and compositional control—balancing fidelity, diversity, efficiency, and reliability across increasingly complex modalities and task requirements.

1. Taxonomy of Controllable Generation Methods

Controllable generation constructions are typically classified by when, where, and how control is injected into the modeling and sampling pipeline. The principal families are:

• Auxiliary-Conditioned Modeling: Models such as Conditional Variational Autoencoders, Conditional Normalizing Flows, Conditional GANs, and conditional tailored Transformers learn $p_\theta(x|c)$, conditioning directly on structured control signals $c$ (labels, attributes, keywords, or property vectors). Extensions include architectural augmentations for explicit order (as in Apex) or multimodal branches for spatial/temporal/video control (Shao et al., 2021, Yao et al., 2024, Wang et al., 2022, Shen et al., 24 Nov 2025, Ma et al., 22 Jul 2025).
• Plug-and-Play/Residual Guidance: Modular, non-intrusive controllers (e.g., Residual Memory Transformer, CAIF, importance sampling plug-ins) act on top of a frozen generative backbone, introducing corrections at decoding via encoders, discriminators, or direct logit reweighting, often exploiting external classifiers or reward functions (Zhang et al., 2023, Sitdikov et al., 2022, Guo et al., 2024).
• Gradient- and MCMC-based Control: Fine-grained constraint satisfaction is achieved by modifying the sampling process in the model’s latent or score space—using gradients from attribute classifiers, energy models, or constraint-induced posteriors. Examples include classifier/energy-based guidance for diffusion, gradient-step updates at each reverse step (Diffusion-LM), or constraint-aware importance sampling for masked/discrete models (Li et al., 2022, Sitdikov et al., 2022, Guo et al., 2024, Li et al., 2022).
• Reinforcement Learning and Goal Conditioning: Policies are trained to maximize task- or user-specific reward functions, with control variables represented as part of the observation or reward structure. This is prevalent for sequential or structured generation, game/level synthesis, and procedural content (Earle et al., 2021, Wang et al., 2022).
• Instructional and Regex Stratification: Control is cast as an instruction or regular expression specifying required outputs—handled either by instruction-finetuned LLMs or by unifying multiple constraints into meta-instruction formats amenable to in-context learning or fine-tuning (Ashok et al., 2024, Zheng et al., 2023).
• Hybrid and Multi-Agent Systems: For settings such as code or multi-modal generation, collaborative agent-based or compositional architectures modularize planning, search/tool use, code generation, and validation, with explicit interfaces for user, tool, and safety control (Liu et al., 9 Oct 2025).

This taxonomy is not exhaustive but captures the mechanisms and loci of control that underpin the state of the art.

2. Key Methodological Mechanisms and Algorithms

Conditional Generation Models

• Conditional VAEs/Transformers: These models (Apex, FineXtrol) encode both control signals and structural information (e.g., ordered keywords or fine-grained per-body-part motion text) into learned embeddings, attended to or fused at each layer in the decoder (Shao et al., 2021, Shen et al., 24 Nov 2025).
• Plug-and-Play Modules: Residual Memory Transformer (RMT) and CAIF sample by adding a “control correction” to the backbone’s logits or hidden states, supporting any control expressible as an embedding or classifier probability. RMT leverages encoder-decoder attention over control signals at all time steps, whereas CAIF reweights the top candidate tokens proportionally to an external classifier's score raised to an exponent (Zhang et al., 2023, Sitdikov et al., 2022).
• Feedback/Control-Theoretic Loops: Apex introduces a PI-controller to actively stabilize the KL-divergence in CVAE objectives at a user-selected setpoint, dynamically manipulating the diversity-accuracy tradeoff and avoiding KL-vanishing. This approach achieves near-perfect control over both diversity and keyword ordering in generated sequences (Shao et al., 2021).
• Importance-Weighted Masked Modeling: For discrete masked models, plug-and-play approaches inject control by reweighting samples during iterative unmasking, requiring no fine-tuning or gradients. The method is agnostic to the form of control (reward, constraint, posterior) and can be deployed with pre-trained backbones (Guo et al., 2024).
• Diffusion and Autoregressive Conditioning: Modern diffusion approaches (RegDiff, Diffusion-LM, VFX Creator, AnimateAnything) offer flexible integration of conditioning via classifier guidance (CG), classifier-free guidance (CFG), regularization during training, or plug-in spatial/temporal modules. For autoregressives, control signals are fused at each latent scale, as in CAR, or via prefix/instruction tokens (Zhou et al., 7 Oct 2025, Li et al., 2022, Liu et al., 9 Feb 2025, Lei et al., 2024, Yao et al., 2024).

Training and Decoding Schemes

• Train-Time Attribute Regularization: In RegDiff, attribute regularization signals are injected exclusively at training, obviating the need for classifiers at inference and enhancing stylistic control and efficiency (Zhou et al., 7 Oct 2025).
• Plug-and-Play Decoding: CAIF, GeDi, FUDGE, DExperts, and NeuroLogic adjust tokens' probabilities using discriminator or classifier feedback at generation time, balancing control strength with fluency and efficiency (Sitdikov et al., 2022, Zhang et al., 2023).
• Preference Optimization: UltraGen uses attribute extraction and a global preference optimization (GPO) phase—to improve constraint satisfaction for extremely fine-grained, multi-attribute steering, mitigating position bias and attention dilution as the number of control axes grows to 50+ (Yun et al., 17 Feb 2025).
• Instruction Tuning and In-Context Learning: LLMs fine-tuned on instruction-following datasets or prompted with regular-expression–encoded constraint instructions can match or exceed custom algorithmic methods on most stylistic control tasks, but frequently fail at strict structural requirements (e.g., precise counts, deep nesting) (Ashok et al., 2024, Zheng et al., 2023).

3. Domains, Evaluation Protocols, and Representative Results

Controllable generation is deployed across text, images, video, protein, time series, code, motion, and structured data domains. Evaluation methodology is unified along three axes:

Metric	Description	Example Usage
Controllability	Attribute/classifier accuracy or constraint coverage	Sentiment, toxicity, keyword presence, motion track (Zhang et al., 2023, Yun et al., 17 Feb 2025, Shao et al., 2021)
Fidelity/Quality	Standard NLG, vision/audio scores	BLEU, ROUGE, FID, MOS, PPL, BERTScore (Wang et al., 2022, Zhou et al., 7 Oct 2025)
Diversity	Distinct-N, Self-BLEU	Ensures multi-modal outputs; e.g., Dist-3↑
Efficiency	Runtime, resource cost	CAIF (≈100ms), Diffusion-LM (slower), CAR (0.3s for 512×512)
Human Alignment	Judgment/preference	Fluency, topicality, preference ranking
Domain-specific	Task/property compliance	Protein stability, video IoU, game playability

Notable empirical findings:

• Apex achieves ≈97% keyword-order control versus 20–40% for baselines, and delivers production gains of +13.17% click-though rate on Taobao (Shao et al., 2021).
• RMT achieves ≈94% coverage on keyword inclusion, 97.6% sentiment accuracy (positive), outperforming prior plug-and-play and prompting methods with minimal fluency degradation (Zhang et al., 2023).
• RegDiff achieves style accuracy up to 0.96 across multiple style transfer domains, at lower inference cost than classifier-guided baselines (Zhou et al., 7 Oct 2025).
• Masked model importance sampling methods achieve near-100% constraint satisfaction in protein property and toy sequence tasks without retraining (Guo et al., 2024).
• FineXtrol achieves FID=0.245, R-Top3=0.685 on motion generation with fine-grained text, matching or outperforming coordinate-conditioned baselines at lower computational cost (Shen et al., 24 Nov 2025).
• Instruction-based prompting methods outperform classic weighting or guided decoding on most stylistic tasks, with structural tasks remaining challenging (Ashok et al., 2024).

4. Applications and Control Signals Across Modalities

Controllable generation is applied to a broad spectrum of domains, with control signals tailored accordingly:

• Text: Keyword inclusion and order (Shao et al., 2021), sentiment, toxicity, formal style, syntax (tree or span) (Zhang et al., 2023, Sitdikov et al., 2022, Li et al., 2022, Zhou et al., 7 Oct 2025), multi-attribute style (Yun et al., 17 Feb 2025), regular expression composition of constraints (Zheng et al., 2023), psychological state chains for story characters (Xie et al., 2022), length/POS/word count (Ashok et al., 2024).
• Vision: Conditional image or video generation controlled by pose, sketch, depth, camera trajectory, spatial/temporal masks, and semantic maps (Zhou et al., 7 Oct 2025, Yao et al., 2024, Ma et al., 22 Jul 2025, Lei et al., 2024, Liu et al., 9 Feb 2025).
• Motion: Fine-grained, temporally explicit natural language ranges for each limb (Shen et al., 24 Nov 2025).
• Game/Content: Goal vectors of designer-specified metrics, with modular reward shaping for level synthesis (Earle et al., 2021).
• Protein/Sequence Design: Arbitrary property/reward optimization, hard constraints on sequence composition (Guo et al., 2024).
• Code: Multi-agent planning, tool use, external safety checks in code generation pipelines (Liu et al., 9 Oct 2025).
• Time Series: Decoupled VAE regression for controllable time series/signal synthesis (Bao et al., 2024).

Control signals can range from simple labels to attribute vectors, natural language instructions, regex-formatted composite expressions, spatial maps, or temporally-aligned sequence-level annotations.

5. Open Problems, Limitations, and Future Research Directions

Despite significant advances, several open challenges remain:

• Fine-Grained, Multi-Attribute Control: As the number or complexity of control axes increases (e.g., >30 attributes in UltraGen), models suffer from position bias and attention dilution. Effective attribute sampling, curriculum design, and preference optimization are required to maintain high constraint satisfaction (Yun et al., 17 Feb 2025).
• Evaluation and Benchmarking: No single metric or benchmark uniformly reflects both multi-attribute controllability and sample quality; domain-specific composite metrics are often required (Wang et al., 2022, Ma et al., 22 Jul 2025).
• Structural and Compositional Constraints: Instructional prompting achieves near-human control for high-level attributes but struggles with hard, compositional, or deeply nested constraints (e.g., “exactly 5 words and 2 verbs”), which require dedicated inference-time decoders or instruction regularizers (Ashok et al., 2024, Zheng et al., 2023, Li et al., 2022).
• Scalability and Efficiency: Diffusion-based and energy-based methods offer fine control but are computationally intensive; plug-and-play and residual methods retain efficiency at some cost in maximum control strength (Zhou et al., 7 Oct 2025, Li et al., 2022, Yao et al., 2024).
• Extensibility to New Modalities and Objectives: There is an ongoing push towards universal, domain-agnostic control architectures that can fuse arbitrary combinations of signals, as in universal video control (VideoComposer, FullDiT) or cross-modal text-motion models (Ma et al., 22 Jul 2025, Lei et al., 2024, Shen et al., 24 Nov 2025).
• Human-Aligning and Safety: Multi-agent code generation frameworks (RA-Gen) integrate user-specified constraints, external tools, and safety checks, but further integration with formal verifiers and self-correcting algorithms remains active research (Liu et al., 9 Oct 2025, Xie et al., 2022).
• Disentanglement and Interpretability: Learning interpretable, disentangled latent controls for high-dimensional generative models is required for reliable, explainable, and user-friendly steering (Wang et al., 2022, Bao et al., 2024).

Anticipated directions include hybrid architectures (crossing AR, diffusion, RL, retrieval), more robust instruction/constraint languages, routine integration of domain knowledge, and universal compositional control across multi-modal and multi-agent settings.

6. Comparative Insights and Practical Guidelines

A comparative synthesis across surveyed approaches yields these practical guidelines:

Scenario	Recommended Paradigm	Cited Example(s)
Frozen backbone, arbitrary control at inference	Plug-and-play/Residual/Importance	RMT, CAIF, PnP masked model (Zhang et al., 2023, Sitdikov et al., 2022, Guo et al., 2024)
Need for hard lexical/structural constraint	Structural or regex instruction	REI, Pointer/CBART, COLD (Zheng et al., 2023, Zhang et al., 2022, Li et al., 2022)
Multi-attribute, high-dimensional control	Attribute regularization + preference	UltraGen, RegDiff (Yun et al., 17 Feb 2025, Zhou et al., 7 Oct 2025)
Multi-modal or temporal/spatial control	Branch/module fusion, ControlNet-style	AnimateAnything, VFX Creator, CAR (Lei et al., 2024, Liu et al., 9 Feb 2025, Yao et al., 2024)
Fast and efficient multi-condition steering	Hierarchical/fused adapters	FullDiT, VideoComposer (Ma et al., 22 Jul 2025)
RL-style property optimization	Policy gradient or reward shaping	RL approaches, PCGRL (Earle et al., 2021, Wang et al., 2022)
Instruction-following, generalist LLMs	Prompt engineering, instruction tuning	ChatGPT, FLAN, ConGenBench (Ashok et al., 2024)

The choice of method must balance controllability, efficiency, sample quality, and scalability—ensuring that the desired control granularity, diversity, and compositional complexity are robustly attained and measurable.

---

In summary, controllable generation approaches constitute a methodological spectrum spanning model architectures, guidance algorithms, sampling regimes, and meta-instruction systems. They underpin fundamental advances across text, vision, audio, game, and protein generation, addressing task-specific requirements for fidelity, precision, compositional flexibility, and user-interactivity. As generative models continue to scale and proliferate, the challenge and importance of fine-grained, efficient, and compositional control will continue to shape both foundational theory and practical deployment across disciplines (Wang et al., 2022, Shao et al., 2021, Zhou et al., 7 Oct 2025, Ma et al., 22 Jul 2025, Shen et al., 24 Nov 2025, Zhang et al., 2023, Ashok et al., 2024, Sitdikov et al., 2022, Zhang et al., 2022).