TVAE: Synthetic Data for Tabular Applications

• TVAE is a specialized variational autoencoder that generates synthetic tabular data by effectively modeling both continuous and categorical features.
• It employs mode-specific normalization for continuous features using Gaussian mixture models and one-hot encoding for categorical features to address complex, non-Gaussian distributions.
• TVAE enables data augmentation, privacy-preserving synthesis, and class balancing, enhancing downstream machine learning performance across diverse applications.

A Tabular @@@@1@@@@ (TVAE) is a specialized form of Variational Autoencoder (VAE) designed to generate synthetic tabular data, accommodating the heterogeneous and mixed-type characteristics typical of real-world datasets. TVAE architectures explicitly address the technical challenge of modeling tabular data containing both continuous and categorical features, often with complex and non-Gaussian marginal distributions. TVAE variants have demonstrated strong performance across tasks including data augmentation, privacy-preserving data synthesis, and imbalanced class oversampling, particularly in domains where regulatory or practical limits restrict the use of real data.

1. Core Architecture and Design Principles

TVAE adopts the canonical VAE structure consisting of an encoder $q_\phi(z|x)$ and a decoder $p_\theta(x|z)$, with both components parameterized by neural networks. The encoder projects each tabular record $x$—potentially containing mixed-type features—into a continuous latent representation $z$. The decoder reconstructs $x$ from $z$, aiming to maximize the fidelity of the generative model relative to the original data distribution (Karst et al., 2024).

For mixed-type data, TVAE employs differentiated treatment for feature types:

• Continuous features: Mode-specific normalization is applied, wherein each continuous column is fitted using a Gaussian mixture model (GMM), and the data are transformed using the component CDFs to the $(0,1)$ interval before being passed to the encoder.
• Categorical features: One-hot encoding is implemented, with embeddings both in the encoder and a multinomial parameterization in the decoder.
• Conditional generation: A short conditioning vector $c$ can be concatenated to both the encoder and decoder inputs to enable class-conditional sampling, which is crucial for tasks like class balancing or targeted oversampling.

2. Mathematical Formulation and Optimization

TVAE is trained by optimizing the Evidence Lower Bound (ELBO) for each data instance $x_i$: $$\mathcal{L}(x_i;\theta,\phi) = \mathbb{E}_{q_\phi(z|x_i)}[\log p_\theta(x_i|z)] - D_{KL}\bigl(q_\phi(z|x_i)\,\|\,p(z)\bigr)$$ where $p(z)$ is typically a standard Normal prior $\mathcal{N}(0, I)$ during training. For mixed-type features, TVAE decomposes the reconstruction term as follows:

• Continuous features: Assumes $$p_\theta(x^c|z) = \mathcal{N}(x^c;\mu_\theta(z), \Sigma_\theta(z))$$, with the loss given by expected negative log-likelihood.
• Categorical features: Uses a multinomial decoder, resulting in standard cross-entropy loss summed over categories.

For each sample,

$$L = \mathbb{E}_{q_\phi(z|x)}[-\log p_\theta(x|z)] + KL(q_\phi(z|x)\|p(z))$$

where

$$KL(q_\phi(z|x)\|p(z)) = \frac{1}{2}\sum_{i=1}^{d_z}(\sigma_{\phi,i}^2(x) + \mu_{\phi,i}^2(x) - 1 - \ln\sigma_{\phi,i}^2(x))$$

(Karst et al., 2024).

3. Model Instantiations and Variants

TVAE has been adopted and extended in various research efforts. Significant implementations include:

• Base TVAE (Karst et al., 2024): Encoder and decoder as multi-layer perceptrons, mode-specific normalization for continuous features, categorical handling via embeddings and multinomial outputs, and optional conditioning via $c$-vectors. Specific hyperparameters and model depth are not detailed in (Karst et al., 2024).
• VAE–GMM integration ("SAVAE", Editor’s term) (Apellániz et al., 2024): Improves over TVAE by replacing the generative prior $p(z)$ with a Bayesian Gaussian Mixture (BGM) model fitted to the set of learned latents $z_i$ after training. This enables latent sampling from a Dirichlet-process GMM with full covariance for each component, increasing flexibility for non-Gaussian latent geometries. SAVAE retains TVAE’s feature-specific head design for modeling heterogeneous data types and follows a similar encoder/decoder architecture.

Model	Latent Prior	Conditioning	Normalization	Decoder Output
Base TVAE	Gaussian	Optional $c$	Mode-specific (GMM)	Feature-specific
SAVAE	Bayesian GMM	Not stated	Not explicit	Feature-specific

4. Generation Process and Inference

After training, TVAE generates synthetic tabular samples by sampling from the latent prior $p(z)$ (standard Normal for base TVAE, Gaussian mixture for SAVAE), then decoding to the data space:

1. Sample $z \sim p(z)$.
   • For SAVAE, $k \sim \mathrm{Categorical}(\pi_1, \ldots, \pi_K)$, $z \sim \mathcal{N}(\mu_k, \Sigma_k)$ with mixture components fitted after VAE training (Apellániz et al., 2024).
2. Decode $x \sim p_\theta(x|z)$.
   • For each feature, sample according to its feature-specific decoder likelihood (Gaussian for continuous, Softmax for categorical, etc.).

The process for SAVAE involves fitting a Bayesian Gaussian mixture on the aggregated $z_i$ latents, with stick-breaking weights and VB-EM updates as detailed in (Apellániz et al., 2024). No closed-form for the KL divergence between $q_\phi(z|x)$ and $\sum_k \pi_k N(\mu_k, \Sigma_k)$ is provided; Monte Carlo or log-sum-exp approximations are recommended.

5. Evaluation Metrics and Comparative Performance

TVAE and its derivatives are evaluated on metrics reflecting both resemblance (distributional similarity) and utility (downstream ML task performance):

• Resemblance
  • Random Forest discriminator accuracy (ideal ≈ 0.5).
  • Marginal+pairwise distribution score (0–1 scale; higher is better).
• Utility
  • Downstream task performance: classification accuracy (for class targets) and C-index (for survival tasks), with comparisons across synthetic → real and real-only training.

Reported metrics from (Apellániz et al., 2024):

Dataset	CTGAN Acc.	TVAE Acc.	SAVAE (SOTA) Acc.
Adult	0.75	0.78	0.68
Metabric	0.73	0.77	0.67
STD	0.94	0.77	0.64

Dataset	CTGAN Score	TVAE Score	SAVAE Score
Adult	0.87	0.87	0.93
Metabric	0.89	0.88	0.92
STD	0.86	0.87	0.95

TVAE displays strong fidelity on real datasets (column-wise KS-statistic 0.90, row-wise Pearson 0.97), high synthesis novelty ($\approx$0.999), rapid training time (401s, fastest tested), and matches or slightly exceeds other models on global graph-structure metrics (NetSimile ≈ 31.2) (Karst et al., 2024). TVAE offers moderate privacy on real data (NNDR ≈ 0.98); privacy improves on synthetic, simpler data.

6. Application Contexts and Tradeoffs

TVAE and SAVAE have been validated on a spectrum of datasets, including the Adult (mixed features, income target), Metabric (binary/decimal, survival), STD (mixed, survival), and proprietary financial transaction data. Their ability to model complex, heterogeneous distributions and support conditional sampling makes them suitable for:

• Data augmentation in class-imbalanced and sensitive settings (e.g., healthcare, banking).
• Synthetic data generation for privacy-preserving data sharing.
• Simulation and benchmarking of ML pipelines in contexts where access to original records is constrained.

In banking applications, TVAE achieves a balance between fidelity and privacy, outperforming GAN variants in training speed and accuracy on complex marginals but offering moderate privacy guarantees relative to GAN models designed for privacy (such as DGAN) (Karst et al., 2024). In healthcare and mixed-type tabular benchmarks, SAVAE outperforms both TVAE and CTGAN on distributional resemblance and utility tasks (Apellániz et al., 2024).

7. Limitations and Future Directions

Key limitations of TVAE include the assumption of a standard Normal latent prior during training and the limited expressiveness for non-Gaussian latent spaces. SAVAE partially remedies these through Bayesian GMM-based latent priors. However, both architectures can show moderate privacy leakage, especially at the cluster level, and may be conservative in synthesis novelty on less complex data. Hyperparameter selection procedures and precise architectural details are often unspecified, especially in high-impact deployments (Karst et al., 2024).

A plausible implication is that further research will focus on latent space priors, advanced per-feature likelihoods, and formal assessment of privacy under more adversarial threat models. The explicit modeling of feature-wise marginals and flexible decoding remains critical for advancing synthetic tabular data generation.