Multimodal Sentiment Association Knowledge Graph

• MSA-KG is a knowledge graph that maps relationships between visual objects, attributes, and emotions, enabling structured reasoning for affective editing.
• It integrates multimodal cues with chain-of-thought reasoning to guide precise, region-aware emotional injection while preserving spatial and semantic integrity.
• Quantitative evaluations show enhanced emotion activation and semantic consistency, demonstrating superior performance over traditional affective editing methods.

The Multimodal Sentiment Association Knowledge Graph (MSA-KG) is a knowledge-centric framework designed to disentangle and model the complex causal relationships between visual entities (objects, scenes, attributes), multimodal cues, and affective states within computational emotion editing and synthesis systems. MSA-KG explicitly encodes object-attribute-emotion chains, providing structured external knowledge that enables chain-of-thought reasoning and supports advanced multimodal editing workflows. As implemented in training-free high-fidelity editing systems—most notably in EmoKGEdit—MSA-KG guides region- and attribute-precise affective transformations while strictly preserving spatial and semantic integrity (Zhang et al., 18 Jan 2026). MSA-KG also integrates with large vision–LLMs (MLLMs) and chain-of-thought prompting to enable context-dependent, knowledge-grounded emotional manipulation.

1. Foundations and Motivation

Existing affective editing methods often conflate structural features, semantic content, and emotional attributes, resulting in weak emotional expressiveness or structural degradation. MSA-KG was introduced to address these limitations by (a) systematizing the causal pathways among objects, their visual/semantic attributes, and corresponding affective outcomes, and (b) facilitating structured reasoning for multimodal models. MSA-KG supports workflows where emotional cues must be injected or manipulated within specific spatial or semantic loci, using knowledge graphs as externalized, queryable reasoning substrates.

The core motivation is to sidestep the entanglement between emotion and layout content in conventional diffusion pipelines. By formally encoding relations such as “dark lighting → isolated chair → sadness,” MSA-KG allows downstream editing modules to select emotion-appropriate cues for injection, leveraging both explicit knowledge and model-generated reasoning chains (Zhang et al., 18 Jan 2026).

2. Knowledge Graph Construction and Structure

MSA-KG is assembled as a multimodal, hierarchical graph whose nodes represent objects, scene elements, fine-grained attributes (e.g., texture, spatial configuration), and emotion categories or scores. Edges encode empirical or heuristic causal relations—from object appearance and attribute composition to predicted emotional response.

Typical construction workflow involves:

• Extraction of object, attribute, and scene candidates from large annotated emotion datasets (e.g., EmoSet, MoodArchive).
• Mining correlations via clustering or model-based attribution (e.g., CLIP clustering on visual embeddings).
• Summarization of clusters into atomic graph nodes, assisted by LLMs for factor description.
• Linking nodes with directed edges capturing observed attribute-to-emotion mappings, augmented by expert judgment or chain-of-thought prompts (Zhang et al., 18 Jan 2026).

In EmoKGEdit, the graph’s object–attribute–emotion chains are extracted and ranked for relevance to the current editing context, yielding candidate cues for emotion injection.

3. Integration with Multimodal Models and Chain-of-Thought Reasoning

MSA-KG serves as an external reasoning scaffold interfaced with multimodal LLMs (e.g., Qwen2.5-VL, LLaVA-NeXT). Chain-of-thought reasoning is performed:

• Querying MSA-KG for causal chains matching target emotions and source scene context.
• Filtering and ranking visual/semantic cues (e.g., “rainy window,” “warm sunlight”) by similarity to the input image and by their causal strength for the desired affect.
• Using the graph as grounding for natural-language instruction generation, which guides the subsequent diffusion editing step.

In practice, chain-of-thought sequences may begin with object candidates, traverse attribute-rich subgraphs, and terminate at emotion nodes, allowing for dynamic, context-aware prompt formation (Zhang et al., 18 Jan 2026, Ye et al., 18 Jul 2025).

4. Technical Implementation in Structure–Emotion Disentanglement

MSA-KG is operationalized in EmoKGEdit through a disentangled structure–emotion editing module. The workflow proceeds in three main steps:

1. Localization: Use an Emotion Region-aware module to segment the affective locus via a binary mask $M$.
2. Affective Injection: Leverage Emotion Cue Transfer to query MSA-KG, generating an emotion-specific text prompt $c_{\mathrm{emo}}$.
3. Editing (DSEE):
   • Generate two parallel latent diffusion streams: a reconstruction path driven by an empty prompt (preserving structure) and an editing path conditioned on $c_{\mathrm{emo}}$ (injecting emotion).
   • Fuse the two streams via mask-guided blending and attention-driven feature injection:

$$x_{t-1}^{\mathrm{edit}} = M \odot \tilde{x}_{t-1}^{\mathrm{edit}} + (1 - M) \odot \tilde{x}_{t-1}^{\mathrm{rec}}$$

$$F_{l,t}^{\mathrm{edit}} \leftarrow F_{l,t}^{\mathrm{edit}} + \lambda_{\mathrm{att}} [A_{l,t}^{\mathrm{rec}} \odot F_{l,t}^{\mathrm{edit}}]$$

This mechanism ensures that only the target region receives affective transformation, and structure is tightly preserved outside $M$ (Zhang et al., 18 Jan 2026).

5. Quantitative and Qualitative Performance

The efficacy of MSA-KG–guided editing has been demonstrated via established and novel metrics:

• Target Emotion Activation (TEA): Quantifies strength of emotion injection.
• Structural preservation (SSIM, PSNR, LPIPS): Measures spatial and semantic consistency with source images.
• Semantic correlation: Evaluated by CLIP-based similarity or human preference studies.

Ablation studies in EmoKGEdit show that incorporating MSA-KG (via ECT and DSEE) increases TEA from 0.10 to 0.3179, semantic-C from 0.5740 to 0.6470, with minimal drop in SSIM, confirming superior performance for both emotion fidelity and layout preservation (Zhang et al., 18 Jan 2026). Similar disentanglement schema in Moodifier and MUSE further support these findings via cross-attention gating and test-time loss optimization (Ye et al., 18 Jul 2025, Xia et al., 26 Nov 2025).

Method	TEA ↑	SSIM ↑	semantic-C ↑
Baseline (SDXL)	0.0600	0.3980	0.5900
+ ERA	0.1000	0.4270	0.5740
+ ERA+ECT+DSEE	0.3179	0.4204	0.6470

6. Comparisons to Alternative Affective Editing Paradigms

Unlike purely prompt-driven or classifier-guided approaches (e.g., MUSE (Xia et al., 26 Nov 2025), Moodifier (Ye et al., 18 Jul 2025)), MSA-KG introduces explicit external structure, enabling explainable and context-adaptive emotional transformations. Other state-of-the-art systems rely on sequential or gradient-based optimization of affective tokens, but lack knowledge-grounded causal modeling. Systems utilizing attention blending and region-aware masking (Moodifier, EmoEditor) realize similar spatial disentanglement but without graph-driven reasoning.

MSA-KG uniquely supports complex, multi-attribute, multi-object scenarios, facilitating grounded, high-precision affective edits while minimizing semantic drift.

7. Applications and Future Directions

MSA-KG-driven editing advances the state-of-the-art in domains requiring precise affective control: therapeutic image transformation, emotional content production, fine-grained video avatar manipulation, and explainable user-driven design. Future research aims to expand graph size, automate graph extraction from richer multimodal corpora, and integrate real-time multimodal reasoning with feedback loops. The paradigm sets a precedent for external knowledge incorporation in affective computing and multimodal reasoning workflows (Zhang et al., 18 Jan 2026, Ye et al., 18 Jul 2025).