MultiM-Poem Systems

• MultiM-Poem Systems are computational architectures that integrate text, images, audio, and concepts to generate and translate poetic expressions using deep learning.
• They employ advanced fusion strategies and neural modules like Transformers, CNNs, and diffusion models to achieve culturally and semantically rich outputs.
• Specialized evaluation protocols utilizing human ratings and metrics such as BLEU, ROUGE, and prosody consistency validate their fluency, coherence, and poetic style.

A MultiM-Poem system is a computational architecture designed to generate, translate, or augment poetry using multiple modalities—text, images, concepts, audio, and sometimes prosody—leveraging advances in deep learning, cross-modal embedding, and generative modeling. These systems surpass template- or rule-based poetry models by incorporating structured neural modules, multi-stage prompt and alignment strategies, and dedicated evaluation protocols to synthesize culturally, semantically, and artistically rich outputs.

1. System Architectures and Modalities

Modern MultiM-Poem systems process heterogeneous inputs, including text, images, artistic conceptions, and raw audio, mapping them to richly structured poetic or visual outputs. State-of-the-art implementations support workflows such as:

• Text→Poem: Direct poetry generation from keywords, phrases, or longer text input using Transformer encoders and hybrid retrieval-generation pipelines. For example, Deep Poetry incorporates both character-level and phrase-level tokenization (Liu et al., 2019).
• Image→Poem and Image→Poetic Image: Extraction of visual themes or objects (typically via ResNet, ViT, or Inception CNNs), mapping to latent or thematic representations, and conditional generation via RNNs or attention-based decoders (Liu et al., 2018, Liu et al., 2018).
• Poem→Image: Semantic, emotion, and entity extraction (e.g., PoeKey algorithm, semantic graphs), iterative prompt engineering, then image synthesis via latent diffusion models and prompt-conditioned U-Nets (Jamil et al., 10 Jan 2025, Jamil et al., 18 Jul 2025, Jamil et al., 17 Nov 2025).
• Translation: Cross-lingual poetic transfer with structure preservation and preference alignment, notably using the Odds Ratio Preference Alignment (ORPO) algorithm to bias generations toward high-quality poetic alignments (Jamil et al., 17 Nov 2025).
• Audio Modality: Precise alignment of text, scansion, and phonetic features to enable multimodal text–prosody interactivity and future poetry-aware TTS (Agirrezabal, 2024).

Modal fusion strategies range from explicit joint embeddings and attention over concatenated features (Liu et al., 2019), to graph-driven semantic clustering for compositional prompt construction (Jamil et al., 17 Nov 2025), to policy gradient/actor-critic optimization with multiple discriminators combining cross-modal relevance and poetic style (Liu et al., 2018).

2. Core Algorithms and Model Components

Multimodal Encoders and Decoders

• Textual Encoding: Typically Transformer or hierarchical self-attention models, handling phrase, character, and sentence-level context (Liu et al., 2019, Liu et al., 2018).
• Visual Encoding: ResNet-50, ViT, or Inception CNNs pretrained on large image sets, sometimes retrained for poetic themes (e.g., mapping paintings to ShiXueHanYing taxonomy) (Liu et al., 2018, Liu et al., 2019).
• Conceptual Encoding: Averaging over embedded phrase vectors corresponding to user/semantic graph-supplied concepts (Liu et al., 2019).
• Audio–Text Alignment: DTW for line–audio, HMM-based forced phonetic alignment, syllabification and scansion via BiLSTM+CRF sequence models (Agirrezabal, 2024).
• Multi-modal Fusion: Concatenation and learned projection of encoded modalities into a shared context token, which is then attended to by downstream generative modules (Liu et al., 2019).

Generation and Postprocessing

• Decoder Architectures: Autoregressive masked self-attention decoders (Transformer-based), hierarchical GRUs with multi-stream (char/phrase/sent) context (Liu et al., 2018, Liu et al., 2019).
• Result Filtering: Beam search for candidate generation, with rule-based screening for metric compliance (length, rhyme, repetition) (Liu et al., 2019).
• Image Generation: Prompt-tuned diffusion models (SDXL, SD-3.5-M) guided by semantic key extraction, graph-based prompt engineering, and iterative prompt refinement for creative alignment (Jamil et al., 10 Jan 2025, Jamil et al., 18 Jul 2025, Jamil et al., 17 Nov 2025).

3. Specialized Training and Alignment Strategies

Data and Annotated Corpora

• Large-scale poetry corpora: Human-authored Chinese quatrains (210K poems), ancient prose (3M lines) (Liu et al., 2019); MorphoVerse (1,570 Indian-language poems + English translations) (Jamil et al., 17 Nov 2025); P4I (1,111 English poems, diverse styles) (Jamil et al., 18 Jul 2025); MiniPo (1,001 children’s poems + generated images) (Jamil et al., 10 Jan 2025).
• Multimodal datasets aligned at line, word, syllable, and phone levels (Shakespeare/Milton) (Agirrezabal, 2024).
• Annotation includes themes, emotional arcs, semantic graphs, and topic-labeled phrase inventories (LDA/Dirichlet components) (Liu et al., 2018, Liu et al., 2019).

Training and Optimization

• Pref. Alignment for Translation: ORPO, operating on the odds of preferred (human) translation over less-preferred, added as a penalty to SFT loss (Jamil et al., 17 Nov 2025).
• Policy Gradient / Reinforcement Learning: Poem generation agents reward cross-modal relevance and poetic style, optimized via REINFORCE over a scalar combination of discriminator outputs (Liu et al., 2018).
• Prompt Tuning: Sequential, reward-guided natural language template selection for optimizing LLM summarization and diffusion-instruction prompts (Jamil et al., 10 Jan 2025).
• Modular Fine-Tuning: LoRA adapters or SFT for LLM backbones, with downstream frozen diffusion weights for image generation (Jamil et al., 17 Nov 2025).

Prompt Engineering and Semantic Extraction

• Graph-Based Prompt Generation: Construction of directed graphs over tokens (lemma, synset pairs), community detection (modularity clustering), and expert-in-the-loop refinement for maximal metaphorical and semantic coverage (Jamil et al., 17 Nov 2025).
• Multi-Stage Refinement: Closed feedback loops using LLMs to iteratively enhance descriptive prompts until convergence in alignment metrics (e.g., Long-CLIP) (Jamil et al., 18 Jul 2025).

4. Evaluation Protocols

MultiM-Poem systems are assessed by a comprehensive suite of metrics:

Dimension	Metric/Protocol	Example Results
Fluency, Coherence, Poeticness	Expert human rating (1–5 scale)	Deep Poetry: 4.2/4.0/3.9 (Liu et al., 2019)
BLEU-n, ROUGE, METEOR, COMET	N-gram/semantic similarity to references	BLEU-4 (Deep Poetry): 0.081 (Liu et al., 2019); BLEU-4 (ORPO): 0.2864 (Jamil et al., 17 Nov 2025)
Rhyme & Rhythm Compliance	Formal/Rule-based classifiers	>98% official rhyme alignment (Liu et al., 2019)
Cross-modal Retrieval	CLIP, Long-CLIP, BLIP, ITM/ITC	BLIP=0.4613 (CP prompt; SD-3.5-M) (Jamil et al., 17 Nov 2025)
Novelty, Imaginativeness	Out-of-vocabulary N-gram %	Novel bigram 60.7%, trigram 89.7% (Liu et al., 2018)
Prosody Consistency	Scansion/stress, syllable/phone metrics	σ(phone)=0.1418s, σ(syll)=0.1527s (Agirrezabal, 2024)
Human Preference	MTurk/expert Turing-style tests	Turing test confusion ≈49% (AMT) (Liu et al., 2018)

5. Representative Systems

Deep Poetry (Chinese Classical Poetry)

• Transformer-based fusion of text, image (ResNet/Vision Transformer), and high-level concept inputs.
• Rule-based screening for prosodic compliance and language quality.
• Deployed in a real-time WeChat mini-program, returning candidate poems with sub-2s latency (Liu et al., 2019).

MultiM-Poem Framework (Indian Poetic Translation & Visualization)

• Two-stage architecture: LLM-based translation (w/ ORPO) followed by semantic graph-driven prompt engineering for latent diffusion image generation (SD-3.5-M).
• MorphoVerse dataset for evaluation across 21 Indian languages, with prompt-derived images scoring highest in both automated and human metrics (Jamil et al., 17 Nov 2025).

PoemToPixel

• Single-image generation from English poems, using multi-level semantic key extraction (PoeKey algorithm) and prompt-tuned SDXL diffusion.
• Modular pipeline validated across adult (PoemSum) and children’s (MiniPo) datasets (Jamil et al., 10 Jan 2025).

PoemTale Diffusion

• Multistage segmentation and refinement loop producing a set of visually and semantically consistent images per poem.
• Consistent self-attention modifies U-Net layers for stable cross-segment identities, optimizing for maximal information retention (Jamil et al., 18 Jul 2025).

Multi-Adversarial Image-to-Poem Generation

• Deep coupled visual-poetic embedding space, GRU-based generator, and dual (cross-modal and poem-style) discriminators trained via policy gradient.
• Human evaluation (including poetry experts) shows generated free-verse is often indistinguishable from human-written poems (Liu et al., 2018).

Multi-Modal Chinese Quatrain Model

• Three-stage sequence: (1) CNN-driven theme phrase, (2) LDA-based title, (3) hierarchy-attention seq2seq poem generation.
• Image–theme phrase mapping provides semantic grounding, while LDA ensures topical cohesion (Liu et al., 2018).

6. Limitations and Future Directions

Notable current challenges and prospects include:

• Genre and Language Generalization: Existing pipelines are primarily tailored to specific poetic forms (e.g., Chinese quatrains, free-verse) and heavily resource-constrained on low-resource languages or genres (Liu et al., 2019, Jamil et al., 17 Nov 2025).
• Semantic Depth and Style Control: Current models rely on surface-level phrase expansion or entity extraction; integration with domain knowledge graphs and fine-grained authorial style embeddings remains underdeveloped (Liu et al., 2019).
• Multi-image Storyboarding: Movement from single-image to multi-stage, narrative-coherent visualizations is ongoing, as in PoemTale Diffusion and proposed expansions for PoemToPixel (Jamil et al., 10 Jan 2025, Jamil et al., 18 Jul 2025).
• On-device and Efficient Inference: Practical deployment necessitates lightweight models or modular pipelines suitable for mobile and edge scenarios (Liu et al., 2019).
• Prosodic and Acoustic Modeling: New multimodal corpora enable integration of phonetic and metrical information for expressive TTS or machine-voiced poetry generation, with potential for using scansion and stress features as conditioning signals (Agirrezabal, 2024).
• Interactive Co-creation: Human–AI collaborative writing, real-time feedback, and editable generations are active areas for interface innovation (Liu et al., 2019).
• Prompt Engineering Automation: Efficiency and scalability of prompt tuning, especially for cross-genre or cross-lingual transfer, will require automated feedback loops or reinforcement learning methodologies (Jamil et al., 10 Jan 2025).

Concerted research across cultural, linguistic, and technical domains continues to expand the capacity and reach of MultiM-Poem systems, towards richer, more semantically grounded, and interactive poetic creation and analysis.