GraphiMind: LLM-Driven Graphics Design

• GraphiMind is an LLM-centric interface that integrates conversational AI with graphical design tools to automate the full visual composition process.
• It employs a tool-augmented library and an interactive canvas to generate, recommend, and refine design elements efficiently.
• User studies demonstrate reduced design time and improved workflow linearity, making it accessible for non-experts.

GraphiMind is an LLM-centric interface and agent system for streamlined, intent-driven information graphics design, targeted at enabling non-experts to generate, recommend, and compose high-quality information graphics using natural language interaction. Distinct from traditional design tools, GraphiMind tightly integrates LLM reasoning with tool-augmented resource generation and a browser-based graphical manipulation canvas. The architecture is built to automate the full pipeline from information curation to visual composition, supported by backend tool orchestration and interactive frontend refinement (Huang et al., 2024).

1. System Architecture and Workflow

GraphiMind consists of two principal components: a Textual Conversational Interface—termed the "Agent"—which manages intent parsing, tool selection, and orchestration; and a Graphical Manipulation Interface ("Canvas") supporting direct editing and layout of generated resources.

Components

• Textual Conversational Interface: Tool-augmented OpenAI ChatGPT in function-calling mode responsible for high-level conversational control, scheduling, and tool invocation.
• Agent-Managed Tool Library:
  • Stable Diffusion XL 1.0: Generation of pivot and background images.
  • ChatGPT: Information extraction and content curation.
  • Iconify API: SVG icon retrieval.
  • GPT-4 with DSL: Automated layout tree generation.
  • InstructPix2Pix: Image editing.
  • SAM: Image clipping.
• Graphical Manipulation Interface: Browser-based canvas with drag-and-drop support, resizing, color and font controls, layering, and a property toolbar.

Workflow

The data flow consists of:

1. User submits a natural language message $m$ via the chat panel.
2. Agent classifies the message. If no design task is detected, a conversational fallback is triggered; otherwise, the agent enters a scheduling routine to select a tool $t\in T$ and synthesize arguments $a$.
3. Function call to tool $t$ (via ChatGPT function-calling API) using arguments $a$.
4. Tool executes and returns one or more resources $R$.
5. Agent embeds $R$ within the dialogue, offering text-based refinement or auto-placing $R$ on the canvas through drag-and-drop hooks.
6. User may regenerate or refine resources via chat, or make local adjustments on the canvas.
7. Final composition is exported.

The agent control flow is formally described as: $$\texttt{OnUserMessage}(m,\,C):\ \text{if}\ \mathrm{NeedsTool}(m,C) = \varnothing\ \mathrm{Reply}( \mathrm{ChatGPT\_Complete}(m,C) )\ \text{else} \ (t,a) = \arg\max_{t\in T,\,a} P(t,a\mid m,C)\ R = \mathrm{CallTool}(t,a)\ \mathrm{EmbedInChat}(R)\ \mathrm{AutoPlaceOnCanvas}(R)$$ where $C$ is the dialog and canvas context, and $P(t,a\mid m,C)$ is the agent’s selection process.

2. Textual Conversational Interface: Function-Calling and Prompt Engineering

Each tool is registered with the agent via JSON function signatures (name, description, argument schema, examples). Example:

{ "name": "generate_pivot_figure", "description": "Generate a central thematic image focusing on a main object or character.", "parameters": {"caption": "string", "style": "string", "effect": "string"}, "example": {"caption": "a smiling dog wearing a hat", "style": "watercolor", "effect": "focused"} }

The LLM autonomously decides, at each user turn, whether to invoke a function or respond in free-form text. Argument selection relies on the LLM’s world knowledge and in-context examples, without explicit scoring or ranking formulas beyond the internal inference of the LLM.

Intent parsing operationally solves

$$(t^*,a^*) = \arg\max_{t,a} P_{\text{LLM}}(t,a \mid \text{dialogue context})$$

Supported tools cover all aspects of information graphics assembly, from asset generation to layout and editing (Huang et al., 2024).

3. Graphical Manipulation Interface and Layout Generation

The browser-based canvas is a drag-and-drop, direct-manipulation environment that supports:

• Object selection and property adjustment (position, size, color, typography, icon stroke)
• Layer management and basic undo/redo
• Snap-to-grid and guide overlays (prototype)
• Inline editing of all rendered assets

Automated layout generation is realized by a tuple-based DSL: $$\begin{aligned} \langle \mathit{Layout} \rangle &\to \langle \mathit{ContainerList} \rangle \ \langle \mathit{ContainerList} \rangle &\to \langle \mathit{Container}\rangle^+ \ \langle \mathit{Container} \rangle &\to [\text{type}:T, x:X, y:Y, w:W, h:H, \mathrm{children}:\langle \mathit{ContainerList}\rangle] \end{aligned}$$ Containers encode layout structure and constraints (e.g., only one icon/headline/content per container, no large overlaps).

4. Automated Resource Curation, Recommendation, and Composition

Content curation is performed by single-prompt ChatGPT invocations that yield structured JSON:

{ "title": "...", "bullet_points": [ {"icon_keyword": "...", "headline": "...", "content": "..."}, ... ] }

Icon keywords are resolved to SVG icons via the Iconify API. Pivot and background images are generated using Stable Diffusion XL 1.0, with template prompts distinguishing between focused (pivot) and scenery (background) images. For layout, the agent calls GPT-4 with a specific prompt and DSL grammar, resulting in an in-memory layout tree that is rendered and edited interactively.

Assets are auto-placed onto the canvas in matching containers as defined by the DSL, and users retain the ability to fine-tune or rearrange as desired.

5. Evaluation: User Studies and Workflow Analysis

A controlled user study compared GraphiMind with a baseline workflow (PowerPoint + Web search) using 16 novice designers across two tasks.

Quantitative results:

• Mean design time: GraphiMind (18.26 ± 8.86min) vs. PowerPoint (33.40 ± 12.24min), $t(14)=3.28, p<0.01$
• Information collection: GraphiMind (2.03min) vs. PowerPoint (10.76 ± 9.47min), $t(14)=3.37, p<0.01$
• Other tasks (background design, layout, visual elements) showed faster completion with GraphiMind, but not all differences were statistically significant.

Workflow patterns:

• GraphiMind users followed a more linear pipeline (Resource → Layout → Info → Local Adjustment); PowerPoint users displayed frequent ad hoc task interleaving.
• GraphiMind users concentrated most fine-tuning at the end, contrasting with the continuous adjustment pattern in PowerPoint.

Subjective evaluation (Likert 5-point scale):

• Information collection efficiency: 4.75 ± 0.58 (highest)
• Layout adjustment: 3.38 ± 0.89 (lowest)
• Other metrics (ease of use, enjoyment, expressiveness): ≥4.0
• Typical positive feedback focused on the smooth integration of chat and canvas, rapid resource creation, and beginner-friendly experience.

6. Case Workflow Example and Practical Usage

A representative user session proceeds as follows:

1. User: “I want to make an infographic about climate change’s effect on polar bears.”
2. Agent: “Would you like a central image (pivot figure) or background scene first?”
3. User selects pivot figure; agent calls the image generator and returns a PNG.
4. Agent offers further resource curation, gathers bullet-point information content, and generates icons.
5. User requests a “flowy” layout; agent calls GPT-4 with the DSL, parses and renders the layout.
6. Agent autopopulates the canvas, user performs minor adjustments, and exports the graphic.

This scenario demonstrates seamless, alternating natural language and GUI-based composition for a focused design task, enabled by agent-mediated tool orchestration and flexible layout generation.

7. Limitations, User Feedback, and Prospective Directions

Several enhancement areas have been identified:

• Context and personalization: The current system lacks persistent memory for canvas state or user style preferences. This suggests user experience could be improved with persistent project state and style templates.
• Text ambiguity: Natural language interfaces excel in usability but may reduce precision. A plausible implication is the integration of hybrid "sketch+text" input for finer control.
• Resource recommendation: Automated font pairing, color palette suggestions, and richer decorative options remain underdeveloped.
• Engineering: Canvas features such as advanced alignment guides, project management, and robust undo/redo functionalities are noted as needed for production-grade scenarios.
• Dependence on model fidelity: The quality and efficiency of generation are dependent on the current state of LLMs and diffusion models; advances in those technologies will directly benefit system capabilities.

GraphiMind demonstrates the viability of LLM-driven multimodal design assistants and points to future research in tightly-coupled conversational graphics generation, adaptive layout, and context-aware personalization (Huang et al., 2024).