From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design

Abstract: Creating digital models using Computer Aided Design (CAD) is a process that requires in-depth expertise. In industrial product development, this process typically involves entire teams of engineers, spanning requirements engineering, CAD itself, and quality assurance. We present an approach that mirrors this team structure with a Vision LLM (VLM)-based Multi Agent System, with access to parametric CAD tooling and tool documentation. Combining agents for requirements engineering, CAD engineering, and vision-based quality assurance, a model is generated automatically from sketches and/ or textual descriptions. The resulting model can be refined collaboratively in an iterative validation loop with the user. Our approach has the potential to increase the effectiveness of design processes, both for industry experts and for hobbyists who create models for 3D printing. We demonstrate the potential of the architecture at the example of various design tasks and provide several ablations that show the benefits of the architecture's individual components.

• The paper introduces a novel multi-agent system that uses VLMs and LLMs to automate the CAD workflow from concept to execution.
• It employs specialized agents for requirements gathering, model generation via CadQuery, and quality assurance to iteratively refine design outputs.
• Experimental results demonstrate improved design compliance and readiness, despite challenges in spatial reasoning and complex geometric constraints.

From Idea to CAD: A LLM-Driven Multi-Agent System for Collaborative Design

This paper presents a comprehensive approach to automating Computer-Aided Design (CAD) through a multi-agent system driven by Vision LLMs (VLMs). The following sections outline the architecture, implementation, and evaluation of this innovative system.

Introduction

The integration of AI, particularly LLMs and Vision LLMs (VLMs), into Computer-Aided Design offers the potential to democratize and automate complex design processes. Traditional CAD demands significant expertise, often involving entire teams for requirement gathering, design, and quality assurance. This paper proposes a system mirroring these human roles with AI agents in a VLM-based Multi-Agent System (MAS), streamlining the workflow from initial design concepts to executable CAD models. The system operates iteratively, allowing for refinement through user interaction and a visual feedback loop.

System Architecture

The proposed system architecture consists of specialized agents representing distinct phases of the engineering design process: requirements engineering, model creation, and quality assurance.

Requirements Engineer

This agent serves as the initial interface with the user, handling ambiguous specifications provided as textual descriptions or sketches. It iteratively refines this input into a comprehensive specification suitable for CAD work.

CAD Engineer

Utilizing the refined specification, this agent generates a CAD model using the CadQuery library. It creates Python code that captures the model's geometry and ensures code validity before execution. This agent leverages documentation and feedback for iterative improvement of the model design.

Figure 1: Engineering team architecture.

Quality Assurance Engineer

This agent verifies the model against the initial specification. It generates multiple views of the model, checking for alignment with design requirements and providing feedback for further refinement if necessary.

Implementation and Experiments

The system was implemented in Python, utilizing libraries like CadQuery for model generation and PyVista for rendering 3D views. Experiments were conducted using a range of input complexities, from simple geometric shapes to more complex components.

Results

The results demonstrated the efficacy of the MAS framework in generating compliant designs with higher readiness levels than traditional, single-shot VLM applications. The iterative approach, combined with specialized agents, enabled more precise adherence to user specifications.

Figure 2: Examples of the iterative design process from the user specification to the resulting model.

Challenges and Discussion

The primary challenges encountered included the VLM's limitations in spatial reasoning and the need for iterative refinement to achieve desired outcomes. Misinterpretations in workplane orientations and complex geometrical constraints proved to be recurring issues. The system's current reliance on standard parametric CAD libraries also limits its ability to represent more intricate operations, such as bending or morphing.

Conclusion and Future Directions

The introduction of a VLM-based MAS for CAD represents a significant advancement in automating design processes. However, further enhancements are needed to address spatial reasoning limitations and integrate more advanced CAD operations. Future research directions include fine-tuning VLMs for better spatial awareness and integrating more complex design features into the MAS framework.

In summary, this system showcases the potential of AI-driven multi-agent systems in transforming traditional CAD workflows, paving the way for broader accessibility and efficiency in the design process.