QueryGenie: Making LLM-Based Database Querying Transparent and Controllable

Abstract: Conversational user interfaces powered by LLMs have significantly lowered the technical barriers to database querying. However, existing tools still encounter several challenges, such as misinterpretation of user intent, generation of hallucinated content, and the absence of effective mechanisms for human feedback-all of which undermine their reliability and practical utility. To address these issues and promote a more transparent and controllable querying experience, we proposed QueryGenie, an interactive system that enables users to monitor, understand, and guide the LLM-driven query generation process. Through incremental reasoning, real-time validation, and responsive interaction mechanisms, users can iteratively refine query logic and ensure alignment with their intent.

• The paper presents QueryGenie’s three-stage framework—Intention Confirmation, Query Generation, and Query Validation—to ensure accurate mapping of natural language to SQL.
• The system employs a chain-of-thought approach to decompose complex queries into interpretable sub-queries, enhancing user understanding and intervention.
• Preliminary evaluations demonstrate a 90.3% query accuracy, outperforming existing tools and significantly strengthening user trust.

QueryGenie: Making LLM-Based Database Querying Transparent and Controllable

The paper "QueryGenie: Making LLM-Based Database Querying Transparent and Controllable" introduces a novel framework aimed at addressing the challenges of opacity and lack of control in LLM-powered database querying systems. QueryGenie enhances the interaction between users and LLMs by implementing a structured, three-module system designed to improve accuracy and user trust in the query generation process.

Introduction

LLM-powered conversational interfaces have democratized access to database querying by enabling users to interact with databases using natural language. However, issues such as intent misinterpretation, hallucinated content generation, and inefficacy in handling user feedback remain significant obstacles to their reliable application. This paper presents QueryGenie, an interactive system that aims to mitigate these challenges by fostering transparent and controllable LLM query generation. The system allows users to monitor and intervene in the query generation process, enhancing both reliability and user trust through a three-stage framework: Intention Confirmation, Query Generation, and Query Validation.

System Framework and Components

QueryGenie's architecture consists of three primary modules, each focusing on a distinct aspect of the querying process:

• Intention Confirmation Module: This module ensures that the user's natural language inputs are correctly mapped to the corresponding database schema elements. This schema linking process involves users reviewing and adjusting mapped entities to improve alignment between user queries and database fields.

  Figure 1: The framework of QueryGenie consists of three key modules: (A) Intention Confirmation Module, (B) Query Generation Module, (C) Query Validation Module.

• Query Generation Module: The core of QueryGenie's innovation lies in its employment of the chain-of-thought (CoT) approach to break down complex queries into smaller, more manageable sub-queries. Each sub-query is transparent and interpretable, providing users with the opportunity to inspect and understand the derivation process.
• Query Validation Module: This module provides a mechanism for real-time validation of SQL statements generated by the LLM. It allows users to execute sub-queries and receive immediate results, facilitating dynamic user feedback and direct SQL statement adjustments when discrepancies occur.

Enhanced User Interface

The user interface of QueryGenie is designed to maximize transparency and user control. It features a database panel where users can connect to their databases and explore their structures visually. This interface supports keyword-based filtering and UML diagram integration for an intuitive understanding of database schemas.

Figure 2: The user interface of QueryGenie.

The conversational interface supports user-LLM interaction by visualizing the schema mappings and reasoning paths. Users have the option to provide corrective feedback to the LLM or manually adjust schema mappings. The interface emphasizes transparency by allowing users to expand intermediate steps of the CoT breakdown and view detailed explanations for each query component.

Evaluation and Performance

Preliminary evaluations indicate that QueryGenie significantly enhances query accuracy compared to existing tools like Vanna. Participants' use of QueryGenie resulted in a performance accuracy of 90.3%, compared to Vanna's 66.7%. While QueryGenie requires more time per task, the trade-off yields higher reliability and insight into LLM reasoning, contributing positively to user confidence and system usability.

Conclusion

QueryGenie represents a step forward in the field of LLM-driven database querying by facilitating a human-AI collaboration paradigm that emphasizes transparency and user control. It addresses fundamental challenges in current LLM applications by integrating visualization and real-time feedback mechanisms. Future work can explore the integration of more sophisticated user guidance systems and the expansion of the framework to accommodate more diverse querying contexts. The implications of this study suggest significant potential for enhancing user trust and effectiveness in LLM-based systems for database interaction.