Papers
Topics
Authors
Recent
Search
2000 character limit reached

Designing with Language: Wireframing UI Design Intent with Generative Large Language Models

Published 12 Dec 2023 in cs.HC | (2312.07755v1)

Abstract: Wireframing is a critical step in the UI design process. Mid-fidelity wireframes offer more impactful and engaging visuals compared to low-fidelity versions. However, their creation can be time-consuming and labor-intensive, requiring the addition of actual content and semantic icons. In this paper, we introduce a novel solution WireGen, to automatically generate mid-fidelity wireframes with just a brief design intent description using the generative LLMs. Our experiments demonstrate the effectiveness of WireGen in producing 77.5% significantly better wireframes, outperforming two widely-used in-context learning baselines. A user study with 5 designers further validates its real-world usefulness, highlighting its potential value to enhance UI design process.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (84)
  1. 2023a. Absolute, Relative, Fixed Positioning: How Do They Differ? https://css-tricks.com/absolute-relative-fixed-positioining-how-do-they-differ/.
  2. 2023b. Adobe Photoshop — Photo and design software. https://www.adobe.com/products/photoshop.html.
  3. 2023c. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt/.
  4. 2023d. DALLE 2, Explained: The Promise and Limitations of a Revolutionary AI. https://towardsdatascience.com/dall-e-2-explained-the-promise-and-limitations-of-a-revolutionary-ai-3faf691be220.
  5. 2023e. Dribbble. https://dribbble.com/.
  6. 2023f. Figma: the collaborative interface design tool. https://www.figma.com/.
  7. 2023g. Fine-tuning from OpenAI. https://platform.openai.com/docs/guides/fine-tuning.
  8. 2023h. Midjourney. https://www.midjourney.com.
  9. 2023i. Online Mockup, Wireframe & UI Prototyping Tool Moqups. https://moqups.com/.
  10. 2023j. Sketch: Design, collaborate, prototype and handoff. https://www.sketch.com/.
  11. Towards conscientiousness-based graphical user interface design guidelines. Personal and Ubiquitous Computing (2022), 1–14.
  12. EmoBalloon-Conveying Emotional Arousal in Text Chats with Speech Balloons. In CHI Conference on Human Factors in Computing Systems. 1–16.
  13. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13642–13652.
  14. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  15. Vins: Visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
  16. From lost to found: Discover missing ui design semantics through recovering missing tags. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–22.
  17. Gallery dc: Design search and knowledge discovery through auto-created gui component gallery. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–22.
  18. Wireframe-based UI design search through image autoencoder. ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 3 (2020), 1–31.
  19. Unveiling the Tricks: Automated Detection of Dark Patterns in Mobile Applications. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–20.
  20. Towards Complete Icon Labeling in Mobile Applications. In CHI Conference on Human Factors in Computing Systems. 1–14.
  21. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
  22. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).
  23. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022).
  24. Ian G Clifton. 2015. Android User Interface Design: Implementing Material Design for Developers. Addison-Wesley Professional.
  25. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 845–854.
  26. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  27. Sidong Feng and Chunyang Chen. 2022. GIFdroid: an automated light-weight tool for replaying visual bug reports. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 95–99.
  28. Sidong Feng and Chunyang Chen. 2023. Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models. arXiv preprint arXiv:2306.01987 (2023).
  29. Gallery DC: Auto-created GUI component gallery for design search and knowledge discovery. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 80–84.
  30. Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial Videos. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–15.
  31. Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs in UI Development. ACM Transactions on Interactive Intelligent Systems 12, 4 (2022), 1–26.
  32. Auto-icon: An automated code generation tool for icon designs assisting in ui development. In 26th International Conference on Intelligent User Interfaces. 59–69.
  33. Efficiency Matters: Speeding Up Automated Testing with GUI Rendering Inference. arXiv preprint arXiv:2212.05203 (2022).
  34. Read It, Don’t Watch It: Captioning Bug Recordings Automatically. arXiv preprint arXiv:2302.00886 (2023).
  35. Wilbert O Galitz. 2007. The essential guide to user interface design: an introduction to GUI design principles and techniques. John Wiley & Sons.
  36. Gabriela Goldschmidt. 1991. The dialectics of sketching. Creativity research journal 4, 2 (1991), 123–143.
  37. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139–144.
  38. Google. 2023. Material Icons. https://material.io/resources/icons/.
  39. Layouttransformer: Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1004–1014.
  40. Getting inspired! Understanding how and why examples are used in creative design practice. In Proceedings of the SIGCHI conference on human factors in computing systems. 87–96.
  41. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851.
  42. Swire: Sketch-based user interface retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–10.
  43. Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models. arXiv preprint arXiv:2110.07775 (2021).
  44. Sketch-Based Creativity Support Tools Using Deep Learning. In Artificial Intelligence for Human Computer Interaction: A Modern Approach. Springer, 379–415.
  45. IBM. 2023. Design Language. https://www.ibm.com/design/language/iconography/ui-icons/library.
  46. Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data (TKDD) 6, 4 (2012), 1–21.
  47. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
  48. James A Landay and Brad A Myers. 2001. Sketching interfaces: Toward more human interface design. Computer 34, 3 (2001), 56–64.
  49. Bryan Lawson. 2006. How designers think. Routledge.
  50. Deep learning. nature 521, 7553 (2015), 436–444.
  51. Enrico: A dataset for topic modeling of mobile UI designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–4.
  52. Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint arXiv:1901.06767 (2019).
  53. Screen2vec: Semantic embedding of gui screens and gui components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
  54. DENIM: Finding a tighter fit between tools and practice for web site design. In Proceedings of the SIGCHI conference on Human factors in computing systems. 510–517.
  55. GPT understands, too. arXiv preprint arXiv:2103.10385 (2021).
  56. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  57. Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7.
  58. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282.
  59. On the Importance of Data Size in Probing Fine-tuned Models. arXiv preprint arXiv:2203.09627 (2022).
  60. GPT-3 models are poor few-shot learners in the biomedical domain. arXiv preprint arXiv:2109.02555 (2021).
  61. Theresa Neil. 2014. Mobile design pattern gallery: UI patterns for smartphone apps. ” O’Reilly Media, Inc.”.
  62. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022).
  63. Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion. 79–80.
  64. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.
  65. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022).
  66. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–7.
  67. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10684–10695.
  68. LoFi Sketch: A Large Scale Dataset of Smartphone Low Fidelity Sketches. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–5.
  69. UISketch: a large-scale dataset of UI element sketches. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
  70. Donna Spencer. 2009. Card sorting: Designing usable categories. Rosenfeld Media.
  71. ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In 26th International Conference on Intelligent User Interfaces. 48–58.
  72. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  73. Attention is all you need. Advances in neural information processing systems 30 (2017).
  74. Enabling Conversational Interaction with Mobile UI using Large Language Models. arXiv preprint arXiv:2209.08655 (2022).
  75. Screen2words: Automatic mobile UI summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. 498–510.
  76. SketchingInterfaces: A Tool for Automatically Generating High-Fidelity User Interface Mockups from Hand-Drawn Sketches. In Proceedings of the 32nd Australian Conference on Human-Computer Interaction. 538–545.
  77. Exploring Designers’ Practice of Online Example Management for Supporting Mobile UI Design. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction. 1–12.
  78. UIED: a hybrid tool for GUI element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1655–1659.
  79. Psychologically-inspired, unsupervised inference of perceptual groups of GUI widgets from GUI images. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 332–343.
  80. Automatic generation of visual-textual presentation layout. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12, 2 (2016), 1–22.
  81. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 3081–3089.
  82. Min Zhang and Juntao Li. 2021. A commentary of GPT-3 in MIT Technology Review 2021. Fundamental Research 1, 6 (2021), 831–833.
  83. Mingyu Zong and Bhaskar Krishnamachari. 2022a. a survey on GPT-3. arXiv preprint arXiv:2212.00857 (2022).
  84. Mingyu Zong and Bhaskar Krishnamachari. 2022b. Solving math word problems concerning systems of equations with gpt-3. In Proceedings of the Thirteenth AAAI Symposium on Educational Advances in Artificial Intelligence.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up