Papers
Topics
Authors
Recent
Search
2000 character limit reached

BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks

Published 10 Apr 2024 in cs.HC and cs.AI | (2404.07387v3)

Abstract: Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on LLMs. However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into computational notebooks that augments LLM-based code generation with an additional ephemeral UI step, offering users UI scaffolds as an intermediate stage between user prompts and code generation. We present this workflow in BISCUIT, an extension for JupyterLab that provides users with ephemeral UIs generated by LLMs based on the context of their code and intentions, scaffolding users to understand, guide, and explore with LLM-generated code. Through a user study where 10 novices used BISCUIT for machine learning tutorials, we found that BISCUIT offers users representations of code to aid their understanding, reduces the complexity of prompt engineering, and creates a playground for users to explore different variables and iterate on their ideas.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. N. Al Madi, “How readable is model-generated code? examining readability and visual inspection of github copilot,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’22.   New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3551349.3560438
  2. M. Kazemitabaar, J. Chow, C. K. T. Ma, B. J. Ericson, D. Weintrop, and T. Grossman, “Studying the effect of ai code generators on supporting novice learners in introductory programming,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI ’23.   New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3544548.3580919
  3. P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Chi conference on human factors in computing systems extended abstracts, 2022, pp. 1–7.
  4. C. Bird, D. Ford, T. Zimmermann, N. Forsgren, E. Kalliamvakou, T. Lowdermilk, and I. Gazit, “Taking flight with copilot: Early insights and opportunities of ai-powered pair-programming tools,” Queue, vol. 20, no. 6, p. 35–57, jan 2023. [Online]. Available: https://doi.org/10.1145/3582083
  5. R. Wang, R. Cheng, D. Ford, and T. Zimmermann, “Investigating and designing for trust in ai-powered code generation tools,” 2023.
  6. B. Johnson, C. Bird, D. Ford, N. Forsgren, and T. Zimmermann, “Make your tools sparkle with trust: The picse framework for trust in software tools,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2023, pp. 409–419.
  7. P. Vaithilingam, E. L. Glassman, J. P. Inala, and C. Wang, “Dynavis: Dynamically synthesized ui widgets for visualization editing,” arXiv preprint arXiv:2401.10880, 2024.
  8. V. Dibia, “Lida: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models,” arXiv preprint arXiv:2303.02927, 2023.
  9. K. Gajos and D. S. Weld, “Supple: automatically generating user interfaces,” in Proceedings of the 9th international conference on Intelligent user interfaces, 2004, pp. 93–100.
  10. K. Z. Gajos, J. O. Wobbrock, and D. S. Weld, “Automatically generating user interfaces adapted to users’ motor and vision capabilities,” in Proceedings of the 20th annual ACM symposium on User interface software and technology, 2007, pp. 231–240.
  11. J. Nichols, B. A. Myers, M. Higgins, J. Hughes, T. K. Harris, R. Rosenfeld, and K. Litwack, “Personal universal controllers: controlling complex appliances with guis and speech,” in CHI’03 Extended Abstracts on Human Factors in Computing Systems, 2003, pp. 624–625.
  12. J. Nichols, B. A. Myers, and B. Rothrock, “Uniform: automatically generating consistent remote control user interfaces,” in Proceedings of the SIGCHI conference on Human Factors in computing systems, 2006, pp. 611–620.
  13. J. Nichols, B. Rothrock, D. H. Chau, and B. A. Myers, “Huddle: automatically generating interfaces for systems of multiple connected appliances,” in Proceedings of the 19th annual ACM symposium on User interface software and technology, 2006, pp. 279–288.
  14. J. Heer, M. Agrawala, and W. Willett, “Generalized selection via interactive query relaxation,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2008, pp. 959–968.
  15. L. Verou, A. X. Zhang, and D. R. Karger, “Mavo: creating interactive data-driven web applications by authoring html,” in Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 2016, pp. 483–496.
  16. P. Vaithilingam and P. J. Guo, “Bespoke: Interactively synthesizing custom guis from command-line applications by demonstration,” in Proceedings of the 32nd annual ACM symposium on user interface software and technology, 2019, pp. 563–576.
  17. Y. Chen, R. Li, A. Mac, T. Xie, T. Yu, and E. Wu, “Nl2interface: Interactive visualization interface generation from natural language queries,” arXiv preprint arXiv:2209.08834, 2022.
  18. E. L. Hutchins, J. D. Hollan, and D. A. Norman, “Direct manipulation interfaces,” Human–computer interaction, vol. 1, no. 4, pp. 311–338, 1985.
  19. P. J. Guo, “Online python tutor: embeddable web-based program visualization for cs education,” in Proceeding of the 44th ACM Technical Symposium on Computer Science Education, ser. SIGCSE ’13.   New York, NY, USA: Association for Computing Machinery, 2013, p. 579–584. [Online]. Available: https://doi.org/10.1145/2445196.2445368
  20. S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer, “Wrangler: interactive visual specification of data transformation scripts,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’11.   New York, NY, USA: Association for Computing Machinery, 2011, p. 3363–3372. [Online]. Available: https://doi.org/10.1145/1978942.1979444
  21. I. Drosos, T. Barik, P. J. Guo, R. DeLine, and S. Gulwani, “Wrex: A unified programming-by-example interaction for synthesizing readable code for data scientists,” in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ser. CHI ’20.   New York, NY, USA: Association for Computing Machinery, 2020, p. 1–12. [Online]. Available: https://doi.org/10.1145/3313831.3376442
  22. N. Shrestha, T. Barik, and C. Parnin, “Unravel: A fluent code explorer for data wrangling,” in The 34th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 198–207. [Online]. Available: https://doi.org/10.1145/3472749.3474744
  23. M. B. Kery, D. Ren, F. Hohman, D. Moritz, K. Wongsuphasawat, and K. Patel, “mage: Fluid moves between code and graphical work in computational notebooks,” in Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, 2020, pp. 140–151.
  24. A. Head, C. Appachu, M. A. Hearst, and B. Hartmann, “Tutorons: Generating context-relevant, on-demand explanations and demonstrations of online code,” in 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015, pp. 3–12.
  25. L. Yan, A. Hwang, Z. Wu, and A. Head, “Ivie: Lightweight anchored explanations of just-generated code,” arXiv preprint arXiv:2403.02491, 2024.
  26. D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, and B. Myers, “Using an llm to help with code understanding,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE).   Los Alamitos, CA, USA: IEEE Computer Society, apr 2024, pp. 881–881. [Online]. Available: https://doi.ieeecomputersociety.org/
  27. A. M. Mcnutt, C. Wang, R. A. Deline, and S. M. Drucker, “On the design of ai-powered code assistants for notebooks,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI ’23.   New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3544548.3580940
  28. R. Ferreira, M. Canesche, P. Jamieson, O. P. V. Neto, and J. A. Nacif, “Examples and tutorials on using google colab and gradio to create online interactive student-learning modules,” Computer Applications in Engineering Education, p. e22729.
  29. S. Barke, M. B. James, and N. Polikarpova, “Grounded Copilot: How Programmers Interact with Code-Generating Models,” Aug. 2022, arXiv:2206.15000 [cs]. [Online]. Available: http://arxiv.org/abs/2206.15000
  30. E. Jiang, E. Toh, A. Molina, K. Olson, C. Kayacik, A. Donsbach, C. J. Cai, and M. Terry, “Discovering the syntax and strategies of natural language programming with generative language models,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI ’22.   New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491102.3501870
  31. R. Cheng, S. Dasgupta, and B. M. Hill, “How interest-driven content creation shapes opportunities for informal learning in scratch: A case study on novices’ use of data structures,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI ’22.   New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491102.3502124
  32. W. Ni, J. Sunshine, V. Le, S. Gulwani, and T. Barik, “recode: A lightweight find-and-replace interaction in the ide for transforming code by example,” in The 34th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 258–269. [Online]. Available: https://doi.org/10.1145/3472749.3474748
  33. X. Zhang and P. J. Guo, “Mallard: Turn the web into a contextualized prototyping environment for machine learning,” in Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’19.   New York, NY, USA: Association for Computing Machinery, 2019, p. 605–618. [Online]. Available: https://doi.org/10.1145/3332165.3347936
  34. A. C. Ryan Holbrook, “Binary classification,” 2023, [Online; accessed 15-March-2024]. [Online]. Available: https://www.kaggle.com/kernels/fork/11887335
  35. ——, “The convolutional classifier,” 2023, [Online; accessed 15-March-2024]. [Online]. Available: https://www.kaggle.com/kernels/fork/10781907
  36. B. Jovanovic, “Generative ui: The future of dynamic user experiences,” March 2024. [Online]. Available: https://bootcamp.uxdesign.cc/generative-ui-the-future-of-dynamic-user-experiences
  37. K. Moran and S. Gibbons, “Is ui dead? how conversational models are taking over,” March 2024. [Online]. Available: https://www.nngroup.com/articles/generative-ui/
  38. M. Beth Kery and B. A. Myers, “Exploring exploratory programming,” in 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2017, pp. 25–29.
  39. M. B. Kery, M. Radensky, M. Arya, B. E. John, and B. A. Myers, “The story in the notebook: Exploratory data science using a literate programming tool,” in Proceedings of the 2018 CHI conference on human factors in computing systems, 2018, pp. 1–11.
  40. A. Rule, A. Tabard, and J. D. Hollan, “Exploration and explanation in computational notebooks,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–12.
  41. S. Chattopadhyay, I. Prasad, A. Z. Henley, A. Sarma, and T. Barik, “What’s wrong with computational notebooks? pain points, needs, and design opportunities,” in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ser. CHI ’20.   New York, NY, USA: Association for Computing Machinery, 2020, p. 1–12. [Online]. Available: https://doi.org/10.1145/3313831.3376729
  42. M. B. Kery and B. A. Myers, “Interactions for untangling messy history in a computational notebook,” in 2018 IEEE symposium on visual languages and human-centric computing (VL/HCC).   IEEE, 2018, pp. 147–155.
  43. A. Rule, I. Drosos, A. Tabard, and J. D. Hollan, “Aiding collaborative reuse of computational notebooks with annotated cell folding,” Proc. ACM Hum.-Comput. Interact., vol. 2, no. CSCW, nov 2018. [Online]. Available: https://doi.org/10.1145/3274419
  44. M. B. Kery, B. E. John, P. O’Flaherty, A. Horvath, and B. A. Myers, “Towards effective foraging by data scientists to find past analysis choices,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–13.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 16 likes about this paper.

Sign Up for Free