Designing Human and Generative AI Collaboration

Abstract: We examined the effectiveness of various human-AI collaboration designs on creative work. Through a human subjects experiment set in the context of creative writing, we found that while AI assistance improved productivity across all models, collaboration design significantly influenced output quality, user satisfaction, and content characteristics. Models incorporating human creative input delivered higher content interestingness and overall quality as well as greater task performer satisfaction compared to conditions where humans were limited to confirming AI's output. Increased AI involvement encouraged creators to explore beyond personal experience but also led to lower aggregate diversity in stories and genres among participants. However, this effect was mitigated through human participation in early creative tasks. These findings underscore the importance of preserving the human creative role to ensure quality, satisfaction, and creative diversity in human-AI collaboration.