Skip to main navigation Skip to search Skip to main content

Human-AI Synergy in Survey Development: Implications from Large Language Models in Business and Research

Research output: Journal article publication › Journal article › Academic research › peer-review

Abstract

This study examines the novel integration of Large Language Models (LLMs) into the survey development process in business and research through the development and evaluation of the Behavioral Research Assistant (BRASS) Bot. We first analyzed the traditional scale development process to identify tasks suitable for LLM integration, including both human-in-the-loop and automated LLM data collection methods. Following this analysis, we developed the details of BRASS Bot, incorporating design principles of falsifiability and reproducibility. We then conducted a comprehensive evaluation of the BRASS Bot across a diverse set of LLMs, including GPT, Claude, Gemini, and Llama, to assess its usability, validity, and reliability. We further demonstrated the practical utility of the BRASS Bot by conducting a user study and a predictive validity simulation. Our research presents both theoretical and practical implications. The augmentation approach of the BRASS Bot enriches the theoretical foundations of behavioral constructs by identifying previously overlooked patterns. Additionally, the BRASS Bot offers significant time and resource efficiency gains while enhancing scale validity. Our work lays the foundation for future research on the broader application of LLMs as both assistants and collaborators in survey analysis and behavioral research design and execution, highlighting their potential for a transformative impact on the field.

Original languageEnglish
Article numberART9
JournalACM Transactions on Management Information Systems
Volume16
Issue number1
DOIs
Publication statusPublished - 8 Feb 2025

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

  1. SDG 8 - Decent Work and Economic Growth
    SDG 8 Decent Work and Economic Growth
  2. SDG 12 - Responsible Consumption and Production
    SDG 12 Responsible Consumption and Production

Keywords

  • behavioral research
  • generative AI
  • Large Language Model
  • scale development

ASJC Scopus subject areas

  • Management Information Systems
  • General Computer Science

Fingerprint

Dive into the research topics of 'Human-AI Synergy in Survey Development: Implications from Large Language Models in Business and Research'. Together they form a unique fingerprint.

Cite this