Is AI the new research scientist? Not so, according to a human-led study

The research, detailed in a paper titled "AI and the advent of the cyborg behavioral scientist," tested how well popular generative AI models including OpenAI's ChatGPT, Microsoft's Copilot and Google's Gemini could handle various stages of the research process.

The team put these AI systems through six stages of academic research -- starting with ideation, literature review and research design, followed by documenting results, extending the research and the final manuscript production -- while limiting any human intervention on their part.

What they discovered was a mixed bag of capabilities and limitations, presumably good news for research scientists wondering if AI will take their job.

"A pervasive fear surrounding these AIs is their ability to usurp human labor," explained Geoff Tomaino, an assistant professor in marketing at the University of Florida Warrington College of Business. "In general, we found that these AIs can offer some assistance, but their value stops there, as assistance. These tools can do a great deal of legwork. However, the researcher still has a vital place in the process, acting as a director and critic of the AI, not an equal partner."

Specifically, the researchers found AI to be a useful tool in the ideation process and research design, including the methods and stimulus design substages. In the literature review, results analysis and manuscript production stages, though, it struggled to produce valuable outputs and required substantial oversight.

Based on their findings, the University of Florida team advises researchers to maintain high skepticism toward AI outputs, treating them as starting points that require human verification and refinement. For journals, the researchers strongly recommend considering policies that call out AI assistance in research papers as well as largely prohibiting the use of AI in the research review process.

While their work primarily focused on if AI could do the job of an academic researcher, the team leading this study encourages researchers to reflect on if AI should perform their role.

"We take a great deal of pride in the work we do as researchers," Tomaino said. "The specific steps that bring us joy (and angst) as researchers are likely as varied as the research in which they are used. As these AI tools evolve, it will be up to each individual researcher to decide for which steps of the research process they want to become a cyborg behavioral researcher, and for which they would like to remain simply human."

This research is published in the Journal of Consumer Psychology.