#201,963 | AsPredicted (original) (raw)

'Human-AI Collaboration & Creative Engagement (Additional Study)'

AsPredicted #: 201,963

Author(s)
Suqing Wu (Zhejiang University) - s.wu@zju.edu.cn
Yukun Liu (Zhejiang University) - liuyk@shanghaitech.edu.cn
Mengqi Ruan (Zhejiang University) - ruanmengqi@zju.edu.cn
Siyu Chen (Zhejiang University) - siyu_chen@zju.edu.cn
Xiao-Yun Xie (School of Management, Zhejiang University) - xiexy@zju.edu.cn

Pre-registered on
2024/11/28 22:11 (PT)

1) Have any data been collected for this study already?
No, no data have been collected for this study yet.

2) What's the main question being asked or hypothesis being tested in this study?
The primary focus of this study is to examine the impact of GenAI's involvement in text generated tasks on human workers' psychological experiences and performance.

Specifically in two consecutive tasks, we investigate the impact of ChatGPT's involvement in text-based tasks on human workers' psychological experiences (sense of control, feelings of boredom, and intrinsic motivation). Also, we examine the effect of ChatGPT's involvement on human workers' current and later-on performance.

We design four combinations that involves human-GPT collaboration and human solo work in two consecutive tasks:

  1. AI to Solo group: transitioning from human-GPT collaboration in Task 1 to human working alone in Task 2.
  2. Solo to AI group: transitioning from human working alone in Task 1 to human-GPT collaboration in Task 2.
  3. AI to AI group: remaining human-GPT collaboration in both Task 1 and Task 2.
  4. Solo to Solo group: remaining human working alone in both Task 1 and Task2.

We hypothesize that:

  1. Participants who collaborate with ChatGPT in Task 1 will show better performance in Task 1, compared to those who work without assistance from ChatGPT in Task 1.
  2. Participants who collaborate with ChatGPT in Task 1 will perform better in Task 2, compared to those who work without assistance in Task 1.
  3. Participants from AI-to-Solo group will show greater regained sense of control, reduced intrinsic motivation, and increased boredom compared to those from Solo-to-Solo group and compared to those from AI-to-AI group.

3) Describe the key dependent variable(s) specifying how they will be measured.
Task Performance: Task performance will be assessed using one of the following evaluation approaches, depending on feasibility in terms of time and budget: (1) human ratings, (2) automated ratings through the Linguistic Inquiry and Word Count (LIWC) software, or (3) AI-based evaluations using a self-developed rating algorithm powered by ChatGPT. The final selection of the evaluation approach will aim to balance accuracy, reliability, and resource constraints.
Perceived Sense of Control: Measured with a three-item scale (Greenaway et al., 2015).
Boredom: Measured with a four-item scale (van Hooft & van Hooff, 2018).
Intrinsic Motivation: Measured with a four-item scale (Shin & Grant, 2019).

4) How many and which conditions will participants be assigned to?
In this study, participants will be assigned to one of four groups:

  1. AI to Solo group: transitioning from human-GPT collaboration in Task 1 to human working alone in Task 2.
  2. Solo to AI group: transitioning from human working alone in Task 1 to human-GPT collaboration in Task 2.
  3. AI to AI group: remaining human-GPT collaboration in both Task 1 and Task 2.
  4. Solo to Solo group: remaining human working alone in both Task 1 and Task2.

To counterbalance the potential effect of task type on key dependent variables, we design two tasks that will be randomly presented to participants as Task 1 or Task 2: a task that requires them to compose a welcome email task, and a task that requires participants to write a Facebook post task.

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
We will primarily employ independent samples t-tests, mixed ANOVA, and regression analyses to examine the effects and relationships of interest in our study. These methods will allow us to test group differences, explore interactions between factors, and analyze the impact of various predictors on the dependent variables.

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
We will use an attention check to identify participants whose data are less reliable and thus should not be included in analyses.

7) How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined.
We have determined that "d = 0.20" represents the smallest effect size of theoretical interest for our study. Therefore, using this effect size, a desired statistical power of 80%, and a significance level of α = 0.05, we calculated that we would need 393 participants per group, for a total sample size of 1,572 participants. This sample size ensures we have sufficient statistical power to detect a small but meaningful effect, balancing the risk of Type I and Type II errors while maintaining rigorous methodological standards.

8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)
We will measure task load (Reid & Nygren, 1988) and task engagement (Rich, LePine & Crawford, 2010) after each task respectively for explorative purpose.

Version of AsPredicted Questions: 2.00