Carol Forsyth - Academia.edu (original) (raw)
Papers by Carol Forsyth
Educational Data Mining, Aug 26, 2021
Lecture Notes in Computer Science, 2015
The current study investigated teacher emotions, student emotions, and discourse features in rela... more The current study investigated teacher emotions, student emotions, and discourse features in relation to learning in a serious game. The experiment consisted of 48 subjects participating in a 4-condition within-subjects counter-balanced pretest-interaction-posttest design. Participants interacted with a serious game teaching research methodology with natural language conversations between the human student and two artificial pedagogical agents. The discourse of the artificial pedagogical agents was manipulated to evoke student affective states. Student emotion was measured via affect grids and discourse features were measured with computational linguistics techniques. Results indicated that learner’s arousal levels impacted learning and that language use is correlated with learning.
Computers in Human Behavior, Mar 1, 2020
Collaborative problem solving (CPS) is a complex construct comprised of skills associated with so... more Collaborative problem solving (CPS) is a complex construct comprised of skills associated with social and cognitive dimensions. The diverse set of skills within these dimensions make CPS difficult to measure. Typically, research on measuring CPS has used highly constrained environments that help narrow the problem space. In the current study, we applied the in-task assessment framework to support the exploration of CPS skills at a deep level in an open digital environment in which three students worked together to solve an electronics problem. The construct of CPS was defined in depth prior to the implementation of the environment through the development of a complex, hierarchical ontology. The features from the ontology were identified in the data and four theoretically-grounded profiles of types of collaborative problemsolvers were produced-high social/high cognitive, high social/low cognitive, low social/high cognitive, and low social/low cognitive. Results showed that students in the low social/low cognitive profile group demonstrated poorer performance than students in other profile groups. Further, having at least one high social/high cognitive member in a team facilitated performance. This study offers groundwork for future studies in measuring CPS with an approach suitable for less constrained collaborative environments.
Educational Data Mining, Jul 1, 2020
Collaborative problem solving (CPS) is considered a necessary skill for students and workers in t... more Collaborative problem solving (CPS) is considered a necessary skill for students and workers in the 21 st century as the advent of technology requires more and more people to frequently work in teams. In the current study, we employed theoretically-grounded data mining techniques to identify four profiles of collaborative problem solvers interacting with an online electronics task. The profiles were created based on 11 theoretically-grounded CPS skills defined a priori. The resulting four profiles correlated in expected directions with in-task performance and had interesting relationships with external measures associated with prior knowledge and CPS skills. These results inform and partially replicate findings from our previous research using a similar approach on a smaller dataset. Implications and comparisons between the two studies will be discussed.
This study relates collaborative problem solving (CPS) behavior and background characteristics of... more This study relates collaborative problem solving (CPS) behavior and background characteristics of three-person student teams completing tasks in an online electronics environment to task performance. Task performance was primarily predicted by classroom membership and minimally impacted by CPS communication types and group dynamics. The online environment and process data measuring CPS behavior substantially add to the field.
Zenodo (CERN European Organization for Nuclear Research), Jul 18, 2022
Conversation-based assessment systems allow for students to display evidence of their knowledge d... more Conversation-based assessment systems allow for students to display evidence of their knowledge during natural language conversations with artificial agents. In the current study, 235 middle-school students from diverse backgrounds interacted with a conversation-based assessment system designed to measure scientific inquiry. There were two versions of the conversations where the initial question was manipulated to examine the relationship between question-framing and conversational discourse. We analyzed the human input during these conversations post-hoc using LIWC to discover linguistic profiles of students that may be related to the type of question asked as well as overall task performance. Furthermore, we compared these linguistic profiles to human ratings as a validity check and to inform our interpretation. Results indicated four separate profiles determined by linguistic features that indeed align to human scores and performance in directions consistent with the effects of question framing. These results offer important implications for improved detection of types of student learners based on linguistic features that do not differ by diverse student characteristics and for designing conversation-based assessments.
Educational Data Mining, Jul 1, 2018
In this paper, we describe a theoretically-grounded data mining approach to identify types of col... more In this paper, we describe a theoretically-grounded data mining approach to identify types of collaborative problem solvers based on students' interactions with an online simulation-based task about electronics concepts. In our approach, we developed an ontology to identify the theoretically-grounded features of collaborative problem solving (CPS). After interaction with the task, students' log files were tagged for the presence of 11 CPS skills from the ontology. The frequencies of the skills were clustered to identify four unique profiles of collaborative problem solvers-Chatty Doers, Social Loafers, Group Organizers, and Active Collaborators. Relationships among cluster membership, task performance, and external ratings of collaboration provide initial validity evidence that these are meaningful profiles of collaborative problem solvers.
Educational research and innovation, Apr 11, 2017
Conversational agents have been developed to help students learn in computerlearning environments... more Conversational agents have been developed to help students learn in computerlearning environments with collaborative reasoning and problem solving. Conversational agents were used in the 2015 Programme for International Student Assessment (PISA) collaborative problem-solving assessments, where a human interacted with one, two or three agents. This chapter reviews advances in conversational agents and how they can help students learn by engaging in collaborative reasoning and problem solving. Using the example of AutoTutor it demonstrates how dialogues can mimic the approaches of expert human tutors. Conversations with intelligent systems are quite different depending on the number of agents involved. A human interacting with only one computer agent during a dialogue needs to continuously participate in the exchange. In trialogues there are two agents, so there are more options available to the human (including social loafing and vicarious observation) and the conversation patterns can be more complex, illustrated by Operation ARIES!, which uses a number of patterns in teaching students the basics of research methodology. Conversational agent systems use online, continuous, formative assessment of human abilities, achievements, and psychological states, tracked during the course of the conversations. Some of these formative assessment approaches are incorporated in the PISA 2015 assessment of collaborative problem solving. However, this chapter focuses on formative assessment in learning environments rather than on summative assessments.
Technology, Knowledge, and Learning, Aug 26, 2019
The current study investigated predictors of shallow versus deep learning within a serious game k... more The current study investigated predictors of shallow versus deep learning within a serious game known as Operation ARA. This game uses a myriad of pedagogical features including multiple-choice tests, adaptive natural language tutorial conversations, case-based reasoning, and an E-text to engage students. The game teaches 11 topics in research methodology across three distinct modules that target factual information, application of reasoning to specific cases, and question generation. The goal of this investigation is to discover predictors of deep and shallow learning by blending Evidence-Centered Design (ECD) with educational data mining. In line with ECD, time-honored cognitive processes or behaviors of time-on-task, discrimination, generation, and scaffolding were selected because there is a large research history supporting their importance to learning. The study included 192 college students who participated in a pretest-interaction-posttest design. These data were used to discover the best predictors of learning across the training experiences. Results revealed distinctly different patterns of predictors of deep versus shallow learning for students across the training environments of the game. Specifically, more interactivity is important for environments contributing to shallow learning whereas generation and discrimination is more important in training environments supporting deeper learning. However, in some training environments the positive impact of generation may be at the price of decreased discrimination.
Current Directions in Psychological Science, Oct 1, 2014
Learning is facilitated by conversational interactions both with human tutors and with computer a... more Learning is facilitated by conversational interactions both with human tutors and with computer agents that simulate human tutoring and ideal pedagogical strategies. In this article, we describe some intelligent tutoring systems (e.g., AutoTutor) in which agents interact with students in natural language while being sensitive to their cognitive and emotional states. These systems include one-on-one tutorial dialogues, conversational trialogues in which two agents (a tutor and a "peer") interact with a human student, and other conversational ensembles in which agents take on different roles. Tutorial conversations with agents have also been incorporated into educational games. These learning environments have been developed for different populations (elementary through high school students, college students, adults with reading difficulties) and different subjects spanning science, technology, engineering, mathematics, reading, writing, and reasoning. This article identifies some of the conversation patterns that are implemented in the dialogues and trialogues.
Elsevier eBooks, 2023
by the primary investigator through observation of the students during PBL sessions. Duration of ... more by the primary investigator through observation of the students during PBL sessions. Duration of study was six months. Results: We found that in our students, development of social dimension skills is facilitated to a greater extent than the development of cognitive dimension skills through the process of PBL. These skills are generally better developed in the leader compared to the scribe and members in a group. They are also more developed in females compared to males. Modification in them is also observed as the year's progress. Conclusion: Although PBLs facilitate development of CPS skills' progression however in our curriculum, PBLs mainly focus on social skills development and have less emphasis on cognitive skill development. Thus, hybrid instructional strategies with components from TBL and mentorship are recommended for better development of CPS skills.
Journal of Intelligence
Competency in skills associated with collaborative problem solving (CPS) is critical for many con... more Competency in skills associated with collaborative problem solving (CPS) is critical for many contexts, including school, the workplace, and the military. Innovative approaches for assessing individuals’ CPS competency are necessary, as traditional assessment types such as multiple -choice items are not well suited for such a process-oriented competency. In a move to computer-based environments to support CPS assessment, innovative computational approaches are also needed to understand individuals’ CPS behaviors. In the current study, we describe the use of a simulation-based task on electronics concepts as an environment for higher education students to display evidence of their CPS competency. We further describe computational linguistic methods for automatically characterizing students’ display of various CPS skills in the task. Comparisons between such an automated approach and an approach based on human annotation to characterize student CPS behaviors revealed above average agr...
Many adult workers need to keep up with advances in technology to remain relevant in the job mark... more Many adult workers need to keep up with advances in technology to remain relevant in the job market. Adults now need 21st century skills including Computational Thinking. It is challenging for adults to find training opportunities that take into account their limited time, educational, and resource constraints. Our approach provides support to adult learners and their tutors to help them reach their goals. This support can take the form of facilitation messages that suggest possible learning activities, hints on study and time management for learners, and instructional suggestions and alerts for tutors based on current information gleaned from the learner and the interaction. We have collected data with adult learners and their tutors, and are designing an automated facilitator system that can provide tutors and learners with feedback according to their needs. CCS Concepts CCS → Applied computing → Education
In this paper, we describe a theoretically-grounded data mining approach to identify types of col... more In this paper, we describe a theoretically-grounded data mining approach to identify types of collaborative problem solvers based on students’ interactions with an online simulation-based task about electronics concepts. In our approach, we developed an ontology to identify the theoretically-grounded features of collaborative problem solving (CPS). After interaction with the task, students’ log files were tagged for the presence of 11 CPS skills from the ontology. The frequencies of the skills were clustered to identify four unique profiles of collaborative problem solvers – Chatty Doers, Social Loafers, Group Organizers, and Active Collaborators. Relationships among cluster membership, task performance, and external ratings of collaboration provide initial validity evidence that these are meaningful profiles of collaborative
Lecture Notes in Computer Science, 2017
Definición de mecánicas de juego a partir de la evaluación de técnicas centradas en la experienci... more Definición de mecánicas de juego a partir de la evaluación de técnicas centradas en la experiencia de usuario Defining game mechanics based on the assessment of user experience techniques
Educational Data Mining, Aug 26, 2021
Lecture Notes in Computer Science, 2015
The current study investigated teacher emotions, student emotions, and discourse features in rela... more The current study investigated teacher emotions, student emotions, and discourse features in relation to learning in a serious game. The experiment consisted of 48 subjects participating in a 4-condition within-subjects counter-balanced pretest-interaction-posttest design. Participants interacted with a serious game teaching research methodology with natural language conversations between the human student and two artificial pedagogical agents. The discourse of the artificial pedagogical agents was manipulated to evoke student affective states. Student emotion was measured via affect grids and discourse features were measured with computational linguistics techniques. Results indicated that learner’s arousal levels impacted learning and that language use is correlated with learning.
Computers in Human Behavior, Mar 1, 2020
Collaborative problem solving (CPS) is a complex construct comprised of skills associated with so... more Collaborative problem solving (CPS) is a complex construct comprised of skills associated with social and cognitive dimensions. The diverse set of skills within these dimensions make CPS difficult to measure. Typically, research on measuring CPS has used highly constrained environments that help narrow the problem space. In the current study, we applied the in-task assessment framework to support the exploration of CPS skills at a deep level in an open digital environment in which three students worked together to solve an electronics problem. The construct of CPS was defined in depth prior to the implementation of the environment through the development of a complex, hierarchical ontology. The features from the ontology were identified in the data and four theoretically-grounded profiles of types of collaborative problemsolvers were produced-high social/high cognitive, high social/low cognitive, low social/high cognitive, and low social/low cognitive. Results showed that students in the low social/low cognitive profile group demonstrated poorer performance than students in other profile groups. Further, having at least one high social/high cognitive member in a team facilitated performance. This study offers groundwork for future studies in measuring CPS with an approach suitable for less constrained collaborative environments.
Educational Data Mining, Jul 1, 2020
Collaborative problem solving (CPS) is considered a necessary skill for students and workers in t... more Collaborative problem solving (CPS) is considered a necessary skill for students and workers in the 21 st century as the advent of technology requires more and more people to frequently work in teams. In the current study, we employed theoretically-grounded data mining techniques to identify four profiles of collaborative problem solvers interacting with an online electronics task. The profiles were created based on 11 theoretically-grounded CPS skills defined a priori. The resulting four profiles correlated in expected directions with in-task performance and had interesting relationships with external measures associated with prior knowledge and CPS skills. These results inform and partially replicate findings from our previous research using a similar approach on a smaller dataset. Implications and comparisons between the two studies will be discussed.
This study relates collaborative problem solving (CPS) behavior and background characteristics of... more This study relates collaborative problem solving (CPS) behavior and background characteristics of three-person student teams completing tasks in an online electronics environment to task performance. Task performance was primarily predicted by classroom membership and minimally impacted by CPS communication types and group dynamics. The online environment and process data measuring CPS behavior substantially add to the field.
Zenodo (CERN European Organization for Nuclear Research), Jul 18, 2022
Conversation-based assessment systems allow for students to display evidence of their knowledge d... more Conversation-based assessment systems allow for students to display evidence of their knowledge during natural language conversations with artificial agents. In the current study, 235 middle-school students from diverse backgrounds interacted with a conversation-based assessment system designed to measure scientific inquiry. There were two versions of the conversations where the initial question was manipulated to examine the relationship between question-framing and conversational discourse. We analyzed the human input during these conversations post-hoc using LIWC to discover linguistic profiles of students that may be related to the type of question asked as well as overall task performance. Furthermore, we compared these linguistic profiles to human ratings as a validity check and to inform our interpretation. Results indicated four separate profiles determined by linguistic features that indeed align to human scores and performance in directions consistent with the effects of question framing. These results offer important implications for improved detection of types of student learners based on linguistic features that do not differ by diverse student characteristics and for designing conversation-based assessments.
Educational Data Mining, Jul 1, 2018
In this paper, we describe a theoretically-grounded data mining approach to identify types of col... more In this paper, we describe a theoretically-grounded data mining approach to identify types of collaborative problem solvers based on students' interactions with an online simulation-based task about electronics concepts. In our approach, we developed an ontology to identify the theoretically-grounded features of collaborative problem solving (CPS). After interaction with the task, students' log files were tagged for the presence of 11 CPS skills from the ontology. The frequencies of the skills were clustered to identify four unique profiles of collaborative problem solvers-Chatty Doers, Social Loafers, Group Organizers, and Active Collaborators. Relationships among cluster membership, task performance, and external ratings of collaboration provide initial validity evidence that these are meaningful profiles of collaborative problem solvers.
Educational research and innovation, Apr 11, 2017
Conversational agents have been developed to help students learn in computerlearning environments... more Conversational agents have been developed to help students learn in computerlearning environments with collaborative reasoning and problem solving. Conversational agents were used in the 2015 Programme for International Student Assessment (PISA) collaborative problem-solving assessments, where a human interacted with one, two or three agents. This chapter reviews advances in conversational agents and how they can help students learn by engaging in collaborative reasoning and problem solving. Using the example of AutoTutor it demonstrates how dialogues can mimic the approaches of expert human tutors. Conversations with intelligent systems are quite different depending on the number of agents involved. A human interacting with only one computer agent during a dialogue needs to continuously participate in the exchange. In trialogues there are two agents, so there are more options available to the human (including social loafing and vicarious observation) and the conversation patterns can be more complex, illustrated by Operation ARIES!, which uses a number of patterns in teaching students the basics of research methodology. Conversational agent systems use online, continuous, formative assessment of human abilities, achievements, and psychological states, tracked during the course of the conversations. Some of these formative assessment approaches are incorporated in the PISA 2015 assessment of collaborative problem solving. However, this chapter focuses on formative assessment in learning environments rather than on summative assessments.
Technology, Knowledge, and Learning, Aug 26, 2019
The current study investigated predictors of shallow versus deep learning within a serious game k... more The current study investigated predictors of shallow versus deep learning within a serious game known as Operation ARA. This game uses a myriad of pedagogical features including multiple-choice tests, adaptive natural language tutorial conversations, case-based reasoning, and an E-text to engage students. The game teaches 11 topics in research methodology across three distinct modules that target factual information, application of reasoning to specific cases, and question generation. The goal of this investigation is to discover predictors of deep and shallow learning by blending Evidence-Centered Design (ECD) with educational data mining. In line with ECD, time-honored cognitive processes or behaviors of time-on-task, discrimination, generation, and scaffolding were selected because there is a large research history supporting their importance to learning. The study included 192 college students who participated in a pretest-interaction-posttest design. These data were used to discover the best predictors of learning across the training experiences. Results revealed distinctly different patterns of predictors of deep versus shallow learning for students across the training environments of the game. Specifically, more interactivity is important for environments contributing to shallow learning whereas generation and discrimination is more important in training environments supporting deeper learning. However, in some training environments the positive impact of generation may be at the price of decreased discrimination.
Current Directions in Psychological Science, Oct 1, 2014
Learning is facilitated by conversational interactions both with human tutors and with computer a... more Learning is facilitated by conversational interactions both with human tutors and with computer agents that simulate human tutoring and ideal pedagogical strategies. In this article, we describe some intelligent tutoring systems (e.g., AutoTutor) in which agents interact with students in natural language while being sensitive to their cognitive and emotional states. These systems include one-on-one tutorial dialogues, conversational trialogues in which two agents (a tutor and a "peer") interact with a human student, and other conversational ensembles in which agents take on different roles. Tutorial conversations with agents have also been incorporated into educational games. These learning environments have been developed for different populations (elementary through high school students, college students, adults with reading difficulties) and different subjects spanning science, technology, engineering, mathematics, reading, writing, and reasoning. This article identifies some of the conversation patterns that are implemented in the dialogues and trialogues.
Elsevier eBooks, 2023
by the primary investigator through observation of the students during PBL sessions. Duration of ... more by the primary investigator through observation of the students during PBL sessions. Duration of study was six months. Results: We found that in our students, development of social dimension skills is facilitated to a greater extent than the development of cognitive dimension skills through the process of PBL. These skills are generally better developed in the leader compared to the scribe and members in a group. They are also more developed in females compared to males. Modification in them is also observed as the year's progress. Conclusion: Although PBLs facilitate development of CPS skills' progression however in our curriculum, PBLs mainly focus on social skills development and have less emphasis on cognitive skill development. Thus, hybrid instructional strategies with components from TBL and mentorship are recommended for better development of CPS skills.
Journal of Intelligence
Competency in skills associated with collaborative problem solving (CPS) is critical for many con... more Competency in skills associated with collaborative problem solving (CPS) is critical for many contexts, including school, the workplace, and the military. Innovative approaches for assessing individuals’ CPS competency are necessary, as traditional assessment types such as multiple -choice items are not well suited for such a process-oriented competency. In a move to computer-based environments to support CPS assessment, innovative computational approaches are also needed to understand individuals’ CPS behaviors. In the current study, we describe the use of a simulation-based task on electronics concepts as an environment for higher education students to display evidence of their CPS competency. We further describe computational linguistic methods for automatically characterizing students’ display of various CPS skills in the task. Comparisons between such an automated approach and an approach based on human annotation to characterize student CPS behaviors revealed above average agr...
Many adult workers need to keep up with advances in technology to remain relevant in the job mark... more Many adult workers need to keep up with advances in technology to remain relevant in the job market. Adults now need 21st century skills including Computational Thinking. It is challenging for adults to find training opportunities that take into account their limited time, educational, and resource constraints. Our approach provides support to adult learners and their tutors to help them reach their goals. This support can take the form of facilitation messages that suggest possible learning activities, hints on study and time management for learners, and instructional suggestions and alerts for tutors based on current information gleaned from the learner and the interaction. We have collected data with adult learners and their tutors, and are designing an automated facilitator system that can provide tutors and learners with feedback according to their needs. CCS Concepts CCS → Applied computing → Education
In this paper, we describe a theoretically-grounded data mining approach to identify types of col... more In this paper, we describe a theoretically-grounded data mining approach to identify types of collaborative problem solvers based on students’ interactions with an online simulation-based task about electronics concepts. In our approach, we developed an ontology to identify the theoretically-grounded features of collaborative problem solving (CPS). After interaction with the task, students’ log files were tagged for the presence of 11 CPS skills from the ontology. The frequencies of the skills were clustered to identify four unique profiles of collaborative problem solvers – Chatty Doers, Social Loafers, Group Organizers, and Active Collaborators. Relationships among cluster membership, task performance, and external ratings of collaboration provide initial validity evidence that these are meaningful profiles of collaborative
Lecture Notes in Computer Science, 2017
Definición de mecánicas de juego a partir de la evaluación de técnicas centradas en la experienci... more Definición de mecánicas de juego a partir de la evaluación de técnicas centradas en la experiencia de usuario Defining game mechanics based on the assessment of user experience techniques