Ido Roll | Technion - Israel Institute of Technology (original) (raw)
Papers by Ido Roll
International Conference of the Learning Sciences, 2010
Abstract Intelligent Tutoring Systems are widely used coached problem-solving environments (Koedi... more Abstract Intelligent Tutoring Systems are widely used coached problem-solving environments (Koedinger, Anderson, Hadley & Mark, 1997; VanLehn, Lynch, Schulze, Shapiro & Shelby, 2005). They are successful, in part, due to their ability to give adaptive feedback (Corbett & Anderson, 2001; Koedinger & Aleven, 2007). More specifically, Intelligent Tutoring Systems adapt to students' behavior and knowledge by tracing students' learning trajectories using a cognitive model of the domain (Corbett & Anderson, 1995). A ...
Abstract. The goal of our research is to investigate whether a Cognitive Tutor can be made more e... more Abstract. The goal of our research is to investigate whether a Cognitive Tutor can be made more effective by extending it to help students acquire help-seeking skills. We present a preliminary model of help-seeking behavior that will provide the basis for a Help-Seeking Tutor Agent. The model, implemented by 57 production rules, captures both productive and unproductive help-seeking behavior. As a first test of the model’s efficacy, we used it off-line to evaluate students ’ help-seeking behavior in an existing data set of student-tutor interactions, We found that 72 % of all student actions represented unproductive help-seeking behavior. Consistent with some of our earlier work (Aleven & Koedinger, 2000) we found a proliferation of hint abuse (e.g., using hints to find answers rather than trying to understand). We also found that students frequently avoided using help when it was likely to be of benefit and often acted in a quick, possibly undeliberate manner. Students’ help-seekin...
Abstract. It has been found in recent years that many students who use intelli-gent tutoring syst... more Abstract. It has been found in recent years that many students who use intelli-gent tutoring systems game the system, attempting to succeed in the educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we introduce a system which gives a gaming student supplementary exercises fo-cused on exactly the material the student bypassed by gaming, and which also expresses negative emotion to gaming students through an animated agent. Stu-dents using this system engage in less gaming, and students who receive many supplemental exercises have considerably better learning than is associated with gaming in the control condition or prior studies. 1
Invention activities are structured tasks in which students create mathematical methods that atte... more Invention activities are structured tasks in which students create mathematical methods that attempt to capture deep properties of data (e.g., variability), prior to receiving instruction on canonical methods (e.g., mean deviation). While experiments have demonstrated the learning benefits of invention activities, the mechanisms of transfer remain unknown. We address this question by evaluating the role of design in invention activities, identifying what knowledge is acquired during invention activities, and how it is applied in transfer tasks. A classroom experiment with 92 students compared the full invention process to one in which students evaluate predesigned methods. Results show that students in the full invention condition acquired more adaptive knowledge, yet not necessarily better procedural knowledge or invention skills. We suggest a mechanism that explains what knowledge invention attempts produce, how that knowledge is productively modified in subsequent instruction, an...
Abstract. Invention activities are Productive Failure activities in which students at-tempt to in... more Abstract. Invention activities are Productive Failure activities in which students at-tempt to invent methods that capture deep properties of given data before being taught expert solutions. The current study evaluates the effect of scaffolding on the invention processes and outcomes, given that students are not expected to succeed in their inquiry and that all students receive subsequent instruction. Two Invention activities related to data analysis concepts were given to 130 undergraduate students in a first-year physics lab course using an interactive learning environment. Students in the Guided Invention condition were given prompts to analyze given data prior to inventing and reflect on their methods after inventing them. These students outperformed Unguided Invention students on delayed measures of transfer, but not on measures of conceptual or proce-dural knowledge. In addition, Guided Invention students were more likely to invent multiple methods, suggesting that they used b...
Intelligent tutoring systems help students acquire cognitive skills by tracing students' kno... more Intelligent tutoring systems help students acquire cognitive skills by tracing students' knowledge and providing relevant feedback. However,
Control of Variables Strategy (CVS) is the process of isolating the effect of single variables wh... more Control of Variables Strategy (CVS) is the process of isolating the effect of single variables when conducting scientific inquiry. We assess how CVS can help student achieve different levels of understanding when implemented in different parts of the inquiry process. 148 students worked with minimally-guided inquiry activities using virtual labs on two different physics topics. The virtual labs allowed for exploration, data collection, and graphical analysis. Using student log data, we identified how CVS manifests itself through these phases of students’ inquiry process. We found that students using CVS during data collection and plotting was associated with students achieving more qualitative and quantitative models, respectively. This did not hold, however, for more complicated mathematical relationships, emphasizing the importance of mathematical and graphical interpretation skills when doing CVS.
International Journal of Artificial Intelligence in Education
Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, 2018
Video is an increasingly popular medium for education. Motivated by the problem of video as a one... more Video is an increasingly popular medium for education. Motivated by the problem of video as a one-way medium, this paper investigates the ways in which learners» active interaction with video materials contributes to active learning. In this study, we examine active viewing behaviors, specifically seeking and highlighting within videos, which may suggest greater levels of participation and learning. We deployed a system designed for active viewing to an undergraduate class for a semester. The analysis of online activity traces and interview data provided novel findings on video highlighting behavior in educational contexts.
Sources of Difficulty in Multi-Step Geometry Area Problems Yvonne S. Kao (ykao@andrew.cmu.edu) De... more Sources of Difficulty in Multi-Step Geometry Area Problems Yvonne S. Kao (ykao@andrew.cmu.edu) Department of Psychology, 5000 Forbes Ave. Pittsburgh, PA 15213 Ido Roll (idoroll@cmu.edu) Human-Computer Interaction Institute, 5000 Forbes Ave. Pittsburgh, PA 15213 Kenneth R. Koedinger (koedinger@cmu.edu) Human-Computer Interaction Institute, 5000 Forbes Ave. Pittsburgh, PA 15213 Abstract Although U.S. students often perform well on basic, single- step math problems, they often struggle with extended, multi- step free-response problems. This study examines the sources of difficulty in multi-step geometry area problems. We found that the presence of distracters creates significant difficulty for students solving geometry area problems, but that practice on composite area problems improves students’ ability to ignore distracters. In addition, this study found some support for the hypothesis that the increased figural analysis requirements of a complex diagram can negatively impact perform...
Video is used extensively as an instructional aid within educational contexts such as blended (fl... more Video is used extensively as an instructional aid within educational contexts such as blended (flipped) courses, self-learning with MOOCs and informal learning through online tutorials. One challenge is providing mechanisms for students to manage their video collection and quickly review or search for content. We provided students with a number of video interface features to establish which they would find most useful for video courses. From this, we designed an interface which uses textbook-style highlighting on a video filmstrip and transcript, both presented adjacent to a video player. This interface was qualitatively evaluated to determine if highlighting works well for saving intervals, and what strategies students use when given both direct video highlighting and the textbased transcript interface. Our participants reported that highlighting is a useful addition to instructional video. The familiar interaction of highlighting text was preferred, with the filmstrip used for int...
Successful instruction should help students acquire robust knowledge and prepare them for future ... more Successful instruction should help students acquire robust knowledge and prepare them for future learning opportunities. However, we are yet to find a winning strategy for systematically achieving robust learning (Bransford & Schwartz, 2001). Accumulated evidence suggests that discovery learning does not help most students acquire the basic foundations, and direct instruction, on the other hand, often leads to a relatively rigid body of knowledge (c.f., Tobias & Duffy, 2009). Instructional technologies are in a similar pursuit of robust learning (Koedinger & Aleven, 2007). However, students working with discovery environments often do not receive adequate support and thus fail to achieve desired learning gains (De Jong & van Joolingen, 1998). Students working with intelligent tutoring systems receive appropriate support, but on tasks that may not prepare them enough to make sense of new situations. Recently, Schwartz and colleagues devised a hybrid method called Invention as Prepara...
Several models were built recently in the metacognitive level of the students’ interaction with C... more Several models were built recently in the metacognitive level of the students’ interaction with Cognitive Tutors, an intelligent tutoring system based on ACT-R theory. After finding suboptimal help-seeking behavior, we built a metacognitive model of desired help-seeking behavior (Aleven et al. in press). In a different Cognitive Tutor, Baker et al. (2004) built a model that identifies misuse of the tutor. Here we take another step and describe a model of students’ goals and strategies, which rely in the basis of their metacognitive actions. By comparing the model’s predictions to students’ log-files we find the correlation between having the goals and learning gains.
The Journal of Interactive Learning Research, 2008
In recent years there has been increasing interest in the phenomena of " gaming the system, ... more In recent years there has been increasing interest in the phenomena of " gaming the system, " where a learner attempts to succeed in an educational environment by exploiting properties of the system's help and feedback rather than by attempting to learn the material. Developing environments that respond constructively and effectively to gaming depends upon understanding why students choose to game. In this article , we present three studies, conducted with two different learning environments, which present evidence on which student behaviors, motivations, and emotions are associated with the choice to game the system. We also present a fourth study to determine how teachers' perspectives on gaming behavior are similar to, and different from, researchers' perspectives and the data from our studies. We discuss what motivational and attitudinal patterns are associated with gaming behavior across studies, and what the implications are for the design of interactive ...
Helping Students Know ‘Further’ – Increasing the Flexibility of Students’ Knowledge Using Symboli... more Helping Students Know ‘Further’ – Increasing the Flexibility of Students’ Knowledge Using Symbolic Invention Tasks Ido Roll (idoroll@cmu.edu) Human Computer Interaction Institute, Carnegie Mellon University 5000 Forbes Avenue, Pittsburgh, PA 15213, USA Vincent Aleven (aleven@cs.cmu.edu) Human Computer Interaction Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA Kenneth R. Koedinger (koedinger@cmu.edu) Human Computer Interaction Institute and Department of Psychology, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA Abstract evaluate a set of examples with regard to one aspect of the data. Figure 1 shows an example of such a task, in which students are asked to invent a method for comparing the variability of two datasets, in order to choose the more consistent one (i.e., where data is “closer together”). Following the invention attempt, students receive direct instruction on canonical methods and practice them. For example...
International Conference of the Learning Sciences, 2010
Abstract Intelligent Tutoring Systems are widely used coached problem-solving environments (Koedi... more Abstract Intelligent Tutoring Systems are widely used coached problem-solving environments (Koedinger, Anderson, Hadley & Mark, 1997; VanLehn, Lynch, Schulze, Shapiro & Shelby, 2005). They are successful, in part, due to their ability to give adaptive feedback (Corbett & Anderson, 2001; Koedinger & Aleven, 2007). More specifically, Intelligent Tutoring Systems adapt to students' behavior and knowledge by tracing students' learning trajectories using a cognitive model of the domain (Corbett & Anderson, 1995). A ...
Abstract. The goal of our research is to investigate whether a Cognitive Tutor can be made more e... more Abstract. The goal of our research is to investigate whether a Cognitive Tutor can be made more effective by extending it to help students acquire help-seeking skills. We present a preliminary model of help-seeking behavior that will provide the basis for a Help-Seeking Tutor Agent. The model, implemented by 57 production rules, captures both productive and unproductive help-seeking behavior. As a first test of the model’s efficacy, we used it off-line to evaluate students ’ help-seeking behavior in an existing data set of student-tutor interactions, We found that 72 % of all student actions represented unproductive help-seeking behavior. Consistent with some of our earlier work (Aleven & Koedinger, 2000) we found a proliferation of hint abuse (e.g., using hints to find answers rather than trying to understand). We also found that students frequently avoided using help when it was likely to be of benefit and often acted in a quick, possibly undeliberate manner. Students’ help-seekin...
Abstract. It has been found in recent years that many students who use intelli-gent tutoring syst... more Abstract. It has been found in recent years that many students who use intelli-gent tutoring systems game the system, attempting to succeed in the educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we introduce a system which gives a gaming student supplementary exercises fo-cused on exactly the material the student bypassed by gaming, and which also expresses negative emotion to gaming students through an animated agent. Stu-dents using this system engage in less gaming, and students who receive many supplemental exercises have considerably better learning than is associated with gaming in the control condition or prior studies. 1
Invention activities are structured tasks in which students create mathematical methods that atte... more Invention activities are structured tasks in which students create mathematical methods that attempt to capture deep properties of data (e.g., variability), prior to receiving instruction on canonical methods (e.g., mean deviation). While experiments have demonstrated the learning benefits of invention activities, the mechanisms of transfer remain unknown. We address this question by evaluating the role of design in invention activities, identifying what knowledge is acquired during invention activities, and how it is applied in transfer tasks. A classroom experiment with 92 students compared the full invention process to one in which students evaluate predesigned methods. Results show that students in the full invention condition acquired more adaptive knowledge, yet not necessarily better procedural knowledge or invention skills. We suggest a mechanism that explains what knowledge invention attempts produce, how that knowledge is productively modified in subsequent instruction, an...
Abstract. Invention activities are Productive Failure activities in which students at-tempt to in... more Abstract. Invention activities are Productive Failure activities in which students at-tempt to invent methods that capture deep properties of given data before being taught expert solutions. The current study evaluates the effect of scaffolding on the invention processes and outcomes, given that students are not expected to succeed in their inquiry and that all students receive subsequent instruction. Two Invention activities related to data analysis concepts were given to 130 undergraduate students in a first-year physics lab course using an interactive learning environment. Students in the Guided Invention condition were given prompts to analyze given data prior to inventing and reflect on their methods after inventing them. These students outperformed Unguided Invention students on delayed measures of transfer, but not on measures of conceptual or proce-dural knowledge. In addition, Guided Invention students were more likely to invent multiple methods, suggesting that they used b...
Intelligent tutoring systems help students acquire cognitive skills by tracing students' kno... more Intelligent tutoring systems help students acquire cognitive skills by tracing students' knowledge and providing relevant feedback. However,
Control of Variables Strategy (CVS) is the process of isolating the effect of single variables wh... more Control of Variables Strategy (CVS) is the process of isolating the effect of single variables when conducting scientific inquiry. We assess how CVS can help student achieve different levels of understanding when implemented in different parts of the inquiry process. 148 students worked with minimally-guided inquiry activities using virtual labs on two different physics topics. The virtual labs allowed for exploration, data collection, and graphical analysis. Using student log data, we identified how CVS manifests itself through these phases of students’ inquiry process. We found that students using CVS during data collection and plotting was associated with students achieving more qualitative and quantitative models, respectively. This did not hold, however, for more complicated mathematical relationships, emphasizing the importance of mathematical and graphical interpretation skills when doing CVS.
International Journal of Artificial Intelligence in Education
Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, 2018
Video is an increasingly popular medium for education. Motivated by the problem of video as a one... more Video is an increasingly popular medium for education. Motivated by the problem of video as a one-way medium, this paper investigates the ways in which learners» active interaction with video materials contributes to active learning. In this study, we examine active viewing behaviors, specifically seeking and highlighting within videos, which may suggest greater levels of participation and learning. We deployed a system designed for active viewing to an undergraduate class for a semester. The analysis of online activity traces and interview data provided novel findings on video highlighting behavior in educational contexts.
Sources of Difficulty in Multi-Step Geometry Area Problems Yvonne S. Kao (ykao@andrew.cmu.edu) De... more Sources of Difficulty in Multi-Step Geometry Area Problems Yvonne S. Kao (ykao@andrew.cmu.edu) Department of Psychology, 5000 Forbes Ave. Pittsburgh, PA 15213 Ido Roll (idoroll@cmu.edu) Human-Computer Interaction Institute, 5000 Forbes Ave. Pittsburgh, PA 15213 Kenneth R. Koedinger (koedinger@cmu.edu) Human-Computer Interaction Institute, 5000 Forbes Ave. Pittsburgh, PA 15213 Abstract Although U.S. students often perform well on basic, single- step math problems, they often struggle with extended, multi- step free-response problems. This study examines the sources of difficulty in multi-step geometry area problems. We found that the presence of distracters creates significant difficulty for students solving geometry area problems, but that practice on composite area problems improves students’ ability to ignore distracters. In addition, this study found some support for the hypothesis that the increased figural analysis requirements of a complex diagram can negatively impact perform...
Video is used extensively as an instructional aid within educational contexts such as blended (fl... more Video is used extensively as an instructional aid within educational contexts such as blended (flipped) courses, self-learning with MOOCs and informal learning through online tutorials. One challenge is providing mechanisms for students to manage their video collection and quickly review or search for content. We provided students with a number of video interface features to establish which they would find most useful for video courses. From this, we designed an interface which uses textbook-style highlighting on a video filmstrip and transcript, both presented adjacent to a video player. This interface was qualitatively evaluated to determine if highlighting works well for saving intervals, and what strategies students use when given both direct video highlighting and the textbased transcript interface. Our participants reported that highlighting is a useful addition to instructional video. The familiar interaction of highlighting text was preferred, with the filmstrip used for int...
Successful instruction should help students acquire robust knowledge and prepare them for future ... more Successful instruction should help students acquire robust knowledge and prepare them for future learning opportunities. However, we are yet to find a winning strategy for systematically achieving robust learning (Bransford & Schwartz, 2001). Accumulated evidence suggests that discovery learning does not help most students acquire the basic foundations, and direct instruction, on the other hand, often leads to a relatively rigid body of knowledge (c.f., Tobias & Duffy, 2009). Instructional technologies are in a similar pursuit of robust learning (Koedinger & Aleven, 2007). However, students working with discovery environments often do not receive adequate support and thus fail to achieve desired learning gains (De Jong & van Joolingen, 1998). Students working with intelligent tutoring systems receive appropriate support, but on tasks that may not prepare them enough to make sense of new situations. Recently, Schwartz and colleagues devised a hybrid method called Invention as Prepara...
Several models were built recently in the metacognitive level of the students’ interaction with C... more Several models were built recently in the metacognitive level of the students’ interaction with Cognitive Tutors, an intelligent tutoring system based on ACT-R theory. After finding suboptimal help-seeking behavior, we built a metacognitive model of desired help-seeking behavior (Aleven et al. in press). In a different Cognitive Tutor, Baker et al. (2004) built a model that identifies misuse of the tutor. Here we take another step and describe a model of students’ goals and strategies, which rely in the basis of their metacognitive actions. By comparing the model’s predictions to students’ log-files we find the correlation between having the goals and learning gains.
The Journal of Interactive Learning Research, 2008
In recent years there has been increasing interest in the phenomena of " gaming the system, ... more In recent years there has been increasing interest in the phenomena of " gaming the system, " where a learner attempts to succeed in an educational environment by exploiting properties of the system's help and feedback rather than by attempting to learn the material. Developing environments that respond constructively and effectively to gaming depends upon understanding why students choose to game. In this article , we present three studies, conducted with two different learning environments, which present evidence on which student behaviors, motivations, and emotions are associated with the choice to game the system. We also present a fourth study to determine how teachers' perspectives on gaming behavior are similar to, and different from, researchers' perspectives and the data from our studies. We discuss what motivational and attitudinal patterns are associated with gaming behavior across studies, and what the implications are for the design of interactive ...
Helping Students Know ‘Further’ – Increasing the Flexibility of Students’ Knowledge Using Symboli... more Helping Students Know ‘Further’ – Increasing the Flexibility of Students’ Knowledge Using Symbolic Invention Tasks Ido Roll (idoroll@cmu.edu) Human Computer Interaction Institute, Carnegie Mellon University 5000 Forbes Avenue, Pittsburgh, PA 15213, USA Vincent Aleven (aleven@cs.cmu.edu) Human Computer Interaction Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA Kenneth R. Koedinger (koedinger@cmu.edu) Human Computer Interaction Institute and Department of Psychology, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA Abstract evaluate a set of examples with regard to one aspect of the data. Figure 1 shows an example of such a task, in which students are asked to invent a method for comparing the variability of two datasets, in order to choose the more consistent one (i.e., where data is “closer together”). Following the invention attempt, students receive direct instruction on canonical methods and practice them. For example...