Teaching the Teacher: Tutoring SimStudent Leads to More Effective Cognitive Tutor Authoring (original) (raw)
Related papers
SimStudent: Building an Intelligent Tutoring System by Tutoring a Synthetic Student
SimStudent is a machine-learning agent that has been developed to help novice authors to build Intelligent Tutoring Systems (ITS) without heavy programming. Integrated into an existing suite of software tools called CTAT (Cognitive Tutor Authoring Tools), SimStudent helps authors to create an expert model for ITS by "teaching" SimStudent how to solve problems. There are two ways for the author to teach SimStudent: by demonstrating solutions or by tutoring SimStudent. In the former approach, the author demonstrates solution steps, and SimStudent attempts to induce underlying domain principles by generalizing those demonstrations. In the latter approach, the author gives SimStudent problems to solve, and provides feedback on the steps performed by SimStudent, and also gives "hints" for steps that SimStudent cannot perform (where the hints consist of demonstrating that particular step). To evaluate which authoring strategy better facilitates authoring, we conducted two studies. The first measured the quality of the expert model; here the authoring processes were simulated by using pre-recorded solutions for demonstrations as well as using an existing Cognitive Tutor for tutoring SimStudent. The second study measured the time for authoring; here a human author was actually using SimStudent to author a Cognitive Tutor. The results show the following: (1) Authoring by demonstration requires more time than authoring by tutoring, because when authoring by demonstration, the author frequently needs to test the quality of the expert model to determine if more demonstration is needed. In contrast, when authoring by tutoring, this decision making can be done naturally by observing SimStudent's performance during tutoring. (2) The expert model generated with authoring by tutoring tends to be more precise than authoring by demonstration. That is, when applied to solve problems, the expert model generated by tutoring makes more correct steps and/or fewer incorrect ones.
The Cognitive Tutor Authoring Tools (CTAT): Preliminary Evaluation of Efficiency Gains
Lecture Notes in Computer Science, 2006
Intelligent Tutoring Systems have been shown to be effective in a number of domains, but they remain hard to build, with estimates of 200-300 hours of development per hour of instruction. Two goals of the Cognitive Tutor Authoring Tools (CTAT) project are to (a) make tutor development more efficient for both programmers and non-programmers and (b) produce scientific evidence indicating which tool features lead to improved efficiency. CTAT supports development of two types of tutors, Cognitive Tutors and Example-Tracing Tutors, which represent different trade-offs in terms of ease of authoring and generality. In preliminary small-scale controlled experiments involving basic Cognitive Tutor development tasks, we found efficiency gains due to CTAT of 1.4 to 2 times faster. We expect that continued development of CTAT, informed by repeated evaluations involving increasingly complex authoring tasks, will lead to further efficiency gains.
Building Cognitive Tutors with Programming by Demonstration
2018
The aim of this study is to incorporate the technique of programming by demonstration (PBD) into an authoring tool for Cognitive Tutors. The primary motivation of using PBD is to facilitate the authoring of Cognitive Tutors by educators, rather than AI programmers. That is, instead of asking authors to build a cognitive model representing a task to be taught, a machine-learning agent – called the Simulated Student – observes the author performing the target task and induces production rules that replicate the author's performance. FOIL is used to learn conditions appearing in the production rules. An evaluation in an example domain of algebra equation solving shows that observing 10 problems solved in 44 steps induced 9 correct and 1 wrong production rules. Two of the correctly induced rules were overly general hence produced redundant solutions.
Authoring Model-Tracing Cognitive Tutors
2009
Intelligent Tutoring Systems (ITSs) that employ a model-tracing methodology have consistently shown their effectiveness. However, what evidently makes these tutors effective, the cognitive model embedded within them, has traditionally been difficult to create, requiring great expertise and time, both of which come at a cost. Furthermore, an interface has to be constructed that communicates with the cognitive model. Together these constitute a high bar that needs to be crossed in order to create such a tutor. We outline a system that lowers this bar on both accounts and that has been used to produce commercial-quality tutors. First, we discuss and evaluate a tool that allows authors who are not cognitive scientists or programmers to create a cognitive model. Second, we detail a way for this cognitive model to communicate with third-party interfaces.
Evaluating an Authoring Tool for Model-Tracing Intelligent Tutoring Systems
2008
We have been creating an authoring tool, the Cognitive Model SDK, which allows non-cognitive scientists and non-programmers to produce a cognitive model for model-tracing tutors [1, 2]. The SDK is in use by developers at Carnegie Learning to produce their commercial Cognitive Tutors for math. However, it has never been evaluated with regards to the strong claim that non-cognitive scientists and non-programmers could, without much effort, produce useful cognitive models with it. The research presented here shows that this can be done, using a task that past researchers have used [3]. The models are evaluated across several metrics to see what characteristics of either them or their creators may distinguish better models from worse models. The goal of this work is to establish a baseline for future work examining how cognitive modeling can be opened up to a wider class of people.
Rapid authoring of intelligent tutors for real-world and experimental use
2006
Authoring tools for Intelligent Tutoring Systems are especially valuable if they not only provide a rich set of options for the efficient authoring of tutoring systems but also support controlled experiments in which the added educational value of new tutor features is evaluated. The Cognitive Tutor Authoring Tools (CTAT) provide both. Using CTAT, real-world "Example-Tracing Tutors" can be created without programming. CTAT also provides various kinds of support for controlled experiments, such as administration of different experimental treatments, logging, and data analysis. We present two case studies in which Example-Tracing Tutors created with CTAT were used in classroom experiments. The case studies illustrate a number of new features in CTAT: Use of Macromedia Flash MX 2004 for creating tutor interfaces, extensions to the Example-Tracing Engine that allow for more flexible tutors, a Mass Production facility for more efficient template-based authoring, and support for controlled experiments.
International Journal of Artificial Intelligence in Education, 2015
The Extensible Problem Specific Tutor (xPST) allows authors who are not cognitive scientists and not programmers to quickly create an intelligent tutoring system that provides instruction akin to a model-tracing tutor. Furthermore, this instruction is overlaid on existing software, so that the learner's interface does not have to be made from scratch. The xPST architecture allows for extending its capabilities by the addition of plug-ins that communicate with additional third-party software. After reviewing this general architecture, we describe three major implementations that we have created using the xPST system, each using different third-party software as the learner's interface. We have conducted three evaluations of authors using xPST to create tutoring content, and these are considered in turn. These evaluations show that xPST authors can quickly learn the system, and can efficiently produce successful embedded instruction.
A Metamodel for Designing an Intelligent Tutoring Systems Authoring Tool
Computer and Information Science, 2014
Previous intelligent tutoring systems (ITS) and ITS authoring studies predominantly simulated and evaluated artificial intelligence (AI) techniques and cognitive architectures/notions in educational domains. Current research focuses on software design that is priori driven by educational theories; it concerns the conception of Augmented Conversation and Cognitive Apprenticeship Metamodel (ACCAM). The pedagogy driven metamodel-ACCAM-forms the basis for a formal (theory based) approach to designing ITS authoring tools for numerical aspect of numerical disciplines. This research, therefore, showcases the convergence of two theoretical perspectives-the Conversation Theory (CT) and Cognitive Apprenticeship (CA)-which were never considered together before now. The novel conceptual platform-the ACCAM-flows and benefited from the synergistic effect of the stated theories through the introduction of the concept of 'augmented conversation' within the resulting integrated framework. Thus, current work draws on the pedagogical import of the mentioned educational theories, elicits new meanings, and lays the foundation as well as opens future evaluation of a pedagogical engineering methodology that flows therefrom.