John Chapman | Brigham Young University-Idaho (original) (raw)

Papers by John Chapman

Research paper thumbnail of Learning Content Analytics: Exploring the Link Between Video, Reading, and Assessment in an Online Course

Research paper thumbnail of What Should Instructional Designers Know About Learning Analytics

Research paper thumbnail of Designing Technology-Enabled Instruction to Utilize Learning Analytics

TechTrends, 2016

A key notion conveyed by those who advocate for the use of data to enhance instruction is an awar... more A key notion conveyed by those who advocate for the use of data to enhance instruction is an awareness that learning analytics has the potential to improve instruction and learning but is not currently reaching that potential. Gibbons (2014) suggested that a lack of learning facilitated by current technology-enabled instructional systems may be due in part to the natural tendency of many designers to focus on the surface layers of instruction (i.e., the content and control layers), while failing to adequately design the internal (less visible) aspects (e.g., the data management layer). In this paper, we outline phases in the design process related to the data management layer that should be considered when integrating learning analytics into a technology-enabled learning system.

Research paper thumbnail of Transaction-level learning analytics in online authentic assessments

Journal of Computing in Higher Education, 2016

This paper presents a case for the use of transaction-level data when analyzing automated online ... more This paper presents a case for the use of transaction-level data when analyzing automated online assessment results to identify knowledge gaps and misconceptions for individual students. Transaction-level data, which records all of the steps a student uses to complete an assessment item, are preferred over traditional assessment formats that submit only the final answer, as the system can detect persistent misconceptions. In this study we collected transaction-level data from 996 students enrolled in an online introductory spreadsheet class. Each student's final answer and step-by-step attempts were coded for misconceptions or knowledge gaps regarding the use of absolute references over four assessment occasions. Overall, the level of error revealed was significantly higher in the step-by-step processes compared to the final submitted answers. Further analysis suggests that students most often have misconceptions regarding non-critical errors. Data analysis also suggests that misconceptions identified at the transaction level persist over time.

Research paper thumbnail of Accelerating Learning with Analytics: Using Transaction Level Data to Identify Student Misconceptions

Research paper thumbnail of Examining Student Learning in Spreadsheet Assignments: The value of activity-trace logs

In most problem-solving assignments, professors evaluate student solutions without the ability to... more In most problem-solving assignments, professors evaluate student solutions without the ability to observe the process students used to arrive at their solutions. This paper presents an approach for allowing professors to have detailed, activity-trace process data about how students arrived at solutions, giving insights into reasoning and student misunderstandings that happen (and are sometimes corrected) prior to submission. By rendering the assignment in Excel and using a template configured to log cell changes, the files submitted by students contain transactional level data for each attempt made. These activity-trace logs also provide a powerful mechanism to tell when students are copying work from other students. An example of how this can help instructors understand the scope of student misconception is presented.

Research paper thumbnail of Researching the Architecture of a Mentoring-Centered Learning Management System: Interim Report

This paper is an interim report of the first two years of a three-year study to design a learning... more This paper is an interim report of the first two years of a three-year study to design a learning management system (LMS) architecture and interface based on a mentoring metaphor. Years 1 and 2, which will be reported, have accomplished: (1) the formulation of a new software architecture based on compact formation, (2) interviews with 100 university students, faculty, and staff for capturing descriptions of the live mentoring process, and (3) abstraction from the interviews of a list of mentoring functions required in a mentoring-centered interface. This paper will also report the creation of a mentoring vignettes book from the interview material for use on-campus to promote a mentoring culture. Paper Summary (500 words): Traditional LMS interfaces, based on a grade book metaphor, offer limited capabilities for adjusting instruction to the needs and desires of the individual, forcing training plans into a standard mold that limits the management of personalized events and sequences ...

Research paper thumbnail of Using transaction-level data to diagnose knowledge gaps and misconceptions

Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, 2015

ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Ass... more ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Assessment instruments often function at the task level. What is rarely considered is the process students go through to reach the final solution. This often allows knowledge component gaps and misconceptions to go undetected. This research identified higher levels of knowledge component gaps and misunderstandings when assessing transaction-level knowledge component data than task-level final solution data. Final solution data showed little evidence that students had any misunderstanding or knowledge gaps about the use of absolute references. However, when analyzing these data at the transaction level we found evidence that far more students struggled than the analysis of the final solutions suggested.

Research paper thumbnail of Analysis of Cognitive Processing in Electronic Problem Solving Activities: Insights on Learning from Examining Activity-Trace Data

Research paper thumbnail of Using transaction-level data to diagnose knowledge gaps and misconceptions

Proceedings of the Fifth International Conference on Learning Analytics And Knowledge - LAK '15, 2015

ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Ass... more ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Assessment instruments often function at the task level. What is rarely considered is the process students go through to reach the final solution. This often allows knowledge component gaps and misconceptions to go undetected. This research identified higher levels of knowledge component gaps and misunderstandings when assessing transaction-level knowledge component data than task-level final solution data. Final solution data showed little evidence that students had any misunderstanding or knowledge gaps about the use of absolute references. However, when analyzing these data at the transaction level we found evidence that far more students struggled than the analysis of the final solutions suggested.

Research paper thumbnail of Researching the Architecture of a Mentoring-Centered Learning Management System: Interim Report

This paper is an interim report of the first two years of a three-year study to design a learning... more This paper is an interim report of the first two years of a three-year study to design a learning management system (LMS) architecture and interface based on a mentoring metaphor. Years 1 and 2, which will be reported, have accomplished: (1) the formulation of a new software architecture based on compact formation, (2) interviews with 100 university students, faculty, and staff for capturing descriptions of the live mentoring process, and (3) abstraction from the interviews of a list of mentoring functions required in a mentoring-centered interface. This paper will also report the creation of a mentoring vignettes book from the interview material for use on-campus to promote a mentoring culture. Paper Summary (500 words): Traditional LMS interfaces, based on a grade book metaphor, offer limited capabilities for adjusting instruction to the needs and desires of the individual, forcing training plans into a standard mold that limits the management of personalized events and sequences ...

Research paper thumbnail of Examining Student Learning in Spreadsheet Assignments: The value of activity-trace logs

In most problem-solving assignments, professors evaluate student solutions without the ability to... more In most problem-solving assignments, professors evaluate student solutions without the ability to
observe the process students used to arrive at their solutions. This paper presents an approach for
allowing professors to have detailed, activity-trace process data about how students arrived at
solutions, giving insights into reasoning and student misunderstandings that happen (and are
sometimes corrected) prior to submission. By rendering the assignment in Excel and using a template
configured to log cell changes, the files submitted by students contain transactional level data for each
attempt made. These activity-trace logs also provide a powerful mechanism to tell when students are
copying work from other students. An example of how this can help instructors understand the scope
of student misconception is presented.

Research paper thumbnail of Learning Content Analytics: Exploring the Link Between Video, Reading, and Assessment in an Online Course

Research paper thumbnail of What Should Instructional Designers Know About Learning Analytics

Research paper thumbnail of Designing Technology-Enabled Instruction to Utilize Learning Analytics

TechTrends, 2016

A key notion conveyed by those who advocate for the use of data to enhance instruction is an awar... more A key notion conveyed by those who advocate for the use of data to enhance instruction is an awareness that learning analytics has the potential to improve instruction and learning but is not currently reaching that potential. Gibbons (2014) suggested that a lack of learning facilitated by current technology-enabled instructional systems may be due in part to the natural tendency of many designers to focus on the surface layers of instruction (i.e., the content and control layers), while failing to adequately design the internal (less visible) aspects (e.g., the data management layer). In this paper, we outline phases in the design process related to the data management layer that should be considered when integrating learning analytics into a technology-enabled learning system.

Research paper thumbnail of Transaction-level learning analytics in online authentic assessments

Journal of Computing in Higher Education, 2016

This paper presents a case for the use of transaction-level data when analyzing automated online ... more This paper presents a case for the use of transaction-level data when analyzing automated online assessment results to identify knowledge gaps and misconceptions for individual students. Transaction-level data, which records all of the steps a student uses to complete an assessment item, are preferred over traditional assessment formats that submit only the final answer, as the system can detect persistent misconceptions. In this study we collected transaction-level data from 996 students enrolled in an online introductory spreadsheet class. Each student's final answer and step-by-step attempts were coded for misconceptions or knowledge gaps regarding the use of absolute references over four assessment occasions. Overall, the level of error revealed was significantly higher in the step-by-step processes compared to the final submitted answers. Further analysis suggests that students most often have misconceptions regarding non-critical errors. Data analysis also suggests that misconceptions identified at the transaction level persist over time.

Research paper thumbnail of Accelerating Learning with Analytics: Using Transaction Level Data to Identify Student Misconceptions

Research paper thumbnail of Examining Student Learning in Spreadsheet Assignments: The value of activity-trace logs

In most problem-solving assignments, professors evaluate student solutions without the ability to... more In most problem-solving assignments, professors evaluate student solutions without the ability to observe the process students used to arrive at their solutions. This paper presents an approach for allowing professors to have detailed, activity-trace process data about how students arrived at solutions, giving insights into reasoning and student misunderstandings that happen (and are sometimes corrected) prior to submission. By rendering the assignment in Excel and using a template configured to log cell changes, the files submitted by students contain transactional level data for each attempt made. These activity-trace logs also provide a powerful mechanism to tell when students are copying work from other students. An example of how this can help instructors understand the scope of student misconception is presented.

Research paper thumbnail of Researching the Architecture of a Mentoring-Centered Learning Management System: Interim Report

This paper is an interim report of the first two years of a three-year study to design a learning... more This paper is an interim report of the first two years of a three-year study to design a learning management system (LMS) architecture and interface based on a mentoring metaphor. Years 1 and 2, which will be reported, have accomplished: (1) the formulation of a new software architecture based on compact formation, (2) interviews with 100 university students, faculty, and staff for capturing descriptions of the live mentoring process, and (3) abstraction from the interviews of a list of mentoring functions required in a mentoring-centered interface. This paper will also report the creation of a mentoring vignettes book from the interview material for use on-campus to promote a mentoring culture. Paper Summary (500 words): Traditional LMS interfaces, based on a grade book metaphor, offer limited capabilities for adjusting instruction to the needs and desires of the individual, forcing training plans into a standard mold that limits the management of personalized events and sequences ...

Research paper thumbnail of Using transaction-level data to diagnose knowledge gaps and misconceptions

Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, 2015

ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Ass... more ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Assessment instruments often function at the task level. What is rarely considered is the process students go through to reach the final solution. This often allows knowledge component gaps and misconceptions to go undetected. This research identified higher levels of knowledge component gaps and misunderstandings when assessing transaction-level knowledge component data than task-level final solution data. Final solution data showed little evidence that students had any misunderstanding or knowledge gaps about the use of absolute references. However, when analyzing these data at the transaction level we found evidence that far more students struggled than the analysis of the final solutions suggested.

Research paper thumbnail of Analysis of Cognitive Processing in Electronic Problem Solving Activities: Insights on Learning from Examining Activity-Trace Data

Research paper thumbnail of Using transaction-level data to diagnose knowledge gaps and misconceptions

Proceedings of the Fifth International Conference on Learning Analytics And Knowledge - LAK '15, 2015

ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Ass... more ABSTRACT The role of assessment in learning is to evaluate student comprehension and ability. Assessment instruments often function at the task level. What is rarely considered is the process students go through to reach the final solution. This often allows knowledge component gaps and misconceptions to go undetected. This research identified higher levels of knowledge component gaps and misunderstandings when assessing transaction-level knowledge component data than task-level final solution data. Final solution data showed little evidence that students had any misunderstanding or knowledge gaps about the use of absolute references. However, when analyzing these data at the transaction level we found evidence that far more students struggled than the analysis of the final solutions suggested.

Research paper thumbnail of Researching the Architecture of a Mentoring-Centered Learning Management System: Interim Report

This paper is an interim report of the first two years of a three-year study to design a learning... more This paper is an interim report of the first two years of a three-year study to design a learning management system (LMS) architecture and interface based on a mentoring metaphor. Years 1 and 2, which will be reported, have accomplished: (1) the formulation of a new software architecture based on compact formation, (2) interviews with 100 university students, faculty, and staff for capturing descriptions of the live mentoring process, and (3) abstraction from the interviews of a list of mentoring functions required in a mentoring-centered interface. This paper will also report the creation of a mentoring vignettes book from the interview material for use on-campus to promote a mentoring culture. Paper Summary (500 words): Traditional LMS interfaces, based on a grade book metaphor, offer limited capabilities for adjusting instruction to the needs and desires of the individual, forcing training plans into a standard mold that limits the management of personalized events and sequences ...

Research paper thumbnail of Examining Student Learning in Spreadsheet Assignments: The value of activity-trace logs

In most problem-solving assignments, professors evaluate student solutions without the ability to... more In most problem-solving assignments, professors evaluate student solutions without the ability to
observe the process students used to arrive at their solutions. This paper presents an approach for
allowing professors to have detailed, activity-trace process data about how students arrived at
solutions, giving insights into reasoning and student misunderstandings that happen (and are
sometimes corrected) prior to submission. By rendering the assignment in Excel and using a template
configured to log cell changes, the files submitted by students contain transactional level data for each
attempt made. These activity-trace logs also provide a powerful mechanism to tell when students are
copying work from other students. An example of how this can help instructors understand the scope
of student misconception is presented.