Natalie Simper - Academia.edu (original) (raw)

Papers by Natalie Simper

Research paper thumbnail of Scaling Up: Value added Cognitive Assessment Redesign Network for the Development of Higher-Order Thinking

Higher Education Quality Council of Ontario, 2019

Research paper thumbnail of Value Added: Demonstrating Student Skill Development at Your Institution

14th Annual Conference of the International Society of Scholarship of Teaching and Learning (ISSOTL) : REACHING NEW HEIGHTS, 2017

Research paper thumbnail of Learning Outcomes Assessment and Program Improvement at Queen’s University

Research paper thumbnail of Guide for Institutional Assessment of Cognitive Skills

Research paper thumbnail of Concept Maps for Creative Development

Research paper thumbnail of Running a Skills Assessment Project: Working with instructors and the utility of tools

This session will highlight the strengths and challenges of multiple approaches to institutional-... more This session will highlight the strengths and challenges of multiple approaches to institutional-scale assessment of transferable learning outcomes. Evidence from the Queen’s University Learning Outcomes Assessment Project (queensu.ca/qloa/home) highlights student engagement, motivation and effort as critical factors for success. In addition, findings show improved student outcomes in courses where instructors embrace assessment processes, and embed skills assessment within their course. The presentation will compare approaches to skills assessment, and highlight some challenges, including faculty engagement, cost, complexity, participation rate, and utility of information gathered. The presenters will share their perception of how each of these impacts thoughtful development and assessment of transferable learning outcomes.

Research paper thumbnail of Informal academic networks and the value of significant social interactions in supporting quality assessment practices

Higher Education Research & Development, 2021

This research investigated social interactions within small significant networks across a range o... more This research investigated social interactions within small significant networks across a range of higher education settings to determine their role in supporting improvements to assessment. Thirty...

Research paper thumbnail of Learning Evaluation and Reflection Narrative (LEARN)- Student’s articulating their workplace learning

This presentation summarizes the use of the Transferable Learning Orientation (TLO) (Author 2015)... more This presentation summarizes the use of the Transferable Learning Orientation (TLO) (Author 2015) as a reflection tool for workplace learning. It has been used for the last two years as a pre-post measure for what some researchers call non-cognitive skills. When the pre and post scores are overlaid, students can see their development visually on the spider diagrams. Students are then provided with a Learning Evaluation and Reflection Narrative (LEARN) framework to respond to a mock interview scenario. The LEARN activity, together with the open-ended responses in the TLO prompt students to effectively communicate their professional experience. The benefits of students engaging in the LEARN activity is twofold: (1) the quantitative results provide evidence of the effectiveness of the summer student program, which can be used for future summer student funding applications and recruitment; (2) students need to apply meta-cognitive processes to complete the survey and LEARN exercise. By ...

Research paper thumbnail of Assessment cultures in higher education: reducing barriers and enabling change

Assessment & Evaluation in Higher Education, 2021

Research paper thumbnail of Rubric Authoring Tool Supporting Cognitive Skills Assessment across an Institution

This paper explores a method to support instructors in assessing cognitive skills in their course... more This paper explores a method to support instructors in assessing cognitive skills in their course, designed to enable aggregation of data across an institution. A rubric authoring tool, “BASICS” (Building Assessment Scaffolds for Intellectual Cognitive Skills) was built as part of the Queen’s University Learning Outcomes Assessment (LOA) Project. It provides a workflow for assessment choices and generates an assessment rubric that can be tailored to individual needs based on user input. The dimensions and criteria in BASICS were adapted from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics, and drew on annotations from over 900 work samples from the LOA project. This paper summarizes the development of the tool, and presents initial reliability and validity data from a pilot study. The pilot found that the BASICS developed rubric was consistent for the assessment of critical thinking and problem solving. The pilot compared assessment data derived from cour...

Research paper thumbnail of Raising the Bar on Assessment and Rubric Design

Research paper thumbnail of Rubric authoring tool for supporting the development an assessment of cognitive skills in higher education

Teaching & Learning Inquiry

This paper explores a method to support instructors in assessing cognitive skills in their course... more This paper explores a method to support instructors in assessing cognitive skills in their course, designed to enable aggregation of data across an institution. A rubric authoring tool, "BASICS" (Building Assessment Scaffolds for Intellectual Cognitive Skills) was built as part of the Queen's University Learning Outcomes Assessment (LOA) Project. It provides a workflow for assessment choices and generates an assessment rubric that can be tailored to individual needs based on user input. The dimensions and criteria in BASICS were adapted from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics, and drew on annotations from over 900 work samples from the LOA project. This paper summarizes the development of the tool, and presents initial reliability and validity data from a pilot study. The pilot found that the BASICS developed rubric was consistent for the assessment of critical thinking and problem solving. The pilot compared assessment data derived from course Teaching Assistants with that of trained Research Assistants. Analysis found moderate intraclass correlation coefficients between the BASICS rubric and corresponding VALUE rubric dimensions, suggesting that the BASICS rubric aligned with the VALUE criteria. Preliminary findings suggest that BASICS is an effective tool for instructors to author rubrics, tailored to their own specifications for assessment of cognitive skills in a course. It is also promising as a method for aggregation of data across the institution. Researchers are conducting further investigation to evaluate the reliability of BASICS rubrics over multiple work samples from a range of disciplinary contexts.

Research paper thumbnail of Assessment thresholds for academic staff: constructive alignment and differentiation of standards

Assessment & Evaluation in Higher Education

Research paper thumbnail of Using the Cla+ to Track Development of Cognitive Skills from First to Third Year: Accessibility and Factors for Success

Proceedings of the Canadian Engineering Education Association (CEEA)

This paper provides results from the first three years of a 4-year longitudinal assessment projec... more This paper provides results from the first three years of a 4-year longitudinal assessment project on the development of cognitive skills using the Collegiate Learning Assessment (CLA+). The CLA+ is an online test, designed to measure cognitive skills (CS) (critical thinking, problem solving and written communication, with sub-scores reporting scientific and quantitative reasoning, critical reading and evaluation, and critiquing an argument). Cognitive skills are fundamental elements of engineering programs and central to the practice of engineering. The Canadian Engineering Accreditation Board’s (CEAB) requires programs to assess their student’s graduate attributes and have a system in place to use the assessment for curriculum improvement. CLA+ assessment constructs relate to the CEAB attributes of problem analysis, investigation and communication. The testing was initiated as part of a strategy to track development of cognitive skills and to inform course improvement efforts. The...

Research paper thumbnail of Network-Based Approach to Assessment of Cognitive Skills

Proceedings of the Canadian Engineering Education Association (CEEA)

Cognitive Assessment Redesign (CAR) project is an institution-wide, network-based approach to the... more Cognitive Assessment Redesign (CAR) project is an institution-wide, network-based approach to the development of cognitive skills in undergraduate education. This project aims to encourage first and fourth-year instructors to align skill development through the design of course assessments, to enhance cognitive skill acquisition and provide a measurement of learning. The learning outcomes for the project are framed and operationalized using the language and dimensions from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics. An assessment redesign network was created, matching assessment facilitators who have disciplinary and educational expertise with instructors to develop authentic assessments of student learning. One of the goals of the network is to encourage sustained participation and collaboration, and to build progression in teaching and learning throughout the institution. The project also includes a standardized test for comparison to course assess...

Research paper thumbnail of An Engineering Design Course to Develop and Assess Critical Thinking and Problem Solving

Proceedings of the Canadian Engineering Education Association (CEEA)

A challenging new engineering design course is developed as part of the Engineering Design and Pr... more A challenging new engineering design course is developed as part of the Engineering Design and Practice Sequence in the Civil Engineering program. This course engages students in a cyclical design process where they plan, build, test, and evaluate a model-scale tidal current turbine. They then use their own observations and analysis to iteratively inform, improve and re-test their design.The two objectives of this paper are to provide a description of the development and structure of this design course, and to assess student learning. The Final Design Reports were externally evaluated using the Valid Assessment of Learning in Undergraduate Education rubrics. Students also completed a standardized test called the Collegiate Learning Assessment as an objective evaluation of longitudinal learning gains. The Civil Engineering students demonstrated significant improvement in critical thinking, problem solving, and written communication skills.

Research paper thumbnail of Student learning in the workplace

Journal of Workplace Learning

Research paper thumbnail of Comparison of standardized assessment methods: logistics, costs, incentives and use of data

Assessment & Evaluation in Higher Education

Research paper thumbnail of Enhancing Development of Competencies by Means of Continuous Improvement Processes

Research paper thumbnail of Formative feedback and scaffolding for developing complex problem solving and modelling outcomes

European Journal of Engineering Education

This paper discusses the use and impact of formative feedback and scaffolding to develop outcomes... more This paper discusses the use and impact of formative feedback and scaffolding to develop outcomes for complex problem solving in a required first-year course in engineering design and practice at a medium-sized research-intensive Canadian university. In 2010, the course began to use team-based, complex, open-ended contextualised problems to develop problem solving, communications, teamwork, modelling, and professional skills. Since then, formative feedback has been incorporated into: task and process-level feedback on scaffolded tasks in-class, formative assignments, and post-assignment review. Development in complex problem solving and modelling has been assessed through analysis of responses from student surveys, direct criterion-referenced assessment of course outcomes from 2013 to 2015, and an external longitudinal study. The findings suggest that students are improving in outcomes related to complex problem solving over the duration of the course. Most notably, the addition of new feedback and scaffolding coincided with improved student performance.

Research paper thumbnail of Scaling Up: Value added Cognitive Assessment Redesign Network for the Development of Higher-Order Thinking

Higher Education Quality Council of Ontario, 2019

Research paper thumbnail of Value Added: Demonstrating Student Skill Development at Your Institution

14th Annual Conference of the International Society of Scholarship of Teaching and Learning (ISSOTL) : REACHING NEW HEIGHTS, 2017

Research paper thumbnail of Learning Outcomes Assessment and Program Improvement at Queen’s University

Research paper thumbnail of Guide for Institutional Assessment of Cognitive Skills

Research paper thumbnail of Concept Maps for Creative Development

Research paper thumbnail of Running a Skills Assessment Project: Working with instructors and the utility of tools

This session will highlight the strengths and challenges of multiple approaches to institutional-... more This session will highlight the strengths and challenges of multiple approaches to institutional-scale assessment of transferable learning outcomes. Evidence from the Queen’s University Learning Outcomes Assessment Project (queensu.ca/qloa/home) highlights student engagement, motivation and effort as critical factors for success. In addition, findings show improved student outcomes in courses where instructors embrace assessment processes, and embed skills assessment within their course. The presentation will compare approaches to skills assessment, and highlight some challenges, including faculty engagement, cost, complexity, participation rate, and utility of information gathered. The presenters will share their perception of how each of these impacts thoughtful development and assessment of transferable learning outcomes.

Research paper thumbnail of Informal academic networks and the value of significant social interactions in supporting quality assessment practices

Higher Education Research & Development, 2021

This research investigated social interactions within small significant networks across a range o... more This research investigated social interactions within small significant networks across a range of higher education settings to determine their role in supporting improvements to assessment. Thirty...

Research paper thumbnail of Learning Evaluation and Reflection Narrative (LEARN)- Student’s articulating their workplace learning

This presentation summarizes the use of the Transferable Learning Orientation (TLO) (Author 2015)... more This presentation summarizes the use of the Transferable Learning Orientation (TLO) (Author 2015) as a reflection tool for workplace learning. It has been used for the last two years as a pre-post measure for what some researchers call non-cognitive skills. When the pre and post scores are overlaid, students can see their development visually on the spider diagrams. Students are then provided with a Learning Evaluation and Reflection Narrative (LEARN) framework to respond to a mock interview scenario. The LEARN activity, together with the open-ended responses in the TLO prompt students to effectively communicate their professional experience. The benefits of students engaging in the LEARN activity is twofold: (1) the quantitative results provide evidence of the effectiveness of the summer student program, which can be used for future summer student funding applications and recruitment; (2) students need to apply meta-cognitive processes to complete the survey and LEARN exercise. By ...

Research paper thumbnail of Assessment cultures in higher education: reducing barriers and enabling change

Assessment & Evaluation in Higher Education, 2021

Research paper thumbnail of Rubric Authoring Tool Supporting Cognitive Skills Assessment across an Institution

This paper explores a method to support instructors in assessing cognitive skills in their course... more This paper explores a method to support instructors in assessing cognitive skills in their course, designed to enable aggregation of data across an institution. A rubric authoring tool, “BASICS” (Building Assessment Scaffolds for Intellectual Cognitive Skills) was built as part of the Queen’s University Learning Outcomes Assessment (LOA) Project. It provides a workflow for assessment choices and generates an assessment rubric that can be tailored to individual needs based on user input. The dimensions and criteria in BASICS were adapted from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics, and drew on annotations from over 900 work samples from the LOA project. This paper summarizes the development of the tool, and presents initial reliability and validity data from a pilot study. The pilot found that the BASICS developed rubric was consistent for the assessment of critical thinking and problem solving. The pilot compared assessment data derived from cour...

Research paper thumbnail of Raising the Bar on Assessment and Rubric Design

Research paper thumbnail of Rubric authoring tool for supporting the development an assessment of cognitive skills in higher education

Teaching & Learning Inquiry

This paper explores a method to support instructors in assessing cognitive skills in their course... more This paper explores a method to support instructors in assessing cognitive skills in their course, designed to enable aggregation of data across an institution. A rubric authoring tool, "BASICS" (Building Assessment Scaffolds for Intellectual Cognitive Skills) was built as part of the Queen's University Learning Outcomes Assessment (LOA) Project. It provides a workflow for assessment choices and generates an assessment rubric that can be tailored to individual needs based on user input. The dimensions and criteria in BASICS were adapted from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics, and drew on annotations from over 900 work samples from the LOA project. This paper summarizes the development of the tool, and presents initial reliability and validity data from a pilot study. The pilot found that the BASICS developed rubric was consistent for the assessment of critical thinking and problem solving. The pilot compared assessment data derived from course Teaching Assistants with that of trained Research Assistants. Analysis found moderate intraclass correlation coefficients between the BASICS rubric and corresponding VALUE rubric dimensions, suggesting that the BASICS rubric aligned with the VALUE criteria. Preliminary findings suggest that BASICS is an effective tool for instructors to author rubrics, tailored to their own specifications for assessment of cognitive skills in a course. It is also promising as a method for aggregation of data across the institution. Researchers are conducting further investigation to evaluate the reliability of BASICS rubrics over multiple work samples from a range of disciplinary contexts.

Research paper thumbnail of Assessment thresholds for academic staff: constructive alignment and differentiation of standards

Assessment & Evaluation in Higher Education

Research paper thumbnail of Using the Cla+ to Track Development of Cognitive Skills from First to Third Year: Accessibility and Factors for Success

Proceedings of the Canadian Engineering Education Association (CEEA)

This paper provides results from the first three years of a 4-year longitudinal assessment projec... more This paper provides results from the first three years of a 4-year longitudinal assessment project on the development of cognitive skills using the Collegiate Learning Assessment (CLA+). The CLA+ is an online test, designed to measure cognitive skills (CS) (critical thinking, problem solving and written communication, with sub-scores reporting scientific and quantitative reasoning, critical reading and evaluation, and critiquing an argument). Cognitive skills are fundamental elements of engineering programs and central to the practice of engineering. The Canadian Engineering Accreditation Board’s (CEAB) requires programs to assess their student’s graduate attributes and have a system in place to use the assessment for curriculum improvement. CLA+ assessment constructs relate to the CEAB attributes of problem analysis, investigation and communication. The testing was initiated as part of a strategy to track development of cognitive skills and to inform course improvement efforts. The...

Research paper thumbnail of Network-Based Approach to Assessment of Cognitive Skills

Proceedings of the Canadian Engineering Education Association (CEEA)

Cognitive Assessment Redesign (CAR) project is an institution-wide, network-based approach to the... more Cognitive Assessment Redesign (CAR) project is an institution-wide, network-based approach to the development of cognitive skills in undergraduate education. This project aims to encourage first and fourth-year instructors to align skill development through the design of course assessments, to enhance cognitive skill acquisition and provide a measurement of learning. The learning outcomes for the project are framed and operationalized using the language and dimensions from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics. An assessment redesign network was created, matching assessment facilitators who have disciplinary and educational expertise with instructors to develop authentic assessments of student learning. One of the goals of the network is to encourage sustained participation and collaboration, and to build progression in teaching and learning throughout the institution. The project also includes a standardized test for comparison to course assess...

Research paper thumbnail of An Engineering Design Course to Develop and Assess Critical Thinking and Problem Solving

Proceedings of the Canadian Engineering Education Association (CEEA)

A challenging new engineering design course is developed as part of the Engineering Design and Pr... more A challenging new engineering design course is developed as part of the Engineering Design and Practice Sequence in the Civil Engineering program. This course engages students in a cyclical design process where they plan, build, test, and evaluate a model-scale tidal current turbine. They then use their own observations and analysis to iteratively inform, improve and re-test their design.The two objectives of this paper are to provide a description of the development and structure of this design course, and to assess student learning. The Final Design Reports were externally evaluated using the Valid Assessment of Learning in Undergraduate Education rubrics. Students also completed a standardized test called the Collegiate Learning Assessment as an objective evaluation of longitudinal learning gains. The Civil Engineering students demonstrated significant improvement in critical thinking, problem solving, and written communication skills.

Research paper thumbnail of Student learning in the workplace

Journal of Workplace Learning

Research paper thumbnail of Comparison of standardized assessment methods: logistics, costs, incentives and use of data

Assessment & Evaluation in Higher Education

Research paper thumbnail of Enhancing Development of Competencies by Means of Continuous Improvement Processes

Research paper thumbnail of Formative feedback and scaffolding for developing complex problem solving and modelling outcomes

European Journal of Engineering Education

This paper discusses the use and impact of formative feedback and scaffolding to develop outcomes... more This paper discusses the use and impact of formative feedback and scaffolding to develop outcomes for complex problem solving in a required first-year course in engineering design and practice at a medium-sized research-intensive Canadian university. In 2010, the course began to use team-based, complex, open-ended contextualised problems to develop problem solving, communications, teamwork, modelling, and professional skills. Since then, formative feedback has been incorporated into: task and process-level feedback on scaffolded tasks in-class, formative assignments, and post-assignment review. Development in complex problem solving and modelling has been assessed through analysis of responses from student surveys, direct criterion-referenced assessment of course outcomes from 2013 to 2015, and an external longitudinal study. The findings suggest that students are improving in outcomes related to complex problem solving over the duration of the course. Most notably, the addition of new feedback and scaffolding coincided with improved student performance.