Marieke van der Schaaf - Academia.edu (original) (raw)

Drafts by Marieke van der Schaaf

Objective: To develop an educational intervention that targets residents' beliefs and attitudes t... more Objective: To develop an educational intervention that targets residents' beliefs and attitudes to quality Improvement (QI) and leadership in order to demonstrate proactive behaviour. Design: Theory-driven, mixed methods study including document analysis, interviews, observations and open-ended questionnaires. Setting: Six Dutch teaching hospitals. Intervention: Using expertise from medicine, psychology, organizational and educational sciences we developed a situated learning programme named Ponder and IMProve (PIMP). The acronym PIMP reflects the original upbeat name in Dutch, Verwonder & Verbeter. It has a modern, positive meaning that relates to improving your current circumstances. In quarterly 1-h sessions residents are challenged to identify daily workplace frustrations and translate them into small-scale QI activities. Main outcome measures: Organizational awareness, beliefs and attitudes to QI and organizational responsibilities, resident behaviour, barriers and facilitators to successful learning and the programme's potential impact on the organization. Results: Overall, 19 PIMP meetings were held over a period of 3 years. Residents defined 119 PIMP goals, resolved 37 projects and are currently working on another 39 projects. Interviews show that PIMP sessions make residents more aware of the organizational aspects of their daily work. Moreover, residents feel empowered to take up the role of change agent. Facilitators for success include a positive cost-benefit trade-off, a valuable group process and a safe learning environment. Conclusion: This article demonstrates the added value of multidisciplinary theory-driven research for the design, development and evaluation of educational programmes. Residents can be encouraged to develop organizational awareness and reshape their daily frustrations in QI work.

During workplace based learning, e.g. clinical practice or during an internship, there is an urge... more During workplace based learning, e.g. clinical practice or during an internship, there is an urgent need for solutions to restore and to guarantee the quality of feedback for (becoming) professionals. In continuing education at the workplace the use of Electronic portfolios (EPs) is a crucial means for acquiring the requisite professional knowledge and skills. Although EPs provide a useful approach to view each trainee's progress, often only limited use is made of the rich contextual learning assessment data to support responsive adaptation for more efficient and rewarding training and hence to provide personalized feedback. This contribution advocates that EPs enhanced with a Learning Analytics engine, may increase the quality and efficiency of workplace-based feedback and assessment. This contribution addresses this by outlining an approach that is applied in a European 7th framework project, called WATCHME (www.project-watchme.eu). The aim of the contribution is to provide insight in underlying rationales to improve workplace-based feedback and assessment and how this is applied in an EP environment that is enhanced with Learning Analytics.

BACKGROUND True competency-based medical education should produce graduates meeting fixed standar... more BACKGROUND True competency-based medical education should produce graduates meeting fixed standards of competence. Current postgraduate programmes are usually based on a pre-determined length of time in training making them poorly suited for an individual approach. The concept of entrustable professional activities (EPAs) enables a more flexible, personalised and less time-dependent approach to training programmes. An EPA is a unit of professional practice, to be entrusted to a trainee to execute without supervision once they demonstrate sufficient competence. As EPAs relate competencies to clinical practice, they structure training and assessment more logically according to the way clinicians actually work. A first step in building an EPA-based curriculum is to identify the core EPAs of the profession.

Electronic portfolios (E-portfolios) are crucial means for workplace-based assessment and feedbac... more Electronic portfolios (E-portfolios) are crucial means for workplace-based assessment and feedback. Although E-portfolios provide a useful approach to view each learner's progress, so far options for personalized feedback and potential data about a learner's performances at the workplace often remain unexploited. This paper advocates that E-portfolios enhanced with learning analytics, might increase the quality and efficiency of workplace-based feedback and assessment in professional education. Based on a 5-phased iterative design approach, an existing E-portfolio environment was enhanced with learning analytics in professional education. First, information about crucial professional activities for professional domains and suited assessment instruments were collected (phase 1). Thereafter probabilistic student models were defined (phase 2). Next, person-alized feedback and visualization of the personal development over time were developed (phase 3). Then the prototype of the E-portfolio—including the student models and feedback and visualization modules—were implemented in professional training-programs (phase 4). Last, evaluation cycles took place and 121 students and 30 supervisors from five institutes for professional education evaluated the perceived usefulness of the design (phase 5). It was concluded that E-portfolios with learning analytics were perceived to assist the development of students' professional competencies and that the design is only successful when developed and implemented through the eyes of the users. Feedback and assessment methods based upon learning analytics can stimulate learning at the workplace in the long run. Practical, technological and ethical challenges are discussed. & Marieke van der Schaaf m.f.vanderschaaf@uu.nl;

Papers by Marieke van der Schaaf

Eye-Tracking Technology Applications in Educational Research, 2000

Shayan, S., Abrahamson, D., Bakker, A., Duijzer, A. C. G., & Van der Schaaf, M. F. (in pr... more Shayan, S., Abrahamson, D., Bakker, A., Duijzer, A. C. G., & Van der Schaaf, M. F. (in press). In A. W. Christopher, F. J. Sansosti, & B. J. Morris (Eds.), Eye-tracking technology applications in educational research. Hershey, PA: IGI Global

Studies in Continuing Education, 2016

Human Development, 2016

The combination of two methodological resources—natural-user interfaces (NUI) and multimodal lear... more The combination of two methodological resources—natural-user interfaces (NUI) and multimodal learning analytics (MMLA)—is creating opportunities for educational researchers to empirically evaluate theoretical models accounting for the emergence of concepts from situated sensorimotor activity. 76 participants (9-12 yo) solved tablet-based presymbolic manipulation tasks designed to foster grounded meanings for the mathematical concept of proportional equivalence. Data gathered in task-based semi-structured clinical interviews included action logging, eye-gaze tracking, and videography. Analysis of these data indicates that successful task performance coincided with spontaneous emergence of stable dynamical gaze-path patterns soon followed by multimodal articulation of strategy. Significantly, gaze patterns included unmanipulated, non-salient screen locations. We present cumulative evidence that these gaze patterns served as ‘attentional anchors’ mediating participants’ problem solving. By way of contextualizing our claim, we also present case studies from the various experimental conditions. We interpret the findings as enabling us to revisit, support, refine, and perhaps elaborate on seminal claims from Piaget’s theory of genetic epistemology and in particular his insistence on the role of situated motor-action coordination in the process of reflective abstraction

Levende Talen Tijdschrift, 2014

Embodied cognition is emerging as a promising approach in educational technology. This study is b... more Embodied cognition is emerging as a promising approach in educational technology. This study is based on the conjecture that a potentially powerful methodology for researching embodied mathematics design would be to gather empirical data on children's shifting visual attention as they learn to operate the technological devices so as to solve the situated problems. The aim of the study is to gain insight into the role of visual attention in the emergence of new sensorimotor schemes underlying mathematical concepts, especially proportion tasks. The research question is: How does visual attention change in the emergence of sensorimotor schemes during proportional reasoning tasks? An exploratory study was conducted amongst eight 5 th -and 6 th -grade students (age 10-12 years). Based on the original Mathematical Imagery Trainer for Proportion (MIT-P) designs for Wii and iPad (Abrahamson), a new tablet app has been developed. While students completed the hands-on proportion tasks, a Tobii x2-30 and an external camera were used for real-time processing and storage of both video and gaze data, resulting in integrated videos of hand-and eye movements. Data analysis revealed cross-participant variation in exploration path, progress rate, and inferences. Yet across all participants insight coincided with a shift from: (a) random finger movements accompanied by gazing at salient figural contour; to (b) new bimanual coordinations accompanied by gazing at new non-salient figural features. The study thus supports classical constructivist claims that mathematical concepts are grounded in operatory schemes. In particular, the data literally show the dynamical emergence of attentional anchors for situated problem solving as mediating the development of mathematical concepts. Thus new research methodologies stand to validate claims from embodiment.

Eesti Haridusteaduste Ajakiri. Estonian Journal of Education, 2016

Scandinavian Journal of Educational Research, Aug 6, 2014

ABSTRACT

ABSTRACT PURPOSE Current radiology practice has become increasingly based on the digital interpre... more ABSTRACT PURPOSE Current radiology practice has become increasingly based on the digital interpretation of volumetric multi-planar-reconstruction images (MPR-images). Nevertheless, assessment of radiological image interpretation skills in medical education and postgraduate radiology training is still mainly based on two-dimensional (2D) images (only one or two slices of a stack are presented). Consequently, the assessment lacks authenticity, which negatively impacts its quality. We hypothesized that using MPR images increases the assessment quality as reflected in its validity (the test assesses what it is intended to measure) and reliability (the accuracy of the test results, its reproducibility and little measurement error). Our aim was to evaluate differences in validity and reliability of assessment with 2D image questions versus MPR image questions. METHOD AND MATERIALS In 2012, 246 medical students, trained with MPR images, took a digital radiology test. There were two versions (A and B), both containing twenty 2D and twenty MPR image questions, concerning anatomy on CT-scans. Participants filled out a questionnaire to judge the authenticity of the assessment as an indication of validity. They also gave their opinion on the difficulty of 2D and MPR image questions. Mean scores and reliabilities (estimated with Cronbach’s alpha) of the 2D and MPR image subtests were compared. RESULTS Cronbach’s alphas on 2D image questions were .49 (A), and .65 (B), and alphas of MPR image questions were .65 (A), and .71 (B). Scores on MPR image questions (M 15.6, SD 2.6; M 14.9, SD 2.9) were lower than scores on 2D image questions (M 15.8, SD 2.2; M 16.8, SD 2.4). This difference between 2D and MPR scores was significant for version B. Assessment based on MPR images was considered more authentic (t (56) = -7.1, p < .001), and less difficult (t (58) = -4.2, p < .001) by the participants. CONCLUSION According to the participants, assessment with MPR images increases authenticity, which can contribute to validity. MPR image questions showed higher reliability than 2D image questions. Scores on MPR image questions in one version were significantly lower, but considered less difficult by participants. CLINICAL RELEVANCE/APPLICATION Valid and reliable assessments of image interpretation skills of radiology trainees, adds to patient safety. MPR image assessment seems to contribute to its validity and to its reliability.

Http Dx Doi Org 10 1080 02602938 2011 576312, Oct 30, 2012

Abstract Student portfolios are increasingly used for assessing student competences in higher edu... more Abstract Student portfolios are increasingly used for assessing student competences in higher education, but results about the construct validity of portfolio assessment are mixed. A prerequisite for construct validity is that the portfolio assessment is based on relevant portfolio content. Assessment criteria, are often used to enhance this condition. This study aims to identify whether assessment criteria can improve content, argumentation and communication during teacher moderation while judging student portfolios. Six teachers ...

European Journal of Radiology, Jan 5, 2015

Current radiology practice increasingly involves interpretation of volumetric data sets. In contr... more Current radiology practice increasingly involves interpretation of volumetric data sets. In contrast, most radiology tests still contain only 2D images. We introduced a new testing tool that allows for stack viewing of volumetric images in our undergraduate radiology program. We hypothesized that tests with volumetric CT-images enhance test quality, in comparison with traditional completely 2D image-based tests, because they might better reflect required skills for clinical practice. Two groups of medical students (n=139; n=143), trained with 2D and volumetric CT-images, took a digital radiology test in two versions (A and B), each containing both 2D and volumetric CT-image questions. In a questionnaire, they were asked to comment on the representativeness for clinical practice, difficulty and user-friendliness of the test questions and testing program. Students' test scores and reliabilities, measured with Cronbach's alpha, of 2D and volumetric CT-image tests were compared. Estimated reliabilities (Cronbach's alphas) were higher for volumetric CT-image scores (version A: .51 and version B: .54), than for 2D CT-image scores (version A: .24 and version B: .37). Participants found volumetric CT-image tests more representative of clinical practice, and considered them to be less difficult than volumetric CT-image questions. However, in one version (A), volumetric CT-image scores (M 80.9, SD 14.8) were significantly lower than 2D CT-image scores (M 88.4, SD 10.4) (p<.001). The volumetric CT-image testing program was considered user-friendly. This study shows that volumetric image questions can be successfully integrated in students' radiology testing. Results suggests that the inclusion of volumetric CT-images might improve the quality of radiology tests by positively impacting perceived representativeness for clinical practice and increasing reliability of the test.

Br Educ Res J, 2005

The increased attention for teacher assessment and current educational reforms ask for procedures... more The increased attention for teacher assessment and current educational reforms ask for procedures to develop adequate content standards. For the development of content standards on teaching research skills, a Delphi method based on stakeholders' judgments has been designed and tested. In three rounds, 21 stakeholders judged and revised content standards. The support for the standards increased over the rounds. The method resulted in nine content standards with a high degree of support and consensus. Qualitative analysis of stakeholders' comments and a Homals analysis showed that the stakeholders differ in the perspectives and preferences from which they judged the standards. The Delphi method proved to be an adequate procedure for developing teaching content standards based on stakeholders' judgments.

Assess Eval High Educ, 2008

Developing and using a design for teacher portfolio assessment is a complex process including sev... more Developing and using a design for teacher portfolio assessment is a complex process including several components: the domain to be assessed (the teacher competences), the content standards or criteria, the portfolio format, the completion of the format (by teachers) with content, and the scoring of the portfolios (by raters). For a portfolio assessment to be valid, these components should be

BMC medical education, Jan 21, 2015

Medical curricula become more and more vertically integrated (VI) to prepare graduates better for... more Medical curricula become more and more vertically integrated (VI) to prepare graduates better for clinical practice. VI curricula show early clinical education, integration of biomedical sciences and focus on increasing clinical responsibility levels for trainees. Results of earlier questionnaire-based studies indicate that the type of the curriculum can affect the perceived preparedness for work as perceived by students or supervisors. The aim of the present study is to determine difference in actual performance of graduates from VI and non-VI curricula. We developed and implemented an authentic performance assessment based on different facets of competence for medical near-graduates in the role of beginning residents on a very busy day. Fifty nine candidates participated: 30 VI (Utrecht, The Netherlands) and 29 non-VI (Hamburg, Germany). Two physicians, one nurse and five standardized patients independently assessed each candidate on different facets of competence. Afterwards, the...

Medical teacher, Jan 14, 2015

This Guide was written to support educators interested in building a competency-based workplace c... more This Guide was written to support educators interested in building a competency-based workplace curriculum. It aims to provide an up-to-date overview of the literature on Entrustable Professional Activities (EPAs), supplemented with suggestions for practical application to curriculum construction, assessment and educational technology. The Guide first introduces concepts and definitions related to EPAs and then guidance for their identification, elaboration and validation, while clarifying common misunderstandings about EPAs. A matrix-mapping approach of combining EPAs with competencies is discussed, and related to existing concepts such as competency milestones. A specific section is devoted to entrustment decision-making as an inextricable part of working with EPAs. In using EPAs, assessment in the workplace is translated to entrustment decision-making for designated levels of permitted autonomy, ranging from acting under full supervision to providing supervision to a junior learn...

Eesti Haridusteaduste Ajakiri = Estonian Journal of Education, 2013

Objective: To develop an educational intervention that targets residents' beliefs and attitudes t... more Objective: To develop an educational intervention that targets residents' beliefs and attitudes to quality Improvement (QI) and leadership in order to demonstrate proactive behaviour. Design: Theory-driven, mixed methods study including document analysis, interviews, observations and open-ended questionnaires. Setting: Six Dutch teaching hospitals. Intervention: Using expertise from medicine, psychology, organizational and educational sciences we developed a situated learning programme named Ponder and IMProve (PIMP). The acronym PIMP reflects the original upbeat name in Dutch, Verwonder & Verbeter. It has a modern, positive meaning that relates to improving your current circumstances. In quarterly 1-h sessions residents are challenged to identify daily workplace frustrations and translate them into small-scale QI activities. Main outcome measures: Organizational awareness, beliefs and attitudes to QI and organizational responsibilities, resident behaviour, barriers and facilitators to successful learning and the programme's potential impact on the organization. Results: Overall, 19 PIMP meetings were held over a period of 3 years. Residents defined 119 PIMP goals, resolved 37 projects and are currently working on another 39 projects. Interviews show that PIMP sessions make residents more aware of the organizational aspects of their daily work. Moreover, residents feel empowered to take up the role of change agent. Facilitators for success include a positive cost-benefit trade-off, a valuable group process and a safe learning environment. Conclusion: This article demonstrates the added value of multidisciplinary theory-driven research for the design, development and evaluation of educational programmes. Residents can be encouraged to develop organizational awareness and reshape their daily frustrations in QI work.

During workplace based learning, e.g. clinical practice or during an internship, there is an urge... more During workplace based learning, e.g. clinical practice or during an internship, there is an urgent need for solutions to restore and to guarantee the quality of feedback for (becoming) professionals. In continuing education at the workplace the use of Electronic portfolios (EPs) is a crucial means for acquiring the requisite professional knowledge and skills. Although EPs provide a useful approach to view each trainee's progress, often only limited use is made of the rich contextual learning assessment data to support responsive adaptation for more efficient and rewarding training and hence to provide personalized feedback. This contribution advocates that EPs enhanced with a Learning Analytics engine, may increase the quality and efficiency of workplace-based feedback and assessment. This contribution addresses this by outlining an approach that is applied in a European 7th framework project, called WATCHME (www.project-watchme.eu). The aim of the contribution is to provide insight in underlying rationales to improve workplace-based feedback and assessment and how this is applied in an EP environment that is enhanced with Learning Analytics.

BACKGROUND True competency-based medical education should produce graduates meeting fixed standar... more BACKGROUND True competency-based medical education should produce graduates meeting fixed standards of competence. Current postgraduate programmes are usually based on a pre-determined length of time in training making them poorly suited for an individual approach. The concept of entrustable professional activities (EPAs) enables a more flexible, personalised and less time-dependent approach to training programmes. An EPA is a unit of professional practice, to be entrusted to a trainee to execute without supervision once they demonstrate sufficient competence. As EPAs relate competencies to clinical practice, they structure training and assessment more logically according to the way clinicians actually work. A first step in building an EPA-based curriculum is to identify the core EPAs of the profession.

Electronic portfolios (E-portfolios) are crucial means for workplace-based assessment and feedbac... more Electronic portfolios (E-portfolios) are crucial means for workplace-based assessment and feedback. Although E-portfolios provide a useful approach to view each learner's progress, so far options for personalized feedback and potential data about a learner's performances at the workplace often remain unexploited. This paper advocates that E-portfolios enhanced with learning analytics, might increase the quality and efficiency of workplace-based feedback and assessment in professional education. Based on a 5-phased iterative design approach, an existing E-portfolio environment was enhanced with learning analytics in professional education. First, information about crucial professional activities for professional domains and suited assessment instruments were collected (phase 1). Thereafter probabilistic student models were defined (phase 2). Next, person-alized feedback and visualization of the personal development over time were developed (phase 3). Then the prototype of the E-portfolio—including the student models and feedback and visualization modules—were implemented in professional training-programs (phase 4). Last, evaluation cycles took place and 121 students and 30 supervisors from five institutes for professional education evaluated the perceived usefulness of the design (phase 5). It was concluded that E-portfolios with learning analytics were perceived to assist the development of students' professional competencies and that the design is only successful when developed and implemented through the eyes of the users. Feedback and assessment methods based upon learning analytics can stimulate learning at the workplace in the long run. Practical, technological and ethical challenges are discussed. & Marieke van der Schaaf m.f.vanderschaaf@uu.nl;

Eye-Tracking Technology Applications in Educational Research, 2000

Shayan, S., Abrahamson, D., Bakker, A., Duijzer, A. C. G., & Van der Schaaf, M. F. (in pr... more Shayan, S., Abrahamson, D., Bakker, A., Duijzer, A. C. G., & Van der Schaaf, M. F. (in press). In A. W. Christopher, F. J. Sansosti, & B. J. Morris (Eds.), Eye-tracking technology applications in educational research. Hershey, PA: IGI Global

Studies in Continuing Education, 2016

Human Development, 2016

The combination of two methodological resources—natural-user interfaces (NUI) and multimodal lear... more The combination of two methodological resources—natural-user interfaces (NUI) and multimodal learning analytics (MMLA)—is creating opportunities for educational researchers to empirically evaluate theoretical models accounting for the emergence of concepts from situated sensorimotor activity. 76 participants (9-12 yo) solved tablet-based presymbolic manipulation tasks designed to foster grounded meanings for the mathematical concept of proportional equivalence. Data gathered in task-based semi-structured clinical interviews included action logging, eye-gaze tracking, and videography. Analysis of these data indicates that successful task performance coincided with spontaneous emergence of stable dynamical gaze-path patterns soon followed by multimodal articulation of strategy. Significantly, gaze patterns included unmanipulated, non-salient screen locations. We present cumulative evidence that these gaze patterns served as ‘attentional anchors’ mediating participants’ problem solving. By way of contextualizing our claim, we also present case studies from the various experimental conditions. We interpret the findings as enabling us to revisit, support, refine, and perhaps elaborate on seminal claims from Piaget’s theory of genetic epistemology and in particular his insistence on the role of situated motor-action coordination in the process of reflective abstraction

Levende Talen Tijdschrift, 2014

Embodied cognition is emerging as a promising approach in educational technology. This study is b... more Embodied cognition is emerging as a promising approach in educational technology. This study is based on the conjecture that a potentially powerful methodology for researching embodied mathematics design would be to gather empirical data on children's shifting visual attention as they learn to operate the technological devices so as to solve the situated problems. The aim of the study is to gain insight into the role of visual attention in the emergence of new sensorimotor schemes underlying mathematical concepts, especially proportion tasks. The research question is: How does visual attention change in the emergence of sensorimotor schemes during proportional reasoning tasks? An exploratory study was conducted amongst eight 5 th -and 6 th -grade students (age 10-12 years). Based on the original Mathematical Imagery Trainer for Proportion (MIT-P) designs for Wii and iPad (Abrahamson), a new tablet app has been developed. While students completed the hands-on proportion tasks, a Tobii x2-30 and an external camera were used for real-time processing and storage of both video and gaze data, resulting in integrated videos of hand-and eye movements. Data analysis revealed cross-participant variation in exploration path, progress rate, and inferences. Yet across all participants insight coincided with a shift from: (a) random finger movements accompanied by gazing at salient figural contour; to (b) new bimanual coordinations accompanied by gazing at new non-salient figural features. The study thus supports classical constructivist claims that mathematical concepts are grounded in operatory schemes. In particular, the data literally show the dynamical emergence of attentional anchors for situated problem solving as mediating the development of mathematical concepts. Thus new research methodologies stand to validate claims from embodiment.

Eesti Haridusteaduste Ajakiri. Estonian Journal of Education, 2016

Scandinavian Journal of Educational Research, Aug 6, 2014

ABSTRACT

ABSTRACT PURPOSE Current radiology practice has become increasingly based on the digital interpre... more ABSTRACT PURPOSE Current radiology practice has become increasingly based on the digital interpretation of volumetric multi-planar-reconstruction images (MPR-images). Nevertheless, assessment of radiological image interpretation skills in medical education and postgraduate radiology training is still mainly based on two-dimensional (2D) images (only one or two slices of a stack are presented). Consequently, the assessment lacks authenticity, which negatively impacts its quality. We hypothesized that using MPR images increases the assessment quality as reflected in its validity (the test assesses what it is intended to measure) and reliability (the accuracy of the test results, its reproducibility and little measurement error). Our aim was to evaluate differences in validity and reliability of assessment with 2D image questions versus MPR image questions. METHOD AND MATERIALS In 2012, 246 medical students, trained with MPR images, took a digital radiology test. There were two versions (A and B), both containing twenty 2D and twenty MPR image questions, concerning anatomy on CT-scans. Participants filled out a questionnaire to judge the authenticity of the assessment as an indication of validity. They also gave their opinion on the difficulty of 2D and MPR image questions. Mean scores and reliabilities (estimated with Cronbach’s alpha) of the 2D and MPR image subtests were compared. RESULTS Cronbach’s alphas on 2D image questions were .49 (A), and .65 (B), and alphas of MPR image questions were .65 (A), and .71 (B). Scores on MPR image questions (M 15.6, SD 2.6; M 14.9, SD 2.9) were lower than scores on 2D image questions (M 15.8, SD 2.2; M 16.8, SD 2.4). This difference between 2D and MPR scores was significant for version B. Assessment based on MPR images was considered more authentic (t (56) = -7.1, p < .001), and less difficult (t (58) = -4.2, p < .001) by the participants. CONCLUSION According to the participants, assessment with MPR images increases authenticity, which can contribute to validity. MPR image questions showed higher reliability than 2D image questions. Scores on MPR image questions in one version were significantly lower, but considered less difficult by participants. CLINICAL RELEVANCE/APPLICATION Valid and reliable assessments of image interpretation skills of radiology trainees, adds to patient safety. MPR image assessment seems to contribute to its validity and to its reliability.

Http Dx Doi Org 10 1080 02602938 2011 576312, Oct 30, 2012

Abstract Student portfolios are increasingly used for assessing student competences in higher edu... more Abstract Student portfolios are increasingly used for assessing student competences in higher education, but results about the construct validity of portfolio assessment are mixed. A prerequisite for construct validity is that the portfolio assessment is based on relevant portfolio content. Assessment criteria, are often used to enhance this condition. This study aims to identify whether assessment criteria can improve content, argumentation and communication during teacher moderation while judging student portfolios. Six teachers ...

European Journal of Radiology, Jan 5, 2015

Current radiology practice increasingly involves interpretation of volumetric data sets. In contr... more Current radiology practice increasingly involves interpretation of volumetric data sets. In contrast, most radiology tests still contain only 2D images. We introduced a new testing tool that allows for stack viewing of volumetric images in our undergraduate radiology program. We hypothesized that tests with volumetric CT-images enhance test quality, in comparison with traditional completely 2D image-based tests, because they might better reflect required skills for clinical practice. Two groups of medical students (n=139; n=143), trained with 2D and volumetric CT-images, took a digital radiology test in two versions (A and B), each containing both 2D and volumetric CT-image questions. In a questionnaire, they were asked to comment on the representativeness for clinical practice, difficulty and user-friendliness of the test questions and testing program. Students' test scores and reliabilities, measured with Cronbach's alpha, of 2D and volumetric CT-image tests were compared. Estimated reliabilities (Cronbach's alphas) were higher for volumetric CT-image scores (version A: .51 and version B: .54), than for 2D CT-image scores (version A: .24 and version B: .37). Participants found volumetric CT-image tests more representative of clinical practice, and considered them to be less difficult than volumetric CT-image questions. However, in one version (A), volumetric CT-image scores (M 80.9, SD 14.8) were significantly lower than 2D CT-image scores (M 88.4, SD 10.4) (p<.001). The volumetric CT-image testing program was considered user-friendly. This study shows that volumetric image questions can be successfully integrated in students' radiology testing. Results suggests that the inclusion of volumetric CT-images might improve the quality of radiology tests by positively impacting perceived representativeness for clinical practice and increasing reliability of the test.

Br Educ Res J, 2005

The increased attention for teacher assessment and current educational reforms ask for procedures... more The increased attention for teacher assessment and current educational reforms ask for procedures to develop adequate content standards. For the development of content standards on teaching research skills, a Delphi method based on stakeholders' judgments has been designed and tested. In three rounds, 21 stakeholders judged and revised content standards. The support for the standards increased over the rounds. The method resulted in nine content standards with a high degree of support and consensus. Qualitative analysis of stakeholders' comments and a Homals analysis showed that the stakeholders differ in the perspectives and preferences from which they judged the standards. The Delphi method proved to be an adequate procedure for developing teaching content standards based on stakeholders' judgments.

Assess Eval High Educ, 2008

Developing and using a design for teacher portfolio assessment is a complex process including sev... more Developing and using a design for teacher portfolio assessment is a complex process including several components: the domain to be assessed (the teacher competences), the content standards or criteria, the portfolio format, the completion of the format (by teachers) with content, and the scoring of the portfolios (by raters). For a portfolio assessment to be valid, these components should be

BMC medical education, Jan 21, 2015

Medical curricula become more and more vertically integrated (VI) to prepare graduates better for... more Medical curricula become more and more vertically integrated (VI) to prepare graduates better for clinical practice. VI curricula show early clinical education, integration of biomedical sciences and focus on increasing clinical responsibility levels for trainees. Results of earlier questionnaire-based studies indicate that the type of the curriculum can affect the perceived preparedness for work as perceived by students or supervisors. The aim of the present study is to determine difference in actual performance of graduates from VI and non-VI curricula. We developed and implemented an authentic performance assessment based on different facets of competence for medical near-graduates in the role of beginning residents on a very busy day. Fifty nine candidates participated: 30 VI (Utrecht, The Netherlands) and 29 non-VI (Hamburg, Germany). Two physicians, one nurse and five standardized patients independently assessed each candidate on different facets of competence. Afterwards, the...

Medical teacher, Jan 14, 2015

This Guide was written to support educators interested in building a competency-based workplace c... more This Guide was written to support educators interested in building a competency-based workplace curriculum. It aims to provide an up-to-date overview of the literature on Entrustable Professional Activities (EPAs), supplemented with suggestions for practical application to curriculum construction, assessment and educational technology. The Guide first introduces concepts and definitions related to EPAs and then guidance for their identification, elaboration and validation, while clarifying common misunderstandings about EPAs. A matrix-mapping approach of combining EPAs with competencies is discussed, and related to existing concepts such as competency milestones. A specific section is devoted to entrustment decision-making as an inextricable part of working with EPAs. In using EPAs, assessment in the workplace is translated to entrustment decision-making for designated levels of permitted autonomy, ranging from acting under full supervision to providing supervision to a junior learn...

Eesti Haridusteaduste Ajakiri = Estonian Journal of Education, 2013

Academic radiology, Jan 12, 2015

Radiology practice has become increasingly based on volumetric images (VIs), but tests in medical... more Radiology practice has become increasingly based on volumetric images (VIs), but tests in medical education still mainly involve two-dimensional (2D) images. We created a novel, digital, VI test and hypothesized that scores on this test would better reflect radiological anatomy skills than scores on a traditional 2D image test. To evaluate external validity we correlated VI and 2D image test scores with anatomy cadaver-based test scores. In 2012, 246 medical students completed one of two comparable versions (A and B) of a digital radiology test, each containing 20 2D image and 20 VI questions. Thirty-three of these participants also took a human cadaver anatomy test. Mean scores and reliabilities of the 2D image and VI subtests were compared and correlated with human cadaver anatomy test scores. Participants received a questionnaire about perceived representativeness and difficulty of the radiology test. Human cadaver test scores were not correlated with 2D image scores, but signifi...

Scandinavian Journal of Educational Research, 2014

Teaching and Teacher Education, 2008

This study concerns the relation between teachers' beliefs towards teaching behaviour and their a... more This study concerns the relation between teachers' beliefs towards teaching behaviour and their actual teaching behaviour in teacher portfolio assessment. We analysed the beliefs and behaviour of 18 teachers as described in their portfolios. In addition, each portfolio was independently assessed by two trained raters on eight content standards and the teachers' classroom behaviour was assessed by their own students in a questionnaire (n ¼ 317). Linear multilevel analysis showed that part of the raters' assessments of the teachers' beliefs and their behaviour as described in their portfolios was significantly related to the students' assessments of their teachers' behaviour. Teachers with high raters' assessments on the content standard about 'choosing and arguing for teaching strategies that meet students' knowledge, abilities and experience' had significantly higher student assessments than teachers who were judged low on this standard. Implications of the results and suggestions for further research are discussed. r

Academic Radiology, 2015

In current practice, radiologists interpret digital images, including a substantial amount of vol... more In current practice, radiologists interpret digital images, including a substantial amount of volumetric images. We hypothesized that interpretation of a stack of a volumetric data set demands different skills than interpretation of two-dimensional (2D) cross-sectional images. This study aimed to investigate and compare knowledge and skills used for interpretation of volumetric versus 2D images. Twenty radiology clerks were asked to think out loud while reading four or five volumetric computed tomography (CT) images in stack mode and four or five 2D CT images. Cases were presented in a digital testing program allowing stack viewing of volumetric data sets and changing views and window settings. Thoughts verbalized by the participants were registered and coded by a framework of knowledge and skills concerning three components: perception, analysis, and synthesis. The components were subdivided into 16 discrete knowledge and skill elements. A within-subject analysis was performed to compare cognitive processes during volumetric image readings versus 2D cross-sectional image readings. Most utterances contained knowledge and skills concerning perception (46%). A smaller part involved synthesis (31%) and analysis (23%). More utterances regarded perception in volumetric image interpretation than in 2D image interpretation (Median 48% vs 35%; z = -3.9; P < .001). Synthesis was less prominent in volumetric than in 2D image interpretation (Median 28% vs 42%; z = -3.9; P < .001). No differences were found in analysis utterances. Cognitive processes in volumetric and 2D cross-sectional image interpretation differ substantially. Volumetric image interpretation draws predominantly on perceptual processes, whereas 2D image interpretation is mainly characterized by synthesis. The results encourage the use of volumetric images for teaching and testing perceptual skills.

International Journal of Educational Research, 2010

This study aims to explore the relationship between the occurrence of reflection (and nonreflecti... more This study aims to explore the relationship between the occurrence of reflection (and nonreflection) and thinking activities (e.g., orientating, selecting, analysing) during portfoliobased conversations. Analysis of 21 transcripts of portfolio-based conversations revealed that 20% of the segments were made up of reflection (content reflection (6%), process reflection (10%), and premise reflection (4%)). The thinking activities comparing, analysing and concluding occurred significantly more often during reflection than during nonreflection. Orientating on the task, selecting and describing, occurred significantly less often during reflection. The outcomes show that the occurrence of certain thinking activities can be an indication of reflection. ß