Evaluation of Serious Games: A Holistic Approach (original) (raw)
Related papers
Study Design and Data Gathering Guide for Serious Games' Evaluation
2013
The objective of this chapter is to provide an overview of the different methods that can be used to evaluate the learning outcomes of serious games. These include Randomised Control Trials (RCT), quasi-experimental designs, and surveys. Case studies of a selection of serious games developed for use in higher education are then presented along with evaluations of these games. The evaluations illustrate the different evaluation methods, along with an assessment of how well the evaluation method performed. Finally, the chapter discusses the lessons learned and compares the experiences with the evaluation methods and their transferability to other games.
Multimedia Tools and Applications
Although serious games are proven to serve as educational tools in many educational domains, there is a lack of reliable, automated and repeatable methodologies that measure their effectiveness: what do players know after playing serious games? Do they learn from them? Previous research shows that the vast majority of serious games are assessed by using questionnaires, which is in stark contrast to current trends in the video game industry. Commercial videogame developers have been learning from their players through Game Analytics for years via non-disruptive game tracking. In this paper, we propose a methodology for assessing serious game effectiveness based on non-disruptive in-game tracking. The methodology involves a design pattern that structures the delivery of educational goals through a game. This structure also allows one to infer learning outcomes for each individual player, which, when aggregated, determine the effectiveness of a serious game. We tested the methodology by having 320 students play a serious game. The proposed methodology allowed us to infer players' learning outcomes, to assess the game effectiveness levels and to identify issues in the game design.
Towards a Comprehensive Methodology for the Research and Evaluation of Serious Games
Procedia Computer Science, 2012
The author presents the methodological backgrounds and underlying research design of an ongoing scientific research project concerned with the scientific evaluation of serious games and/or computer-based simulation-games (SG) for advanced learning. The main questions of this research project are: 1. what are the requirements and design principles for a comprehensive social-scientific methodology for the evaluation of SG? 2. To what extend does SG contribute to advanced learning? 3. What factors contribute to, or determine this learning? 4. To what extend and under what conditions can SGbased learning be transferred to the real world (RW)? Between 2004 and 2012, several hundreds of SG-sessions in the Netherlands with twelve different SG were evaluated systematically, uniformly and quantitatively to give a data-set of 2100 respondents in higher education and in work-organizations. The author presents the research model, the quasi-experimental design and evaluation instruments. This focus in this article is on methodology and data-set to establish a proper foundation for forthcoming publications on empirical results.
The research and evaluation of serious games: Toward a comprehensive methodology
2013
The authors present the methodological background to and underlying research design of an ongoing research project on the scientific evaluation of serious games and/or computer-based simulation games (SGs) for advanced learning. The main research questions are: (1) what are the requirements and design principles for a comprehensive social scientific methodology for the evaluation of SGs?; (2) to what extent do SGs contribute to advanced learning?; (3) what factors contribute to or determine this learning?; and (4) to what extent and under what conditions can SG-based learning be transferred to the real world? In the Netherlands between 2005 and 2012, several hundred SG sessions with 12 SGs were evaluated systematically, uniformly and quantitatively to create a dataset, which comprises data on 2488 respondents in higher education or work organizations. The authors present the research model, the quasi-experimental design and the evaluation instruments. This focus in this paper is on the methodology and dataset, which form a sound foundation for forthcoming publications on the empirical results.
Informing design and evaluation methodologies for serious games for learning
2007
It is widely believed that virtual and gaming environments for learning, otherwise known as serious games, have vast potential at all educational levels. Their immediacy, accessibility and portability could complement existing teaching tools, methods and resources, and do so economically, reaching students in all communities, including the underrepresented and underprivileged. While there has been considerable interest from researchers, teachers and academics as to whether serious games can be developed to support the curriculum, until recently there has been only moderate research literature describing human-computer interaction (HCI) design and evaluation methodologies to inform the development of serious games and to demonstrate their learning and educational value. Although research addressing this is now beginning to appear, there remains a lack of widely accepted serious approaches and methodologies for serious games. This paper will describe concerns and problematic findings from our serious game research, empirical evaluations and development work, some of the challenges facing us, and will make recommendations towards creating HCI design and evaluation methodologies for serious games-based learning.
Groundwork for a new perspective on the evaluation of serious games
2017
Serious games are games that contribute to a purpose other than pure entertainment. The serious games industry and research community are growing rapidly. However, research on their effectiveness has long been based on theoretical assumptions about their potential, but has been missing empirical evidence. Studies attempting to evaluate serious games show inconclusive, inconsistent and even contradictory results. This is caused by a lack of standardized methods and guidelines to avoid common mistakes in the evaluation of serious games, and the unquestioned adoption of methodologies from other domains. This thesis attempts to identify issues in common methodologies used in the evaluation of serious games. This includes the identification of mediating variables as well as some peculiarities regarding the evaluation of games. The use of randomized controlled trials - promoted as method of choice in the field - is found to be unnecessary or even inappropriate in many cases. The common us...
Success factors for serious games to enhance learning: a systematic review
Virtual Reality, 2016
There is no doubt that an abundance of factors exists that makes learning with serious games successful. Research articles reporting on these factors, however, tend to focus on select serious game elements and do not combine all salient factors for successful learning with serious games. Addressing this gap is a necessity for the success of serious games and may even alleviate long-standing debates about pedagogy over enjoyment, how much realism is enough or whether artificial intelligence is worth the cost. This article examines existing academic literature from 2000 to 2015, extracting shared serious game success factors that have had an encouraging impact on gameful learning experiences. As such, we subsequently aim to withdraw the field from a perpetual spiral of doesmy-game-work research toward more worthwhile whydoes-my-game-not-work research. Qualitative content analysis through the constant comparison method (CCM) analyzed a total of 63 articles from a variety of recognized electronic libraries and databases. Through this analysis, we reveal five central serious game themes: backstory and production; realism; artificial intelligence and adaptivity; interaction; and feedback and debriefing, all of which require deliberate intertwining with pedagogical content to ensure successful learning. This review unravels each of the five themes into their constituent factors and consequently presents the factors as practical guidelines that serious games producers should strive to include in their game productions. Applying these recommendations whenever serious games are considered will provide a foundation for effective gameful learning experiences.
Serious Games: A New Paradigm for Education?
2011
This chapter explores the context for the new paradigm of learning emerging in education, in relation to key critical concepts that centre around gamification, immersion, interface and social interactivity. The chapter provides an extensive literature review as part of the context for the paradigm shift, including considering serious games and gamification, and social learning as key constructs for considering the changes to educational practices and infrastructure faced by educationalists and instructors over the coming years. The chapter also provides an historical background section and highlights some of the conceptual work that has been done already to frame the changes, firstly in relation to the notion of ‘gamification’ through the lens of an historical overview of serious games and secondly in a section exploring the need for an overall model for serious game design based upon four models and frameworks developed in past research work including the four dimensional framework, exploratory learning model, multimodal interface architecture model and the game-based learning framework. The chapter aims to set out the key conceptual territory for serious game design and bring together the main theoretical areas under consideration for future development of effective serious game content.
A Brief Methodology for Researching and Evaluating Serious Games and Game-Based Learning
Psychology, Pedagogy, and Assessment in Serious Games, 2014
In this chapter, the authors present a methodology for researching and evaluating Serious Games (SG) and digital (or other forms of) Game-Based Learning (GBL). The methodology consists of the following elements: 1) frame-reflective analysis; 2) a methodology explicating the rationale behind a conceptual-research model; 3) research designs and data-gathering procedures; 4) validated research instruments and tools; 5) a body of knowledge that provides operationalised models and hypotheses; and 6) professional ethics. The methodology is intended to resolve the dilemma between the “generality” and “standardisation” required for comparative, theory-based research and the “specificity” and “flexibility” needed for evaluating specific cases.