ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity (original) (raw)
Related papers
Online Journal of Communication and Media Technologies
The use of artificial intelligence (AI) in education is becoming increasingly prevalent, and its encroachment and impact on online education and assessment is a topic of interest to researchers and lecturers. ChatGPT is one such AI model that has been trained on a large corpus of text data to generate human-like responses to questions and prompts. Using the theory of disruptive innovation as a foundation for our argument, this conceptual article explores the potential and possible disruption of ChatGPT in online assessment. This article also considers the ethical and pedagogical implications of using ChatGPT, particularly in relation to online assessment in distance education. While the use of AI in online assessment presents a myriad of limitations and possibilities, it is crucial to approach its use with caution and consider the ethical implications of academic integrity for online assessment. This article aims to contribute to the ongoing discussion and debate around the use of A...
Framing Assessment Questions in the Age of Artificial Intelligence: Evidence from ChatGPT 3.5
Emerging Science Journal, 2024
With the rise of artificial intelligence (AI), higher education faces a significant challenge in learning assessment. The emergence of tools like ChatGPT raises concerns regarding the potential for cheating and the reliability of assessment outcomes. This paper aims to address these concerns by proposing a methodology for framing questions that effectively measures learning outcomes while reducing the risk of AI-enabled cheating. To achieve this objective, we employ a methodological approach that involves getting responses from ChatGPT 3.5 to various question prompts across different domains. These responses are then evaluated by faculty members specializing in management education. Through this process, we aim to identify question-framing strategies that effectively assess learning outcomes while minimizing susceptibility to AI Cheating. Our analysis reveals several key findings. Certain question Types (Decision Making, Recent Events, and Experiential Learning) demonstrate greater resilience against AI-generated responses, indicating their potential effectiveness in assessing student learning. This study offers original insights into the challenges and opportunities associated with learning assessment in the context of AI integration. The paper tries to provide valuable guidance for Policymakers, educators & students seeking to enhance the integrity and reliability of their assessment practices.
AI in Higher Education: A Literature Review of ChatGPT and Guidelines for Responsible Implementation
International Journal of Research and Innovation in Social Science (IJRISS), 2023
Specifically, natural language models like ChatGPT present numerous advantages and disadvantages for higher education. It was trained on a massive dataset of text from the internet, allowing it to respond to a broad range of prompts. This article covers the use and potential implications of ChatGPT since its release in November 2022. One of the main benefits is the potential of Artificial Intelligence (AI) to address challenges in learning, such as improving the transfer of knowledge, dispelling misconceptions, and promoting critical thinking skills among students. It also acknowledges concerns about its use in assessments and the potential for academic dishonesty, integrity, and malpractices. In this article, the authors discuss the high level of interest in ChatGPT and its use in education by reviewing the literature about ChatGPT usage. The article provides a set of guidelines and emphasizes the need for further research to fully understand the current practices, challenges, and opportunities of ChatGPT in higher education. The authors conducted a thorough and systematic review of peer-reviewed journal articles to present a theoretical and conceptual perspective on ChatGPT. They acknowledge the possibility of ChatGPT's hype mirroring that of previous advancements in artificial intelligence, automation, and AI algorithms. The article offers a summary of the research findings, highlighting both the benefits and drawbacks, while also providing practical guidelines for students and teachers on how to utilize ChatGPT effectively.
Challenges and Opportunities of ChatGPT in Higher Education
ChatGPT and similar text-based AI applications are set to radically reinvent how we go about almost any aspects of our lives that include a language component. This reinvention poses particular challenges for educational environments, where proof of authorship is fundamental to the integrity of assessment. AI undermines this proof of authorship and thus potentially the fundamental validity of current assessment practices. A re-evaluation of higher education assessment practices is therefore unavoidable. At the same time, the unprecedented linguistic, reasoning and stylistic abilities of current AI language models provide an opportunity to reimagine how we conduct our work on many levels, with potentially groundbreaking productivity and performance benefits. This paper explores these challenges and opportunities in relation to the higher education sector, primarily regarding assessment but also looking more broadly at the impacts on teaching and learning, research, and administration.
Educator and Student Perspectives on the Impact of Generative AI on Assessments in Higher Education
Proceedings of the Tenth ACM Conference on Learning @ Scale
The sudden popularity and availability of generative AI tools, such as ChatGPT that can write compelling essays on any topic, code in various programming languages, and ace standardized tests across domains, raises questions about the sustainability of traditional assessment practices. To seize this opportunity for innovation in assessment practice, we conducted a survey to understand both the educators' and students' perspectives on the issue. We measure and compare attitudes of both stakeholders across various assessment scenarios, building on an established framework for examining the quality of online assessments along six dimensions. Responses from 389 students and 36 educators across two universities indicate moderate usage of generative AI, consensus for which types of assessments are most impacted, and concerns about academic integrity. Educators prefer adapted assessments that assume AI will be used and encourage critical thinking, but students' reaction is mixed, in part due to concerns about a loss of creativity. The findings show the importance of engaging educators and students in assessment reform efforts to focus on the process of learning over its outputs, higher-order thinking, and authentic applications.
Higher education assessment practice in the era of generative AI tools
Journal of Applied Learning & Teaching , 2024
The higher education (HE) sector benefits every nation's economy and society at large. However, their contributions are challenged by advanced technologies like generative artificial intelligence (GenAI) tools. In this paper, we provide a comprehensive assessment of GenAI tools towards assessment and pedagogic practice and, subsequently, discuss the potential impacts. This study experimented using three assessment instruments from data science, data analytics, and construction management disciplines. Our findings are twofold: first, the findings revealed that GenAI tools exhibit subject knowledge, problem-solving, analytical, critical thinking, and presentation skills and thus can limit learning when used unethically. Secondly, the design of the assessment of certain disciplines revealed the limitations of the GenAI tools. Based on our findings, we made recommendations on how AI tools can be utilised for teaching and learning in HE.
Rethinking online assessment strategies: Authenticity versus AI chatbot intervention
Journal of Applied Learning & Teaching, 2023
As artificial intelligence (AI) and chatbot technologies like ChatGPT continue to evolve, educators grapple with the risks and benefits these advances bring to online assessment. The democratisation of AI-based technologies, while offering personalised learning experiences, threatens online assessment legitimacy and academic integrity. This paper critically examines the intersection of AI chatbots and online assessments, in the context of their impact on the design of authentic online assessments. The widespread usage of AI chatbots has caused serious problems for the validity of online tests because of the possibility of student abuse. This underlines the need for 'authentic assessments' that concentrate on higher-order cognitive skills, problem-solving, creative thinking, and collaborative talents and calls for a reevaluation of conventional assessment methods. These types of assessments not only align with the evolving pedagogical needs of the 21st century but also present tasks that are significantly challenging for AI chatbots to replicate, thereby preserving their integrity. Conversely, the paper also explores how AI can facilitate the assessment process by automating certain tasks, providing personalised learning experiences, and supporting collaborative assessments. The era of AI chatbots presents an opportunity to rethink and enhance online assessments, making them more authentic, meaningful, and resistant to AI-assisted malpractice.
The impact of artificial intelligence on online assessment: A preliminary review
Journal of Educational Technology and Online Learning, 2023
The purpose of this study is to examine the impact of artificial intelligence (AI) on online assessment in the context of opportunities and threats based on the literature. To this end, 19 articles related to the AI tool ChatCPT and online assessment were analysed through rapid literature review. In the content analysis, the themes of “AI's assistance role”, “automatic grading and feedback”, “improving assessment” and “time benefit” were obtained in the opportunities category, while the themes of “academic integrity concern”, “reliability issues” and “adaptability issues” were obtained in the threats category. The impact of AI on online assessment was explained within the scope of these themes. The results revealed that the most emphasis was placed on "improving assessment" themes in the opportunities category, and "academic integrity concern" themes in the threats category. At the end of this preliminary review, it was revealed that more studies investigating the integration of AI to online assessment are needed and all educational institutions, especially distance education institutions, should take measures to ensure the ethical use of AI.
AI-Driven Exam Evaluation Systems: Challenges, Innovations, and Future Directions
International Journal of Electronics Automation, 2024
A proposed AI system is used to grade exams automatically. It addresses inefficiencies in human assessment. A GPT model trained on graded replies is used for evaluation, and TrOCR is used for precise handwritten text recognition. Efficiency and less bias are provided by this method, although there are still issues. More work is needed to assess open-ended questions and make sure they are understandable. To automate many aspects of exam evaluation, including grading, feedback, and plagiarism detection, it first examines the evolution of AI technologies, including machine learning, deep learning, and natural language processing. It also examines the potential for AI-driven assessment tools to enhance learning outcomes, reduce teacher workloads, and provide students with personalized feedback. Additionally, the study highlights several challenges, such as addressing. Our algorithm makes use of developments in two important fields of AI. To reduce bias, careful curation of training data is required. In its conclusion, the study emphasizes how important it is that the system be able to handle different question formats, deal with ambiguities, and incorporate human assessment. A promising first step toward an efficient, equitable, and AI-powered exam grading system is this research.
Computers and Education: Artificial Intelligence
The growing use of generative AI tools built on large language models (LLMs) calls the sustainability of traditional assessment practices into question. Tools like OpenAI's ChatGPT can generate eloquent essays on any topic and in any language, write code in various programming languages, and ace most standardized tests, all within seconds. We conducted an international survey of educators and students in higher education to understand and compare their perspectives on the impact of generative AI across various assessment scenarios, building on an established framework for examining the quality of online assessments along six dimensions. Across three universities, 680 students and 87 educators, who moderately use generative AI, consider essay and coding assessments to be most impacted. Educators strongly prefer assessments that are adapted to assume the use of AI and encourage critical thinking, while students' reactions are mixed, in part due to concerns about a loss of creativity. The findings show the importance of engaging educators and students in assessment reform efforts to focus on the process of learning over its outputs, alongside higher-order thinking and authentic applications.