Human and automated assessment of oral reading fluency (original) (raw)
Related papers
Performance of Automated Scoring for Children’s Oral Reading
2011
For adult readers, an automated system can produce oral reading fluency (ORF) scores (e.g., words read correctly per minute) that are consistent with scores provided by human evaluators (Balogh et al., 2005, and in press). Balogh's work on NAAL materials used passage-specific data to optimize statistical language models and scoring performance. The current study investigates whether or not an automated system can produce scores for young children's reading that are consistent with human scores. A novel aspect of the present study is that text-independent rule-based language models were employed (Cheng and Townshend, 2009) to score reading passages that the system had never seen before. Oral reading performances were collected over cell phones from 1st, 2nd, and 3rd grade children (n = 95) in a classroom environment. Readings were scored 1) in situ by teachers in the classroom, 2) later by expert scorers, and 3) by an automated system. Statistical analyses provide evidence th...
Automatic Assessment of Children's L2 Reading for Accuracy and Fluency
This project targets using state-of-the-art in automatic speech recognition technology, coupled with new work in predicting the relevant prosody ratings, to build an oral reading assessment tool. A reliable automatic system can prove invaluable in helping children acquire basic reading skills apart from facilitating the monitoring of literacy programs at large scale. In the present work, we target middle-school learners of English as a second language in a rural Indian setting. We present the design and observed characteristics of our field-collected oral reading dataset to outline the research challenges faced. Recently proposed solutions to the training of robust acoustic models in the face of limited task specific data are evaluated for the prediction of the child's word decoding accuracy and for achieved word-level alignments for prosody scoring. A language model is designed to exploit the known text and observed reading errors while being flexible enough to adapt to new reading material without further training. Based on a scoring rubric proposed by a national mission on literacy assessment in India, we present an automatic system that detects reading miscues and computes fluency indicators at the sentence level which are then correlated with fine-grained subjective ratings by an expert.
This paper describes a study exploring automated assessment of reading proficiency, in terms of oral reading and reading comprehension, for a middle school population including students with reading disabilities and low reading proficiency, utilizing automatic speech recognition technology. We build statistical models using features related to fluency, pronunciation, and reading accuracy to predict three dependent variables: two are related to accuracy and speed of reading, the third is a reading comprehension measure from a state assessment of reading. The correlation coefficients of the best-performing linear regression models range from r = 0.64 (reading comprehension score) to 0.98 (correctly read words per minute). We further look at the features with the highest absolute regression weights in the three models and find that most of them fall into the classes of reading accuracy and reading speed. Still, features from the pronunciation class and other fluency features, e.g., relating to silences in the read speech, are also represented in the regression models but with less emphasis.
Automatically Assess Children’s Reading Skills
2020
Assessing reading skills is an important task teachers have to perform at the beginning of a new scholastic year to evaluate the starting level of the class and properly plan next learning activities. Digital tools based on automatic speech recognition (ASR) may be really useful to support teachers in this task, currently very time consuming and prone to human errors. This paper presents a web application for automatically assessing fluency and accuracy of oral reading in children attending Italian primary and lower secondary schools. Our system, based on ASR technology, implements the Cornoldi’s MT battery, which is a well-known Italian test to assess reading skills. The front-end of the system has been designed following the participatory design approach by involving end users from the beginning of the creation process. Teachers may use our system to both test student’s reading skills and monitor their performance over time. In fact, the system offers an effective graphical visual...
Grantee Submission, 2015
Assessing reading fluency is critical because it functions as an indicator of comprehension and overall reading achievement. Although theory and research demonstrate the importance of ORF proficiency, traditional ORF assessment practices are lacking as sensitive measures of progress for educators to make instructional decisions. The purpose of this study is to compare traditional ORF measures/administration to a computerized ORF assessment system based on speech recognition software (CORE). Using WCPM scores as the outcome, we compare: (a) traditional ORF passages to CORE passages, (b) CORE passage lengths, and (c) scoring methods. We used a general linear model with two within-subject factors, to test the mean WCPM score differences between passage length, scoring method, and their interaction. We found that CORE passages, whether short, medium, or long, tended to have higher WCPM means than the Traditional ORF passages. Real-time scoring tended to have higher WCPM means than both Audio Recording and ASR scoring types, and the ASR and Recording scores were quite similar across passage length and grade, providing preliminary evidence that the speech recognition scoring engine can score passages as well as human administrators in real settings.
Automatic Assessment of Reading with Speech Recognition Technology
2016
In this paper, we describe ongoing research towards building an automatic reading assessment system that emulates a human expert in a spoken language learning scenario. Audio recordings of read aloud English stories by children of grades 6-8 are acquired on an available tablet application that facilitates guided oral reading and recording. The created recordings, uploaded to a web-based ratings panel, are currently evaluated by human experts on four relevant dimensions. Observations of typical learner progress patterns will form the bases of a system that applies Automatic Speech Recognition (ASR) techniques to obtain robust automatic predictions of reading fluency and word decoding accuracy.
Automatic assessment of children’s oral reading skills
2020
Fluent reading is a critical component of literacy skills and necessary for overall personality development. Proper assessment and feedback to students are therefore essential. This project is an effort to automate reading skill evaluation. We collected data of English stories read by Indian children who are L2 speakers of English, and asked teachers to rate it on different lexical and prosodic attributes. The ratings were then predicted using a machine learning model trained with different acousticprosodic features. The work aims at determining the optimal set of predictive features for rating the paragraphs for the scoring attributes. The trained machine learning models are expected to accurately mimic the teachers’ ratings on unseen test data.
Behavioral Research and Teaching, 2016
There is strong theoretical support for oral reading fluency (ORF) as an essential building block of reading proficiency. The current and standard ORF assessment procedure requires that students read aloud a grade-level passage (≈ 250 words) in a one-to-one administration, with the number of words read correctly in 60 seconds constituting their ORF performance. The current study was part of a larger project to develop and validate a computerized ORF assessment system-Computerized Oral Reading Evaluation (CORE)-to reduce limitations in current ORF measures and procedures. The purposes of this technical report are to: (a) document whether the CORE system was accessible and useful for teachers, (b) explore potential differences between CORE and traditional ORF (i.e., easyCBM) passages, and (c) identify potential deficits in the three CORE lengths (≈ 25, 50, or 85 words). This information contributes to the response-process evidence for the CORE system's validity. Our results suggest that delivering, scoring, and storing ORF assessments online may be feasible, and desirable, for classroom teachers across Grades 2 through 4. In addition, although there were no distinct differences between CORE and traditional ORF passages, teacher reports suggest that the CORE short passages are most appropriate for Grade 2 students while CORE long and medium passages are preferred by teachers for students in Grades 3 and 4.
Automatic assessment of children's reading level
… Annual Conference of …, 2007
In this paper, an automatic system for the assessment of reading in children is described and evaluated. The assessment is based on a reading test with 40 words, presented one by one to the child by means of a computerized reading tutor. The score that expresses the ...
Automatic evaluation of reading aloud performance in children
Speech Communication
Evaluating children's reading aloud proficiency is typically a task done by teachers on an individual basis, where reading time and wrong words are marked manually. A computational tool that assists with recording reading tasks, automatically analyzing them and outputting performance related metrics could be a significant help to teachers. Working towards that goal, this work presents an approach to automatically predict the overall reading aloud ability of primary school children by employing automatic speech processing methods. Reading tasks were designed focused on sentences and pseudowords, so as to obtain complementary information from the two distinct assignments. A dataset was collected with recordings of 284 children aged 6-10 years reading in native European Portuguese. The most common disfluencies identified include intra-word pauses, phonetic extensions, false starts, repetitions, and mispronunciations. To automatically detect reading disfluencies, we first target extra events by employing task-specific lattices for decoding that allow syllable-based false starts as well as repetitions of words and sequences of words. Then, mispronunciations are detected based on the log likelihood ratio between the recognized and target words. The opinions of primary school teachers were gathered as ground truth of overall reading aloud performance, who provided 0-5 scores closely related to the expected performance at the end of each grade. To predict these scores, various features were extracted by automatic annotation and regression models were trained. Gaussian process regression proved to be the most successful approach. Feature selection from both sentence and pseudoword tasks give the closest predictions, with a correlation of 0.944 compared to the teachers' grading. Compared to the use of manual annotation, where the best models obtained give a correlation of 0.949, there was a relative decrease of only 0.5% for using automatic annotations to extract features. The error rate of predicted scores relative to ground truth also proved to be smaller than the deviation of evaluators' opinion per child.