Methodological Synthesis in Quantitative L2 Research: A Review of Reviews and a Case Study of Exploratory Factor Analysis (original) (raw)
Related papers
This study provides a systematic review of the methodological features of meta-analyses in second language learning. The synthesis aims to inform how metaanalyses in L2 learning have been conducted, evaluate whether methodological decisions are aligned with norms and standards, identify issues, and suggest solutions based on expert advice and statistical guides. A total of 120 meta-analyses were retrieved and coded for key features related to bibliographic and demographic characteristics, search and selection, publication bias, quality control, data coding, data analysis, and effect size interpretation. The synthesis showed that 98 metaanalyses examined the effectiveness of instructional treatments, 21 investigated correlations, and one explored the occurrence of events. These meta-analyses included an average of 37 primary studies (range = 9-302). Common selection criteria the meta-analyses applied included publication type, availability of data for effect size calculation, learner traits, learners' target languages, publication dates, publication language, independent variables, dependent variables, and so on. Major strategies used to detect publication bias included creating a funnel plot, using trimand-fill analysis, and calculating a fail-safe N. Typical moderators examined in the meta-analyses related to research context, treatment features, sample characteristics, and outcome measures. The synthesis also identified a number of issues, including failure to report key features such as model selection (fixed-vs. randomeffects model), effect size weighting, and so on; conducting moderator analysis based on very small cell sizes (e.g., only one study in a subgroup); lack of justification for certain methodological decisions such as using d instead of g, using confidence intervals rather than Q-tests to identify significant moderators; lack of quality
Factor analysis has been frequently exploited in applied research to provide evidence about the underlying factors in various measurement instruments. A close inspection of a large number of studies published in leading applied linguistic journals shows that there is a misconception among applied linguists as to the relative merits of exploratory factor analysis and principal components analysis (PCA) and the kind of interpretations that can be drawn from each method. In addition, it is argued that the widespread application of orthogonal, rather than oblique, rotations and also the criteria used for factor selection are not in keeping with the findings in psychometrics. It is further argued that the current situation is partly due to the fact that PCA and orthogonal rotation are default options in mainstream statistical packages such as SPSS and the guidebooks on such software do not provide an explanation of the issues discussed in this article.
Exploratory Factor Analysis (EFA) in Quantitative Researches and Practical Considerations
Explanatory factor analysis (EFA) is a multivariate statistical method frequently used in quantitative research and has begun to be used in many fields such as social sciences, health sciences and economics. With EFA, researchers focus on fewer items that explain the structure, instead of considering too many items that may be unimportant and carry out their studies by placing these items into meaningful categories (factors). However, for over sixty years, many researchers have made different recommendations about when and how to use EFA. Differences in these recommendations confuse the use of EFA. The main topics of discussion are sample size, number of items, item extraction methods, factor retention criteria, rotation methods and general applicability of the applied procedures. The abundance of these discussions and opinions in the literature makes it difficult for researchers to decide which procedures to follow in EFA. For this reason, it would be beneficial for researchers to ...
STATISTICAL REPORT REFORM IN SECOND LANGUAGE RESEARCH: A CASE OF EXPERIMENTAL DESIGNS
JEELS (Journal of English Education and Linguistics Studies), 2021
This survey aims to review statistical report procedures in the experimental studies appearing in ten SLA and Applied Linguistic journals from 2011 to 2017. We specify our study on how the authors report and interpret their power analyses, effect sizes, and confidence intervals. Results reveal that of 217 articles, the authors reported effect sizes (70%), Apriori power and Post Hoc power consecutively (1.8% and 6.9%), and confidence intervals (18.4%). Additionally, it reveals that the authors interpret those statistical terms counted 5.5%, 27.2%, and 6%, respectively. The call for statistical report reform recommended and endorsed by scholars, researchers, and editors is inevitably echoed to shed more light on the trustworthiness and practicality of the data presented.
Exploratory factor analysis (EFA) is a complex, multi-step process. The goal of this paper is to collect, in one article, information that will allow researchers and practitioners to understand the various choices available through popular software packages, and to make decisions about " best practices " in exploratory factor analysis. In particular, this paper provides practical information on making decisions regarding (a) extraction, (b) rotation, (c) the number of factors to interpret, and (d) sample size. Exploratory factor analysis (EFA) is a widely utilized and broadly applied statistical technique in the social sciences. In recently published studies, EFA was used for a variety of applications, including developing an instrument for the evaluation of school principals (Lovett, Zeiss, & Heinemann, 2002), assessing the motivation of Puerto Rican high school students (Morris, 2001), and determining what types of services should be offered to college students (Majors & Sedlacek, 2001). A survey of a recent two-year period in PsycINFO yielded over 1700 studies that used some form of EFA. Well over half listed principal components analysis with varimax rotation as the method used for data analysis, and of those researchers who report their criteria for deciding the number of factors to be retained for rotation, a majority use the Kaiser criterion (all factors with eigenvalues greater than one). While this represents the norm in the literature (and often the defaults in popular statistical software packages), it will not always yield the best results for a particular data set. EFA is a complex procedure with few absolute guidelines and many options. In some cases, options vary in terminology across software packages, and in many cases particular options are not well defined. Furthermore, study design, data properties, and the questions to be answered all have a bearing on which procedures will yield the maximum benefit. The goal of this paper is to discuss common practice in studies using exploratory factor analysis, and provide practical information on best practices in the use of EFA. In particular we discuss four issues: 1) component vs. factor extraction, 2) number of factors to retain for rotation, 3) orthogonal vs. oblique rotation, and 4) adequate sample size. BEST PRACTICE Extraction: Principal Components vs. Factor Analysis PCA (principal components analysis) is the default method of extraction in many popular statistical software packages, including SPSS and SAS, which likely contributes to its popularity. However, PCA is
QUESTIONNAIRES IN SECOND LANGUAGE RESEARCH
Questionnaires in Second Language Research, 2023
Administration/Dornyei-Dewaele/p/book/9781032364315?fbclid=IwAR3t26N-e09sefiX51CRiTNpPR7Atuf_VhpR0BvX_SGeIaD7w0jRsAu6_rY Book Description Questionnaires in Second Language Research is the first state-of-the-art methodological guide for producing and using questionnaires as reliable and valid research instruments in second language studies. Zoltán Dörnyei and Jean-Marc Dewaele provide a comprehensive, reader-friendly overview of the theory of questionnaire design, administration, and processing, made accessible with a detailed how-to guide and concrete, real-life applications. This new edition is thoroughly updated to reflect developments in the field and with recent example studies that focus on considerations, challenges, and opportunities raised at all stages of the research process by online questionnaires. There is also expanded, detailed guidance on how to use the IRIS database and how to clean, process, and analyze questionnaire data prior to determining and reporting findings. This is an invaluable resource to students and researchers of SLA, applied linguistics, psychology, and education who are interested in understanding and conducting quantitative L2 research using questionnaires and surveys.