Relating language examinations to the common European framework of reference for languages: learning, teaching, assessment (CEFR) (original) (raw)

2009, A Manual. Strasbourg: …

The Linking Process 2.1. Approach Adopted 2.2. Quality Concerns 2.3. Stages of the Process 2.4. Use of the CEFR 2.5. Use of the Manual 2.1. Approach Adopted Relating an examination or test to the CEFR is a complex endeavour. The existence of a relationship between the examination and the CEFR is not a simple observable fact, but is an assertion for which the examination provider needs to provide both theoretical and empirical evidence. The procedure by which such evidence is obtained is in fact the "validation of the claim". Relating (linking) examinations or tests to the CEFR presupposes standard setting, which can be defined as a process of establishing one or more cut scores on examinations. These cut scores divide the distribution of examinees' test performances into two or more CEFR levels. Appropriate standards can be best guaranteed if the due process of standard setting is attended to from the beginning. Standard setting involves decision making which requires high-quality data and rigorous work. As these decisions may have important consequences, they need to be fair, open, valid, efficient and defensible. This can be facilitated by the use of well-tried systematic processes and explicit criteria. In standard setting, it is usual to refer to content standards and performance standards. Content standards describe the content domain from which the examination can be or has been constructed. Very frequently this description refers to performance levels. Such descriptions are by necessity general and usually formulated in qualitative terms. In standard setting literature they are referred to as "Performance Level Descriptors" (PLDs: See Section 6.7.) and act as a general reference system against which particular examinations can be described. Performance standards refer to specific examinations and express the minimum performance on that specific test or examination; in this sense they are synonymous to "cut scores". There is, however, one major point which needs to be stressed. The Common European Framework of Reference for Languages (CEFR) provides the content and Performance Level Descriptors. The PLDs are given, unlike the situation in most standard setting in other contexts, where the PLDs first need to be defined. This means that the CEFR needs to be referred to at all stages of the linking process as illustrated in Figure 2.1. The approach adopted in this Manual is such that thorough familiarity with the CEFR is a fundamental requirement. Standard Setting (Chapter 6): The crucial point in the process of linking an examination to the CEFR is the establishment of a decision rule to allocate students to one of the CEFR levels on the basis of their performance in the examination. Usually this takes the form of deciding on cutoff scores, borderline performances. The preceding stages of Familiarisation, Specification and Standardisation can be seen as preparatory activities to lead to valid and rational decisions. Chapter 6 describes procedures to arrive at the final decision of setting cut scores. The material presented there draws on an extensive literature on standard setting, and the procedures presented in Chapter 6 are a selection from the many available procedures deemed to be suitable in the context of language testing. Additional procedures based on the exploitation of teacher judgments and IRT to incorporate an external criterion (e.g. CEFR illustrative items, or teacher assessments with CEFR illustrative descriptors) into a linking study are presented in Extra Material provided by Brian North and Neil Jones. Validation (Chapter 7): While the preceding stages of Familiarisation, Specification, Standardisation and Standard Setting can be conceived roughly to represent a chronological order of activities, it would be naïve to postpone validation activities until everything has been done, and to conceive it as an ultimate verdict on the quality of the linking process. Validation must rather be seen as a continuous process of quality monitoring, giving an answer to the general question: "Did we reach the aims set for this activity?" A simple, but nevertheless important example has already been referred to: it is important to provide CEFR familiarisation and standardisation training, but it is equally important to check if such activities have been successful; this is precisely what is meant by validation. Aspects of validity and procedures to collect validity evidence are described in this final chapter. Aspects of validity and procedures on how to collect validity evidence have been put together in the final chapter (Chapter 7) of this Manual. 2.4. Use of the CEFR A common framework of reference enables different examinations be to related to each other indirectly without any claim that two examinations are exactly equivalent. The focus of examinations may vary but their coverage can be profiled with the categories and levels of the framework. In the same way that no two learners at Level B2 are at Level B2 for the same reason, no two examinations at Level B2 have completely identical profiles. The parts of the CEFR most relevant for linking examinations are:  Chapter 3 "The Common Reference Levels";  Chapter 4 "Language Use and the Language User"with scales for Communicative Language Activities and for Communicative Language Strategies;  Chapter 5 "The User/Learner's Competences", particularly Section 5.2 "Communicative Language Competences" with the illustrative scales for aspects of linguistic, pragmatic and socio-linguistic competence. Users of this Manual will find the full text of the CEFR and related documents, plus a number of useful tools on the Council of Europe website, including the following: Documents  The CEFR in English and French, including appendices.  Links to other language versions on the Council of Europe website (www.coe.int/lang; www.coe.int/portfolio)  The Manual, including appendices.  The forms and reference Grids included in the Manual. 12  The Reference Supplement. Content Analysis Grids  CEFR Content Analysis Grid for listening and reading (sometimes referred to as "the Dutch CEFR Grid"): Appendix B1.  CEFR Content Analysis Grids for speaking and writing, developed by ALTE: Appendix B2.