Determining the Level of a Language Test with English Profile: A Forensic Linguistics Case Study
AbstractThis article deals with a forensic linguistics case study of the determination of the level of a B1 English multiple-choice test that was challenged in court by numerous candidates on the grounds that it was not of the appropriate level. A control corpus comprising 240 analogous multiple-choice questions from B1 exams aligned with the Common European Framework of Reference for Languages (CEFR) was compiled in order to establish a threshold for the percentage of questions of a level higher than that being tested which can be expected in such exams. The analysis was carried out following a combination of qualitative and quantitative methods, with the help of the tool English Profile, which provides Reference Level Descriptions (RLDs) for the English language within the CEFR. The results of the analysis of the control corpus established a baseline of 5 to 7% of questions that include key items classified as higher than B1, while the percentage was 68% in the case of the disputedexam. Thus, the present study proposes a further application of the tool English Profile within the field of forensic linguistics and puts forward the concept of Level Appropriateness Threshold (LAT), analogous to other thresholds established in forensic linguistics, which can serve as a baseline for determining the appropriateness of B1 English multiple-choice exams and a model for other levels and skill areas.
Archer, Dawn et al., eds. 2003. Proceedings of the Corpus Linguistics 2003 Conference. UCREL Technical Paper, 16. Lancaster: Lancaster University.
Arsland, Abdullah and Ali Erarslan. 2019. “Lexical Analysis of a Textbook Based on the EVP.” International Journal of Languages’ Education and Teaching 7 (1): 1-12.
Banerjee, Jayanti and Dina Tsagari, eds. 2016. Contemporary Second Language Assessment. London: Bloomsbury.
Barker, Fiona. 2010. “How Can Corpora Be Used in Language Testing?” In O’Keeffe and McCarthy 2010, 633-45.
—. 2014. “Using Corpora to Design Assessment.” In Kunnan 2014, 1013-28.
Callies, Marcus and Sandra Götz, eds. 2015. Learner Corpora in Language Testing and Assessment. Amsterdam and Philadelphia: John Benjamins.
Cambridge Assessment English. 2020. “Validity and Validation.” [Accessed online on October 30, 2019].
Cambridge University Press. 2015. English Profile – What the CEFR Means for English. [Accessed online April 2, 2019].
Carlsen, Cecilie. 2012. “Proficiency Level: A Fuzzy Variable in Computer Learner Corpora.” Applied Linguistics 33 (2): 161-83.
Coulthard, Malcolm. 2004. “Author Identification, Idiolect and Linguistic Uniqueness.” Applied Linguistics 25 (4): 431-47.
Coulthard, Malcolm and Alison Johnson, eds. 2010. The Routledge Handbook of Forensic Linguistics. London and New York: Routledge.
Council of Europe. 2001a. The Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge UP. [Accessed online on October 30, 2019].
—. 2001b. Structured Overview of all CEFR Scales. [Accessed online on October 30, 2019].
—. Reference Level Descriptions (RLDs) (Language by Language). [Accessed online on October 30, 2019].
—. 2020. Reference Level Descriptions (RLDs) Developed so Far. [Accessed online on October 30, 2020].
Cushing, Sara T. 2017. “Corpus Linguistics in Language Testing Research.” Language Testing 34 (4): 441-49.
Escola Oficial d’Idiomes Barcelona Drassanes. “Mission, Vision and Values.” [Accessed online on October 30, 2019].
Garayzábal Heinze, Elena, Sheila Queralt Estévez and Mercedes Reigosa Riveiros. 2019. Fundamentos de la lingüística forense. Madrid: Síntesis.
Gibbons, John and M. Teresa Turell, eds. 2008. Dimensions of Forensic Linguistics. Amsterdam and Philadelphia: John Benjamins.
Grant, Tim. 2007. “Quantifying Evidence in Forensic Authorship Analysis.” The International Journal of Speech, Language and the Law 14 (1): 1-25.
Harrison, Julia and Fiona Barker, eds. 2015. English Profile Studies 5. English Profile in Practice. Cambridge: CUP.
Knight, Ben. 2015. “Applications of English Profile.” In Harrison and Barker 2015, 93-105.
Kunnan, Antony John, ed. 2014. The Companion to Language Assessment. Vol 2, Approaches and Development. Hoboken, NJ: Wiley-Blackwell.
Lavery, Urban A. 1921. “The Language of the Law”. American Bar Association Journal, 7 (6), 277-83.
Leńko-Szymańska, Agnieszka. 2015. “The English Vocabulary Profile as a Benchmark for Assigning Levels to Learner Corpus Data.” In Callies and Götz 2015, 115-40.
Ley Orgánica 8/2013. Boletín Oficial del Estado, December 10.
Marquina, Montse and Sheila Queralt. 2014. “Similarity Threshold to Detect Plagiarism in Spanish.” RAEL: Revista Electrónica de Lingüística Aplicada 13 (1): 79-95.
Montolío, Estrella. 2012. “La modernización del discurso jurídico español impulsada por el Ministerio de Justicia. Presentación y principales aportaciones del Informe sobre el lenguaje escrito.” Revista de llengua i dret 57: 95-121.
Nicholls, Diane. 2003. “The Cambridge Learner Corpus: Error Coding and Analysis for Lexicography and ELT.” In Archer et al. 2003, 572–82.
O’Keeffe, Anne and Michael McCarthy, eds. 2010. The Routledge Handbook of Corpus Linguistics. London and New York: Routledge.
Park, Kwanghyun. 2014. “Corpora and Language Assessment: The State of the Art.” Language Assessment Quarterly 11 (1): 27-44.
Poblete, Claudia, Lisbeth Arenas, Alejandro Córdova, Emmy González and Daniela Tapia. 2018. Estrategias en comprensión del discurso escrito en contextos jurídicos. Valparaíso: Ediciones Universitarias de Valparaíso.
Stygall, Gail. 2010. “Legal Writing: Complexity”. In Coulthard and Johnson 2010, 51-64.
Tiersma, Peter. 1999. Legal Language. Chicago, IL: U of Chicago P.
Turell, M. Teresa. 2007. “Plagio y traducción literaria.” Vasos Comunicantes 37 (1): 43-54.
—. 2008. “Plagiarism”. In Gibbons and Turell 2008, 265-99.
—. 2010. “The Use of Textual, Grammatical and Sociolinguistic Evidence in Forensic Text Comparison.” The International Journal of Speech, Language and the Law 17 (2): 211-50.
University of Cambridge Local Examinations Syndicate (UCLES) and Cambridge University Press (CUP). 2011. English Profile: Introducing the CEFR for English Version 1.1. [Accessed online on April 2, 2019].
University of Cambridge ESOL Examinations. 2011. Using the CEFR: Principles of Good Practice. [Accessed online on April 2, 2019].
Weigle, Sara Cushing and Sarah Goodwin. 2016. “Applications of Corpus Linguistics in Language Assessment.” In Banerjee and Tsagari 2016, 209-23.
The authors retain copyright of articles. They authorise AEDEAN to publish them in its journal Atlantis and to include them in the indexing and abstracting services, academic databases and repositories the journal participates in.
Under the terms of the Creative Commons Attribution NonCommercial ShareAlike 4.0 International Licence (CC BY-NC-SA 4.0), for non-commercial (i.e., personal or academic) purposes only, users are free to share (i.e., copy and redistribute in any medium or format) and adapt (i.e., remix, transform and build upon) articles published in Atlantis, free of charge and without obtaining prior permission from the publisher or the author(s), as long as they give appropriate credit to the author, the journal (Atlantis) and the publisher (AEDEAN), provide the relevant URL link to the original publication and indicate if changes were made. Such attribution may be done in any reasonable manner, but not in any way that suggests the journal endorses the user or their use of the material published therein. Users who adapt (i.e., remix, transform or build upon the material) must distribute their contributions under the same licence as the original.
Self-archiving is also permitted, so that authors are allowed to deposit the published PDF version of their articles in academic and/or institutional repositories, without fee or embargo. Authors may also post their individual articles on their personal websites, again on condition that the original link to the online edition is provided.
Authors are expected to know and heed basic ground rules that preclude simultaneous submission and/or duplicate publication. Prospective contributors to Atlantis commit themselves to the following when they submit a manuscript:
- That no concurrent consideration of the same, or almost identical, work by any other journal and/or publisher is taking place.
- That the potential contribution has not appeared previously, in any form whatsoever, in another journal, electronic format or as a chapter/section of a book.
Seeking permission for the use of copyright material is the responsibility of the author.