The incremental validity of three tests of academic literacy in the context of a South African university of technology

Abstract

English: This study focuses on the incremental validity of three assessments of academic language readiness, compared to Grade 12 English results: the National Benchmark Test in Academic Literacy (NBT AL), the Proficiency Test English Second Language Advanced Level (PTESLAL) and the Test of Academic Literacy Levels (TALL) at the end of the first year of academic study. More specifically, the study investigates the ability of any of the four assessments to predict first year academic performance better than the others. Where those that are examined do not possess this ability, the further question is asked: can they at least add to the predictive power of the best predictor? Ultimately, the aim is to determine if the assessments designed to provide additional information about first year academic preparedness are valid for this purpose, and the extent to which this is the case. The study starts with a brief exploration of the literature on the reportedly low levels of academic language ability among first time entrants to higher education in South Africa in recent years, and the consequent need for the development and use of valid tests of academic language ability for channelling these students into academic language interventions that are aimed at dealing with this challenge. The literature on the current theories of validity is also explored in relation to the hypothesis of the study, which is that as a test designed to provide additional information about the academic language readiness of first year students, TALL will possess better incremental validity in relation to the best predictor of first year academic performance. Subsequently, an attempt is made to account for the ability of Grade 12 English results to predict first year academic performance better than the other three assessments investigated in the present study. Similarly, an effort is made to account for the ability of TALL to show evidence of incremental validity in relation to Grade 12 English results and the inability of NBT AL and PTESLAL to do the same. Furthermore, on the basis of the results of previous studies and the current one, a recommendation is made that Grade 12 results in general and Grade 12 English results in particular be used together with those of academic literacy tests to make access and placement decisions. The basis for this recommendation resides in the psychometric and other shortcomings of Grade 12 results that have been identified by previous studies as well as the evidence that similar studies have produced to show that tests of academic literacy possess better ability to partition test taker performance from different school backgrounds and at different levels of performance. Finally, the implications of the results of the study for current theories of test validity are discussed. In the main, the discussion focuses on demonstrating on the basis of these results and those of previous studies that the currently popular theory of validity wherein a unitary approach to validity is upheld and the interpretation and use of test scores are regarded as the essence of validation does not hold. At the same time, the discussion focuses on demonstrating that the traditional theory of validity, wherein validity is believed to reside in the objective ability of a test to produce valid scores and a distinction is made between the three traditional types of validity, namely construct, content and criterion-related validity is, with certain obvious qualifications, still defensible. Finally, the implications of the results of the study for validity theory are dealt with in relation to the validity of courses of academic literacy.

Afrikaans: Hierdie studie fokus op die inkrementele geldigheid van drie assesserings van akademiese taalgereedheid, te wete die Nasionale Normtoets vir Akademiese Geletterdheid (NBT AL), die Proficiency Test English Second Language Advanced Level (PTESLAL), en die Test of Academic Literacy Levels (TALL), in vergelyking met die Graad 12-resultate vir Engels. Die studie ondersoek in die besonder die vermoë van elk van die genoemde toetse om die akademiese prestasie van eerstejaars beter te kan voorspel as die ander. Waar die toetse wat ondersoek word nie daardie vermoë het nie, word daar 'n verdere vraag gevra, naamlik of die toets ten minste bydra tot die voorspellingsvermoë van die beste voorspeller? Die uiteindelike doel is om te bepaal of assesserings wat ontwerp is om addisionele inligting te bied oor die voorbereidheid van studente vir eerstejaarstudie geldig is vir hierdie doel, en tot watter mate dit die geval is. Die ondersoek begin met 'n bondige verkenning van die literatuur oor die beweerde lae vlakke van akademiese taalvaardigheid by nuwelingstudente aan instellings van hoër onderwys in Suid-Afrika die afgelope aantal jaar, sowel as die gevolglike behoefte aan die ontwikkeling en aanwending van geldige toetse van akademiese taalvermoë ten einde daardie studente te kan kanaliseer na akademiese taalintervensies wat ten doel het om hierdie uitdaging te bowe te kom. Die literatuur oor huidige teorieë oor geldigheid word ook verken en binne konteks van die hipotese van die ondersoek. Dit is naamlik dat as 'n toets wat ontwerp is om addisionele inligting oor die akademiese taalgereedheid van studente te verskaf, TALL 'n groter moontlikheid bied om inkrementele geldigheid te vertoon in verhouding tot die beste aanwyser van eerstejaars se akademiese prestasie. Vervolgens word daar gepoog om 'n verklaring te verky vir die vermoë van die Graad 12-resultate in Engels om eerstejaars se akademiese prestasie beter te kan voorspel as enige ander assessering wat in hierdie studie gebruik is. Op dieselfde wyse word daar gepoog om die nodige bewyse te vind vir die vermoë van TALL om inkrementele geldigheid toe te voeg tot die Graad 12-resultate, asook vir waarom die ander twee assesserings, NBT AL en PTESLAL, dit nie het nie. Op basis van die resultate van vorige studies, asook van hierdie ondersoek, word die aanbeveling gemaak dat die Graad 12-resultate in die algemeen, en die Graad 12-resultate vir Engels in die besonder, tesame met die resultate van 'n akademiese geletterdheidtoets gebruik moet word om besluite te neem oor toegang en plasing. Die gronde vir hierdie aanbeveling lê in die psigometriese en ander tekortkomings van die Graad 12-resultate, wat geblyk het uit vorige studies, en wat ook geïdentifiseer is deur ondersoeke soortgelyk aan hierdie een. Uit hierdie ondersoeke blyk dit dat toetse van akademiese geletterdheid veral goed kan vaar om te onderskei tussen diegene wat die toets deurloop het se onderskeie opvoedkundige agtergronde en hul verskillende vlakke van prestasie. Ten besluite word die implikasies van die studie se resultate oorweeg betreffende huidige teorieë van toetsgeldigheid. Die bespreking fokus hoofsaaklik daarop om aan te toon dat hierdie resultate en die van soortgelyke studies ’n basis bied waarteen populêre geldigheidsteorie bevraagteken sou kon word, in die besonder aannames onderliggend aan die idee van een saambindende aanpak om geldigheid te bepaal, asook om die interpretasie van toetsresultate as die essensie van geldigmaking voor te hou. Terselfdertyd dui die bespreking daarop dat die tradisionele siening van geldigheid, waarin dit geag word verbind te wees aan die objektiewe vermoë van 'n toets om geldige resultate te lewer, steeds met sekere voorbehoude regverdigbaar is, asook die onderskeid wat getref kan word tussen die tradisionele drietal tipes geldigheid, naamlik konstruk-, inhouds- en kriteriumgeldigheid. Ter afsluiting word die implikasies van die resultate van hierdie ondersoek vir geldigheidsteorie in verband gebring met die geldigheid van akademiese geletterdheidskursusse.