The incremental validity of three tests of academic literacy in the context of a South African university of technology

Loading...
Thumbnail Image
Date
2016
Authors
Sebolai, Kabelo Wilson
Journal Title
Journal ISSN
Volume Title
Publisher
University of the Free State
Abstract in other languages 𝘚𝘤𝘳𝘰𝘭𝘭 𝘥𝘰𝘸𝘯 𝘧𝘰𝘳 𝘈𝘧𝘳𝘪𝘬𝘢𝘢𝘯𝘴, 𝘚𝘦𝘚𝘰𝘵𝘩𝘰 𝘢𝘯𝘥 𝘐𝘴𝘪𝘡𝘶𝘭𝘶
Abstract
𝑬𝒏𝒈𝒍𝒊𝒔𝒉 This study focuses on the incremental validity of three assessments of academic language readiness, compared to Grade 12 English results: the National Benchmark Test in Academic Literacy (NBT AL), the Proficiency Test English Second Language Advanced Level (PTESLAL) and the Test of Academic Literacy Levels (TALL) at the end of the first year of academic study. More specifically, the study investigates the ability of any of the four assessments to predict first year academic performance better than the others. Where those that are examined do not possess this ability, the further question is asked: can they at least add to the predictive power of the best predictor? Ultimately, the aim is to determine if the assessments designed to provide additional information about first year academic preparedness are valid for this purpose, and the extent to which this is the case. The study starts with a brief exploration of the literature on the reportedly low levels of academic language ability among first time entrants to higher education in South Africa in recent years, and the consequent need for the development and use of valid tests of academic language ability for channelling these students into academic language interventions that are aimed at dealing with this challenge. The literature on the current theories of validity is also explored in relation to the hypothesis of the study, which is that as a test designed to provide additional information about the academic language readiness of first year students, TALL will possess better incremental validity in relation to the best predictor of first year academic performance. Subsequently, an attempt is made to account for the ability of Grade 12 English results to predict first year academic performance better than the other three assessments investigated in the present study. Similarly, an effort is made to account for the ability of TALL to show evidence of incremental validity in relation to Grade 12 English results and the inability of NBT AL and PTESLAL to do the same. Furthermore, on the basis of the results of previous studies and the current one, a recommendation is made that Grade 12 results in general and Grade 12 English results in particular be used together with those of academic literacy tests to make access and placement decisions. The basis for this recommendation resides in the psychometric and other shortcomings of Grade 12 results that have been identified by previous studies as well as the evidence that similar studies have produced to show that tests of academic literacy possess better ability to partition test taker performance from different school backgrounds and at different levels of performance. Finally, the implications of the results of the study for current theories of test validity are discussed. In the main, the discussion focuses on demonstrating on the basis of these results and those of previous studies that the currently popular theory of validity wherein a unitary approach to validity is upheld and the interpretation and use of test scores are regarded as the essence of validation does not hold. At the same time, the discussion focuses on demonstrating that the traditional theory of validity, wherein validity is believed to reside in the objective ability of a test to produce valid scores and a distinction is made between the three traditional types of validity, namely construct, content and criterion-related validity is, with certain obvious qualifications, still defensible. Finally, the implications of the results of the study for validity theory are dealt with in relation to the validity of courses of academic literacy. ___________________________________________________________________
𝑨𝒇𝒓𝒊𝒌𝒂𝒂𝒏𝒔 Hierdie studie fokus op die inkrementele geldigheid van drie assesserings van akademiese taalgereedheid, te wete die Nasionale Normtoets vir Akademiese Geletterdheid (NBT AL), die Proficiency Test English Second Language Advanced Level (PTESLAL), en die Test of Academic Literacy Levels (TALL), in vergelyking met die Graad 12-resultate vir Engels. Die studie ondersoek in die besonder die vermoë van elk van die genoemde toetse om die akademiese prestasie van eerstejaars beter te kan voorspel as die ander. Waar die toetse wat ondersoek word nie daardie vermoë het nie, word daar 'n verdere vraag gevra, naamlik of die toets ten minste bydra tot die voorspellingsvermoë van die beste voorspeller? Die uiteindelike doel is om te bepaal of assesserings wat ontwerp is om addisionele inligting te bied oor die voorbereidheid van studente vir eerstejaarstudie geldig is vir hierdie doel, en tot watter mate dit die geval is. Die ondersoek begin met 'n bondige verkenning van die literatuur oor die beweerde lae vlakke van akademiese taalvaardigheid by nuwelingstudente aan instellings van hoër onderwys in Suid-Afrika die afgelope aantal jaar, sowel as die gevolglike behoefte aan die ontwikkeling en aanwending van geldige toetse van akademiese taalvermoë ten einde daardie studente te kan kanaliseer na akademiese taalintervensies wat ten doel het om hierdie uitdaging te bowe te kom. Die literatuur oor huidige teorieë oor geldigheid word ook verken en binne konteks van die hipotese van die ondersoek. Dit is naamlik dat as 'n toets wat ontwerp is om addisionele inligting oor die akademiese taalgereedheid van studente te verskaf, TALL 'n groter moontlikheid bied om inkrementele geldigheid te vertoon in verhouding tot die beste aanwyser van eerstejaars se akademiese prestasie. Vervolgens word daar gepoog om 'n verklaring te verky vir die vermoë van die Graad 12-resultate in Engels om eerstejaars se akademiese prestasie beter te kan voorspel as enige ander assessering wat in hierdie studie gebruik is. Op dieselfde wyse word daar gepoog om die nodige bewyse te vind vir die vermoë van TALL om inkrementele geldigheid toe te voeg tot die Graad 12-resultate, asook vir waarom die ander twee assesserings, NBT AL en PTESLAL, dit nie het nie. Op basis van die resultate van vorige studies, asook van hierdie ondersoek, word die aanbeveling gemaak dat die Graad 12-resultate in die algemeen, en die Graad 12-resultate vir Engels in die besonder, tesame met die resultate van 'n akademiese geletterdheidtoets gebruik moet word om besluite te neem oor toegang en plasing. Die gronde vir hierdie aanbeveling lê in die psigometriese en ander tekortkomings van die Graad 12-resultate, wat geblyk het uit vorige studies, en wat ook geïdentifiseer is deur ondersoeke soortgelyk aan hierdie een. Uit hierdie ondersoeke blyk dit dat toetse van akademiese geletterdheid veral goed kan vaar om te onderskei tussen diegene wat die toets deurloop het se onderskeie opvoedkundige agtergronde en hul verskillende vlakke van prestasie. Ten besluite word die implikasies van die studie se resultate oorweeg betreffende huidige teorieë van toetsgeldigheid. Die bespreking fokus hoofsaaklik daarop om aan te toon dat hierdie resultate en die van soortgelyke studies ’n basis bied waarteen populêre geldigheidsteorie bevraagteken sou kon word, in die besonder aannames onderliggend aan die idee van een saambindende aanpak om geldigheid te bepaal, asook om die interpretasie van toetsresultate as die essensie van geldigmaking voor te hou. Terselfdertyd dui die bespreking daarop dat die tradisionele siening van geldigheid, waarin dit geag word verbind te wees aan die objektiewe vermoë van 'n toets om geldige resultate te lewer, steeds met sekere voorbehoude regverdigbaar is, asook die onderskeid wat getref kan word tussen die tradisionele drietal tipes geldigheid, naamlik konstruk-, inhouds- en kriteriumgeldigheid. Ter afsluiting word die implikasies van die resultate van hierdie ondersoek vir geldigheidsteorie in verband gebring met die geldigheid van akademiese geletterdheidskursusse. ___________________________________________________________________
𝑺𝒆𝑺𝒐𝒕𝒉𝒐 Diphuputso tsena di fuputsa bokgoni ba keketso ba ditlhahlobo tse tharo tsa boemo bo lokileng ba puo e sebediswang thutong, ho bapiswa le diphetho tsa Kereiti ya 12 tsa English: National Benchmark Test in Academic Literacy (NBT AL), the Proficiency Test English Second Language Advanced Level (PTESLAL) esita le Test of Academic Literacy Levels (TALL) qetellong ya selemo sa pele sa boithuto. Ka ho qolleha, diphuputso tsena di batlisisa bokgoni ba e nngwe le e nngwe ya ditlhahlobo tsena tse nne ho ka totobatsa bokgoni ba thuto ba selemong sa pele ho feta ditlhahlobo tse ding. Moo tse hlahlobilweng di se nang bokgoni bona, potso e latelang e ba hore: na bonyane di ka eketsa matla a ho lepa ho molepi wa mmakgonthe? Qetellong sepheo ke ho fumana hore na ditlhahlobo tsena di ka fana ka dintlha tse eketsehileng mabapi le ho lokela selemo sa pele sa dithuto hakae bakeng sa morero oo, le hore na seo se ka lekanyetswa boemong bofe. Diphuputso tsena di qala ka tekodiso e kgutshwanyane ya dingolwa ka dipehelo tsa boemo bo tlase ba bokgoni ba puo ho ba qalang selemong sa pele thutong e phahameng mona Afrika Borwa dilemong tsena tsa jwale, le lebaka la ho etswa le ho sebediswa ha diteko tse tshwanelehang tsa bokgoni ba puo e le ho tataisa baithuti bana ho kena mererong ya ho thuswa e reretsweng ho rarolla bona bothata kapa phephetso ena. Dingolwa tsa diteori tsa hona jwale tsa ho tshwaneleha le tsona di a hlahlojwa mabapi le menahano ya diphuputso tsena, e leng hore jwaloka teko e etseditsweng ho fana ka dintlha tse ekeditsweng mabapi le boitokiso le ho lokela puo ha baithuti ba selemo sa pele puong, TALL e tla ba le bokgoni bo betere ba keketso ho amana le molepi wa mmakgonthe wa tshebetso ya selemo sa pele. Ho latela mona, ho entswe mekutu ya ho fumana bonnete ba bokgoni ba Kereiti ya 12 ho diphetho tsa English e le ho nahanela pele diphetho tsa tshebetso ya selemo sa pele thutong hore di tla ba betere ho feta ditlhahlobo tse ding tse tharo tse fupuditsweng diphuputsong tsona tsena. Ka tsela e jwalo, ho entswe mekutu ya ho hlalosa bakeng sa bokgoni ba xv TALL e le ho fana ka bopaki ba bokgoni ba keketso mabapi le Kereiti ya 12 diphethong tsa puo ya English le ho se kgone ha NBT AL le PTESLAL ho etsa jwalo. Ho feta mona, ka lebaka la diphetho tsa diphuputso tse fetileng le tsa jwale, ho entswe kgothaletso ya hore diphetho tsa Kereiti ya 12 ka kakaretso le diphetho tsa Kereiti ya 12 tsa English ka ho qolleha di sebetswe hammoho le tsane tsa ho be bobebe ho fihlellwa le ho etsa diqeto tsa ho behwa sebakeng se itseng. Lebaka la kgothaletso ena le amana le bofokodi ba diphetho tsa Kereiti ya 12 tse hlwailweng ke diphuputso tse fetileng esita le bopaki boo diphuputso tse jwalo di bo hlahisitseng e le ho bontsha hore diteko tsa ho bala le ho ngola di na le bokgoni bo betere ba ho bontsha tshebetso ya motho ya entseng teko e le motho ya tswang maemong a fapaneng a dikolo le maemong a fapaneng a tshebetso. Qetellong mehopolo e meng ya diphetho tsa diphuputso tsa diteori tsa jwale tsa tshwaneleho ya diteko di ile tsa hlaloswa. Tabeng ya bohlokwa, dipuisano di bontsha ho ya ka diphetho tsena le tsa diphuputso tse fetileng hore ha jwale teori e tlwaelehileng ya tiisetso ya bokgoni moo katamelo ya bokgoni e nkwang e le tlhaloso le tshebediso ya diphetho tsa teko mme e nkwa e le sesosa sa tiiso, le hore ha e tshwarelle hakaalo. Ka yona tsela e jwalo, dipuisano di bontsha hore teori ya sekgalekgale ya tiiso ya bongoni kapa bohlokwa e leng moo boleng ho dumelwang hore bo bakwa ke bokgoni ba nnetefatso ka ho etsa diteko ho hlahisa matshwao a kgodisang mme ho etswa phapano pakeng tsa mefuta e meraro ya kgodiso, e leng kaho, dikateng le kgodiso e hlahiswang ka tekanyetso, e ntse e ka tshehetswa. Qetellong, mehopolo e meng ya diphetho tsa diphuputso bakeng sa teori ena ya kgodiso kapa boleng e sebetswa ho ya ka boleng ba dithuto tsa ho bala le ho ngola. ___________________________________________________________________
𝑰𝒔𝒊𝒁𝒖𝒍𝒖 Lolu cwaningo lugxile ekwenyukeni kokufaneleka kokuhlolwa okuthathu kokulungela ulimi lwezemfundo, uma kuqhathaniswa nemiphumela yesiNgisi yeBanga le-12: i-National Benchmark Test in Academic Literacy (NBT AL), i-Proficiency Test English Language Second Advanced Level (PTESLAL) kanye ne-Test of Academic Literacy Level (TALL) ekupheleni konyaka wokuqala wokufunda. Ngokucacile, ucwaningo luphenya ikhono lanoma yikuphi ukuhlola kokune okubikezela ukusebenza kahle kwezemfundo konyaka wokuqala kunokunye. Lapho labo abahlolwayo bengenalo leli khono, umbuzo olandelayo uyabuzwa: ingabe bangakwazi yini ukwengeza emandleni okubikezela ngesibikezeli esingcono kakhulu? Ekugcineni, inhloso ukuthola ukuthi ukuhlola okuklanyelwe ukuhlinzeka ngolwazi olwengeziwe mayelana nokulungela izifundo zonyaka wokuqala kuyasebenza yini kule njongo, kanye nezinga okwenzeka ngalo lokhu. Ucwaningo luqala ngokuhlolwa okufushane kwemibhalo emayelana namazinga okubikwa ukuthi aphansi ekhono lolimi ezifundweni phakathi kwabaqalayo ukungena emfundweni ephakeme eNingizimu Afrikha eminyakeni yamuva nje, kanye nesidingo esiwumphumela sokuthuthukiswa nokusetshenziswa kwezivivinyo ezisebenzayo zekhono lokufunda ulimi ukuze ukufaka laba bafundi ekungeneleleni kolimi lwezemfundo okuhloswe ngalo ukubhekana nale nselelo. Izincwadi ezikhuluma ngenjulalwazi yamanje yokuba semthethweni ziyahlolwa futhi maqondana nenkolelo-mbono yocwaningo, okungukuthi njengovivinyo oluklanyelwe ukunikeza ulwazi olwengeziwe mayelana nokulungela kolimi lwezemfundo lwabafundi abenza unyaka wokuqala, i-TALL izoba nokuqinisekiswa okungcono okukhuphukayo maqondana nesibikezelo esihle kakhulu sokusebenza konyaka wokuqala. Kamuva, kwenziwa umzamo wokulandisa ngekhono lemiphumela yesiNgisi yeBanga le-12 lokubikezela ukusebenza kahle kwezemfundo onyakeni wokuqala kunokunye ukuhlola okuthathu okuphenywe ocwaningweni lwamanje. Ngokufanayo, kwenziwa umzamo wokulandisa ngekhono le-xv TALL ukukhombisa ubufakazi bokufaneleka okukhulayo ngokuphathelene nemiphumela yesiNgisi yeBanga le-12 kanye nokungakwazi kwe-NBT AL kanye ne-PTESLAL ukwenza okufanayo. Ukwengeza, ngokwesisekelo semiphumela yezifundo zangaphambilini kanye nalolu lwamanje, kwenziwa isincomo sokuthi imiphumela yeBanga le-12 jikelele kanye nemiphumela yesiNgisi yeBanga le-12 ikakhulukazi isetshenziswe kanye naleyo yokuhlolwa kokukwazi ukufunda nokubhala kwezemfundo ukuze kwenziwe izinqumo zokufinyelela nokuzibeka endaweni. Isisekelo salesi sincomo sisekelwe emiphumeleni ye-psychometric kanye nokunye ukushiyeka kwemiphumela yeBanga le-12 okuhlonzwe izifundo zangaphambilini kanye nobufakazi bokuthi ucwaningo olufanayo luye lwaveza ukukhombisa ukuthi izivivinyo zolwazi lokufunda nokubhala zinamandla angcono okuhlukanisa abenzi bokuhlolwa abahlukene, izizinda zesikole kanye namazinga ahlukene okusebenza. Okokugcina, kuxoxiswana ngemiphumela yemiphumela yocwaningo lwenjulalwazi yamanje yokufaneleka kokuhlolwa. Ngokuyinhloko, ingxoxo igxile ekuboniseni ngesisekelo sale miphumela kanye naleyo yezifundo zangaphambilini ukuthi inkolelo-mbono ethandwayo njengamanje yokufaneleka lapho indlela eyodwa yokuqinisekisa isekelwa futhi ukuchazwa nokusetshenziswa kwamaphuzu okuhlola kuthathwa njengengqikithi yokuqinisekisa. Ngesikhathi esifanayo, ingxoxo igxile ekuboniseni ukuthi injulalwazi yendabuko yokufaneleka, lapho ukufaneleka kukholakala ukuthi kuhlezi ekhonweni lenhloso yokuhlola ukuze kukhiqizwe amaphuzu avumelekile futhi kwenziwa umehluko phakathi kwezinhlobo ezintathu zendabuko zokufaneleka, okungukuthi ukwakha, okuqukethwe kanye nokufaneleka okuhlobene nemibandela, ngeziqu ezithile ezisobala, kusangavikeleka. Okokugcina, imiphumela yemiphumela yocwaningo lwenjulalwazi yokuqinisekisa kubhekwana nayo ngokuphathelene nokuba semthethweni kwezifundo zokufunda nokubhala. ___________________________________________________________________
Description
Thesis (Ph.D.(Linguistics and Language Practice))--University of the Free State, 2016
Keywords
Academic literacy, Incremental validity, Validity theory, NBT AL, PTESLAL, TALL, Grade 12 English, Academic performance, Academic under-preparedness, Literacy -- South Africa, Students -- South Africa
Citation