Nel, M. M.Burch, V. C.Schoeman, Frans Hendrik Scarpa2015-11-162015-11-162015-01http://hdl.handle.net/11660/1663English: Setting defensible and fair pass standards for high-stakes postgraduate specialist certification examinations is a critical quality assurance component of assessment. Doing so in a feasible and sustainable way, within a resource-constrained context such as South Africa, is challenging. Traditionally the 28 member Colleges of the Colleges of Medicine of South Africa (CMSA), the national specialist licensing examination body in South Africa, have used a fixed pass mark of 50%. This practice does not acknowledge the inherent variance in examination difficulty and so increases the risk of failing competent candidates (false negative outcome) and passing incompetent examinees (false positive outcome). In 2011, the College of Physicians (CoP), a large CMSA member College, addressed the matter by implementing a standard setting process for the written components of their specialist physician certification examinations. The aim of this study was twofold: i) To evaluate the knowledge, attitudes, views and perspectives of CoP examiners regarding standard setting, and ii) compare the performance and utility of the Cohen and Angoff methods to advise the CoP regarding an appropriate standard setting method in a resource-constrained setting. A literature review was done to conceptualise standard setting as it pertains to assessment in medical education. In addition, policies and regulatory systems relevant to specialist certification examinations in South Africa were reviewed to provide the context for this study. Two research components were concurrently conducted between 2012 - 2014: A prospective study evaluated the knowledge, attitudes, views and perspectives of CoP examiners regarding standard setting before and after training and 30 months of practical experience using both the Cohen and Angoff methods of standard setting. A comparative study evaluated the performance (pass marks and failure rates) and utility (according to a framework derived from the literature review) of the Cohen and Angoff methods using five cycles of examination data, including multiple choice questions (MCQ), short answer questions and short essay questions. The introduction of standard setting was successful and widely supported by the CoP examiners. The Cohen method performed well when used for test data with a reasonable number of test items (30 or more) in homogeneous exit-level cohorts of more than 50 candidates. Tests containing few test items (i.e. short essay questions) performed poorly. The performance of the Cohen method was variable for smaller cohorts (less than 100) of candidates drawn from heterogeneous populations, such as entry-level Part I MCQ test takers. The Angoff method yielded unacceptable outcomes regardless of test format. The utility comparison identified the Cohen method as the preferred standard setting method for the CoP. The findings of this study support the introduction and ongoing use of the Cohen method as a feasible and sustainable method of setting pass marks for the written components of the CoP certification examinations. Education and training in the use of standard setting methods, as part of a change management strategy, improved examiners’ understanding of the role, importance and basic methodology of standard setting and strengthened their support for the use of standard setting in certification examinations. More data are needed to evaluate the true impact of cohort size on the stability of the Cohen method for entry-level, heterogeneous cohorts of examinees. The purist Angoff strategy, used in this study due to resource limitations, performed poorly and was deemed ‘not fit for purpose’ by the CoP examiners. The usefulness of the novel standard setting utility framework developed in this study warrants further research in other examination settings such as performance–based examinations.Afrikaans: Om verdedigbare en regverdige slaagstandaarde te bepaal vir nagraadse spesialissertifiseringseksamens, wat belangrike gevolge het, is ʼn kritiese deel van gehalteversekering van assessering. Om dit te doen op ʼn haalbare en volhoubare manier, in ʼn hulpbronbeperkte konteks soos Suid-Afrika, is ʼn uitdaging. Tradisioneel het die 28 lid-kolleges van die Kolleges van Geneeskunde van Suid-Afrika (KGSA), die nasionale spesialis-lisensiëringseksamenliggaam in Suid-Afrika, ʼn vaste slaagsyfer van 50% gebruik. Hierdie praktyk reflekteer nie die inherente variasie in eksamenmoeilikheidsgraad nie en verhoog so die risiko om bevoegde kandidate verkeerdelik te druip (vals-negatiewe uitkoms) en om onbevoegde kandidate te laat slaag (vals-positiewe uitkoms). In 2011 het die Kollege van Interniste (KvI), ʼn groot KGSA lid-kollege, hierdie situasie aangespreek deur ʼn proses te implementeer om die slaagstandaard (slaagsyfer) van die skriftelike komponente van hulle spesialis-internis sertifiseringseksamens te bepaal. Die doel van hierdie studie was tweeledig: i) evalueer die kennis, houding, sienings en perspektiewe van KvI eksaminatore ten opsigte van slaagstandaard-bepaling, en ii) vergelyk die prestasie en nuttigheid van die Cohen- en Angoff-metodes om sodoende die KvI te adviseer ten opsigte van ʼn toepaslike metode vir slaagstandaard-bepaling, in ʼn hulpbronbeperkte omgewing. ʼn Literatuuroorsig is gedoen om slaagstandaard-bepaling te konseptualiseer, soos dit van toepassing is op assessering in mediese opvoedkunde. Daarbenewens, is beleide en regulatoriese sisteme, relevant tot spesialis-internis sertifiseringseksamens in Suid- Afrika, ook hersien en bygewerk om die konteks van die studie te skets. Twee navorsingskomponente was gelyktydig uitgevoer tussen 2012 – 2014: ʼn Prospektiewe studie het die kennis, houdings, sienings en perspektiewe van KvI eksaminatore geëvalueer ten opsigte van slaagstandaard-bepaling, voor en na opleiding en 30 maande se praktiese ervaring in die gebruik van beide die Cohen- en Angoff-metodes van slaagstandaard-bepaling. ʼn Vergelykende studie het die prestasie (slaagsyfers en druipkoerse) en nuttigheid (‘n raamwerk wat uit die literatuuroorsig ontwikkel is) van die Cohen- en Angoff-metodes ondersoek deur vyf siklusse van eksamendata te evalueer, insluitende veelvuldige keusevrae (VKV), kort-antwoordvrae en kort-opstelvrae. Die ingebruikneming van slaagstandaard-bepaling was suksesvol en het wye steun geniet onder die KvI eksaminatore. Die Cohen-metode het goed presteer op toetsdata met ʼn redelike aantal toetsitems (30 of meer) in homogene, uittreevlak kohorte van 50 kandidate of meer. Toetse met min items (kort-opstelvrae) het swak vertoon. Die prestasie van die Cohen-metode was variërend vir kleiner kohorte (minder as 100) van kandidate uit heterogene populasies, soos die intreevlak Deel I VKV toetsnemers. Die Angoff-metode het onaanvaarbare uitkomste gelewer, ongeag die toetsformaat. Die nuttigheidsvergelyking het die Cohen-metode geïdentifiseer as die voorkeurmetode vir slaagstandaard-bepaling in die KvI. Die bevindinge van hierdie studie ondersteun die ingebruikneming en voortgesette gebruik van die Cohen-metode as ʼn haalbare en volhoubare metode om die slaagsyfers van die skriftelike dele van die KvI se sertifiseringseksamens te bepaal. Onderrig en opleiding in die gebruik van slaagstandaard-bepalingsmetodes, as deel van ʼn veranderingsbestuurstrategie, verbeter eksaminatore se begrip van die rol, belangrikheid en basiese metodiek van slaagstandaard-bepaling en versterk hulle steun vir die gebruik van slaagstandaard-bepaling in sertifiseringseksamens. Meer data is nodig om die ware impak van kohortgrootte op die stabiliteit van die Cohen-metode vir intreevlak-heterogene kohorte van kandidate te evalueer. Die puristiese Angoffstrategie, wat weens hulpbrontekorte in hierdie studie gebruik is, het swak presteer en is as ‘nie-geskik vir die doel’ deur die KvI eksaminatore geag. Die bruikbaarheid van die nuwe slaagstandaard-bepaling nuttigheidsraamwerk, wat ontwikkel is in hierdie studie, regverdig verdere navorsing in ander eksamenomgewings, soos prestasiegebaseerde eksamens.enAngoff methodAssessmentChange managementCohen methodLicensing examinationsMedical specialist certificationPostgraduate medical educationQuality assuranceResource-limited assessmentStandard settingWritten assessmentPhysiciansMedicine -- Study and teachingThesis (Ph.D. (Health Professions Education))--University of the Free State, 2015ThesisUniversity of the Free State