Standard setting for specialist physician examinations in South Africa

English: Setting defensible and fair pass standards for high-stakes postgraduate specialist certification examinations is a critical quality assurance component of assessment. Doing so in a feasible and sustainable way, within a resource-constrained context such as South Africa, is challenging. Traditionally the 28 member Colleges of the Colleges of Medicine of South Africa (CMSA), the national specialist licensing examination body in South Africa, have used a fixed pass mark of 50%. This practice does not acknowledge the inherent variance in examination difficulty and so increases the risk of failing competent candidates (false negative outcome) and passing incompetent examinees (false positive outcome). In 2011, the College of Physicians (CoP), a large CMSA member College, addressed the matter by implementing a standard setting process for the written components of their specialist physician certification examinations. The aim of this study was twofold: i) To evaluate the knowledge, attitudes, views and perspectives of CoP examiners regarding standard setting, and ii) compare the performance and utility of the Cohen and Angoff methods to advise the CoP regarding an appropriate standard setting method in a resource-constrained setting. A literature review was done to conceptualise standard setting as it pertains to assessment in medical education. In addition, policies and regulatory systems relevant to specialist certification examinations in South Africa were reviewed to provide the context for this study. Two research components were concurrently conducted between 2012 - 2014: A prospective study evaluated the knowledge, attitudes, views and perspectives of CoP examiners regarding standard setting before and after training and 30 months of practical experience using both the Cohen and Angoff methods of standard setting. A comparative study evaluated the performance (pass marks and failure rates) and utility (according to a framework derived from the literature review) of the Cohen and Angoff methods using five cycles of examination data, including multiple choice questions (MCQ), short answer questions and short essay questions. The introduction of standard setting was successful and widely supported by the CoP examiners. The Cohen method performed well when used for test data with a reasonable number of test items (30 or more) in homogeneous exit-level cohorts of more than 50 candidates. Tests containing few test items (i.e. short essay questions) performed poorly. The performance of the Cohen method was variable for smaller cohorts (less than 100) of candidates drawn from heterogeneous populations, such as entry-level Part I MCQ test takers. The Angoff method yielded unacceptable outcomes regardless of test format. The utility comparison identified the Cohen method as the preferred standard setting method for the CoP. The findings of this study support the introduction and ongoing use of the Cohen method as a feasible and sustainable method of setting pass marks for the written components of the CoP certification examinations. Education and training in the use of standard setting methods, as part of a change management strategy, improved examiners’ understanding of the role, importance and basic methodology of standard setting and strengthened their support for the use of standard setting in certification examinations. More data are needed to evaluate the true impact of cohort size on the stability of the Cohen method for entry-level, heterogeneous cohorts of examinees. The purist Angoff strategy, used in this study due to resource limitations, performed poorly and was deemed ‘not fit for purpose’ by the CoP examiners. The usefulness of the novel standard setting utility framework developed in this study warrants further research in other examination settings such as performance–based examinations.