About MEAJO | Editorial board | Search | Ahead of print | Current Issue | Archives | Instructions to authors | Online submission | Subscribe | Advertise | Contact | Login 
Middle East African Journal of Ophthalmology Middle East African Journal of Ophthalmology
Users Online: 102   Home Print this page Email this page Small font sizeDefault font sizeIncrease font size


 
  Table of Contents 
OPHTHALMIC EDUCATION UPDATE
Year : 2014  |  Volume : 21  |  Issue : 2  |  Page : 109-113  

Assessment principles and tools


University of Cincinnati and the Cincinnati Eye Institute, Cincinnati, Ohio, USA

Date of Web Publication1-Apr-2014

Correspondence Address:
Karl C Golnik
University of Cincinnati and the Cincinnati Eye Institute, Cincinnati, Ohio - 45242
USA
Login to access the Email id

Source of Support: Research to Prevent Blindness,, Conflict of Interest: None


DOI: 10.4103/0974-9233.129746

Rights and Permissions
   Abstract 

The goal of ophthalmology residency training is to produce competent ophthalmologists. Competence can only be determined by appropriately assessing resident performance. There are accepted guiding principles that should be applied to competence assessment methods. These principles are enumerated herein and ophthalmology-specific assessment tools that are available are described.

Keywords: Assessment, Education, Feedback


How to cite this article:
Golnik KC. Assessment principles and tools. Middle East Afr J Ophthalmol 2014;21:109-13

How to cite this URL:
Golnik KC. Assessment principles and tools. Middle East Afr J Ophthalmol [serial online] 2014 [cited 2019 Aug 25];21:109-13. Available from: http://www.meajo.org/text.asp?2014/21/2/109/129746


   Introduction Top


Assessing resident performance is an essential part of the training process and integral to the development of a competent ophthalmologist. Without good assessments in place, one cannot be sure if the resident has learned what is expected. Appropriate tools are needed for assessment, which should be based on accepted principles. This article will review relevant principles of assessment and ophthalmology-specific assessment tools that have already been published and are available for use. Many variations of the principles of assessment have been described. Williams previously described ten specific guidelines for good assessment and I describe those I believe to be most relevant to medical training assessment. [1]


   Assessment Principles Top


0Principle
"Time is valuable and limited". Ophthalmologist teachers are typically very busy and are paid little or nothing for their teaching activities. Thus, any assessment should be as time efficient as possible but still provide the needed information. The need for brevity must be balanced and at the same time gather adequate valid and reliable information. Too brief an assessment may not provide what is needed and if it is too long, the assessor may not complete the assessment. The most important performance aspects only should be assessed and one should be ready to act on the assessment results to maximize their effect.

Principle
"Test application of knowledge/skills rather than just facts". Although there is clearly a need to assess basic memorized ophthalmic knowledge, assessment of surgical and clinical skill is also essential. A multiple-choice test may be adequate for measuring medical knowledge but it is not a good measure of procedural skill. This means that multiple assessment tools must be used to assess important aspects of the resident's performance. Ideally, multiple assessors will also be involved.

Principle
"Performance is situation specific". Good performance in one situation (e.g. medical knowledge multiple-choice test) does not mean there will be good performances in other situations (e.g. surgical skill or examination techniques). Nevertheless, sometimes it is assumed that the "smart" resident is also professional or a good communicator. Clearly this is not always the case and thus assessment tools must be used to evaluate multiple aspects of the resident's performance.

Principle
"Define assessments clearly in advance". Assessments should not be kept a secret. Let the residents know exactly how they will be assessed and be sure the assessments are aligned with your objectives and performance expectations. Doing this will minimize unnecessary assessments and assure relevant assessments are performed.

Principle
"Make every assessment a learning experience". This may not always be possible if the assessment is an examination that leads to passing or failing a course, rotation or residency. This type of assessment is known as summative feedback. On the contrary, formative feedback is assessment that is designed to improve performance and thus must be shared with the resident. The vast majority of assessments should be formative. When the assessment is shared with the resident on a frequent basis and is designed to improve performance it is called "teaching" Good formative feedback is timely and specific and ideally given every day. It should be used to reinforce good behavior (make a good resident even better) and used to change undesirable behavior.

Good assessment methods and tools have become increasingly important as the spectrum of desired competencies and abilities are more clearly defined and emphasized. Both the Accreditation Council for Graduate Medical Education (ACGME) [2] in the United States and the Royal College of Physicians and Surgeons in Canada [3] have developed and emphasized competencies thought essential to become a "good" physician. These competencies are similar to certain extent and are listed in [Table 1] and [Table 2]. Assessment tools must be available to assure the resident that he/she has achieved these competencies. Fortunately, variety of ophthalmology-specific assessment tools has already been developed to assess these competencies.
Table 1: The ACGME core competencies

Click here to view
Table 2: The CanMEDS competencies

Click here to view



   Assessment Tools Top


Many ophthalmology programs have unpublished and very subjective methods of resident assessment. Typically, these will be evaluations carried out by the teaching faculty where residents are rated on a Likert scale (e.g. 1-5 or 1-10). This form of assessment is probably better than not having a rating scale but it does not meet many of the assessment principles described above. Additionally, medical knowledge is usually assessed by multiple-choice tests at a program or national level. The International Council of Ophthalmology (ICO) offers the only internationally developed test of ophthalmic medical knowledge. The ICO examinations include both basic and advanced levels. [4]

However, other forms of assessment tools are necessary to assess procedures and skill. These tools usually consist of rubrics ideally with behavioral descriptors. A rubric is a tool that can help one give timely, specific, structured feedback and is defined as an explicit set of criteria used to assess a particular skill. Rubrics can be used to assess virtually any skill including playing a musical instrument, lecturing, or performing a surgical procedure. Good rubrics consist of three parts: (1) Dimensions (e.g. steps of a surgical procedure); (2) Levels (e.g. score of 1-5 or novice, beginner, advanced beginner, competent, expert); and (3) Behavioral Descriptors (what it means to perform at a certain level for any of the dimensions). For example, an assessment rubric for cataract surgery might include dimensions of prepping and draping the patient; and levels from 1-5; and descriptions of exactly what behavior is necessary to score 1-5. Specific examples of procedural rubrics will be described below and are available at (www.icoph.org). Rubrics should be given to the resident in advance so they will serve both, as a tool for teaching and assessing. The resident can read the rubric to learn what is required to be competent in various stages of the procedure. The rubrics should be completed by the teacher immediately after the observed performance and then reviewed with the resident to provide structured, timely and specific formative feedback as described above. Additionally, the rubric allows more objective, structured assessment of resident performance thus allowing areas of deficiency to be identified and remediated.

The first of these ophthalmology-specific assessment rubrics was developed to assess patient care and called the Ophthalmic Clinical Evaluation Exercise (OCEX). [5] The specified tool is completed by a teaching physician as they observe the resident performing a patient history, examination and then listens to the case presentation. The teaching physician completes scoring in 33 categories that rate the residents' ability to communicate effectively, perform a history and examination, and synthesize the information into a differential diagnosis and plan. Importantly, a rubric that describes the behavior necessary to achieve each grade on the OCEX was developed. The OCEX has been shown to have content validity and inter-rater reliability. [5],[6] It is available on the ICO's website in multiple languages (www.icoph.org). It was not developed by an international panel and thus may need to be modified to reflect cultural differences. The OCEX is a valid and reliable tool for assessing the competencies of patient care, medical knowledge and communication skills.

Several methods of surgical skill assessment have been devised. Cremers and associates developed the "Objective Assessment of Skills in Intraocular Surgery" (OASIS), a one-page objective evaluation form to assess residents' skills in cataract surgery. [7] The form is completed by an evaluator who directly observes the surgical procedure and includes objective data such as wound placement and size, phacoemulsification time, and total surgical time, etc., They showed that the OASIS had both face and content validity. To complement this objective assessment the same group developed a subjective rating of surgical skills named "Global Rating Assessment of Skills in Intraocular Surgery" (GRASIS). [8] This one-page form allows the evaluator to assign scores from 1-5 based on a behaviorally anchored rubric to domains such as preoperative knowledge, microscope use, instrument handling, and tissue treatment in addition to seven other areas. Thus the use of the combination of the OASIS and GRACIS provides objective and subjective evaluation of surgical skills. Feldman and Geist described the Subjective Phacoemulsification Skills Assessment as an evaluative instrument designed specifically for intraoperative assessment of resident phacoemulsification cataract extraction (PCE) surgery. [9] This form delineates PCE into overall performance and specific steps of the procedure (e.g. capsulorrhexis, hydrodelineation, IOL implantation, etc). The performance was graded with a rubric defining a good outcome at each step and asking the evaluator to rate on a 1-5 spectrumfrom 'strongly agree' to 'strongly disagree'. They were able to show a degree of inter-rater reliability.

Saleh and colleagues described an assessment tool called the "Objective Structured Assessment of Cataract Surgical Skill" (OSACSS). [10] This tool breaks down the phacoemulsification procedure into 20 steps that are scored on a 5-point Likert scale. The scale anchors are: 1= "poorly or inadequately performed", 3= "performed with some errors or hesitation" and 5= "performed well with no prompting or hesitation". There are no scale anchors for scores of 2 or 4. Further, there is no behavioral or skill-based rubric for the evaluators to use when assessing the resident's competence. An international panel of author's modified the OSACSS by producing a globally-applicable rubric with levels based on the Dreyfus model of skill acquisition (novice, beginner, advanced beginner, competent, and expert) and with behavioral anchors for each level in each step of the surgical procedure was created. [11] Once drafted, the content and face validity were achieved by having an international panel of 15 experts review the draft instrument and provide feedback. The international panel was composed of representatives from North America, Latin America, Asia, the Middle East, Europe and Africa. After incorporating suggestions from the international panel, a final document, the ICO-Ophthalmology Surgical Competency Assessment Rubric (OSCAR) - phacoemulsification was produced. [11] Furthermore, it was subsequently shown to have inter-rater reliability. [12] In a similar fashion internationally applicable assessment tools for extracapsular cataract surgery (ICO-OSCAR: ECCE), [11] small incision cataract surgery (ICO-OSCAR: SICS), [13] lateral tarsal strip surgery (ICO-OSCAR: LTS), [14] and strabismus surgery (ICO-OSCAR: Strabismus) [15] were developed. Additional ICO-OSCARS are being developed for panretinal photocoagulation, corneal transplant, vitrectomy, trabeculectomy and epiretinal membrane peeling. Recently, another cataract surgery assessment tool was developed in Canada and was shown to have a degree of validity and reliability. [16] It was not created by an international panel nor does it has a specific behaviorally grounded rubric.

The ICO-OSCAR assessment tools serve a variety of purposes: (1) They are internationally applicable, as comments from an international panel of experts were used to adapt it and make it flexible to any setting; (2) they will decrease subjectivity of the assessment by clearly defining for the assessor what behavior must be observed for each level of proficiency; (3) the rubric will clearly communicate to the learner what is expected to attain competence, and thus this tool can be used for both assessment and teaching.

Other authors have investigated surgical skills outside of actual human surgery. Fisher and associates developed the Eye Surgical Skills Assessment Test (ESSAT); a 3-station (skin suturing, muscle recession, phacoemulsification/wound construction and suturing technique) wet laboratory surgical skills obstacle course for ophthalmology residents. [17] In contrast to other surgical assessments, the ESSAT is designed to evaluate residents' basic skills before entering the operating room. Lee and associates developed an ophthalmology wet laboratory curriculum for teaching and assessing cataract surgical competency. [18] The specified curriculum includes pre- and post-tests of cognitive skills in addition to a structured wet lab curriculum with observed ratings of surgical skill. The same group from Iowa has shown that changes in their surgical curriculum have decreased resident complications during cataract surgery. [19] Finally, surgical simulation may afford non-patient methods of surgical skill assessment. Le and associates have shown that the anti-tremor and anterior forceps modules of the Eyesi simulator's have construct validity and may be valuable assessment (and teaching) tool in the future. [20]

Medical educators have also been devising methods to teach and assess the competencies other than patient care and medical knowledge. Lee and associates described an assessment tool involving a structured assessment of journal club that leads to assessment of practice-based learning and improvement. [21],[22] Golnik and associates developed and showed validity of a tool to assess resident on-call performance, the "On Call Assessment Tool" (OCAT). [23] The OCAT is a one page checklist to be used retrospectively during random chart review of on-call consultations. Finally, a "360 degree" evaluation tool can be very helpful in evaluating professionalism and communication skills. The intention is to get feedback of the residents from all the groups they interact. Thus, faculty evaluations can be considered 90 degrees and the patients' surveys, ancillary staff and peer evaluations constitute the other 270 degrees. Self-evaluations may be utilized in addition. Of course, patients and staff are not asked about the resident's medical knowledge but rather about professionalism and communication skills. Jagadeesan and associates have shown that their patients' satisfaction surveys can discriminate levels of resident communication skill and thus may be useful to assess this competency. [24] Internationally, cultural differences may produce difficulties with this type of tool. The information must be anonymous and the evaluators must believe this as the case. Emphasizing to the assessors that this is an attempt to improve the young doctors' performance can facilitate acceptance of this method. To my knowledge, no internationally valid 360-degree assessment tool exists and thus the ICO is currently developing one in a manner similar to the ICO-OSCARs.


   Conclusion Top


There are accepted principles of assessment that should guide our assessment of resident performance. A number of ophthalmology-specific resident assessment tools has been developed. Ideally, every resident training program would utilize a minimum test of medical knowledge, rubrics to assess procedural and clinical skill and a 360-degree evaluation. Assessment of the resident's performance is essential in both teaching and demonstrating the graduating residents, capable to function as competent ophthalmologists.

 
   References Top

1.Williams Ten Commandments of Principle Assessment. Available from: http://www.facs.org/education/rap/williams.html. [Last accessed on 2013 Oct 26].  Back to cited text no. 1
    
2.ACGME Common Program Requirements. Pgs. 8-11. Available from: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf. [Last accessed on 2013 Oct 26].  Back to cited text no. 2
    
3.The CanMEDS 2005 Physician Competency Framework. Available from: http://www.ub.edu/medicina_unitateducaciomedica/documentos/CanMeds.pdf. [Last accessed on 2013 Oct 26].  Back to cited text no. 3
    
4.The ICO examinations. Available from: http://icoph.org. [Last accessed on 2013 Oct 26].  Back to cited text no. 4
    
5.Golnik KC, Goldenhar LM, Gittinger JW Jr, Lustbader JM. The Ophthalmic Clinical Evaluation Exercise (OCEX). Ophthalmology 2004;111:1271-4.  Back to cited text no. 5
    
6.Golnik KC, Goldenhar L. The Ophthalmic Clinical Evaluation Exercise (OCEX): Interrater reliability determination. Ophthalmology 2005;112:1649-54.  Back to cited text no. 6
    
7.Cremers SL, Ciolino JB, Ferrufino-Ponce ZK, Henderson BA. Objective assessment of skills in intraocular surgery (OASIS). Ophthalmology 2005;112:1236-41.  Back to cited text no. 7
    
8.Cremers SL, Lora AN, Ferrufino-Ponce ZK. Global rating assessment of skills in intraocular surgery (GRASIS). Ophthalmology 2005;112:1655-60.  Back to cited text no. 8
    
9.Feldman BH, Geist CE. Assessing residents in phacoemulsification. Ophthalmology 2007;114:1586.  Back to cited text no. 9
[PUBMED]    
10.Saleh GM, Gauba V, Mitra A, Litwin AS, Chung AK, Benjamin L. Objective structured assessment of cataract surgical skill. Arch Ophthalmol 2007;125:363-6.  Back to cited text no. 10
    
11.Golnik KC, Beaver H, Gauba V, Lee AG, Mayorga E, Palis G, et al. Cataract surgical skill assessment. Ophthalmology 2011;118:427.e1-5.  Back to cited text no. 11
    
12.Golnik KC, Beaver H, Gauba V, Lee AG, Mayorga E, Palis G, et al. Development of a new valid, reliable, and internationally applicable assessment tool of residents′ competence in ophthalmic surgery (An American Ophthalmological Society Thesis). Trans Am Ophthalmol Soc 2013;111:24-33.  Back to cited text no. 12
    
13.Golnik KC, Haripriya A, Beaver H, Gauba V, Lee AG, Mayorga E, et al. Cataract surgical skill assessment. Ophthalmology 2011;118:2094-2094.e2.  Back to cited text no. 13
    
14.Golnik KC, Gauba V, Saleh GM, Collin R, Naik MN, Devoto M, et al. The ophthalmology surgical competency assessment rubric for lateral tarsal strip surgery. Ophthal Plast Reconstr Surg 2012;28:350-4.  Back to cited text no. 14
    
15.Golnik KC, Motley WW, Atilla H, Pilling R, Reddy A, Sharma P, et al. The ophthalmology surgical competency rubric for strabismus surgery. J AAPOS 2012;16:318-21.  Back to cited text no. 15
    
16.Rootman DB, Lam K, Sit M, Liu E, Dubrowski A, Lam WC. Pyschometric properties of a new tool to assess task-specific and global competency in cataract surgery. Ophthalmic Surg Lasers Imaging 2012;43:229-34.  Back to cited text no. 16
    
17.Fisher JB, Binenbaum G, Tapino P, Volpe NJ. Development and face and content validity of an eye surgical skills assessment test for ophthalmology residents. Ophthalmology 2006;113:2364-70.  Back to cited text no. 17
    
18.Lee AG, Greenlee E, Oetting TA, Beaver HA, Johnson T, Boldt C, et al. The Iowa ophthalmology wet laboratory curriculum for teaching and assessing cataract surgical competency. Ophthalmology 2007;114:e21-6.  Back to cited text no. 18
    
19.Rogers GM, Oetting TA, Lee AG, Grignon C, Greenlee E, Johnson AT, et al. Impact of a structured surgical curriculum on ophthalmic resident cataract surgery complication rates. J Cataract Refract Surg 2009;35:1956-60.  Back to cited text no. 19
    
20.Le TDB, Adatia FA, Lam WC. Virtual reality surgical simulation as a feasible training and assessment tool: Result of a multicenter study. Can J Ophthalmol 2011;46:56-60.  Back to cited text no. 20
    
21.Lee AG. Using the American Journal of Ophthalmology′s website for assessing residency subcompetencies in practice-based learning. Am J Ophthalmol 2004;137:206-7.  Back to cited text no. 21
[PUBMED]    
22.Lee AG, Boldt CH, Golnik KC, Arnold AC, Oetting TA, Beaver HA, et al. Using the journal club to teach and assess competence in practice-based learning and improvement: A literature review and recommendation for implementation. Surv Ophthalmol 2005;50:542-8.  Back to cited text no. 22
    
23.Golnik KC, Lee AG, Carter K. Assessment of ophthalmology resident on-call performance. Ophthalmology 2005;112:1242-6.  Back to cited text no. 23
    
24.Jagadeesan R, Kaylan DN, Lee P, Stinnet S, Challa P. Use of a standardized patient satisfaction questionnaire to assess the quality of care provided by ophthalmology residents. Ophthalmology 2008;115:738-43.e3.  Back to cited text no. 24
    



 
 
    Tables

  [Table 1], [Table 2]


This article has been cited by
1 One-Year Progress in Ophthalmic Education
Eduardo Mayorga,Karl Golnik,Gabriela Palis
Asia-Pacific Journal of Ophthalmology. 2015; 4(6): 388
[Pubmed] | [DOI]



 

Top
  
 
  Search
 
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

 
  In this article
    Abstract
   Introduction
    Assessment Princ...
   Assessment Tools
   Conclusion
    References
    Article Tables

 Article Access Statistics
    Viewed4058    
    Printed70    
    Emailed3    
    PDF Downloaded234    
    Comments [Add]    
    Cited by others 1    

Recommend this journal