• Users Online: 380
  • Home
  • Print this page
  • Email this page
Home About us Editorial board Ahead of print Current issue Search Archives Submit article Instructions Subscribe Contacts Login 

 Table of Contents  
ORIGINAL ARTICLE
Year : 2016  |  Volume : 2  |  Issue : 1  |  Page : 60-64

An investigation of various quality indicators of final exams in specialized courses of Bachelor of Nursing


1 Department of Nursing Health, School of Nursing and Midwifery, Isfahan University of Medical Sciences, Isfahan, Iran
2 Medical Education Research Center, Student's Research Committee, Isfahan University of Medical Sciences, Isfahan, Iran
3 School of Medical Education, Shahid Beheshti University of Medical Sciences, Tehran, Iran

Date of Web Publication25-Jan-2016

Correspondence Address:
Mahsa Shakour
Medical Education Research Center, Student's Research Committee, Isfahan University of Medical Sciences, Isfahan
Iran
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/2395-2296.168506

Rights and Permissions
  Abstract 

Aim: Different evidence has revealed that the quality of exams significantly affects the quality of training. Therefore, the present study, with the purpose of evaluating final exams of specialized theoretical courses of Nursing, was conducted based on standard indicators of classical model of test evaluation. Methods: This is a cross-sectional study. Sampling was census and samples were 372 final exam questions of eight specialized theoretical courses of Nursing. Variables in exams were evaluated: Reliability, validity, difficulty, discrepancies, type and level of questions by two experts of education, and eight professors of courses. Results: The correlation between sessions and the overall number of designed questions for all courses was significant (r = 0.34, P < 0.001). For Bloom's level, 56.2% of all the questions were at knowledge level and 3 (0.8%) at synthesis level. Conclusions: Exam questions of Nursing major in the present study and many other studies are not of high quality. It is recommended that Evaluation Committees of universities carry out continuous supervision over educational and training plans be offered for question designers.

Keywords: Assessment, meta-evaluation, Nursing educationAddress for Correspondence:


How to cite this article:
Hoseini H, Shakour M, Dehaghani AR, Abdolmaleki MR. An investigation of various quality indicators of final exams in specialized courses of Bachelor of Nursing. Int J Educ Psychol Res 2016;2:60-4

How to cite this URL:
Hoseini H, Shakour M, Dehaghani AR, Abdolmaleki MR. An investigation of various quality indicators of final exams in specialized courses of Bachelor of Nursing. Int J Educ Psychol Res [serial online] 2016 [cited 2024 Mar 19];2:60-4. Available from: https://www.ijeprjournal.org/text.asp?2016/2/1/60/168506


  Introduction Top


Evaluation is one essential basis for every curriculum which helps education to navigate through the right path.[1] The evaluation of students' achievement involves measuring the performance of students and comparing the obtained results with educational objectives. The outcome of this evaluating procedure helps to make sure if teachers' training activities and students' efforts for learning have achieved the desired ends.[2],[3] Among the objectives of exams are the determination of sufficient evidence on advancements in order to set off for the next level of teaching, and classification of learners as well as modification of teaching methods.[4] On the other hand, different evidence has revealed that the quality of exams significantly affects the quality of training and teaching-learning procedures.[5],[6],[7] Therefore, the quality of questions in such exams is of a great significance.

An appropriate exam of medical academic achievements is one that best reflects the entire educational objectives and curriculum content. It also helps to determine the distribution and type of questions on the basis of size and verify the significance and value of educational content such that all educational objectives are considered.[8] Among exam, evaluation methods is the utilization of scales such as discrimination index, difficulty index, questions level, questions validity, and reliability. Discrimination index is associated with the capacity of question to distinguish between strong and weak students and its scale varies between 1 and − 1. Best questions are those with scores between 0.5 and 1. Questions with negative discrimination index show that a strong group student has performed poorly compared to a weak group student. These sorts of questions are basically flawed and should be discarded or fully revised.[9] Difficulty index is also used to determine the easiness of questions for students. The less the score of index is, the more complex the question will be, and the more the score increases, the easier the question will become. Therefore, most appropriate questions are those with scores between 0.3 and 0.7. In order to evaluate the level of questions, Bloom's taxonomy of cognitive learning is used. Thus, questions are classified from bottom to top in six cognitive levels: Knowledge, comprehension, application, analysis, synthesis, and evaluation.[10] To assess the validity of questions, predictive validity, content validity, criterion validity and face validity can be examined, depending upon the purpose of evaluation. Reliability is also evaluated in various ways. The most common methods for its assessment are Cronbach's alpha coefficient and split-half.

Given the inevitable importance of specialized courses and their evaluation for Nursing students in order to enter clinical educations, also lack of essential information on how to conduct educational evaluation in Isfahan University of Medical Sciences, this study attempts to provide a suitable situation to improve student's assessment through the determination of current status of exams evaluation. Therefore, the present study, with the purpose of evaluating final exams of specialized theoretical courses of Nursing major and showing the true position of exams, was conducted based on standard indicators of classical model of test evaluation.


  Methods Top


This is a descriptive cross-sectional study which was conducted on final exam questions of specialized theoretical courses for students majoring at Bachelor of Nursing at Isfahan University of Medical Sciences in the first semester of academic year 2012–2013. Sampling was census and samples were 372 final exam questions of all specialized theoretical courses of undergraduate Nursing. The lack of access to professors or their unwillingness to share the exam questions of relevant subjects, were among the exclusion criteria. The evaluated courses were eight, including: Neonatal Nursing, patient training, emergency, community healthcare (1) mental health (2) community healthcare (3) and medical-surgical Nursing (4). The instrument for collecting data were a questionnaire previously employed by Kazemi in a survey on the educational evaluation of specialized theoretical courses of undergraduate Midwifery in 2007, which was reliable with a correlation coefficient of 75%.[11] This questionnaire consisted of sections dedicated to general characteristics of each course (name, number, and units of the course), evaluation method, number of questions regarding teaching hours devoted to each subject in the curriculum, the level introduced in course objectives (in the curriculum), and other details of questions (level, type, difficulty index, and discrimination index of each question). Questions with difficulty index scoring more than 0.7 were concluded easy, below 0.3, difficult and between 0.3 and 0.7 suitable. Discrimination index score between 1 and 0.5 proved suitable, <0.5 up to − 0.5 indicated low discrimination, and below − 0.5 up to − 1 indicated an unsuitable question.[12]

The study began as professors of each subject willingly provided the researcher with final exams questions and the syllabus of specialized course. At the first step, a questionnaire was considered for each course. Next, questions assessed by two experts in Medical Education who know classical model of test evaluation and professors of courses who designed questions. Questions and the syllabus were delivered to two experts of Medical Education in order to determine the level of questions and the level introduced in course objectives according to Bloom. Then, relevant information and the data concerning the number of questions were entered into the questionnaire by the researcher. Face validity and content validity of questions were reviewed with the help of course experts. Difficulty index and discrimination index of multiple choice questions for per course were also entered into the questionnaire by the researcher and with the assistance of Word Processing Center at University of Medical Sciences. Statistical Package for the Social Sciences (SPSS) version 13 software was employed to analyze the data. To assess the correlation between questions level and educational content level of each question in lesson plan, also to evaluate the correlation between the number of questions for each educational content and the predicted number of questions for each educational content, Pearson and Spearman correlation coefficient was used. For ethical issues and safekeeping, exam results were reported with new artificial names (A, B, C, D,…) for courses.


  Results Top


In the current study, we assessed final exam questions of eight specialized theoretical courses of Nursing in order to review their final exams evaluation capacity. The overall number of questions for all courses was 372.

First, the relation between course sessions, in terms of lesson plan topics for each session, and the number of designed questions for final exams was examined. Results demonstrated that the correlation between sessions and the overall number of designed questions for all courses was significant (r = 0.34, P < 0.001) [Table 1].
Table 1. Correlation between the number of sessions and the number of questions

Click here to view


By reviewing questions level in terms of cognitive domain, 209 (56.2%) of all the questions were found to be at knowledge level, 108 (29%) at comprehension level, 38 (10.2%) at application level, 14 (3.8%) at analysis level, and 3 (0.8%) at synthesis level. Most of designed questions for every single course were also at knowledge level which are presented in results for each course separately [Table 2].
Table 2: Descriptive statistics of questions according to level of knowledge in every courses

Click here to view


Regarding difficulty index, 197 (58.6%) of questions were ranked easy, 108 (31.9%) were suitable with a moderate difficulty index and 32 (9.5%) were difficult [Figure 1].
Figure 1: Frequency of questions in terms of difficulty index

Click here to view


In addition, the analysis of discrimination index demonstrated that 121 (36%) of questions occupied a suitable discrimination level, 170 (50.6%) were at a low discrimination level, and 45 (13.4%) were at an unsuitable discrimination level.

By reviewing face and content validity of all questions, it was found out that 97.3% of questions enjoyed content validity and 93.3% of them were with face validity.


  Discussion Top


Based on Bloom's taxonomy, assessing eight reviewed courses the data reflected that questions at first level of learning (knowledge) had the maximum frequency (56.2%) among all courses and questions at higher levels had smaller percentage of frequency. Analysis and synthesis levels were only 6.4% of all questions. These results are consistent with that of Khoddam et al., in that, as he concluded, the scores for evaluated levels of learning were also small in exam questions designed by Nursing and Midwifery professors. According to him, 65% of questions were at the three lowest levels regarding cognitive learning.[13] Furthermore, in a study by Kazemi, on the state of theoretical courses evaluation of Undergraduate Midwifery in Isfahan, in 83.33% of exams 50% of questions were placed at the first level of difficulty.[11] However, one of the educational objectives of Medical Departments is the ability of reasoning and inferring and gaining the ability to recognize and utilize the acquired knowledge of specialized subjects which are essential capabilities for every nurse. Furthermore, the required and sufficient knowledge of nurses and an increase in their decision-making ability would result in improvement of their performance.[14] Based on results concerning the evaluation of final exams for specialized courses of Nursing, the assessing of knowledge levels has got priority over the estimation and evaluation of reasoning ability. Unfortunately, the assessment of cognitive levels of specialized courses of Nursing for the current level of questions may not evaluate students in terms of their future responsibilities.

Taking difficulty index and discrimination index, questions with moderate difficulty index and high discrimination index are reported to be most suitable. It is recommended to determine the difficulty index and discrimination index after each exam enabling the question designer to modify questions based on results from difficulty and discrimination levels.[9] The level of difficulty must agree with teaching objectives, content, teaching methods, and learners' acquiring ability.[15],[16] In the present study, the assessment of obtained results for difficulty index of questions reveals that in 58.6% of exam, easy questions had the maximum frequency and only 31.9% of questions were suitable in terms of difficulty index. In the study by Kazemi, 73.33% of exams were easy and enjoyed the highest frequency. Only two cases of suitable questions in terms of difficulty index had the highest frequency.[14] In another study by Dadgari et al., the analysis of final exam comprehensive questions of Nursing demonstrated that 57.78% of questions were easy, 26.94% of them were suitable, and 15.28% were difficult.[17] Heidari in a study on exam questions for Nursing and Midwifery students of Shahed University showed that 60% of questions had a difficulty index over 0.7 and 10% of questions had a difficulty index lower than 0.3.[18] However, reviewing Nursing exams occurring in Malaysia demonstrated that the difficulty index of 40% of questions was above 80%.[19] The analysis of discrimination index showed that among the entire evaluated questions only 36% of them had high discrimination index. In the study by Kazemi, 44.27% of all evaluated questions had suitable discrimination index.[19] This rate is slightly higher but still far from the optimum state. In the study by Dadgari, only 5.4% of final exam comprehensive questions in Shahroud had high discrimination index.[17] It should be acknowledged that low levels of discrimination index in all of these studies indicate the failure of designing process to reach suitable discrimination index. However, in a study by Mitra et al. in Malaysia, where suitable discrimination index was assumed above 0.2, the results reflect that the situation is slightly better but not yet good and only half of questions had the ability to discriminate between strong and weak students.[15] The designed questions should seek to measure the acquired behaviors precisely.[20] If a question is to be a beneficial tool for the student's improvement, first the teacher should perceive the relationship between the teaching objective and evaluation and then design the questions. This is because the quality of evaluation is estimated with regard to its relevance and accuracy.[21] In this study, the designers of evaluated exam questions had considered face validity and content validity in their questions. When reviewing questions, more than 97.3% of them contained content validity and 93.3% had face validity. Thus, it is concluded that these exams had suitable content and face validity. Unlike the discussed results, Mc Corby et al.[22] at Bristol University reported a considerable percentage of questions had encountered problems in this respect. The findings of the study by Kazemi and Ehsanpour revealed that 75% of questions of all the exams had content validity, while 28.78% of exams had more than 90% face validity.[19]

In the present study, only 30% of exams contained criterion validity. Lack of criterion validity in 70% of evaluations reflects a poor evaluation system in terms of criterion validity examination in specialized theoretical courses of Nursing major. In the study by Kazemi and Ehsanpour, 61.11% of exams on specialized theoretical courses of Midwifery major had criterion validity and compared to the current study, specialized theoretical courses of Nursing major had lower levels of criterion validity.[19]

Although multiple choice questions are more popular because of their high reliability, they are not much applicable in evaluation of cognitive learning at high levels. Therefore, in order to make optimum use of all features of questions, experts suggest applying various types of questions in a single test.[14] Some studies mentioned that multiple choices questions with proper validity and reliability could be a good tool for evaluating clinical knowledge.[8] However, for evaluation of exams enrolled in the current study, professors had substantially employed multiple choice questions. The flaws in designing procedure of these questions are accompanied with a decrease in criterion validity. It must be noted that in multiple choice questions, stronger learners, compared to ordinary learners, are more probable to find faults, complexities, and unusual opinions in questions. However, since there exist only one correct answer for each question, lack of accuracy in designing process would not only cause the vigilance of strong students to fail to encourage them, but also might expose them to punishment for the wrong answer and bring about lower grades for them.

Finally, besides admitting the existence of various problems in evaluation of exams, it must be added that the present study encountered some limitations, for instance, a limited number of exams were investigated, while it is necessary for a proper judgment to include a more comprehensive study on the continuous evaluation of specialized theoretical courses of Nursing major (such as: Midterm exams, quizzes and …). It is recommended that practical actions be taken in order to resolve the mentioned problems and the effect of them on improving evaluation process be studied.


  Conclusions Top


Exam questions of Nursing major in the present study and many other studies do not have the desire quality, and regarding how vital exam questions can be in students' success particularly in their future jobs. Evaluation Committees of universities should carry out continuous supervision over educational evaluation of specialized courses and offer sufficient training plans for question designers who are not much familiar with evaluation principles and procedures.

Acknowledgments

The researchers are very grateful to Vice Chancellor for Research Affairs of Isfahan University of Medical Sciences, and to the professors who made their exam questions, lesson plans, and relevant question analyses available for the researchers.

Financial support and sponsorship

This article was derived from a thesis submitted by Isfahan University of Medical Sciences, No: 288088.

Conflicts of interest

There are no conflicts of interest.

 
  References Top

1.
Newble DI, Jaeger K. The effect of assessments and examinations on the learning of medical students. Med Educ 1983;17:165-71.  Back to cited text no. 1
[PUBMED]    
2.
Rust C, Price M, O'Donovan B. Improving students learning by developing their understanding of assessment criteria and processes. Assess Eval High Educ 2003;28:147-164.  Back to cited text no. 2
    
3.
Roschelle J, Penuel WR, Yarnall L, Shechtman N, Tatar D. Handheld tools that 'Informate' assessment of student learning in science: A requirements analysis. J Comput Assist Learn 2005;21:190-203.  Back to cited text no. 3
    
4.
Wink DM. Using questioning as a teaching strategy. Nurse Educ 1993;18:11-5.  Back to cited text no. 4
[PUBMED]    
5.
Brady AM. Assessment of learning with multiple-choice questions. Nurse Educ Pract 2005;5:238-42.  Back to cited text no. 5
    
6.
Larsen DP, Butler AC, Roediger HL 3rd. Test-enhanced learning in medical education. Med Educ 2008;42:959-66.  Back to cited text no. 6
    
7.
Reid WA, Duvall E, Evans P. Relationship between assessment results and approaches to learning and studying in Year Two medical students. Med Educ 2007;41:754-62.  Back to cited text no. 7
    
8.
Duff JP, Cheng A, Bahry LM, Hopkins J, Richard M, Schexnayder SM, et al. Development and validation of a multiple choice examination assessing cognitive and behavioural knowledge of pediatric resuscitation: a report from the EXPRESS pediatric research collaborative. Resuscitation 2013;84:365-8.  Back to cited text no. 8
    
9.
Tavakol M, Dennick R. Post-examination analysis of objective tests. Med Teach 2011;33:447-58.  Back to cited text no. 9
    
10.
Momsen J, Offerdahl E, Kryjevskaia M, Montplaisir L, Anderson E, Grosz N. Using assessments to investigate and compare the nature of learning in undergraduate science courses. CBE Life Sci Educ 2013;12:239-49.  Back to cited text no. 10
    
11.
Kazemi A, Ehsanpour S, Hassanzadeh A. Investigating the academic achievement evaluation of specialized theoretical courses of midwifery BS. Iran J Med Educ 2010;9:346-55.  Back to cited text no. 11
    
12.
Saif AA. Educational assessment, measurment, and evaluation. 5th ed. Tehran: Doran; 2010.  Back to cited text no. 12
    
13.
Khoddam H, Jouybari L, Sanagoo A. Nursing clinical instructors' skills in designing question. Iran J Med Educ 2005;5:191-2.  Back to cited text no. 13
    
14.
Stoner M. Critical Thinking for Nursing. 4th ed. St. Louis: WB Saunders Co.; 1999.  Back to cited text no. 14
    
15.
Mitra N, Nagaraja H, Ponnudurai G, Judson J. The levels of difficulty and discrimination indices in type a multiple choice questions of pre-clinical semester 1, multidisciplinary summative tests. IeJSME 2009;3:2-7.  Back to cited text no. 15
    
16.
Fisher K. Demystifying critical reflection: Defining criteria for assessment. High Educ Res Dev 2003;22:313-25.  Back to cited text no. 16
    
17.
Dadgari A, Ebrahimi H, Hassani M, Mashyekhi Y. Analysis of the comprehensive nursing final examin Shahrod faculty of medical sciences. Shahrekord Univ Med Sci J 2003;4:4-10.  Back to cited text no. 17
    
18.
Heidari M. The analysis of the achievement test performed in shahed nursing and midwifery faculty. Iran J Med Educ 2004;Suppl 10:120.  Back to cited text no. 18
    
19.
Kazemi A, Ehsanpour S. Evaluation of validity of midwifery special courses in Isfahan University of Medical Sciences. Iran J Nurs Midwifery Res 2011;16:153-7.  Back to cited text no. 19
    
20.
Ruhe V, Boudreau JD. The 2011 Program Evaluation Standards: A framework for quality in medical education programme evaluations. J Eval Clin Pract 2013;19:925-32.  Back to cited text no. 20
    
21.
Harlen W. Trusting teachers' judgement: Research evidence of the reliability and validity of teachers' assessment used for summative purposes. Res Pap Educ 2005;20:245-70.  Back to cited text no. 21
    
22.
McCoubrie P. Improving the fairness of multiple-choice questions: A literature review. Med Teach 2004;26:709-12.  Back to cited text no. 22
    


    Figures

  [Figure 1]
 
 
    Tables

  [Table 1], [Table 2]



 

Top
 
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
Abstract
Introduction
Methods
Results
Discussion
Conclusions
References
Article Figures
Article Tables

 Article Access Statistics
    Viewed3748    
    Printed265    
    Emailed0    
    PDF Downloaded297    
    Comments [Add]    

Recommend this journal


[TAG2]
[TAG3]
[TAG4]