How Can We Use Course Evaluations to Improve Teaching and the Curriculum? - Workshop Report
Wie können wir mit Hilfe von Lehrveranstaltungsevaluierung Lehre und Lehrpläne verbessern? - Bericht aus dem Workshop 5
To find an answer to the question posed by the workshop we considered the following related questions:
- What are the most important characteristics of effective teaching?
- Based on our understanding of effective teaching, can we use course evaluations to identify teaching deficiencies?
- If we can identify deficiencies, how can we use this information to improve teaching and the curriculum?
Our strategy was thus to determine the main attributes of effective instructors, use those characteristics to design an evaluation questionnaire that would enable us to identify deficiencies in teaching, and then consider how we can use this information to help instructors improve their teaching and make program changes to improve the curriculum. We addressed these points in turn.
Determine Characteristics of Effective Teaching
To test our intuitions about effective teaching, we began with an informal, ungraded quiz based on an empirical study of the characteristics of effective instructors (Murray, 1983). In the study, MURRAY used student course evaluations to identify the instructors who were most effective. He then looked at different characteristics of teaching to determine which ones were correlated with the group of most effective instructors. He thus found a set of characteristics that were reliable discriminators of effective instructors.
In the exercise, the workshop participants were given the following list of 10 characteristics of teaching, five of which were among those identified by MURRAY as reliable discriminators and five of which were not:
- Smiles or laughs
- Puts outline of lecture on board
- Shows energy/excitement
- Directs questions to individuals
- Directs questions to whole class
- Uses humor
- States teaching objectives
- Addresses students by name
- Shows strong interest in subject
- Uses a variety of media. (ROSS, 1988)
If you want to test your own intuitions, please do so now. Circle the five characteristics that you think would be most highly correlated with effective teaching. Continue reading only after making your choices.
Several interesting points about effective teaching were made in the discussion of this list of characteristics. Smiling and laughing may or may not be good things to do, depending on the context. They are not sufficient by themselves to have a positive effect. To be effective they must be the concomitants of something genuinely humorous. Hence (6) turns out to be among the reliable discriminators but (1) does not.
Putting an outline of one's lecture on the board is certainly a good thing to do, but does not qualify as a reliable discriminator. It is fairly simple to write an outline of one's lecture on the board - any instructor could do this. It is not so easy to structure lectures clearly and communicate that structure to students in an understandable way.
Showing energy and excitement is a discriminating characteristic. An instructor's enthusiasm is infectious and motivates students to pay attention and become actively involved in the course work. The same holds for (9). By showing a strong interest in the subject, the instructor motivates students to find out what it is about the subject that makes it so fascinating. Both of these related characteristics thus turn out to be reliable indicators of effective instructors.
Of the next two options, one is a reliable discriminator and one is not. Asking questions keeps students actively involved in the class and tells the instructor how well they are following the lecture. But should one direct questions to the class as a whole or to specific individuals. Two advantages of putting questions to the entire class are that it is less threatening to students and one will generally get more informed answers. A disadvantage, however, is that the answers generally come from the same students. By posing questions to individuals we can involve the entire class, and, even more important, we thereby hold the entire class accountable at all times for all of the material covered during the class. This helps motivate students to prepare for classes, which in turn improves the level of discussion during class. It is, however, important to put questions in a non-threatening way, in order to create a relaxed and productive class atmosphere. The empirical evidence thus shows that (5) is a reliable discriminator and (4) is not.
Stating teaching objectives, like putting an outline of the lecture on the board, is a good thing to do, but it is also easy to do and not a reliable discriminator of effective instructors. Once again, it is nonetheless important to have clear teaching objectives and to communicate them to students.
We now have two items left to examine and one outstanding reliable discriminator. After adjusting their answers in light of the remaining choices, most of our group opted, correctly, for addressing students by name.
We agreed that this is not feasible in every situation, for example in a lecture class with several hundred students, but that it is certainly good to call on students by name where possible. By learning students' names, instructors demonstrate their genuine interest in interacting with each individual student, which is of course highly motivating. Conversely, when students expect to be called on by name - in small classes - they are disappointed and de-motivated if the instructor does not make this effort.
That leaves the use of a variety of media as another good thing to do, where appropriate and beneficial, but not a reliable discriminator of effective instructors.
We closed our first workshop session by considering two additional lists of characteristics of effective instructors. The first is from a study by T. R. WOTRUBA and P. L. WRIGHT (1974) in which they asked faculty members, students and administrators the question, "What are characteristics of good teachers?" The resulting list of 9 characteristics is as follows:
A more recent survey of national academic leaders by Kathleen S. SMITH and Ronald D. SIMPSON (1995) yielded a list of 27 competencies considered to be important or very important for university instructors. The top 10 competencies, in rank order, are the following:
- Favorable attitudes towards students
- Knowledge of subject
- Communication skills
- Good speaking ability
- Willingness to experiment
- Enthusiasm about subject
- Good organization of subject matter and course
- Fairness in exams and grading
- Encouragement of students to think. (Quoted in CENTRA, 1987, p. 50)
In our second session we discussed the similarities between the characteristics on these two lists and Murray's five reliable discriminators of effective instructors. We then developed the following list of characteristics that we felt were important for an effective instructor, without any rank order:
- Provide helpful feedback to students in a variety of ways.
- Exhibit respect and understanding for all students.
- Demonstrate mastery of the subject.
- Communicate effectively in both written and oral formats in English.
- Develop a reflective approach to teaching through collecting feedback and continually modifying instructional approaches.
- Promote individual involvement of students through learner-centered teaching methods.
- Enhance motivation of students through personal enthusiasm for the subject.
- Communicate and manage appropriate expectations for achievement in the course.
- Demonstrate a general belief that all students are capable of learning.
- Encourage cooperation and collaboration among students. (Quoted in GREEN, 1995, p. 3)
Having agreed on this list of characteristics, we turned our attention to the next question - can we use course evaluations to identify teaching deficiencies?
- Treat students and patients [in a medical school environment] with respect and understanding; serve as a role model.
- Demonstrate mastery of the subject.
- Communicate effectively.
- Be willing to experiment with different teaching methods.
- Be enthusiastic.
- Respond positively to criticism or suggestions for improvement.
- Explain the subject matter in a clear and understandable way.
- Encourage class activities involving work in teams.
- Structure the course and material presented clearly.
- Be fair in dealing with students, especially with exams and grades.
- Encourage critical thinking.
- Define learning goals clearly and adhere to them.
Identify Teaching Deficiencies
After some discussion, we agreed that we could use the above list of teaching attributes to formulate questions that would indicate the extent to which a given quality is exemplified by a given instructor. For example, to determine the extent to which an instructor treats students and patients with respect we could ask students/patients to indicate on a five point scale the degree to which they agree with the statement, "The instructor shows respect for students/patients." After considering several such examples we concluded that it was possible in each case to formulate appropriate questions for a course evaluation questionnaire.
We then looked in some detail at the features of an actual course evaluation form, taking the Webster Vienna questionnaire as an example. The current form is the third one we have used in 15 years. Each form is structured in the same way; it consists of two parts, a block of questions about the course and the instructor, and space for written comments. Students are asked to rate different aspects of the course or the instructor's performance on a five point scale, and are encouraged to comment on the course or the instructor in the space provided.
The original form came from the home office in St. Louis and had 7 questions about the course and 8 relating to the instructor. It was used from the first classes in 1981 until 1991. Neither the faculty nor the administration was satisfied with it, however, because it provided too little specific information about instructors, so it was replaced by an expanded questionnaire that had 5 questions about the course, 15 that related to the instructor, and one general question.
The first version of this revised form had 5 additional questions and prompted students to write comments on different aspects of the course and the instructor's performance. Unfortunately, this more extensive version of the questionnaire only lasted one term because of the amount of effort required to process the additional information. It was then shortened to 21 questions, as indicated above, and the section soliciting specific comments became an open invitation to write comments about the course and the instructor on the back of the form. This questionnaire was used from 1991 to 1995.
Last year, a student-lead initiative to review the form resulted in the current version, which has 5 questions about the course, 16 relating to the instructor, and one general question. But it also asks students to comment on four specific aspects of the course and the instructor, and invites them to make additional comments on anything that may have been missed in the questionnaire. We just started using this form in Fall, 1995. All three of these questionnaires are provided as appendices to this report, along with the faculty form that instructors fill out during the evaluation process.
In my experience, student comments supplement the ratings part of the evaluation is an important way. The comments typically give the instructor the best idea of what students thought of the course, providing insight into her or his strengths or weaknesses, and suggesting changes to improve the course. Any significant deficiencies in the instruction that are not addressed by specific questions on the form invariably turn up in student comments.
Having satisfied ourselves that we could use a course evaluation questionnaire to identify teaching deficiencies, we turned our attention to the question of the validity of the evaluation process.
Since the course evaluation process is justified only if its results are valid, we must satisfy ourselves that this is the case. We discussed typical objections that instructors make about the validity of course evaluations and considered the research findings relating to these objections. We agreed that the evidence clearly supported the validity of the process, and thus proceeded to the last question - how can we use the information about teaching deficiencies to improve teaching and the curriculum?
Improve Teaching and the Curriculum
We began this discussion by considering the fundamental question whether a good teacher, like a good poet, can be made as well as born. Ben Jonson maintained that a good poet can also be made, but Peter Drucker believes that good teachers must be born and cannot be made:
Teaching is the only major occupation of man for which we have not yet developed tools that make an average person capable of competence and performance. In teaching we rely on the "naturals," the ones who somehow know how to teach. (quoted in CHARLTON, 1992, p. 65)
But if poets can be made, why not teachers? To help us answer this question, we once again turned to the research literature.
The research evidence on teaching improvement suggests that some aspects of teaching can be improved by almost all instructors but other aspects are much more resistant to change. In an extensive study of teaching improvement, Robert WILSON found that the percentage of instructors who showed improvement in specific areas varied from 90% to 39% (WILSON, 1987, p. 14, 18).
This study was carried out at the University of California, Berkeley (UCB), from 1979 - 1982, and was based on course evaluation feedback and consultations with experienced instructors. When the course evaluations indicated a weakness in a given area, teaching consultants worked with the instructor and gave her or him several suggestions to try to overcome the deficiency. These suggestions were drawn from the experience of instructors recognized for their teaching effectiveness. WILSON then compared the course evaluations from the next time the course was taught with the original results to determine what improvement, if any, was made.
The area that was easiest for instructors to improve their performance was "states objectives for each class session." 90% of all instructors deficient in this area showed improvement the second time the course was taught. The most difficult area to improve was "varies the speed and tone of his/her voice." Only 39% of all instructors who were weak in this area were able to improve their performance. In terms of overall effectiveness, 52% of all instructors participating in the study were able to improve their performance (WILSON, 1987, pp. 9 - 18).
After discussing the conclusions reached in this study, we agreed that there was indeed some truth in Drucker's contention that effective teachers are born not made. Some characteristics of effective teaching that relate to an instructor's fundamental attitudes, personality, or intellectual capabilities are certainly difficult to change. However, the UCB study also shows that significant improvements can be made in many areas.
We agreed that the key to improvement was information. Beyond identifying areas for improvement, the course evaluation process should include some mechanism to support instructors in making changes. This may be individual consultations with experienced instructors, training in different teaching methods or approaches, or simply informal exchange of information between instructors.
Taking this last approach, we then had an informal exchange of ideas about different teaching methods. We took as our starting point Professor Bardolin's welcoming remarks at the first plenary session of the workshop, in which he argued for a practical approach to teaching that combines theory and practice. He said that traditional lecturing ("Frontalberieselung") is a very ineffective method of learning since students typically retain only 20% of the material presented. He then described an intensive learning experience in which students worked together in a real-life environment, applying their knowledge under the supervision of the instructor.
We agreed with Professor Bardolin that it is good to involve students actively in the work of the course, and discussed several ways of doing this, as follows:
- Class discussion: an easy way to involve students and allow them to express and refine their understanding of central concepts.
- Case study method: this technique is commonly used in management courses, in which students analyze cases based on real business situations, and it can also be used effectively in a medical school.
- Project work: students work on projects that allow them to apply their knowledge in practical situations.
- Group work: students work together in small teams on projects or other tasks, applying their knowledge in practical ways; two added benefits of group work are that students develop their interpersonal skills and learn the value of working in teams.
- Real-life projects: within the context of a course students carry out real projects for companies or institutions and present their results to the sponsoring organization.
- Internships: an intensive work experience in which one or more students works on a project for a company or institution under the supervision of a supervisor in the organization.
- Study trips: an ideal supplement to classroom instruction.
- Interdisciplinary learning: courses that combine two or more disciplines and are co-taught by instructors from each discipline; such courses promote greater interaction among and between instructors and students.
It was pointed out that most of these approaches are applicable only in relatively small classes and would be difficult to use in large lecture classes. However, two of the most effective methods, real-life experience and team work are integral parts of the training in medical schools during the internship or residency (Turnus or Facharztausbildung).
We considered different ways of using these methods and decided that this depends very much on the individual instructor. We agreed that training about different teaching methods could be useful in helping instructors find an approach that best suits their own interests and abilities.
Having decided that the course evaluation process could indeed be used to improve teaching, we then considered how it might help us improve the curriculum.
In my experience, students' comments on course evaluations often point to curricular problems, such as a lack of coordination between courses, which could be either too much overlap in the content of related courses or large gaps between courses - topics that should be covered in one or another course but are not. They can also identify inconsistencies in the level at which courses are taught or the demands of the instructors within a given department. At Webster, for example, we discovered through the course evaluations that some instructors were requiring far less work from students than others in the same area. We then discussed this concern at an area meeting and agreed on guidelines for the kind of work that should be required at each course level.
It was generally agreed that departmental communication and coordination is necessary to improve programs and keep them up to date, and that the course evaluation results could provide useful information for this ongoing process.
An effective instructor, we decided, is one who is an expert in the subject, has the ability to communicate it to students enthusiastically and explain it clearly, treats students and patients with respect and understanding, is fair in grading, and encourages team work and critical thinking. Furthermore, she or he takes a keen interest in didactics, being receptive to suggestions for improvement and willing to experiment with different teaching methods.
These last qualities will make the effective instructor responsive to feedback from course evaluations, which will result in further improvements in her or his teaching performance.
Teaching is a difficult job - few instructors have all of the qualities listed above - and thus one that is eminently suited to a process of continuous improvement.
Course evaluations can be designed to identify teaching deficiencies and thus provide the basis for a program of continuous improvement in teaching. Such a program should, however, include some mechanism to help instructors understand and overcome the deficiencies identified. Consultations with experienced instructors, training about alternative teaching methods or techniques, or informal sharing of experiences with colleagues, are three ways of facilitating improvement.
The important thing is to help an instructor find an approach that best suits her or his own interests and strengths. In discussing effective teaching, John CENTRA argues that there is a danger in looking at individual teaching characteristics in isolation. He says, "good teaching occurs when the instructor uses a method that is best suited to his or her abilities and also best suited to accomplishing what the course should accomplish" (CENTRA, 1987, p. 50).
Course evaluation results can also be used to improve the curriculum by highlighting program deficiencies, which can then be dealt with in departmental meetings.
We thus concluded that the course evaluation process should not be an end in itself but can indeed be used by an institution to improve teaching and the curriculum by raising the level of consciousness about what constitutes effective teaching and why it is important, and by generating greater interest in didactics as a means to achieving our learning objectives.
- CENTRA, John A. (1987).
- "Formative and Summative Evaluation: Parody or Paradox?" In Lawrence M. ALEAMONI (ed.), Techniques for Evaluating and Improving Instruction, 47-55. San Francisco: Jossey-Bass.
- CHARLTON, James (1994).
- A Little Learning is a Dangerous Thing. New York: St. Martin's Press.
- GREEN, Doris (1995).
- "Seeking Consensus: What is Good Teaching?" Academic Leader, 11/4, 3.
- MURRAY (1983).
- Referenced in ROSS (1988).
- ROSS, Steven (1988).
- "Evaluating Lectures From the Student's Perspective." Reproduced in Appendix A.
- SMITH, Kathleen S., and SIMPSON, Ronald D. (1995).
- Innovative Higher Education. New York: Human Sciences Press.
- WILSON, Robert C. (1987).
- "Toward Excellence in Teaching." In Lawrence M. ALEAMONI (ed.), Techniques for Evaluating and Improving Instruction, 9-24. San Francisco: Jossey-Bass.
- WOTRUBA, T. R., and WRIGHT, P.L. (1974).
- "How to Develop a Teacher-Rating Instrument: A Research Approach." Journal of Higher Education, 46/6, 653-663.
Appendix A: Exercise, "Evaluating Lectures From the Student's Perspective."
Appendix B: Webster University Vienna Course Evaluation Questionnaire, used from 1981 - 1991
Appendix C: Webster University Vienna Course Evaluation Questionnaire, used from 1991 - 1995
Appendix D: Webster University Vienna Course Evaluation Questionnaire, used from 1995 - present
Appendix E: Webster University Vienna Faculty Survey Form
EVALUATING LECTURES FROM THE STUDENT'S PERSPECTIVE
The following are characteristics of lecturers. Which discriminate between teachers that students identify as effective or ineffective? Examine them from the student's perspective and rate each as "Important" or "Unimportant" based on those judgments. Note: In a classic study by Murray (1983), half were reliable discriminators and half were not. Test your intuitions against the actual findings.