How Can We Use Course Evaluations to Improve Teaching and the Curriculum? - Workshop Report
Wie können wir mit Hilfe von Lehrveranstaltungsevaluierung Lehre und Lehrpläne verbessern? - Bericht aus dem Workshop 5

Introduction

• What are the most important characteristics of effective teaching?
• Based on our understanding of effective teaching, can we use course evaluations to identify teaching deficiencies?
• If we can identify deficiencies, how can we use this information to improve teaching and the curriculum?

Our strategy was thus to determine the main attributes of effective instructors, use those characteristics to design an evaluation questionnaire that would enable us to identify deficiencies in teaching, and then consider how we can use this information to help instructors improve their teaching and make program changes to improve the curriculum. We addressed these points in turn.

Determine Characteristics of Effective Teaching

To test our intuitions about effective teaching, we began with an informal, ungraded quiz based on an empirical study of the characteristics of effective instructors (Murray, 1983). In the study, MURRAY used student course evaluations to identify the instructors who were most effective. He then looked at different characteristics of teaching to determine which ones were correlated with the group of most effective instructors. He thus found a set of characteristics that were reliable discriminators of effective instructors.

In the exercise, the workshop participants were given the following list of 10 characteristics of teaching, five of which were among those identified by MURRAY as reliable discriminators and five of which were not:

1. Smiles or laughs
2. Puts outline of lecture on board
3. Shows energy/excitement
4. Directs questions to individuals
5. Directs questions to whole class
6. Uses humor
7. States teaching objectives
8. Addresses students by name
9. Shows strong interest in subject
10. Uses a variety of media. (ROSS, 1988)

If you want to test your own intuitions, please do so now. Circle the five characteristics that you think would be most highly correlated with effective teaching. Continue reading only after making your choices.

Several interesting points about effective teaching were made in the discussion of this list of characteristics. Smiling and laughing may or may not be good things to do, depending on the context. They are not sufficient by themselves to have a positive effect. To be effective they must be the concomitants of something genuinely humorous. Hence (6) turns out to be among the reliable discriminators but (1) does not.

Putting an outline of one's lecture on the board is certainly a good thing to do, but does not qualify as a reliable discriminator. It is fairly simple to write an outline of one's lecture on the board - any instructor could do this. It is not so easy to structure lectures clearly and communicate that structure to students in an understandable way.

Showing energy and excitement is a discriminating characteristic. An instructor's enthusiasm is infectious and motivates students to pay attention and become actively involved in the course work. The same holds for (9). By showing a strong interest in the subject, the instructor motivates students to find out what it is about the subject that makes it so fascinating. Both of these related characteristics thus turn out to be reliable indicators of effective instructors.

Of the next two options, one is a reliable discriminator and one is not. Asking questions keeps students actively involved in the class and tells the instructor how well they are following the lecture. But should one direct questions to the class as a whole or to specific individuals. Two advantages of putting questions to the entire class are that it is less threatening to students and one will generally get more informed answers. A disadvantage, however, is that the answers generally come from the same students. By posing questions to individuals we can involve the entire class, and, even more important, we thereby hold the entire class accountable at all times for all of the material covered during the class. This helps motivate students to prepare for classes, which in turn improves the level of discussion during class. It is, however, important to put questions in a non-threatening way, in order to create a relaxed and productive class atmosphere. The empirical evidence thus shows that (5) is a reliable discriminator and (4) is not.

Stating teaching objectives, like putting an outline of the lecture on the board, is a good thing to do, but it is also easy to do and not a reliable discriminator of effective instructors. Once again, it is nonetheless important to have clear teaching objectives and to communicate them to students.

We now have two items left to examine and one outstanding reliable discriminator. After adjusting their answers in light of the remaining choices, most of our group opted, correctly, for addressing students by name.

We agreed that this is not feasible in every situation, for example in a lecture class with several hundred students, but that it is certainly good to call on students by name where possible. By learning students' names, instructors demonstrate their genuine interest in interacting with each individual student, which is of course highly motivating. Conversely, when students expect to be called on by name - in small classes - they are disappointed and de-motivated if the instructor does not make this effort.

That leaves the use of a variety of media as another good thing to do, where appropriate and beneficial, but not a reliable discriminator of effective instructors.

We closed our first workshop session by considering two additional lists of characteristics of effective instructors. The first is from a study by T. R. WOTRUBA and P. L. WRIGHT (1974) in which they asked faculty members, students and administrators the question, "What are characteristics of good teachers?" The resulting list of 9 characteristics is as follows:

1. Favorable attitudes towards students
2. Knowledge of subject
3. Communication skills
4. Good speaking ability
5. Willingness to experiment
6. Enthusiasm about subject
7. Good organization of subject matter and course
8. Fairness in exams and grading
9. Encouragement of students to think. (Quoted in CENTRA, 1987, p. 50)

1. Provide helpful feedback to students in a variety of ways.
2. Exhibit respect and understanding for all students.
3. Demonstrate mastery of the subject.
4. Communicate effectively in both written and oral formats in English.
5. Develop a reflective approach to teaching through collecting feedback and continually modifying instructional approaches.
6. Promote individual involvement of students through learner-centered teaching methods.
7. Enhance motivation of students through personal enthusiasm for the subject.
8. Communicate and manage appropriate expectations for achievement in the course.
9. Demonstrate a general belief that all students are capable of learning.
10. Encourage cooperation and collaboration among students. (Quoted in GREEN, 1995, p. 3)

1. Treat students and patients [in a medical school environment] with respect and understanding; serve as a role model.
2. Demonstrate mastery of the subject.
3. Communicate effectively.
4. Be willing to experiment with different teaching methods.
5. Be enthusiastic.
6. Respond positively to criticism or suggestions for improvement.
7. Explain the subject matter in a clear and understandable way.
8. Encourage class activities involving work in teams.
9. Structure the course and material presented clearly.
10. Be fair in dealing with students, especially with exams and grades.
11. Encourage critical thinking.
12. Define learning goals clearly and adhere to them.

Identify Teaching Deficiencies

After some discussion, we agreed that we could use the above list of teaching attributes to formulate questions that would indicate the extent to which a given quality is exemplified by a given instructor. For example, to determine the extent to which an instructor treats students and patients with respect we could ask students/patients to indicate on a five point scale the degree to which they agree with the statement, "The instructor shows respect for students/patients." After considering several such examples we concluded that it was possible in each case to formulate appropriate questions for a course evaluation questionnaire.

We then looked in some detail at the features of an actual course evaluation form, taking the Webster Vienna questionnaire as an example. The current form is the third one we have used in 15 years. Each form is structured in the same way; it consists of two parts, a block of questions about the course and the instructor, and space for written comments. Students are asked to rate different aspects of the course or the instructor's performance on a five point scale, and are encouraged to comment on the course or the instructor in the space provided.

The original form came from the home office in St. Louis and had 7 questions about the course and 8 relating to the instructor. It was used from the first classes in 1981 until 1991. Neither the faculty nor the administration was satisfied with it, however, because it provided too little specific information about instructors, so it was replaced by an expanded questionnaire that had 5 questions about the course, 15 that related to the instructor, and one general question.

The first version of this revised form had 5 additional questions and prompted students to write comments on different aspects of the course and the instructor's performance. Unfortunately, this more extensive version of the questionnaire only lasted one term because of the amount of effort required to process the additional information. It was then shortened to 21 questions, as indicated above, and the section soliciting specific comments became an open invitation to write comments about the course and the instructor on the back of the form. This questionnaire was used from 1991 to 1995.

Last year, a student-lead initiative to review the form resulted in the current version, which has 5 questions about the course, 16 relating to the instructor, and one general question. But it also asks students to comment on four specific aspects of the course and the instructor, and invites them to make additional comments on anything that may have been missed in the questionnaire. We just started using this form in Fall, 1995. All three of these questionnaires are provided as appendices to this report, along with the faculty form that instructors fill out during the evaluation process.

In my experience, student comments supplement the ratings part of the evaluation is an important way. The comments typically give the instructor the best idea of what students thought of the course, providing insight into her or his strengths or weaknesses, and suggesting changes to improve the course. Any significant deficiencies in the instruction that are not addressed by specific questions on the form invariably turn up in student comments.

Having satisfied ourselves that we could use a course evaluation questionnaire to identify teaching deficiencies, we turned our attention to the question of the validity of the evaluation process.

Since the course evaluation process is justified only if its results are valid, we must satisfy ourselves that this is the case. We discussed typical objections that instructors make about the validity of course evaluations and considered the research findings relating to these objections. We agreed that the evidence clearly supported the validity of the process, and thus proceeded to the last question - how can we use the information about teaching deficiencies to improve teaching and the curriculum?

Improve Teaching and the Curriculum

We began this discussion by considering the fundamental question whether a good teacher, like a good poet, can be made as well as born. Ben Jonson maintained that a good poet can also be made, but Peter Drucker believes that good teachers must be born and cannot be made:

Teaching is the only major occupation of man for which we have not yet developed tools that make an average person capable of competence and performance. In teaching we rely on the "naturals," the ones who somehow know how to teach. (quoted in CHARLTON, 1992, p. 65)

But if poets can be made, why not teachers? To help us answer this question, we once again turned to the research literature.

The research evidence on teaching improvement suggests that some aspects of teaching can be improved by almost all instructors but other aspects are much more resistant to change. In an extensive study of teaching improvement, Robert WILSON found that the percentage of instructors who showed improvement in specific areas varied from 90% to 39% (WILSON, 1987, p. 14, 18).

This study was carried out at the University of California, Berkeley (UCB), from 1979 - 1982, and was based on course evaluation feedback and consultations with experienced instructors. When the course evaluations indicated a weakness in a given area, teaching consultants worked with the instructor and gave her or him several suggestions to try to overcome the deficiency. These suggestions were drawn from the experience of instructors recognized for their teaching effectiveness. WILSON then compared the course evaluations from the next time the course was taught with the original results to determine what improvement, if any, was made.

The area that was easiest for instructors to improve their performance was "states objectives for each class session." 90% of all instructors deficient in this area showed improvement the second time the course was taught. The most difficult area to improve was "varies the speed and tone of his/her voice." Only 39% of all instructors who were weak in this area were able to improve their performance. In terms of overall effectiveness, 52% of all instructors participating in the study were able to improve their performance (WILSON, 1987, pp. 9 - 18).

After discussing the conclusions reached in this study, we agreed that there was indeed some truth in Drucker's contention that effective teachers are born not made. Some characteristics of effective teaching that relate to an instructor's fundamental attitudes, personality, or intellectual capabilities are certainly difficult to change. However, the UCB study also shows that significant improvements can be made in many areas.

We agreed that the key to improvement was information. Beyond identifying areas for improvement, the course evaluation process should include some mechanism to support instructors in making changes. This may be individual consultations with experienced instructors, training in different teaching methods or approaches, or simply informal exchange of information between instructors.

Taking this last approach, we then had an informal exchange of ideas about different teaching methods. We took as our starting point Professor Bardolin's welcoming remarks at the first plenary session of the workshop, in which he argued for a practical approach to teaching that combines theory and practice. He said that traditional lecturing ("Frontalberieselung") is a very ineffective method of learning since students typically retain only 20% of the material presented. He then described an intensive learning experience in which students worked together in a real-life environment, applying their knowledge under the supervision of the instructor.

We agreed with Professor Bardolin that it is good to involve students actively in the work of the course, and discussed several ways of doing this, as follows:

• Class discussion: an easy way to involve students and allow them to express and refine their understanding of central concepts.
• Case study method: this technique is commonly used in management courses, in which students analyze cases based on real business situations, and it can also be used effectively in a medical school.
• Project work: students work on projects that allow them to apply their knowledge in practical situations.
• Group work: students work together in small teams on projects or other tasks, applying their knowledge in practical ways; two added benefits of group work are that students develop their interpersonal skills and learn the value of working in teams.
• Real-life projects: within the context of a course students carry out real projects for companies or institutions and present their results to the sponsoring organization.
• Internships: an intensive work experience in which one or more students works on a project for a company or institution under the supervision of a supervisor in the organization.
• Study trips: an ideal supplement to classroom instruction.
• Interdisciplinary learning: courses that combine two or more disciplines and are co-taught by instructors from each discipline; such courses promote greater interaction among and between instructors and students.

It was pointed out that most of these approaches are applicable only in relatively small classes and would be difficult to use in large lecture classes. However, two of the most effective methods, real-life experience and team work are integral parts of the training in medical schools during the internship or residency (Turnus or Facharztausbildung).

We considered different ways of using these methods and decided that this depends very much on the individual instructor. We agreed that training about different teaching methods could be useful in helping instructors find an approach that best suits their own interests and abilities.

Having decided that the course evaluation process could indeed be used to improve teaching, we then considered how it might help us improve the curriculum.

In my experience, students' comments on course evaluations often point to curricular problems, such as a lack of coordination between courses, which could be either too much overlap in the content of related courses or large gaps between courses - topics that should be covered in one or another course but are not. They can also identify inconsistencies in the level at which courses are taught or the demands of the instructors within a given department. At Webster, for example, we discovered through the course evaluations that some instructors were requiring far less work from students than others in the same area. We then discussed this concern at an area meeting and agreed on guidelines for the kind of work that should be required at each course level.

It was generally agreed that departmental communication and coordination is necessary to improve programs and keep them up to date, and that the course evaluation results could provide useful information for this ongoing process.

Conclusion

An effective instructor, we decided, is one who is an expert in the subject, has the ability to communicate it to students enthusiastically and explain it clearly, treats students and patients with respect and understanding, is fair in grading, and encourages team work and critical thinking. Furthermore, she or he takes a keen interest in didactics, being receptive to suggestions for improvement and willing to experiment with different teaching methods.

These last qualities will make the effective instructor responsive to feedback from course evaluations, which will result in further improvements in her or his teaching performance.

Teaching is a difficult job - few instructors have all of the qualities listed above - and thus one that is eminently suited to a process of continuous improvement.

Course evaluations can be designed to identify teaching deficiencies and thus provide the basis for a program of continuous improvement in teaching. Such a program should, however, include some mechanism to help instructors understand and overcome the deficiencies identified. Consultations with experienced instructors, training about alternative teaching methods or techniques, or informal sharing of experiences with colleagues, are three ways of facilitating improvement.

The important thing is to help an instructor find an approach that best suits her or his own interests and strengths. In discussing effective teaching, John CENTRA argues that there is a danger in looking at individual teaching characteristics in isolation. He says, "good teaching occurs when the instructor uses a method that is best suited to his or her abilities and also best suited to accomplishing what the course should accomplish" (CENTRA, 1987, p. 50).

Course evaluation results can also be used to improve the curriculum by highlighting program deficiencies, which can then be dealt with in departmental meetings.

We thus concluded that the course evaluation process should not be an end in itself but can indeed be used by an institution to improve teaching and the curriculum by raising the level of consciousness about what constitutes effective teaching and why it is important, and by generating greater interest in didactics as a means to achieving our learning objectives.

References

CENTRA, John A. (1987).

"Formative and Summative Evaluation: Parody or Paradox?" In Lawrence M. ALEAMONI (ed.), Techniques for Evaluating and Improving Instruction, 47-55. San Francisco: Jossey-Bass.

CHARLTON, James (1994).

A Little Learning is a Dangerous Thing. New York: St. Martin's Press.

GREEN, Doris (1995).

"Seeking Consensus: What is Good Teaching?" Academic Leader, 11/4, 3.

MURRAY (1983).

Referenced in ROSS (1988).

ROSS, Steven (1988).

"Evaluating Lectures From the Student's Perspective." Reproduced in Appendix A.

SMITH, Kathleen S., and SIMPSON, Ronald D. (1995).

Innovative Higher Education. New York: Human Sciences Press.

WILSON, Robert C. (1987).

"Toward Excellence in Teaching." In Lawrence M. ALEAMONI (ed.), Techniques for Evaluating and Improving Instruction, 9-24. San Francisco: Jossey-Bass.

WOTRUBA, T. R., and WRIGHT, P.L. (1974).

"How to Develop a Teacher-Rating Instrument: A Research Approach." Journal of Higher Education, 46/6, 653-663.

Appendices

EVALUATING LECTURES FROM THE STUDENT'S PERSPECTIVE

Characteristic		Rating (Circle one)
		Important	Unimportant
1.	Smiles or laughs	I	U
2.	Puts outline of lecture on board	I	U
3.	Shows energy / excitement	I	U
4.	Directs questions to individuals	I	U
5.	Directs questions to whole class	I	U
6.	Uses humor	I	U
7.	States teaching objectives	I	U
8.	Adresses students by name	I	U
9.	Shows strong interest in subject	I	U
10.	Uses a variety of media	I	U

EVALUATION QUESTIONNAIRE

As part of our continuing effort to evaluate and improve the quality of the graduate program, we ask you to complete this questionnaire on your class. Since all responses to this questionnaire are treated in a confidential manner, you may feel free to make any comments you see fit. Any suggestions you have for improvement of the instruction will be appreciated. the responses from this questionnaire are tabulated; one copy of the tabulation is sent to the instructor, one copy is put in the instructor's file, and one copy is retained by the site coordinator. Thank you for your response.

1 = Highest rating; superior
2 = Above average rating; good
3 = Average rating; adequate
4 = Below average rating; poor
5 = Unsatisfactory rating

On a scale of one to five, with one being the highest rating, evaluate the course in each of the following areas:

THE COURSE:

___ 1. The content of the class reflected the catalog description.
___ 2. The instructor used the text and other learning materials effectively.
___ 3. Course objectives were clearly defined.
___ 4. Assignments were appropriate for course objectives.
___ 5. Course requirements and grading procedures were included in the syllabus and were explained at the first class meeting.
___ 6. Evaluation prcedures reflected the material and subject matter covered in the course.
___ 7. How do you rate the course on a scale of one to five? (See explanation of rating scale above.)

On a scale of one to five, with one being the highest rating, evaluate the instructor in each of the following areas:

THE COURSE:

__ 8. The instructor was organized.
__ 9. The instructor evidenced knowledge of subject matter.
__ 10. The instructor showed an ability to teach the subject.
__ 11. The instructor was helpful in responding to class and individual needs.
__ 12. The instructor provided feedback on student's progress during the course.
__ 13. The class atmosphere was one in which student felt free to ask questions and initiate discussion.
__ 14. How would you rate the instructor on a scale of one to five? (See explanation of rating scale above.)

Read the following question carefully since the responses are slightly different from above: >P>__ 15. The class met for a full four-hour instructional block, including the break period,
(1) always; (2) most of time; (3) half of the time; (4) some of the time; (5) never.

COMMENTS:
Please write any comments you have about the instructor (strong or weak points) or about the course (likes or dislikes) in space provided:

COURSE EVALUATION QUESTIONNAIRE

Please fill out this questionnaire objectively, making any comments you feel are appropriate. All responses will be treated confidentially. After the course grades are turned in, the instructor will receive a computer printout with all ratings and comments made. The instructor will never see this form with your written responses. Thank you. >P>Using the scale given below, indicate the extent to which you agree or disagree with the following statements.

Strongly				Strongly	Not
agree	Agree	Neutral	Disagree	Disagree	Applicable
1	2	3	4	5	NA

The Course

___ 1. The course objectives were clearly defined.
___ 2. The scope of the course was good and the topics were covered in sufficient depth.
___ 3. Taking this course has increased my knowledge of the subject matter.
___ 4. The textbook was a good choice for the course.
___ 5. I would recommend this course to a friend.

The Instructor

___ 6. has a clear and objective grading system.
___ 7. is enthusiastic about the subject.
___ 8. speaks English clearly and fluently.
___ 9. has a thorough knowledge of the subject matter.
___ 10. explains clearly.
___ 11. is well prepared.
___ 12. uses class time effectively.
___ 13. encourages questions and class discussion.
___ 14. gives interesting and stimulating assignments.
___ 15. has a genuine interest in students.
___ 16. gives examinations that require understanding of the subject matter.
___ 17. returned homework and exams promptly.
___ 18. motivates students to do their best work.
___ 19. discusses points of view different from own.
>P>Answer the following questions according to the scale given:

___ 20. Compared with other instructors you have had, how effective has this instructor been?

1 One of the most effective
2 More effective than most
3 Average NA Not applicable
4 Not as effective as most
5 One of the least effective

___ 21. How much time did you spend doing homework for this course?

1 More than 16 hours/week
2 12 - 16 hours/week
3 8 - 12 hours/week
4 4 - 8 hours/week
5 Less than 4 hours/week

Comments: Please write any comments you have about the instructor (strengths/weaknesses), the course (likes/dislikes), the facilities (space, lighting, cleanliness, temperature, noise, etc.), or anything else on the back of this questionnaire.

Course Evaluation Questionnaire

1 Strongly agree
5 Strongly Disagree
NA Not Applicable

	The Course	1	2	3	4	5	NA
1.	The course objectives were clearly defined at the	O	O	O	O	O	O
	beginning of the course.
2.	The scope of the course was good.	O	O	O	O	O	O
3.	The textbook was a good choice for the course.	O	O	O	O	O	O
4.	The level of the course satisfied my expectations.	O	O	O	O	O	O

What material covered in this course do you find most valuable? ________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

What material was least relevant to the course? ________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

	The Instructor	1	2	3	4	5	NA
5.	creates an atmosphere conductive to learning.	O	O	O	O	O	O
6.	encourages questions and class discussion, which leads to a better understanding of the material.	O	O	O	O	O	O
7.	has a genuine interest in students.	O	O	O	O	O	O
8.	motivates students to do their best work.	O	O	O	O	O	O
9.	is well prepared.	O	O	O	O	O	O
10.	explains clearly.	O	O	O	O	O	O
11.	has a good command of the English language.	O	O	O	O	O	O
12.	has a presentation style that engages student' attention	O	O	O	O	O	O
13.	uses teaching aids, such as overhead transparencies or handouts, that facilitate understanding.	O	O	O	O	O	O
14.	uses class time effectively.	O	O	O	O	O	O
15.	has a thorough knowledge of the subject.	O	O	O	O	O	O
16	covered the topics in sufficiat depth.	O	O	O	O	O	O
17.	gives assignments that facilitate learning.	O	O	O	O	O	O
18.	has a clear and objective grading system.	O	O	O	O	O	O
19.	gives exams that require understanding of the course content.	O	O	O	O	O	O

Which aspects of the teaching and course activities do you find most useful?

________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

What would you like to see changed?

________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

	General	1	2	3	4	5	NA
20.	My interest in the subject has been stimulated bythis course.	O	O	O	O	O	O
21.	Overall, I am satisfied with this course.	O	O	O	O	O	O
22.	Overall, the instructor is an effective teacher.	O	O	O	O	O	O

Please comment on any aspect of the course that you feel that we missed in this questionnaire.

________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

Thank you for your time.

The Service Committee

FACULTY SURVEY

1.	Did the administration provide the assistance you needed for this course?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
2.	Were the classroom facilities satisfactory?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
3.	Were the textbooks available for students in time to prepare for the first night's assignment?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
4.	Were students capable of working at the level you expected?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
5.	Did you think you offered adequate feedback to students?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
6.	Were you able to achieve the objectives detailed in your syllabus?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
7.	Does the catalog accurately describe what you think should be covered in this course?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
8.	If you taught your course in a four hour block, did you have any problem with this format?
	YES ____ PARTLY ____ NO ____
	COMMENTS:
9.	How would you rate the textbook used?
	EXCELLENT __ GOOD __ AVERAGE __ BELOW AVERAGE __ POOR __
	COMMENTS:

Your comments are welcomed. Please use the back of this page for additional space.