TO: The faculty

FROM: Leslie Miller-Bernal for the Advisory Committee

RE: Report on Evaluating the Evaluation Process

DATE: June, 1999

As you may recall, in April you received a questionnaire from the Advisory Committee that asked for your opinions of the evaluation system currently in use. Twenty-eight faculty filled out the form; 20 of these 28 faculty had been involved with an evaluation subcommittee (some just for the evaluation of particular faculty, i.e., not as appointed members); 20 had been evaluated under the current system (13 of whom had also worked with an evaluation subcommittee); 9 had been a member of the Advisory Committee since we instituted our current evaluation process (all but one of whom had also worked with an evaluation subcommittee); and 19 had written letters for a colleague[s] who was being evaluated. Thus it is fair to say that most of the faculty who responded to our questionnaire had had at least some experience with our evaluation process, and many had had a variety of experiences with it.

Overall, faculty appear to have a favorable opinion of the evaluation process at Wells. No one believed that the prior, department-based system was superior to what we have now (a committee of 12 faculty -- divided into two subcommittees -- comprised of the Chairs of the Divisions and 8 other appointed, tenured faculty). That does not mean, however, that faculty believe the system is perfect or could not be improved upon. To discuss the ways faculty think our system should be improved, I have summarized their major areas of concern, rather than going through a question-by-question analysis of faculty's responses. At the end of this report, I report general comments faculty made about our evaluation system.

The aspect of our evaluation system that the highest number of faculty feel needs to be improved is the time we allot to the different stages of the process. One of the specific questions on the questionnaire related to this issue was, "Do you think there are any ways of improving how much time we are able to give to each stage of the evaluation process?" Nineteen faculty felt able to answer this question; all but four said they thought there was a way of improving the time frame. Some of those who thought we could improve this aspect of our evaluation system suggested we should start earlier, perhaps by announcing evaluations in the spring and having all materials due by the end of the first week of fall semester. Starting earlier is especially important for tenure cases, one faculty member noted. Getting materials to an outside evaluator earlier was another suggestion related to improving the timetable.

Faculty who have been evaluated under the current system were asked whether they had sufficient time to put together their dossier. Only 5 of the 17 faculty who answered this question said that the time was insufficient, and yet specific comments made on this part of the questionnaire made it clear that a larger proportion had actually experienced difficulty. For instance, one faculty member who said that the time had been adequate noted that this was because s/he has been evaluated several times and hence knows to get the material ready over the summer. Another faculty member who checked that s/he had received the request for the dossier in adequate time noted that s/he had been given an extension. A couple of faculty said that it took them longer to put together their dossiers than it would have if Wells had computerized (machine-readable) course evaluation forms. In general, faculty who answered this question seemed to agree with a frequent comment made about the general timetable: it would be helpful if the evaluation process got started at the end of the spring semester -- faculty who are to be evaluated in the fall could be notified so that they could begin assembling materials for their files over the summer.

Although few faculty had concerns about the class visitations we do as part of the evaluation process, one of the concerns sometimes mentioned had to do with not being notified about them sufficiently ahead of time. Some faculty have had to change their plans for a class because there was such a short time between being approached about a class visitation and the date of their class being observed. In one case a faculty member felt unable to conduct a review for the final exam because that was the only day left for his or her class to be visited; in another case a faculty member changed class plans from student presentations to a discussion of an article.

Twenty-three faculty answered the questions about whether they believed the subcommittees were the right size and composition. A majority said they think they are the right size, but a sizeable minority, 7 of the 23, thought that the subcommittees are too large or that they probably are (another two faculty's answers were unclear, but may have indicated dissatisfaction with the subcommittee's size).

Those faculty who believe the subcommittees are an appropriate size often mentioned that having six faculty (or more when, as usual, other faculty are brought in for a particular evaluation) ensures a fairer evaluation. One faculty member wrote: "It's helpful that there are fewer members of the major than previous methods." Some of the faculty who approve of the subcommittees' size did note that it is difficult to find enough people to serve, however.

Faculty who criticized the size of the evaluation subcommittees often said that it is hard to find meeting times for so many people. Having so many faculty involved is a waste of faculty time, one faculty member commented; this faculty member also said that s/he would not serve on an evaluation subcommittee again unless the size were reduced. On the other hand, a faculty member who thought that the subcommittees "seem rather large," questioned whether their size could be reduced "without substantially changing the composition," and s/he approved of their composition.

Other comments related to this section of the questionnaire included a recommendation that we reduce the number of evaluations of tenure-track faculty and that it is important to have an odd number of faculty on the evaluation subcommittees in case of tie votes.

One faculty member's comment succinctly sums up arguments for and against decreasing the size of the evaluation subcommittees: "Large size helps give fair treatment but sure puts a large strain on faculty resources."

A minority of faculty had concerns about the letters routinely used in the evaluation process to solicit information about the candidate, but some faculty had very strong feelings about this topic. The questionnaires asked faculty who had been evaluated under our current system whether the letters they received about the process and the materials they were expected to submit were clear. Only two faculty members said they were not clear, but others who said they were added qualifications. One faculty member pointed out that the letters candidates receive about the process arrive at different times and have confusing layouts; this faculty member suggested having one letter that clearly lays out the entire process and what the candidate is expected to submit. Another faculty member said that for one evaluation, s/he had been approached by the chair of the evaluation subcommittee and asked who would be appropriate to write letters for the evaluation file. Yet for another evaluation, this same faculty member was not asked such questions; instead, s/he received a letter that said who would write letters but was given the opportunity to add a name.

Other suggestions about ways to improve the letters used during evaluations (from faculty members who indicated that they found the letters clear) included being given more information about what counts as community service, and being given guidelines about "less obvious things," such as, should faculty include in their dossiers thank you notes from admissions for participating in open houses, or does a statment in PARs about such participation suffice? One faculty member commented on the letters requesting faculty to write a letter for a colleague, saying that s/he would like more information on what is suitable to include.

When faculty undergoing an evaluation have questions about the process, they do not always know whom to consult. Although not many faculty reported being unclear about who could answer their questions, a couple said that in their first evaluations, they did not know. One faculty member said that s/he had asked several people (the chair of the Advisory Committee, the divisional chair, and the dean), and the only response they gave was to quote the Faculty Manual, not what this person felt s/he needed.

About ten faculty expressed strong feelings about candidates' ability to have sufficient input into the evaluation process while it is ongoing and/or the kind of feedback they receive when the evaluation has concluded. Underlying concerns appear to involve issues of justice (knowing when negative, perhaps incorrect statements have been made about you and having the opportunity to correct them), issues of faculty development (ensuring that the evaluations help faculty members improve), and issues related to faculty's feelings (ensuring that they do not go "insane" with worry during the process, and giving them appropriate acknowledgement and appreciation). What follows are some of the specific comments faculty made on their questionnaires about these issues:

A few faculty addressed seven-year reviews of tenured faculty, specifically. One faculty member thought insufficient materials are required (in particular, course syllabi have been optional). Other sources of dissatisfaction with these reviews were that the letters written to faculty at the conclusion of these reviews are brief, and connected to this, they do not assist faculty in their development or improvement of their performance.

Many faculty expressed the opinion that our evaluation system is generally fair and consistent. As one person wrote, consistency and fairness are the "best reasons for maintaining this model." Another faculty member echoed this sentiment, saying, "I was treated fairly and consistently by the faculty."

Perhaps because fairness is a key criterion for any evaluation system, many faculty members commented on times that our system did not live up to this ideal. Remarks were far-ranging, with some stressing perceived causes of inconsistencies, others placing blame on particular groups, and still others addressing ways our system might be improved.

One reason evaluations are not as fair as they might be is inconsistencies in the ways different subcommittees function. Some faculty said that the committees appear to function by word of mouth, procedures are not written down and followed as often as they should be, and therefore consistency depends on the continuity of personnel, in particular, experienced divisional chairs. As one faculty member wrote, "guidelines" have "emerged through experience" but they are "not always known." An example provided by one faculty member is being told that a particular colleague had been chosen to write a letter for the evaluation file because otherwise the letters would appear to be stacked in his or her favor. This faculty member questioned whether this was really a criterion applied to everyone. Other ways in which evaluation subcommittees may contribute to unfair evaluations are that chairs of committees may have too much influence on votes, or members may be affected by their like or dislike of a candidate or by how much the candidate "socializes" with evaluators. One faculty member recommended that all votes be conducted by secret ballot and tallied by someone outside of the committee.

Another form of inconsistency some faculty members commented on is differences between evaluation letters received from different parties (the subcommittee, advisory committee, and dean). One faculty member had also experienced inconsistency between letters of evaluation and what the chair of the division said to him or her at the conclusion of the evaluation. A couple of faculty noted the difficulty of interpreting letters that report a split vote on one of the criteria (teaching, scholarship, or community service) but then say only positive things about the faculty member in that area. The frustration that such inconsistencies can cause are reflected in the following comment of a faculty member: "I feel we need to develop a process that helps us develop and grow as faculty. That's what I would like. Right now, I find the process punishing and lacking in constructive criticism to build on."

Five faculty members expressed concerns about the role of the dean (not necessarily the current dean) in the evaluation process. One faculty member, in fact, was positive about the way our evaluation system works but only "at the level of the faculty." Inconsistencies between letters written by evaluation subcommittees and by the dean were noted; one faculty member felt the dean's letters were "more grudging." There was some concern that the dean may use information that does not appear in a candidate's file. One faculty member reported questioning the dean about the source of information for a comment the dean had made in her letter to the candidate and being told that it was based on private conversations other faculty had had with her (the dean).

A few faculty commented on the need to use outside evaluators more consistently. One faculty member said that at least in the past, there were "inconsistencies and even abuses" of outside peer review. Another noted the importance of outside reviews for objectivity in an institution as small as Wells.

One way that some faculty saw for improving fairness and consistency was for the community to discuss standards; for instance, what it means for teaching to be judged "excellent." One faculty member argued that Wells is insufficiently rigorous in its standards and that not everyone should expect to receive tenure. Another faculty member noted that we do not have clear procedures for cases when there are disagreements at different levels of a review.

Some faculty cautioned that we should not be unrealistic in our expectations for consistency and fairness since all evaluation systems are "human." Another faculty member opined that while evaluation committees may be inconsistent, since they depend on their "whole cast of characters," it is up to the advisory committee and dean to apply the same standards from candidate to candidate. And still another faculty member expressed his or her belief in the system by saying that it is "difficult to convince the Dean, President and some faculty that it is a very fair and consistent process which should be seen as thorough and sound and worth their trust."

Faculty enthusiastically endorse our evaluation system, despite criticisms and suggestions for improvement. Those faculty who experienced our previous department-based system seem particularly impressed by our current system. To convey these sentiments, I conclude with quotes from the questionnaires themselves.