This posting discusses what some evaluators have experienced as destructive behaviors by stakeholders during evaluation projects.
For example, my experience and the experiences of my associate while evaluating a safety program for a school district is not discussed in our article, “How the School Built Environment Exacerbates Bullying and Peer Harassment (Fram and Dickmann 2012).” Our experiences were not as extreme as we have heard, but have similarities to such experiences by Bechar & Mero-Jaffe (2014), who stated in their article:
Overtly, the program head had agreed to the evaluation and recognized its importance, covertly, his attitude showed inconsistency in his willingness to support the evaluation, which was expressed during the course of the evaluation and in his reaction to the final report; we interpret these as sign of fear of evaluation (p. 369).
We did not interpret our experience as a sign of fear, but a serious issue involving the cohesiveness of district leaders involving the superintendent and all of the principals of the schools. Our evidence pointed towards the leadership style of the superintendent and the politics embedded in the school system. We even stated in our article that politics at the district level had a major negative impact on safety program at the schools. If any fear of evaluation from the stakeholders existed if was purely concerning the possible loss of their job. Any fear we witnessed was a symptom of the dysfunctional relationships among the stakeholders.
Bechar & Mero-Jaffe refer to Donaldson’s (2007) introduction of a new term, excessive evaluation anxiety (XEA). I am inclined to say that our experiences had little connection if any to this phenomenon.
I am proposing two ways to lower the possibilities of experiencing destructive behavior from stakeholders.
1. More thoroughly plan the initial stakeholder meeting and incorporate an informative session (whether on another day or same day) to discuss the perceptions of and concerns of all of the stakeholders involved. Meet with individual stakeholders afterwards to further address their concerns in private.
We did have an informative and well-planned meeting with the stakeholders together and my associate and I both met separately with each stakeholder to address any concerns that they had. For the most part, we believe that the individual meetings improved our chances of collecting data effectively with support from the stakeholders. One of the principals wanted to be involved and we made this happen even though the superintendent did not want any principal involved at any stage. We asked the principal to help us orchestrate the informed consent meeting involving the teachers and I asked the principal to walk with me as I took photographs to offer me insight about particular spaces of the school. My associate was able to show the superintendent the benefit of having the principal involved.
2. Before the initial stakeholder meeting schedule and orchestrate a focus group designed from a social constructivist perspective or what Ryan, Gandha et al. (2014) called a Type B focus group. The style of focus group reveals tacit knowledge during social participation. This focus group is effective at getting at what is hidden during social interactions. With targeted questions intermingled among conversation, a moderator, can tease out underlying politics, beliefs and facades of social relationships. In addition, targeted and informative comments by the moderator can help to better inform the stakeholders about the evaluation process and eliminate confusion and misunderstandings. Ultimately, this additional data adds richness.
I believe that had we completed such a focus group with all of the stakeholders, we would have had a more effective evaluation; which would have benefited all of the stakeholders. We were not able to collect all of the necessary data needed to complete a thorough evaluation of the safety program in all of the schools. We were only allowed to focus on the elementary schools for collecting data.
The Bechar & Mero-Jaffe article does concern me. As an outsider regarding the experiences, I have to say that while reading the article, I felt that the authors were still too sensitive about their experiences. Their description and choice of information to present regarding the program head as one of the stakeholders was unsettling to me. I question whether or not this article should have been published at all.
Regarding the use of the term, excessive evaluation anxiety, I have to add that just because destructive behavior exists does not mean that it always points to some anxiety about evaluation. In our case, the destructive behavior pointed to a dysfunctional relationship among the stakeholders and how such relationships had a negative impact on the safety program.
Bechar, Shlomit & Mero-Jaffe, Irit. (2014) Who is afraid of evaluation? Ethics in evaluation research as a way to cope with excessive evaluation anxiety: Insights from a case study. American Journal of Evaluation, 35(3): 364-376.
Donaldson, S. I. (2007). Program theory-driven evaluation science: strategies and applications. London, England: Routledge.
Ryan, K. E., Gandha, T., Culbertson, K. J. and Carlson, C. (2014). Focus group evidence: Implications for design and analysis. American Journal of Evaluation, 35(3): 328-345.