According to BetterEvaluation (n.d.) there are currently 17 possible approaches to program evaluation. These include such approaches as case study, contribution analysis, horizontal evaluation, positive deviance, and the approach where this analysis will focus, appreciative inquiry. As remarked by BetterEvaluation (n.d.), “Appreciative Inquiry is about the coevolutionary search for the best in people, their organizations, and the relevant world around them. In its broadest focus, it involves systematic discovery of what gives ‘life’ to a living system when it is most alive, most effective, and most constructively capable in economic, ecological, and human terms” (para. 2). Yet the process for such an approach can be difficult to hold universal. This, as appreciative inquiry by definition is neither specifically quantitative nor qualitative by requirement, rather it seeks to utilize the methods necessary to collect what data can be found meaningful among that which is available, and the design which proves most fruitful in the pursuit of positive deviants among an organization’s ecosystem. With the understanding this approach is not suitable to every situation, Pascale, Sternin, & Sternin (2010) specify, “the process excels over most alternatives when addressing problems that, to repeat, (1) are enmeshed in a complex social system, (2) require social and behavioral change, and (3) entail solutions that are rife with unforeseeable or unintended consequences” (p. 10).
Example 1 – Organizational Culture in Higher Education
Knowing appreciative inquiry holds a special value among program evaluators, yet is one which can introduce a host of considerations for the processes, procedures, measures, rationale, and theoretical bases of studies, three instances of appreciative inquiry are introduced. First, Niemann (2010) who studied organizational culture in higher education, when discussing the research question explored noted, “it is necessary to know how to create that sense of belonging, what the vision and mission are, what the people value and expect from their leaders and colleagues, what they identify with, and what will make them move collectively towards taking united ownership of the future of their institution or at least part of their institution” (p. 1004). To explore this transformation, the design included first a theoretical basis founded in Geertz’s semiotic approach to culture, as well as Thompson & Luthans psycho-social interpretive framework (Niemann, 2010, p. 1005). The population consisted of the surveying of 40 full-time faculty members of the Faculty of Education at the University of the Free State. The survey consisted of a number of open-ended questions, seeking the establishment of responses which could indicate ‘life-giving’ moments among the university’s ecosystem, whereby best practice could additionally flourish and the best version of the university could be furthered. This method was chosen to permit such exploration as well as the opportunity for further probing if necessary. 27 narratives were collected in all among the 40 surveyed. Questions asked included those such as “Tell the story of one of your best experiences when you felt most involved in your work environment”, and “What do you value most in terms of the faculty’s values, norms, philosophy, mission and vision” (Niemann, 2010, p. 1010). The factors affecting this design included organizational structure, access to participants, theoretical basis of the questions themselves, as well as the research question driven by sponsorship from the Ministerial Committee. An alternative could have been perhaps to exchange open-ended surveying with transcription to closed-ended multiple choice survey items. Yet in sum total the strength of this study was based on a combination of the population’s organizational ecosystem in common, the totality of the questions asked, the open forum in which all were permitted to respond, and the direction taken by the inquiry leading to the pursuit of a best version of the university moving forward.
Example 2 – Online Courses and Knowledge Assimilation
A second study to consider involved an exploration to see if graphic enhancements and navigation could increase learning and reduce cognitive load to make it easier for at-risk, lower socioeconomic, and ethnic self-identified students to have a positive experience in online courses and increase the likelihood of their success in online courses (Cook, 2009, p. 303). Based on the study of semiotics, or the study of ‘patterned human communication behavior’, Cook sought to determine whether enhancements to courses could permit for reduced barriers to interaction for learners. On the design of the study Cook (2009) states, “This study used an exploratory survey research model and several qualitative methodologies, appreciative inquiry and development design, to examine whether embedded semiotics and carefully designed metaphors helped students in the online courses to feel more comfortable in assimilating new knowledge online, reinforce their learning, and increase the potential for their course completion” (Cook, 2009, p. 304). Appreciative inquiry was chosen as the basis of the design for its positive approach, and the theory underpinning the 23 open-ended interviews conducted of students to identify positive deviants were established by research borne of the work of Richey & Klein (2007) on a developmental research design founded on data collected via practice. The practice of open-ended interviews, seeking answers to positive questions, for the purpose of identifying best practice is all founded in extant theory. Factors affecting this design notably include the desire to gather actionable data from those experiencing the phenomena first-hand, while emphasizing what is working amid this phenomena. And while closed-ended surveying of experience could have been performed, this would have needed to be predicated on existing theory regarding what factors most impact the navigation of online courses and facilitate the reduction of cognitive load. Until these themes emerge, and positive deviants discovered, open-ended questioning permitted for the most actionable data regarding what can be built upon to optimize online courses. Conclusions in this instance thus included as remarked by Cook (2009), “The findings suggest that the students’ need to be heard was an important factor, and more relevant to the students than had been discerned in the generic student assessments conducted by the university after each course” (p. 305).
Example 3 – Strategic Planning at the ISPI
Finally, we have Van Tiem & Rosenzweig (2008) who, with the cooperation of the Board of Directors for the International Society for Performance Improvement, “undertook a study to uncover the ‘best of ISPI’ to enhance their strategic planning” (p. 5). The methodology employed was to ask a series of positive questions which were designed to draw what was indeed working of the society, to then perpetuate these ‘wins’ across the remainder of ISPI practices. The questions were founded on established appreciative inquiry methodology, and spanned a very short window from a June 2007 board meeting to an established cutoff of August 15th of the same year. As remarked by Van Time et al. (2008), “the response time was short, but the entire timeframe for the project was very limited” (p. 6). In order to draw a series of conclusions based upon reliable answers from amid the ISPI population, the researchers first divided society membership into four equal parts. Among those parts systematic random sampling was employed, and of the four questions constituting the aforementioned positive questions, each group received one of these four questions. Regarding analysis, Rosenzweig and Van Tiem, with advice from Thomas (research advisor), analyzed the data, Rosenzweig and Van Tiem analyzed the member responses and determined category descriptors, and they collaborated on the individual question-findings pages, summary pages, conclusions, and recommendations (Van Tiem et a., 2008, p. 6). The process thus used was a single cluster systematic random sampling of ISPI members. This sampling received one of four survey questions regarding best practice among the ISPI. Responses were then coded and themed by the researchers with the guidance of their research advisor. Finally, the theoretical basis of study was founded upon prior appreciative inquiry research spanning the work of Cooperrider, Whitney, & Stavros, 2003 as well as Watkins & Mohr, 2001. Factors influencing this design included available data while at the same time a lack of available time. Factors also included the means by which the entirety of society membership could be surveyed regarding the same topic. An alternative approach in this instance would have been to ask all members the same four questions, thus removing the clustering among the sampling strategy. This would permit the ability to detect trend across groups, rather than simply categorize data among groups.
These studies, while only a small sample of those available regarding appreciative inquiry, reflect an opportunity to further consider the possibilities of appreciative inquiry among program evaluation. Said of sound designs and analyses, Yarbrough et al. (2011) note, “Every evaluation requires an overall design that is responsive to features of the program and program components, context factors, and the purposes of the evaluation” (p. 201). I believe appreciative inquiry can provide this responsiveness if designed well, and I am excited by the prospect of exploring this very possibility further.
BetterEvaluation. (n.d.). Appreciative inquiry. Retrieved September 29, 2013 from http://betterevaluation.org/plan/approach/appreciative_inquiry.
Cook, R. (2009). Lessons learned from parietal art: Transformative pansemiotics for elearning. Proceedings of the IADIS International Conference on Cognition & Exploratory Learning In Digital Age, 303-306.
Niemann, R. R. (2010). Transforming an institutional culture: An appreciative inquiry. Catalyst (21519390), 39(3), 1003-1022.
Pascale, R., Sternin, J., & Sternin, M. (2010). The power of positive deviance: How unlikely innovators solve the world’s toughest problems. Boston, MA: Harvard Business Press.
Van Tiem, D., & Rosenzweig, J. (2008). How are we doing? “Best of ISPI” Appreciative inquiry member survey. Performance Improvement, 47(7), 5-11. doi:10.1002/pfi.20011
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.