Complicity Revisited: Balancing Stakeholder Input and Roles in Evaluation Use. American Journal of Evaluation. (original) (raw)

Reflecting on evaluation research: Intersections of academy, community, and identity

Communicative evaluation is a type of community-engaged scholarship that encourages collaboration between stakeholders and evaluators as they develop an action plan about a social problem. However, extant research has failed to adequately explore issues of power and identity encountered by communication evaluators in the field. Doing so could enrich assessment processes and outcomes to develop more nuanced theory and practice. Thus, we reflexively develop and integrate our personal stories and experiences of conducting communication evaluation research to highlight four dialectic identity tensions: (a) insider/outsider; (b) expert/novice; (c) program sustainer/impeder; and (d) researcher/friend. By displaying these tensions, we reveal potential opportunities for new insights that could offer pragmatic applications more attuned to the people and contexts of evaluation research. These tensions highlight the need for critical reflection in the pursuit of program sustainability and offer points for transformation. We conclude with pragmatic recommendations for engaging reflexivity in communication evaluation research.

Working with evaluation stakeholders: A rationale, step-wise approach and toolkit

Evaluation and Program Planning, 2011

Attention to stakeholders has gained prominence for both practical and ethical reasons. 3 Attention to, and involvement of, key stakeholders is presumed to enhance the design and implementation of evaluations and the use of evaluation results in decision-making. Beyond that, it would appear to be difficult to adhere to the standards for judging evaluations put forward by the Joint Committee on Standards for Educational Evaluations (1994)utility, feasibility, propriety, and accuracy-without careful attention to stakeholders. Similarly, it would be hard to follow the Guiding Principles for Evaluators of the American Evaluation Association (1995) without attending to stakeholders. The principles include: systematic inquiry, providing competent performance to stakeholders, integrity and honesty, respect for people, and responsibility for the general and public welfare. While most members of the evaluation community would agree that attention to stakeholders is important, they might not agree on how to define the term. The definition is consequential as it affects who and what counts (Alkin, Hofstetter & Ai, 1998; Mitchell, Agle, & Wood, 1997). For example, some definitions of stakeholders focus on program stakeholders (e.g. Rossi, Lipsey, and Freeman, 2003, pp. 18, 435). But starting with a program focus seems unduly restrictive. We propose a broader and more inclusive approach and define stakeholders as individuals, groups, or organizations that can affect or are affected by an evaluation process and/or its findings. The definition is purposefully broad so that the full range of possible stakeholders is considered before narrowing the focus to the primary intended users of an evaluation. This broad approach thus helps support the process of narrowing the focus to those stakeholders who are going to be the major audience for a specific evaluation effort-the primary intended users-while also identifying other stakeholders and their interests, powers, perspectives, and other related information to inform the evaluation effort (Bryson & Patton, 2010; Patton, 2008). Such inclusive thinking about stakeholders early on is consistent with (but broader than) the Encyclopedia of Evaluation definition of stakeholders as ''people who have a stake or a vested Evaluation and Program Planning 34 (2011) 1-12

Evaluation for What Purpose? Findings From Two Stakeholder Groups

Approaches to Global Sustainability, Markets, and Governance, 2018

A host of reasons exist for the pursuit of evidence in the public sector, including to support good governance and policy development. As the expectations for program evaluation from policymakers have evolved, so too has evaluation practice and a great deal of experimentalism has ensued. There is a risk that these developments and the complexity inherent in them, may lead to conflicting expectations about why program evaluation is done, or even a loss of purpose. This prompts the meso-level analysis of two types of stakeholders in a governance network, explored in this chapter. This chapter presents the findings of an ongoing study which explores the perceptions of program evaluators and policy implementers towards the purpose of evidence. The findings suggest that program evaluators and policy implementers have divergent expectations of why and how evaluation data might be used. The findings suggest that program evaluators aspire to support change and enhance the policy domains they serve, whereas policy implementers perceive program evaluation as serving a more governance-/managementorientated role. The chapter demonstrates the complexity of both program evaluation and policy and may have implications for the twin pillars of governance and responsibility at the heart of the book. If governance and responsibility are the twin pillars of sustainability then the complex networks of relationships, expectations, values, and outcomes may need to be considered. The findings also have implications for evaluation commissioners and practitioners, demonstrating the need for the purpose and expectations of program evaluation to be agreed early. The use of program evaluation as a symbolic, aesthetic or structural mechanism also emerges, prompting opportunity for further research, for instance, to explore legitimacy and program evaluation.

Evaluation: emergence, mode of inquiry, theory and practice

Research in Comparative and International Education, 2007

The aim of this article is to present the reader with an accessible and practical account of evaluation as a mode of inquiry within the broad domain of social science. The starting point is the presentation of a general outline of the main milestones relating to the emergence of evaluation as a mode of inquiry and some of the prominent advocates of contemporary evaluation. Particular attention is directed towards what is termed as 'programme evaluation'. Then the author endeavours to explore whether evaluation as a mode of inquiry is practice or theory led. A definitive answer to this question is not presented here in this short article; instead, the author makes assumptions based on his interpretation of the reviewed discourse within the field and personal reflection from practice. The case of the European University Association's quality review of the seven Irish universities and the Dublin Institute of Technology is cited as an example of evaluation in practice in the Irish higher education sector. Issues such as the politics and power, decision making, and the use, misuse or non-use of findings of evaluations are briefly introduced. A long shadow has fallen across the United States. The American tradition of open government is being seriously eroded. We have a regime in Washington under President George Bush that strives to control information beyond anything in my experience. Of course, all governments hide their mistakes and misdeeds and spin news to their advantage. Of course. But I am talking about control of information that threatens evaluation-and perhaps democracy.

Theoretical Synthesis of Evaluation Theory and Practice

States are increasingly looking to research and evidence to support capacity building decision-making in education policy (Akey, 2016). In many cases, the assessments and findings will be directly tied to accountability measures required to access federal funding by states. As a result, states and local education authorities are increasingly looking at innovative and alternative evaluative processes that can move beyond discipline-specific approaches (Akey, 2016). With this purpose in mind, 21st Century education leaders need to have an understanding of the theoretical foundations that underpin evaluation and the corresponding paradigms, branches, theories and approaches available. Mertens and Wilson (2012) have organized four models of evaluations: the post-positivist, constructivist, pragmatic and the transformative. These models link to branches that originate from various perspectives, and the assertion is made that each branch is well matched to a type of study and attendant methodology that reflect the evaluator’s philosophical assumptions (Shadish, 1998).

Communicating About Evaluation: A Conceptual Model and Case Example

2017

Background: Despite consensus within the evaluation community about what is distinctive about evaluation, confusion among stakeholders and other professions abounds. The evaluation literature describes how those in the social sciences continue to view evaluation as applied social science and part of what they already know how to do, with the implication that no additional training beyond the traditional social sciences is needed. Given the lack of broader understanding of the specialized role of evaluation, the field struggles with how best to communicate about evaluation to stakeholders and other professions. Purpose: This paper addresses the need to clearly communicate what is distinctive about evaluation to stakeholders and other professions by offering a conceptual tool that can be used in dialogue with others. Specifically, we adapt a personnel evaluation framework to map out what is distinctive about what evaluators know and can do. We then compare this map with the knowledge ...