Description

Equity-focused evaluation is defined as conducting evaluation to ensure equitable results and has its roots in the international development sector while equitable evaluation is rooted in the domestic context, has been particularly focused in philanthropy, and is defined as conducting evaluation in service of advancing equity. Equitable evaluation drew from equity-focused evaluation in its earliest conceptualization. Given their mutual inspiration across contexts, we discuss them together but note important distinctions in the international and U.S. domestic context.

Equity-focused evaluation was initially defined as “a judgment made of the relevance, effectiveness, efficiency, impact, and sustainability — and, in humanitarian settings, coverage, connectedness, and coherence — of policies, programs, and projects concerned with achieving equitable development results.In 2020, the Development Assistance Committee Network on Development Evaluation recognized the applicability of the “coherence” criterion across all settings, not just humanitarian ones (Organisation for Economic Co-operation and Development, 2020). Equity-focused evaluation involves a rigorous, systematic, and objective process in the design, analysis, and interpretation of information to answer specific questions, including those of concern to worst-off groups. It provides assessments of what works and what does not work to reduce inequity, and it highlights intended and unintended results for worst-off groups as well as the gap between best-off and worst-off groups. It provides strategic lessons to guide decision-makers and to inform stakeholders. Equity-focused evaluations provide evidence-based information that is credible, reliable, and useful, enabling the timely incorporation of findings, recommendations, and lessons into the decision-making process” (Bamberger & Segone, 2011 as cited in Dean-Coffey et al., 2014).

In the U.S., a 2017 framing paper (Center for Evaluation Innovation et al.) called on evaluators and funders to conduct and contribute to equity through evaluation. This paper introduced the Equitable Evaluation Framework™ (EEF), “a conceptual framing for the practice of evaluation,” that aims to encourage and help funders and practitioners to “challenge cultural norms that continue to promote preferences for a singular type of truth, knowing, and evidence” (EEI & GEO, 2021). It also identified and sought to highlight specific orthodoxies, tightly held and often unchallenged beliefs, about what is considered valid and credible evaluative practice. While not a specific approach or method, the EEF™ highlights three principles to ground the evolving practice of equitable evaluation (EEI and GEO, 2021): (1) evaluative work is in service of and contributes to equity, (2) evaluative work should be designed and implemented with the values underlying equity work, and (3) evaluative work can and should answer critical questions. It also warns of the risk of “reinforcing or even exacerbating the very inequities the change initiative seeks to address” if evaluations do not adhere to these principles.

Unique Contributions

Equitable evaluation in the international context was focused on achieving equitable development results for children (primarily girls) through policies, programs, and projects. It has since expanded to include a focus on individuals with disabilities. A primary contribution has been ensuring that vulnerable groups are centered in evaluation processes.

In the U.S. domestic context, equitable evaluation initially focused on how philanthropies impact evaluation. Foundations hold “love of person kind” as core and are uniquely positioned to create new norms and expectations for evaluation through their evaluation requests and funding. To that end, the Equitable Evaluation Framework™ (EEF) has explicit axiological, ontological, and epistemological underpinnings, which have implications for methodology. The EEF asks for clarity and transparency. It asks all engaged in the evaluative process to make clear and visible the mindsets, tensions, and sticking points that exist in evaluative practice in U.S. philanthropy. It asks evaluators to be more honest about the limitations and delimitations of evaluation efforts, and to move away from declarations of certainty and causality in contexts and in work where those are not only unattainable but also not useful aspirations. EEF has since expanded to address practice in broader contexts such as local and non-profit agencies.

History

Notions of equitable evaluation developed in both the international and domestic contexts. Tony Lake, a previous director of the United Nations Children’s Fund (UNICEF), was among the first to promote a focus on achieving equity over addressing inequalities. The 2011 UNICEF publication “How to design and manage equity-focused evaluations” sought to provide guidance on how to design and manage evaluations to assess the contribution of policies, programs, and projects to equitable development results for children. This was precipitated by a push for a stronger focus on equity in human development, especially the achievement of equitable development results for children. More recently, evaluators working in international settings have called for evaluation to move beyond accountability and contribute to transformational changes that address global crises (Van den Berg et al., 2021).

In the U.S., Dean-Coffey, Casey, and Caldwell (2014) conducted work supported by the Annie E. Casey Foundation to examine the role of philanthropy and equity in evaluation. Dean-Coffey expanded the work with support from the W.K. Kellogg Foundation, The California Endowment, and the California Health Care Foundation. Dean-Coffey reminds us that evaluation was initially conducted by academic researchers and the federal government to inform the allocation of public dollars and the effectiveness of those investments. It was not intended to explore and inform the development of solutions that addressed structural and social policies and practices that perpetuate inequality or promote equity (2018). Equitable evaluation in the U.S. started out as a capacity-building approach, with a focus on organizational learning and change (2014) and evolved into the Equitable Evaluation Framework™, which leads with three principles and 12 orthodoxies to challenge views on merit, value, and worth in philanthropic evaluation (EEI 2020; EEI and GEO, 2021). Additional orthodoxies for nonprofits and evaluation consultants are under development. Whereas equitable evaluation initially focused on methodology, the EEF explicitly takes an axiological stance and has broader epistemological and ontological underpinnings.

PRINCIPLES

Promotion of equity and social justice; attendance to issues of power

Equitable evaluation as practiced in the international and domestic contexts is rooted in social justice. The primary aim of equitable evaluation in the U.S. “is not only to shed light on the factors that impede equity, but also to analyze and assess interventions, investments, and strategies through a lens of promoting equity” (Dean-Coffey et al., 2014). Dean-Coffey and Equitable Evaluation Initiative (EEI) collaborators believe that “evaluators have a moral imperative to contribute to equity. Evaluators who work with foundations and nonprofits who are working on equity have a special obligation to ensure that their evaluation practices don’t reinforce or even exacerbate the inequities that efforts seek to address” (Center for Evaluation Innovation et al., 2017). The approach requires practitioners to call out institutional and structural policies that create inequities to attain equity, justice, and liberation. It also requires that evaluators pay attention to power throughout the entire evaluation process. “Power [is] the understanding of how privilege is attached to some cultural signifiers and prejudice to others. It involves paying attention to equity and social justice, and avoiding perpetuating discrimination, disparity, or condescension” (EEI & GEO, 2021). While likely in agreement with these goals, those interested in the practice of the Equitable Evaluation Framework™ (EEF) in international contexts have not been able to be as proactive in the conduct of evaluation in service of equity, justice, and liberation due to political limitations. International evaluators have, however, begun to shift their focus from the most marginalized populations (e.g., women) to the power dynamics that contribute to marginalized statuses (e.g., between men and women).

Engagement of partners and community members, particularly those with less social power, during all phases of the evaluation

Engagement of partners and community members, particularly those with less social power, is a key component of equitable evaluation. “Involvement in the design, planning and implementation of the evaluation of those stakeholders (especially worst-off groups) directly affected by an intervention (be they the implementers or intended beneficiaries), is a fundamental principle of any equity-focused evaluation … guaranteeing stakeholder participation strengthens accountability, builds trust and agreement in the evaluation process, generates credibility, and can itself contribute to equity building” (Bamberger & Segone, 2011). In equitable evaluation, stakeholder participation can be facilitated by paying attention to group dynamics, development, needs, and effectiveness, as well as ensuring that materials are translated “into languages and formats designed to ensure that everyone can participate fully in the evaluation and plan to share data in meaningful ways to all populations” (Dean-Coffey et al., 2014).

Composition of evaluation team and reflection on assumptions and biases

Equitable evaluation in both the international and U.S. domestic contexts focuses on having a diverse and inclusive team to reflect the situations and contexts. “Selecting a strong team to conduct an equity-focused evaluation is a key step in a successful evaluation process. A good team must have an appropriate mix of skills and perspectives” (Bamberger & Segone, 2011). When assembling teams, it is important to “identify and include individuals with shared experiences related to the issue at hand when considering evaluation design and implementation, meeting structures, time durations, and locations” (Dean-Coffey et al., 2014). The practice of the Equitable Evaluation Framework™ (EEF) “calls upon individuals [evaluators] to understand how race and ethnicity operate with respect to equity and how race and ethnicity intersect with other socially defined characteristics, such as sexuality, class, nationality, and age“ (Dean-Coffey et al., 2014) as well as reflection on “our history because it sheds light on the implicit and explicit beliefs, values, and intentions that frame our current approaches and frameworks“ (Dean-Coffey, 2018). Three mindset shifts have been identified as important for practitioners of the Equitable Evaluation Framework™: moving from doing to being, from scarcity to abundance, and from fixed to growth (EEI & GEO, 2021). These mindset shifts are key to being able to be in service of equity.

Consideration of cultural and historical contexts and different worldviews

In the international development sector, consideration of cultural and historical contexts and different worldviews is critically important. Bamberger & Segone (2011) note that “when designing an equity-focused evaluation it is important to understand the context within which the intervention has been implemented, and the factors that affected implementation and accessibility to the different worst-off groups (including economic, political, socio-cultural, environmental, legal, institutional, and other factors)”. Even when practiced in the US Domestic context, “equitable evaluation explicitly accounts for the context of structural barriers to social change and recognizes that culture is bound to this context. We are evolving toward evaluation that more fully recognizes context and cultural validity and that explores questions that deepen understanding of historic and present-day structural and systematic barriers to desired outcomes” (Dean-Coffey, 2018). The Equitable Evaluation Framework™ (EEF) notes that consideration of historical and structural decisions has contributed to the condition to be addressed and understanding that the cultural context is tangled up in both the structural conditions and the change initiative itself.”

Intentional methods and thoughtful data collection

Appreciation for mixed methods has increased in both the international and U.S. domestic contexts. In their seminal publication on equity-focused evaluations in international contexts, Bamberger and Segone (2011) noted, “irrespective of the size and nature of the intervention, an evaluation design which applies a mixed-method approach will usually be the most appropriate to generate an accurate and comprehensive picture of how equity is integrated into an intervention. Mixing qualitative and quantitative approaches, while ensuring the inclusion of different stakeholders (including the worst-off groups), will offer a wide variety of perspectives and a more reliable picture of reality.” International evaluations now almost exclusively rely on qualitative methods such as key informant interviews and document review, as quantitative surveys can be expensive and contribute to assessment fatigue. In both international and U.S. domestic contexts, “existing methodology may not be sufficient. …through experimentation and creativity, evaluators can show how using a different set of tools or tests might allow for new questions to be asked and answered.” (Dean-Coffey et al., 2014). Dean-Coffey urges evaluators to “move beyond methodological approaches and evaluator demographics to address culture and context, and in so doing, unpack our definitions of evidence, knowledge, and truth, so that we may create new ones grounded in this time, place, and set of intentions” (2018). The Equitable Evaluation Framework™ calls for consideration of history, location, power, voice, relationship, time, plasticity, and reflexivity to ensure validity of evaluation findings (EEI, 2020; EEI & GEO, 2021).

Intentional analysis and inclusive interpretation

Equitable evaluation and the practice of the Equitable Evaluation Framework™ (EEF) examines what works, for whom, and under what conditions (Bamberger & Segone, 2011; Center for Evaluation Innovation et al., 2017). Both Dean-Coffey (2018), and Bamberger and Segone, (2011) provide guidance on how data should be interpreted with an eye toward equity. Dean-Coffey (2018) states, “data about the problem being addressed m—as well as about the outcomes of the change initiative — should at minimum be disaggregated, so that differential effects by race, ethnicity, gender, language, or a myriad of other dimensions can be spotted and accounted for.” Furthermore, stakeholders play a prominent role in the interpretation of findings. Bamberger and Segone (2011) believe that discussing the results with them provides “an opportunity to explain how their contributions were used, and to provide them with the chance to correct any inaccuracies and to clarify any doubts.” In international settings, implementation partners and program managers are more likely than community members receiving services to have time to contribute to interpreting findings. Nevertheless, involving community members in interpreting data is important, as “community members or implementers of the intervention studied can also provide important context for and interpretation of the results of quantitative or qualitative data analysis” (Stern et al. 2019). Additionally, the Equitable Evaluation Framework™ identifies cultural responsiveness and context as key components of rigor (EEI & GEO, 2021). Further, one of the principles of the EEF states evaluative work can and should answer critical questions about the effect of a strategy on different populations and the underlying systemic drivers of inequity (Center for Evaluation Innovation et al. 2017; EEI, 2020).

Accessible and actionable evaluation findings

Critical to equitable evaluation and the practice of the Equitable Evaluation Framework™ (EEF) in both the international and domestic contexts is the use of evaluation findings for policy and systems change. Bamberger and Segone (2011) note that, “knowledge generated through an equity-focused evaluation provides evidence to influence major policy decisions to ensure that existing and future policies will enhance equity and improve the well-being of worst-off groups.” Indeed, policy development is often in service of trying to provide for communities at large, thus, “evaluation findings or the process itself should be used to move policy and practice that can either promote or inhibit equity” (LaFrance & Nichols, 2008 as cited in Dean-Coffey et al., 2014). The Equitable Evaluation Framework™ “requires the ability to influence conversations that shift culture and practice, and impact resource allocation” (EEI & GEO, 2021).