Overview
The Eval Matrix
You can use this interactive tool to compare and contrast how the philosophies and approaches apply the seven key principles of culturally responsive and equity-focused evaluation. Use this tool to select which principles and approaches you would like to examine; you can select up to three at one time. The results of your query will automatically display in a table below. This tool was created to help easily pull information from the matrix that you may be most interested in exploring.
If you would like to explore the full evaluation matrix, you can download it using the link below. The downloadable matrix is a table in excel format that includes each of the philosophies and approaches, a brief description of each, some historical background, and their unique contributions, as well as how they apply each of the key principles. Additionally, the excel file includes references to seminal publications associated with each approach.
Approaches
Choose up to 3 approaches to compare.
If limit is reached, please uncheck to choose more.
Principles
Choose your principles
View Your Results
| PRINCIPLES | APPROACHES | ||||||
|---|---|---|---|---|---|---|---|
| Culturally Competent Evaluation | Culturally Responsive Evaluation | Culturally Responsive and Equitable Evaluation | Culturally Responsive Indigenous Evaluation | Empowerment Evaluation | Equity-Focused Evaluations | Transformative Evaluation | |
 Promotion of equity and social justice; attendance to issues of power |
A culturally competent evaluation should pay attention to “how race, power, inclusion, politics, and privilege may be affecting the context, and discuss their relevance to the program” (CDC, 2014). Culturally competent evaluator(s) “represent the values and standards of marginalized stakeholders [and] use their power to promote equality and self-determination” (AEA, 2011). However, the culturally competent evaluation approach has been critiqued when power is not addressed. Thomas et al. (2018) argue that “the emphasis on the multiple dimensions of culture has been inadequate in addressing many of the systemic racial issues related to social programs … attention to cultural responsiveness in evaluation has stalled thoughtful deliberation of how we could integrate issues of race and racism in our work.” | A key characteristic of culturally responsive evaluation (CRE) is “attention to issues of power differentials among people and systems” (Hood et al., 2015). CRE requires the use of methods and tools that best serve the community (Hood et al., 2015) and the use of evaluation to promote social agendas that challenge current power structures and dynamics rooted in white supremacy and oppression (Caldwell & Bledsoe, 2019; Chouinard & Cram, 2019; Hopson 2009; Public Policy Associates, 2015). | As indicated by its name, a focus on equity is inherent to culturally responsive and equitable evaluation (CREE). As part of the CREE Learning Series, Tamara Cadet addresses why equity matters and draws from Chimamanda Adichie who explains, if we only hear about a people, place, or situation from one point of view, we risk accepting one experience as the whole truth (Expanding the Bench, 2021, video 1.3). In video 2.4 of the Learning Series (Expanding the Bench, 2021), Michael P. Arnold addresses the importance of being intentional and transparent in our work and our definitions of equity to help create a better understanding of equity goals. | Social justice itself is not specifically mentioned; instead, it is assumed to be “baked“ into the theory of culturally responsive Indigenous evaluation (CRIE) and tribal critical theory (TCT). The call for CRIE evaluators and researchers to be intimately involved in social change and calling out injustices is specifically mentioned (Bowman et al., 2015). Bowman and colleagues (2007, as cited in 2015) discuss indigenizing research and evaluation via seven steps, including (1) utilizing traditional-knowledge councils and elders; (2) using oral and written traditional knowledge and Indigenous institutions; (3) providing traditional gifts as part of the evaluation process to show gratitude for allowance into the community; (4) understanding that Indigenous intellectual knowledge, approval of evaluation, and ownership of data by the tribal community is controlled by the Indigenous community and is formalized through memoranda of understanding; (5) understanding that evaluation data is used to inform and improve (not pathologize) Indigenous communities and institutions; (6) understanding that evaluation work is examined by traditional knowledge councils to prohibit racism and colonization, and to promote value and use of Indigenous ways of knowing, knowledge, and processes; and (7) making evaluation useful to community needs, and through Indigenous control/ownership of data. That said, neither social justice nor equity can exist without true decolonization. Decolonization — which would include reparations and the return of stolen lands and property — is a foundation of equity, social justice, and transformation. | Social justice is one of the approach’s 10 principles — evaluation can and should be used to address social inequities in society. A key component is inclusion, ensuring that all voices, particularly those most impacted, are heard and valued (Fetterman et al., 2015). Indeed, Fetterman et al., (2015) expanded this to “transformative empowerment evaluation,” which specifically focuses on psychological, social, and what he refers to as the “political power of liberation.” Empowerment Evaluation has long addressed the ability to engage in social redress (Mithaug as cited in Fetterman et al.,1996) by enabling capacity building and increased opportunity. This is prominent throughout empowerment evaluation, which additionally seeks to close the gap between evaluator and partners; empowerment evaluators and partners learn from one another, thereby closing the power gap between them (Andrews, 1996). | Equitable evaluation as practiced in the international and domestic contexts is rooted in social justice. The primary aim of equitable evaluation in the U.S. “is not only to shed light on the factors that impede equity, but also to analyze and assess interventions, investments, and strategies through a lens of promoting equity” (Dean-Coffey et al., 2014). Dean-Coffey and Equitable Evaluation Initiative (EEI) collaborators believe that “evaluators have a moral imperative to contribute to equity. Evaluators who work with foundations and nonprofits who are working on equity have a special obligation to ensure that their evaluation practices don’t reinforce or even exacerbate the inequities that efforts seek to address” (Center for Evaluation Innovation et al., 2017). The approach requires practitioners to call out institutional and structural policies that create inequities to attain equity, justice, and liberation. It also requires that evaluators pay attention to power throughout the entire evaluation process. “Power [is] the understanding of how privilege is attached to some cultural signifiers and prejudice to others. It involves paying attention to equity and social justice, and avoiding perpetuating discrimination, disparity, or condescension” (EEI & GEO, 2021). While likely in agreement with these goals, those interested in the practice of the Equitable Evaluation Framework™ (EEF) in international contexts have not been able to be as proactive in the conduct of evaluation in service of equity, justice, and liberation due to political limitations. International evaluators have, however, begun to shift their focus from the most marginalized populations (e.g., women) to the power dynamics that contribute to marginalized statuses (e.g., between men and women). | Power is a central issue that must be addressed at each stage of a transformative evaluation (Mertens, 2007). Evaluators who subscribe to the transformative paradigm “consciously and explicitly position themselves side by side with the less powerful in a joint effort to bring about social transformation” (Mertens, 2020). The transformative evaluator focuses on “unequal distributions of power and the resultant oppression of subjugated groups,” and “a preset goal of the research is to empower participants to transform the status quo and emancipate themselves from ongoing oppression” (Mertens, 2012). “Evaluators have a responsibility to make visible the dynamics of discrimination and oppression that are relevant in the evaluation context” (axiological assumption) (Mertens, 2016). “The transformative paradigm has relevance for people who experience discrimination and oppression on [the basis of any] characteristics that are associated with less access to social justice” (Mertens, 2010). |
 Engagement of partners and community members, particularly those with less social power, during all phases of the evaluation |
Culturally competent evaluation requires engaging stakeholders that reflect the diversity of the community (in terms of demographics, beliefs, institutions, ideologies, etc.) and working to ensure these stakeholders can participate fully in the evaluation (CDC, 2014). When identifying key stakeholders, it may be helpful to talk to community leaders or others who can serve as bridge builders or cultural translators. These individuals can help the evaluator become familiar with cultural norms (Lee, 2007) and help “assess the community’s readiness and willingness to address the issues that the evaluation may raise” (CDC, 2014). A smaller group of key stakeholders may serve on an advisory team that is able to bring in a larger group of stakeholders as needed. It is important to build trust with all stakeholders. This can be achieved by talking with stakeholders “about the evaluation, why it is being done, and how the community can expect to benefit” (CDC, 2014). Defining clear roles and setting ground rules can also aid in establishing equality among stakeholders and evaluators. Once a group of stakeholders has been identified and prepared to participate, they can ensure that evaluation questions reflect the values of their community and that their input shapes decisions, judgments, interpretations and recommendations (CDC, 2014). | “Stakeholders play a critical role in all evaluations, especially culturally responsive ones, providing sound advice from the beginning (framing questions) to the end (disseminating the evaluation results)” (Frierson et al., 2002). It is important to acknowledge that “people are at the center” of culturally responsive evaluation (CRE) and they know what they value and need (McBride, 2018). This approach recommends that evaluators “develop a stakeholder group representative of the populations the project serves, assuring that individuals from all sectors [and varying levels of power, status, and resources (Hood et al., 2015)] have the chance for input” (Frierson et al., 2002). One important way of doing this is to listen and hear the questions of stakeholders so that they are incorporated into the evaluation design (Frierson et al., 2002). As in culturally competent evaluation, it is critical to establish a climate of trust and respect among all parties involved (Hood et al., 2015). | In keeping with the definition of culturally responsive and equitable evaluation (CREE), Lemos and Garcia (2020) assume that community members are partners and are an essential part of the evaluative process, from conceptualization to dissemination. They argue that the community must be involved in the change process, with data made available to community members in order to bring about change. Evaluators need to spend adequate time during the design phase to align their understanding, expectations, and strategies. Lemos and Garcia (2020) state that evaluators who work collaboratively with diverse Latinx and other diverse communities should ensure that these perspectives are integrated throughout the process and address unanticipated challenges. | In the best-case scenario in culturally responsive Indigenous evaluation (CRIE), the team would be led by Indigenous evaluators and stakeholders. In their article focusing on tribal populations’ relationship with federal governments, Waapalaneexkweew (Nicole Bowman, Mohican/Lunaape) and Dodge-Francis (2018) provided an example in which the evaluation team asked tribal stakeholders for help in guiding the activities of the evaluation. The project team used multiple strategies to receive varied and diverse stakeholder feedback throughout the study design, testing, and full data-collection process. This included meeting documents and formal administrative responses/reactions to the project and team, including production and dissemination of the final report. | Specifically, the 10 principles of empowerment evaluation are geared toward empowering the key stakeholder communities to conceptualize, lead, and perform their own evaluation work. Key principles include (1) community ownership – empowerment evaluation values and facilitates community control, and use and sustainability are dependent on a sense of ownership; (2) inclusion – empowerment evaluation invites involvement, participation, and diversity and contributions come from all levels and walks of life; (3) community knowledge – empowerment evaluation respects and values community knowledge; (4) and capacity building – empowerment evaluation is designed to enhance stakeholders’ ability to conduct an evaluation and to improve program planning and implementation (Fetterman et al., 2015). | Engagement of partners and community members, particularly those with less social power, is a key component of equitable evaluation. “Involvement in the design, planning, and implementation of the evaluation of those stakeholders (especially worst-off groups) directly affected by an intervention (be they the implementers or intended beneficiaries), is a fundamental principle of any equity-focused evaluation … guaranteeing stakeholder participation strengthens accountability, builds trust and agreement in the evaluation process, generates credibility, and can itself contribute to equity building” (Bamberger & Segone, 2011). In equitable evaluation, stakeholder participation can be facilitated by paying attention to group dynamics, development, needs, and effectiveness, as well as ensuring that materials are translated “into languages and formats designed to ensure that everyone can participate fully in the evaluation and plan to share data in meaningful ways to all populations” (Dean-Coffey et al., 2014). | “The evaluator has an ethical responsibility to engage with stakeholders to examine the assumptions about the problem, solution, and evaluation methods in order to increase the potential for social change” (Mertens, 2016). “Community participation is needed at the beginning, throughout, and at the end of each research study” (Mertens, 2007). “The evaluation focus, purpose, design, implementation, and utilization should be developed through a cooperative process between the evaluator and community members” (Mertens, 2012). Benefits of an inclusive approach include “aiding in impartiality, gaining new information, and improving the sensitivity of evaluation” (Mertens, 1999). “Evaluators need to establish trusting relationships with the full range of stakeholder groups in order to obtain an accurate picture of the problem under study” (epistemological assumption) (Mertens, 2016). Coalitions are one means of ensuring community members inform the evaluation process and are positioned to support sustainability when the evaluation ends. |
 Composition of evaluation team and reflection on assumptions and biases |
Assessment of cultural self-awareness is important for effective communication and establishment of trust and respect with stakeholders (CDC, 2014). Culturally competent evaluators reflect on their backgrounds and how those may shape their assumptions and frames of reference for evaluation (Hopson, 2003). They are “willing to engage in a dialogue about how culture, social identity, and privilege and power affect them personally and in their work” (Lee, 2007), and “work to avoid reinforcing cultural stereotypes and prejudice” (AEA, 2011). | Social investments often target populations of color and yet are evaluated by white evaluators (Public Policy Associates, 2015). Culturally responsive evaluation (CRE) argues that data collection, quality, and interpretation can be improved when evaluators share a lived experience with program participants (Frierson et al., 2002; Hopson, 2009; Hood et al., 2015). However, this does not mean that racial congruence automatically leads to cultural congruence (Frierson et al., 2002). A diverse evaluation team allows for multiple perspectives in the evaluation (Public Policy Associates, 2015) and genuine connection with the local context (Hood et al., 2015). All evaluators should critically reflect on their own personal cultural preferences and biases (Hood et al., 2015; Hopson 2009; Public Policy Associates, 2015) and “make a conscious effort to restrict any undue influence they might have on the work” (Frierson et al., 2002). | Culturally responsive and equitable evaluation (CREE) advocates for diverse evaluation teams that provide multiple voices and perspectives. “Evaluators must recognize that different life experiences lead to different views of the world, and that linguistic, historical, and socioeconomic differences can be contributors and/or barriers to understanding” (Public Policy Associates, 2015). Suggestions include hiring staff from diverse backgrounds and life experiences. To understand the community perspective, evaluation team members should include evaluators who share similar experiences. “For areas where the evaluation team does not have shared life experiences with the priority population and cannot compensate for that fact, be very conscious of these differences and how they might impact key evaluation findings” (Public Policy Associates, 2015). CREE also stresses the importance of evaluators’ awareness of their own biases and recommends cultural competence training (Anderson et al., 2020). | Although the team might be composed of non-Indigenous evaluators, the team and evaluation plan must be led by and reflective of Indigenous practices (Cram et al., 2015). Evaluators must be prepared to include, if not be grounded in, multicultural validity (Bowman et al., 2015). | Empowerment evaluation is designed to be community and stakeholder owned, in which communities and partners self-determine processes and outcomes, and measure against those. Thus, the composition of the evaluation team is comprised primarily of community partners/stakeholders and an evaluation coach/facilitator. Empowerment evaluation is described as being driven by group process — the group serves as its own review and accountability group. The group learns from each other, serving as their own peer review group (Fetterman & Wandersman, 2004). Empowerment evaluation is designed to empower stakeholders to be able to do their own evaluation, capacity building, and thinking through of the values (and considerations) that underlie the evaluation. | Equitable evaluation in both the international and U.S. domestic contexts focuses on having a diverse and inclusive team to reflect the situations and contexts. “Selecting a strong team to conduct an equity-focused evaluation is a key step in a successful evaluation process. A good team must have an appropriate mix of skills and perspectives” (Bamberger & Segone, 2011). When assembling teams, it is important to “identify and include individuals with shared experiences related to the issue at hand when considering evaluation design and implementation, meeting structures, time durations, and locations” (Dean-Coffey et al., 2014). The practice of the Equitable Evaluation Framework™ (EEF) “calls upon individuals [evaluators] to understand how race and ethnicity operate with respect to equity and how race and ethnicity intersect with other socially defined characteristics, such as sexuality, class, nationality, and age“ (Dean-Coffey et al., 2014) as well as reflection on “our history because it sheds light on the implicit and explicit beliefs, values, and intentions that frame our current approaches and frameworks“ (Dean-Coffey, 2018). Three mindset shifts have been identified as important for practitioners of the Equitable Evaluation Framework™: moving from doing to being, from scarcity to abundance, and from fixed to growth (EEI & GEO, 2021). These mindset shifts are key to being able to be in service of equity. | Evaluators are not viewed as unbiased or detached from the evaluation process. Transformative evaluators expect that “their values in regard to social justice and human rights will influence the process and outcomes of their work” (Mertens, 2012). Additionally, to do their work well, evaluators must be aware of culture and power dynamics and be respectful. “To know realities, it is necessary to have an interactive link between the researcher and the participants in a study. Knowledge is socially and historically located within a complex cultural context. Respect for culture and awareness of power relations is critical (epistemological assumption). Respect is critically examined in terms of the cultural norms of interaction within a community and across communities (axiological assumption) (Mertens, 2007). |
 Consideration of cultural and historical contexts and different worldviews |
When conducting culturally competent evaluations it is important to “understand the program’s social, cultural, and historical contexts as well as the culture of the program itself” (CDC, 2014). Evaluators should familiarize themselves with culturally determined mores, values, and ways of knowing in the population being evaluated (AEA, 2011). This may require “deliberately setting aside time and resources in the evaluation timeline and budget to learn about differences and similarities” among stakeholders, as well as between the evaluator and stakeholders, and designing processes that take these similarities and differences into account (Lee, 2007). | Culturally responsive evaluation (CRE) requires particular attention to the historical, sociopolitical, community, and organizational contexts in which an evaluation will be conducted (McBride, 2018). This includes the history of the location, the program, and the people (Hood et al., 2015). “As much as possible, evaluators should understand the realities and challenges to the daily lives of the priority population based on political positioning, leadership, power dynamics, peer agency coordination, etc. These conditions and more could influence the success of an investment and the outcomes continuing, discontinuing, or expanding support based on evaluation results and findings” (Public Policy Associates, 2015). | Cultural and historical context are acknowledged and addressed in culturally responsive and equitable evaluations. “Evaluators should also recognize the continuing reality of white privilege and structural oppression that perpetuate racial inequity in America today and develop evaluation protocols that account for these factors. Finally, evaluators using this lens must recognize that each community has its own history and context, which must be acknowledged and considered when assessing the impact of social investments and developing findings and recommendations” (Public Policy Associates, 2015). | Understanding historical context in this field is essential; researchers must acknowledge and address the dynamics of power (Gitlin, 1994 as cited in Bowman et al., 2015) and disempowerment when creating research or evaluation studies conducted with Indigenous people. Prior to European contact, Indigenous people inhabiting North America used their own systems of self-governance to sustain high levels of health, education, and social and community welfare of tribal people. Each tribe was unique in its culture, customs, worldview, and traditions, and other teachings were grounded in a way of life that was distinct to each tribe. In a culturally responsive Indigenous evaluation (CRIE) framework, it is important to understand and consider the unique worldview(s) of the tribe(s) impacted by the evaluation (University of Toronto Press, 2020). | Although culture and context are not specifically mentioned, empowerment evaluation’s grounding has been in ethnography and ethnographic research. Additionally, the overarching and guiding principles of being community-owned and based upon community knowledge are closely aligned with grounding in culture and context. To that end, the history and background of communities must be part of the evaluation, if honoring community knowledge and capacity. In an empowerment evaluation framework, centering community knowledge and community ownership is foundational to self-determination (Fetterman et al., 2018). | In the international development sector, consideration of cultural and historical contexts and different worldviews is critically important. Bamberger and Segone (2011) note that “when designing an equity-focused evaluation it is important to understand the context within which the intervention has been implemented, and the factors that affected implementation and accessibility to the different worst-off groups (including economic, political, socio-cultural, environmental, legal, institutional, and other factors)”. Even when practiced in the U.S. domestic context, “equitable evaluation explicitly accounts for the context of structural barriers to social change and recognizes that culture is bound to this context. We are evolving toward evaluation that more fully recognizes context and cultural validity and that explores questions that deepen understanding of historic and present-day structural and systematic barriers to desired outcomes” (Dean-Coffey, 2018). The Equitable Evaluation Framework™ (EEF) notes that consideration of historical and structural decisions has contributed to the condition to be addressed and understanding that the cultural context is tangled up in both the structural conditions and the change initiative itself.” | “The use of a social justice lens highlights the need for careful contextual analysis as the starting point for evaluations that are focused on social transformation designed to improve human rights” (Mertens, 2016). Contextual analyses are an essential stage of the evaluation. Evaluators are charged with acknowledging “the historical and social location of knowledge, combined with an understanding of the effect of a legacy of discrimination and oppression. The evaluator’s role is to bring to visibility historical knowledge of this nature, link it to the social positioning of those who create the knowledge, and critically examine the consequences of accepting one version of reality over another” (Mertens, 2016). The transformative ontological assumption holds that “there are multiple realities that are socially constructed” (Mertens, 2007); that is, different realities can emerge and be experienced based on one’s lived experience. Evaluation should explore different versions of reality for people from different social positionalities in “determining the reality that holds potential for social transformation and increased social justice” (Mertens, 2007). |
 Intentional methods and thoughtful data collection |
The CDC (2014) notes that culturally competent evaluations should be sensitive to preferences for nonlinearity in logic model design, integrate multiple perspectives when determining what counts as credible evidence, rely on culturally appropriate data collection instruments, consider cultural and linguistic needs when planning data collection, and ensure that data collection processes are respectful of cultural mores (e.g., communication styles, conceptualizations of time, social hierarchies). | Social science methodologies are socially constructed and centered on the Western white male perspective (Chouinard & Cram, 2019). Culturally responsive evaluation (CRE) originally favored qualitative over quantitative methodologies to allow for the emergence of cultural nuances but now recommends mixed methods (Hood et al., 2015). For the purposes of CRE, the evaluation design does not need to be elaborate — it just needs to be appropriate for context. Instruments should be culturally valid; that is, they should resonate with the culture of the participants participating in the program being evaluated (Public Policy Associates, 2015). This may require pilot testing and adaptation to ensure cultural sensitivity and responsiveness. The process of data collection should also be responsive to cultural context. Failure to ensure that those collecting qualitative data are attuned to cultural context may lead to invalid data (Frierson et al., 2002). | Culturally responsive and equitable evaluation (CREE) is an approach that should be infused into all evaluation methodologies. Because CREE is a compilation of culturally responsive and equity-focused evaluation approaches, intentional methodology focusing on cultural validity and credible evidence is highlighted. As part of the CREE Learning Series, Donna Mertens describes CREE methods that include “building of relationships, contextual analysis, having inclusion and diversity, addressing power differences, and focusing on the use of the data for transformative purposes” (Expanding the Bench, 2021, video 2.2). | In regard to international methods and data collection, Bowman et al. (2015) states, “Tribal communities must not lose sight of the quest to create or attain a culturally responsive evaluation system that embraces their hegemonic ability to dictate the mission, infrastructure, or organizational framework” (pg. 354). To that end, the context of Indigenous evaluation addresses issues related to the cultural/traditional context and that define the design of the evaluation, including gaining access to the right of knowledge and understanding how that knowledge is transmitted (Bowman et al., 2015). | Methods and data collection are designed to place evaluation in the hands of community and staff members to facilitate ownership, enhance credibility, and promote action. Communities are involved in the development of tools designed to help with assessment, planning, implementation, and self-evaluation of their program. All methods, including those that are innovative, are considered viable if appropriate for the situation (Fetterman et al., 2015). Although the use of quasi-experimental designs is considered possible and probable, experimental designs are not necessarily mentioned. That said, there is no expectation that under the appropriate condition, an experimental design would be discounted. | Appreciation for mixed methods has increased in both the international and U.S. domestic contexts. In their seminal publication on equity-focused evaluations in international contexts, Bamberger and Segone (2011) noted, “irrespective of the size and nature of the intervention, an evaluation design which applies a mixed-method approach will usually be the most appropriate to generate an accurate and comprehensive picture of how equity is integrated into an intervention. Mixing qualitative and quantitative approaches, while ensuring the inclusion of different stakeholders (including the worst-off groups), will offer a wide variety of perspectives and a more reliable picture of reality.” International evaluations now almost exclusively rely on qualitative methods such as key informant interviews and document review, as quantitative surveys can be expensive and contribute to assessment fatigue. In both international and U.S. domestic contexts, “existing methodology may not be sufficient. … through experimentation and creativity, evaluators can show how using a different set of tools or tests might allow for new questions to be asked and answered” (Dean-Coffey et al., 2014). Dean-Coffey urges evaluators to “move beyond methodological approaches and evaluator demographics to address culture and context, and in so doing, unpack our definitions of evidence, knowledge, and truth, so that we may create new ones grounded in this time, place, and set of intentions” (2018). The Equitable Evaluation Framework™ (EEF) calls for consideration of history, location, power, voice, relationship, time, plasticity, and reflexivity to ensure validity of evaluation findings (EEI, 2020; EEI & GEO, 2021). | Evaluators should use mixed methods to capture the complexity of the problem under study (Mertens, 2020). “The transformative methodological assumption includes the concepts of inclusion of the full range of stakeholders in the process of decision-making about the evaluation methods, conduct of contextual analyses that identify cultural factors and issues of power as a basis for building trusting relationships, use of mixed methods to capture the cultural complexity and the data necessary to be appropriately responsive and inclusive of diverse stakeholders, building on community assets, and designing the evaluation to challenge assumptions that impede progress toward positive social change. … a mixed methods approach has the potential to provide a fuller picture of the complexities of problems, as well as the design, implementation, and determination of the effectiveness of solutions” (Mertens, 2016). Mertens proposes a cyclical model of mixed-methods research “as a way of involving the community in research decisions and the collection of data that can be used for social justice purposes” (Mertens, 2007; Mertens, 2009). |
 Intentional analysis and inclusive interpretation |
Culturally competent evaluators “carefully examine the data to understand how contextual conditions and structural inequities affect the outcomes” (Lee, 2007). Mixed methods allow for the detection of patterns of statistical significance through quantitative analysis and contextual understanding of the patterns through qualitative analysis (Lee, 2007). “Culture serves an intrinsic role in influencing how we define categories for organization purposes, classify and interpret data, and determine which comparisons have meaning. Therefore, engaging stakeholders during the analysis is essential” (CDC, 2014). Culturally competent evaluators also consider the intersection of multiple cultural and social identities (Thomas & Campbell, 2020). | Culturally responsive analysis requires “sensitivity to, and understanding of, the cultural context in which data are gathered” (Frierson et al., 2002) and culture is assumed to frame validity, rather than validity framing culture (Kirkhart, 2013). Deriving meaning from data in program evaluations that are culturally responsive requires people who understand the context in which the data were gathered (Frierson et al., 2002; Hood et al., 2015). Stakeholder review panels are one way to examine evaluative findings gathered by the principal evaluator and/or an evaluation team (Frierson et al., 2002). Special attention should be paid to differences by groups (disaggregation) and outliers in the analysis process (Frierson et al., 2002; Hood et al., 2015; Public Policy Associates, 2015). | Culturally responsive and equitable evaluation (CREE) is a compilation of culturally responsive and equity-focused evaluation approaches. Analysis and dissemination would follow those of culturally responsive evaluation, culturally responsive Indigenous evaluation, and other participatory approaches that lead to both accessible and actionable findings. It includes engagement of the priority population in the analysis and interpretation of the data in order to develop contextualized and sustainable recommendations (Mendez & Taniuchi, 2020; Anderson et al., 2020). | Bowman et al. (2015) notes that the evaluation work is designed to allow for continuous program evaluation and quality improvement, and is built upon human and infrastructure capacities for future evaluations. Analysis and interpretation are guided by cultural perspective and context. Member checking, a method that improves the credibility of qualitative research by matching the evaluator/researcher’s data and analysis with participants’ and community interpretations, is often used in decolonizing research to improve credibility of qualitative data (Smith, 2012 as cited in Bowman et al., 2015). Tribal, community elders, and other representatives are key participants in the analysis and interpretation of the data. | Through an empowerment process, people learn to think evaluatively (Patton, 2002 as cited in Fetterman et al., 2015). This process makes them more likely to make decisions and take actions based on their evaluation data. Empowerment evaluation is designed to “improve their programs using a form of self-evaluation and reflection. Participants continually assess their progress toward self-determined goals and shape plans and strategies in accordance with the community assessment” (Thomas & Campbell, 2020). | Equitable evaluation and the practice of the Equitable Evaluation Framework™ (EEF) examines what works, for whom, and under what conditions (Bamberger & Segone, 2011; Center for Evaluation Innovation et al., 2017). Both Dean-Coffey (2018) and Bamberger and Segone (2011) provide guidance on how data should be interpreted with an eye toward equity. Dean-Coffey (2018) states, “aata about the problem being addressed — as well as about the outcomes of the change initiative — should at minimum be disaggregated, so that differential effects by race, ethnicity, gender, language, or a myriad of other dimensions can be spotted and accounted for.” Furthermore, stakeholders play a prominent role in the interpretation of findings. Bamberger and Segone (2011) believe that discussing the results with them provides “an opportunity to explain how their contributions were used, and to provide them with the chance to correct any inaccuracies and to clarify any doubts.” In international settings, implementation partners and program managers are more likely than community members receiving services to have time to contribute to interpreting findings. Nevertheless, involving community members in interpreting data is important, as “community members or implementers of the intervention studied can also provide important context for and interpretation of the results of quantitative or qualitative data analysis” (Stern et al. 2019). Additionally, the Equitable Evaluation Framework™ identifies cultural responsiveness and context as key components of rigor (EEI & GEO, 2021). Further, one of the principles of the EEF states evaluative work can and should answer critical questions about the effect of a strategy on different populations and the underlying systemic drivers of inequity (Center for Evaluation Innovation et al. 2017; EEI, 2020). | Most germane to analysis is the transformative ontological assumption which deals with the nature of reality. This assumption “holds that there are diversities of viewpoints with regard to many social realities, but we need to place those viewpoints within a political, cultural, and economic value system to understand the basis for the differences. And then, we need to struggle with revealing those multiple constructions, as well as with the decisions about privileging one perspective over another” (Mertens, 1999). “We are led to ask questions such as, ‘Whose reality is privileged in this context?’ ‘What is the mechanism for challenging perceived realities that sustain an oppressive system?’ ‘What are the consequences in terms of who is hurt if we accept multiple versions of reality or if we accept the ‘wrong/privileged’ version?’” (Mertens, 2010). |
 Accessible and actionable evaluation findings |
The dissemination of evaluation results should be tailored to stakeholder needs and assessment of risks and benefits (Lee, 2007). Deliverables of culturally competent evaluations target multiple audiences and are creative in presentation style, product, language, tone, and graphics (CDC, 2014; Endo Inouye et al., 2005). | Results from culturally responsive evaluations should be widely shared and presented clearly for all of the intended audiences (Frierson et al., 2002). This may require multiple, audience-specific communication formats (Hood et al., 2015). | Culturally responsive and equitable evaluation (CREE) is a compilation of culturally responsive and equity-focused evaluation approaches. Analysis and dissemination would follow those of culturally responsive evaluation, culturally responsive Indigenous evaluation, and other participatory approaches that lead to both accessible and actionable findings (Mendez & Taniuchi, 2020; Anderson et al., 2020). | A culturally responsive Indigenous evaluation (CRIE) is designed to be useful within the community and community context, and be reflective of the communities, stories, traditions, and culture. Results and use are designed to underscore the political and legal context of Indigenous communities (Bowman et al., 2015). Thus, dissemination of results and findings are reflective of the culture and may include traditions unique to Indigenous populations. Using community and experiential knowledge to document evidence-based practices that guide decision-making and a future sustainable vision are considered key features of CRIE (Waapalaneexkweew, 2018). Key questions to consider in the evaluation of findings are: How can capacity among community members be built? What did community members learn? What did they do with what they learned? (Chouinard & Cousins, 2007) | Empowerment evaluation has three steps to guide communities through their own self-assessment process and an evaluation dashboard to help them monitor their own progress toward desired goals: (1) establishing the mission in which groups come to agreement about their values and visions; (2) taking stock, in which the group evaluates its vision and mission; and (3) planning for the future in which the group plans for future activities, monitoring, and what will be considered as credible evidence (Fetterman, 2012). | Critical to equitable evaluation and the practice of the Equitable Evaluation Framework™ (EEF) in both the international and domestic contexts is the use of evaluation findings for policy and systems change. Bamberger and Segone (2011) note that, “knowledge generated through an equity-focused evaluation provides evidence to influence major policy decisions to ensure that existing and future policies will enhance equity and improve the well-being of worst-off groups.” Indeed, policy development is often in service of trying to provide for communities at large, thus, “evaluation findings or the process itself should be used to move policy and practice that can either promote or inhibit equity” (LaFrance & Nichols, 2008 as cited in Dean-Coffey et al., 2014). The Equitable Evaluation Framework™ “requires the ability to influence conversations that shift culture and practice, and impact resource allocation” (EEI & GEO, 2021). | The transformative paradigm means an “explicit connection is made between the process and outcomes of research and furtherance of a social justice agenda” (axiology assumption) (Mertens, 2007). Ensuring that evaluation can and will further a social justice agenda requires consideration of the products that will be developed and how they will be utilized. “The challenge is to approach the evaluation in a way that builds in a greater likelihood of utilization. However, the challenge goes further, and asks, ‘What is the evaluator’s role in facilitating utilization in a way that holds greater promise that the social ills that formed the basis for the need for the program can actually be ameliorated?’ ‘How can we insure or increase the probability that the bridge between evaluation findings and social transformation can be crossed?’ Transformation can only occur if this information is used to inform policies that effectively address the inequities that create the need for social programs” (Mertens, 1999). |
© 2022 SLP4i and The Colorado Trust, authored by Katrina Bledsoe, Felisa Gonzales, and Blanca Guillen-Woods. This work is protected by copyright laws. No permission is required for its non-commercial use, provided that the authors are credited and cited.
For full citation use: Bledsoe, K., Gonzales, F., & Guillen-Woods, B*. (2022). The Eval Matrix©. Strategy Learning Partners for Innovation https://slp4i.com/the-eval-matrix.
*These authors contributed equally to this work with support from the Annie E. Casey Foundation and The Colorado Trust.
The Eval Matrix site designed by KarBel Multimedia