Investing in What Works

The Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation (HHS/ASPE) has contracted with the American Institutes for Research (AIR) to conduct the Investing in What Works (IWW) project. The goal of this project is to continue ASPE’s efforts to build knowledge, tools, and supports that evidence-based interventions (EBIs) and initiatives can use to improve the quality and outcomes of interventions funded through federal investments. Through developing various products and hosting several events, we have tried to deepen knowledge regarding the effective implementation and evaluation of federally funded prevention and intervention programs.

Issue Briefs

Three issue briefs are available to support the development, implementation, and evaluation of future initiatives and federal investments:

Infographics

Three interactive infographics were developed to complement the issue briefs:

Willing, Able -> Ready: Basics and Policy Implications of Readiness As A Key Component for Implementation of Evidence-Based Interventions

The Importance of Contextual Fit When Implementing Evidence-Based Interventions

Using Evidence-Based Constructs to Assess Extent Of Implementation Of Evidence-Based Interventions

Videos

Videos of the presentations are available for those who were unable to attend. Please note that the contents of these videos do not necessarily represent the policy or views of the U.S. Department of Health and Human Services.

Investing in What Works Project Background

What is Implementation Science?

Implementation science is the study of methods to promote the integration of research findings and evidence into policy and practices to improve health.¹ Implementation science helps us answer questions such as Why do some research based practices easily transfer from one place to another?, What are the supports necessary to promote successful adoption, implementation, and scaling up?, and How can we combine multiple interventions effectively to be more cost efficient and less duplicative? These questions and others are important for various audiences when grappling with how to best serve and improve outcomes for children, youth, and their families.

Who Can Benefit from Resources, Including Briefs Developed During the IWW Project?

Several audiences can benefit from the issue briefs, including federal and state program administrators and staff, federal and state policy makers, human services practitioners, researchers and evaluators—essentially any individual, organization, or community working to successfully implement and scale up EBIs, or evaluate these efforts. Examples of topics reviewed include:

The importance of implementation quality, monitoring if a program is being carried out according to plan, and how implementing EBIs with quality can lead to stronger outcomes;
The challenges to implementing EBIs in new contexts and with existing personnel, particularly when staff, agencies, and even consumers are also implementing other programs;
How to determine if an organization is ready to implement a particular EBI;
The importance of understanding context and contextual fit to improve the selection, implementation, scaling, and sustainability of EBIs;
How to measure the implementation process and milestones with attention to the needs and concerns of practitioners; and
How policy makers can include and require indicators/measures of implementation quality over the life of a project.

What do the IWW Issue Briefs tell us about Implementing EBIs?

Willing, Able -> Ready: Basics and Policy Implications of Readiness As A Key Component for Implementation of Evidence-Based Interventions (PDF, 17 pages)
Authors: Allison Dymnicki, American Institutes for Research, Abe Wandersman, University of South Carolina, David Osher, American Institutes for Research, Violanda Grigorescu, Centers for Disease Control and Prevention, and Larke Huang, Substance Abuse and Mental Health Services Administration
As more funders require grantees to use EBIs and practices, it is critical to know whether an organization is ready—willing and able—to implement a particular EBI or practice. Drivers of readiness include motivation of staff to adopt new EBIs/practices; general organizational capacity; and intervention specific capacities. This brief establishes key components of readiness and describes what readiness looks like during different phases of implementation.
The Importance of Contextual Fit When Implementing Evidence-Based Interventions (PDF, 16 pages)
Authors: Gene Hall, University of Nevada Las Vegas, Allison Dymnicki, American Institutes for Research, Jennifer Coffey, Office of Special Education Programs, and Melissa Brodowski, Children’s Bureau
An intervention involves three major phases of change—creating, implementing, and sustaining. Understanding what happens during the implementation phase of an EBI requires monitoring progress and establishing milestones. Several constructs can be used to monitor progress as staff take on new obligations and master new practices, including the quality and fidelity of implementation, competence in use, feelings and perception, organizational and community context, and support for the intervention. This brief introduces several constructs that can be used to describe, monitor, and facilitate implementation.
Using Evidence-Based Constructs to Assess Extent Of Implementation Of Evidence-Based Interventions (PDF, 18 pages)
Authors: Robert H. Horner, University of Oregon, Caryn Blitz, Administration on Children, Youth & Families, and Scott W. Ross, Utah State University
Contextual fit—the match between an intervention and the local context in which it is being implemented—affects the quality of implementation and whether or not the intervention produces the desired outcomes. This is important to consider in selecting, implementing, adapting, and scaling up EBIs to allow for more effective monitoring and facilitating. Eight key elements to consider are described, including need, precision, an evidence-base, efficiency, skills/competencies, cultural relevance, resources, and administrative and organizational support.

The matters addressed by these briefs are both distinct and connected. The issues of readiness, contextual fit, and monitoring implementation exist at all stages of implementation, and they will vary based on the type of intervention and the context in which it is being implemented.

What are the Key Take Home Messages from All Three Briefs?

Organizations should assess aspects of readiness before and during the implementation of an EBI;
Technical assistance should focus on building readiness and strong contextual fit before investing in direct implementations efforts;
Policymakers should consider including questions about readiness, selecting appropriate EBIs based on contextual fit criteria, and monitoring implementation in funding opportunity announcements and decisions;
Policymakers should support the use of implementation milestones as a key monitoring strategy across the life cycle of a program; and
Practitioners should assess implementation progress (including increased confidence and mastery to implement the EBI, and deeper engagement with the target recipient of the EBI) with a variety of indicators from different perspectives (e.g., leadership, frontline staff).

What’s Next in Improving the Implementation Quality of EBIs?

As more and more funders recommend the use of EBIs, a better understanding of how to implement and scale up EBIs with quality becomes critical. The IWW project is an important first step in paying attention to the importance of implementation quality when adapting and scaling up EBIs and underscores the role of federal staff, policymakers, program administrators, and youth serving organizations. Important next steps in advancing this work include: (1) identifying the biggest facilitating factors for readiness, implementation milestones, and contextual fit; (2) identifying the training and technical assistance needs relevant to grantees in different topic areas; and (3) providing suggested guidance to funders on requirements/criterion for programs to meet before and during a project period.

¹ Fogarty International Center, National Institutes of Health. (n.d.) Implementation Science: Information and resources. Retrieved from http://www.fic.nih.gov/researchtopics/pages/implementationscience.aspx

Breadcrumb