Implications of COVID-19 on the Research and Evaluation Provisions of the Family First Act

Background

The prevention provisions of the Family First Act are optional: States that choose to opt in must file (and have approved) Title IV-E prevention program five-year plans that outline their proposed work.[1] To qualify for Title IV-E funds, prevention programs must meet certain evidentiary criteria and be evaluated and rated by the Title IV-E Prevention Services Clearinghouse (Clearinghouse) as promising, supported, or well-supported.[2] The Family First Act requires that a certain proportion of state expenditures be used for programs that receive the highest rating from the Clearinghouse, while the subsequent Family First Transition Act phases in these requirements. As of December 2020, eight states and the District of Columbia have approved plans and six states and two tribal IV-E agencies have submitted plans to the Children’s Bureau for review and approval.[3] Kinship navigator programs, which connect kin who are caring for children to services and supports, can also qualify for Title IV-E funds if the Clearinghouse rates the program as promising, supported, or well-supported. The Clearinghouse is required to review “culturally specific, or location- or population-based adaptations” to ensure that programs designed to address the needs of communities that meet the Clearinghouse standards qualify for Title IV-E funds.

In March 2020, most areas of the United States began to practice social distancing measures to reduce the spread of COVID-19. As in other public sectors, child welfare services drastically changed to protect children and families, as well as agency staff. Every facet of the child welfare system was impacted; as case workers shifted to remote service delivery, parent-child visitation schedules were disrupted, reunification services experienced disruptions, and court timelines were delayed. Although it is unclear as to whether, or how, rates of maltreatment may have changed, the closures of schools and daycares also led to a deep reduction in child welfare hotline reporting.

Implementation of the Family First Act, including aspects of its evidence-based provisions, has similarly been impacted by the pandemic and resulting recession. In addition to the significant scope of changes to services, programs, and budgets required during the pandemic, state and local budgets are in crisis, resulting in limited capacity and reduced resources for agency leaders and staff to focus on a new initiative. From a practical perspective, implementing a major initiative requires a great deal of time, energy, and funding from multiple stakeholders. Implementing new prevention programs will likely cause a huge strain on child welfare systems already in crisis due to the pandemic.

While the pandemic has unfolded, child welfare leaders and stakeholders have reexamined existing inequities for families and begun to strategize new ways to support all families and children, with a particular focus on those who have been marginalized by the child welfare system. There are persistent racial and ethnic disparities and disproportionality in the child welfare system. Nationally, Black and American Indian/Alaska Native children are more likely to be removed from their families than white, Asian, and Hispanic children; once in foster care, children of color experience poorer outcomes than their white peers. These same families are also facing disproportionate health and economic obstacles due to the impact of COVID-19 and the resulting recession. As a component of policy and programmatic responses to these challenges, evidence-based prevention and kinship navigator programs can serve as key resources for creating a more equitable child welfare system.

Although we do not yet know the full extent of the impact of the pandemic, or know how child welfare systems may adapt to be more racially equitable, we describe implications for programs along the evidence-building continuum below.

The following key evaluation terms will aid readers’ understanding of this brief’s implications for program evaluations, as well as our recommendations to evaluators, program leaders, and policymakers.

Implications for Evidence-based Programs that Have Been Rated by the Clearinghouse

As of December 2020, 48 programs have been reviewed by the Clearinghouse. Of those, 20 did not currently meet criteria for having evidence of effectiveness, one was not eligible for review, and the remaining 27 were reviewed and received ratings of promising (13 programs), supported (5 programs), or well-supported (9 programs). Two additional programs have been approved for transitional payments through independent systematic reviews until the Clearinghouse completes its review. As the Clearinghouse continues its reviews, it is likely that more programs will be rated as promising or higher, given existing research evidence. However, as states continue to respond to COVID-19, implementers of programs currently meeting evidence requirements (i.e., rated as promising, supported, or well-supported) may be making:

• Changes in service delivery and content. Program purveyors and implementers are changing the ways in which they serve children and families to keep children, families, and staff safe during the pandemic. The Children’s Bureau issued guidance in November 2020 to allow states and tribes with approved prevention plans to claim federal funding for prevention services adapted for remote service delivery during the pandemic, when these services would otherwise not be available to families due to pandemic-related obstacles such as stay-at-home orders or office closures. Such adjustments to delivery formats are also promoted in joint guidance from the Administration for Children and Families and the Health Resources and Services Administration; this guidance describes the ways programs are using remote service delivery, such as video conferencing, to continue serving families. Program purveyors have responded quickly to support families and staff. For example, Family Centered Treatment, a home-based family therapy program, released a variety of resources to their providers on engaging families and completing certain program activities (e.g., structured family assessment) via telehealth. While these changes are necessary and important, they do change the way services are delivered and have the potential to change their effectiveness. The changes may also raise unique equity challenges when children or families lack ready access to the technology or high-speed internet needed to fully participate in services. Although some programs may rely on evidence of the effectiveness of telehealth in medicine and behavioral health, many prevention programs are being delivered remotely for the first time, and some changes have not yet been tested. Beyond service delivery, program purveyors and implementers may find that certain program components cannot be delivered effectively remotely, leading to adaptations to program content.
• Changes that may interfere with fidelity. Changes to service delivery and/or content may have fidelity implications, and the Family First Act requires documented processes for implementing programs with fidelity. Several program purveyors (for example, in the home visiting field) have released guidance that allows services to be delivered remotely during an emergency while still maintaining program fidelity. Federal programs such as the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program have accepted these changes to modality. Additionally, changes to the core components of a program may impact fidelity—and, in turn, impact anticipated child or family outcomes.

Implications for Programs Planning or Currently Undergoing Evaluation

The Family First Act promotes evaluation by tying Title IV-E funds to evidence-based programs reviewed by the Clearinghouse and by requiring ongoing evaluation of programs that are being implemented under approved state prevention plans.[4] As a result, prevention programs and kinship navigator programs across the country are in various stages of evaluation (including planning stages), as required by their state’s prevention plans or with the goal of receiving or improving a Clearinghouse rating. The pandemic and resulting recession are likely to affect existing and future program evaluations in myriad ways, including the following:

• Evaluation findings may be influenced by changes in families’ needs. Contextual factors related to COVID may influence families’ outcomes in ongoing studies. For example, if a study is measuring whether a program reduces parenting stress, the introduction of a range of other stressors (health, economic, social, etc.) into a family’s life during the pandemic may influence outcomes. These challenges are particularly salient for Black, Hispanic, and American Indian/Alaskan Native families, who are experiencing greater impacts from COVID-19—both in health outcomes and economic stability. Research also points to the interconnections between disasters and child, parent, and family stress responses. Therefore, comparing parenting stress levels pre- and post-COVID may cloud our understanding of whether the program achieves its intended goal.
• Changes to programs may impact evaluation. As the response to COVID-19 creates programmatic shifts, evaluators need to consider whether and how the programs being examined have changed. For example, if an evaluation sample includes families served both pre- and post-COVID and the program employed different service delivery methods and/or content pre- and post-COVID, implementers and evaluators should jointly determine what changes to their evaluation plans may be necessary. While the Children’s Bureau guidance allows remote service delivery when necessitated by the pandemic, it remains to be seen whether the Clearinghouse will consider evaluations of programs currently being delivered remotely as contributing to the evidence base for programs originally designed for in-person delivery. The Children’s Bureau’s guidance indicates that programs operated remotely—when not in accordance with the manual and documentation reviewed and rated by the Clearinghouse—may be considered formal adaptations. Under the current Clearinghouse Handbook of Standards and Procedures, such a designation would require a separate evidence review. It is not yet known how many programs might fall into this category or how states will respond. As described above, many program purveyors are striving to develop guidance on how to implement their programs remotely; the Children’s Bureau requires states and tribes to notify their regional office of any virtual adaptations and encourages them to coordinate these adaptations with the program purveyor.
• Practical considerations shift evaluation plans. Many planned evaluation activities have slowed or stopped due to COVID-19. Programs may be receiving fewer referrals for their services, which lengthens the time needed to build sufficient samples for different evaluation designs. Traditional evaluation activities that rely on in-person data collection—such as focus groups—have been delayed due to changes in service delivery and/or health concerns. Researchers using administrative data collected by public agencies need to consider any delays, as well as the larger context for the data. For example, evaluators examining whether participation in a program shortens the time it takes for a child to achieve permanency will need to consider whether, and how, temporary court closures or reduced court hearings brought about by the pandemic have affected time to permanency for children in that jurisdiction. Additionally, as state leaders struggle with limited budgets, evaluations may be delayed in order to redirect funding to direct services. For programs that do not meet Clearinghouse criteria (including all kinship navigator programs reviewed to date), evaluation delays mean it will take more time to build the evidence base and access Title IV-E funds under the Family First Act.
• New circumstances may open the door to new evaluation opportunities. The sudden shift to remote service delivery for large populations of children and families presents a unique opportunity to better understand whether service delivery mode (i.e., remote or in-person) matters—and if so, for whom and under what conditions it will matter. Implementers and evaluators can assess parent and child preferences, the technological needs of families, and whether service uptake improves when services are delivered remotely. In addition, evaluators can utilize designs that leverage existing data (e.g., through an interrupted time series evaluation design) to explore the effectiveness of programs now offered through remote service delivery.

Recommendations

As they plan their responses to COVID-19, state and federal leaders, program purveyors, program implementers, and evaluators must be aware of the aforementioned implications related to the Family First Act’s evaluation provisions. These implications also play a role in how states develop prevention programs that support the families disproportionately impacted by COVID-19 and leverage their prevention work to address racial disparities and inequities within the child welfare system. Although there are challenges, the prevention provisions of the Family First Act provide a new funding opportunity for states to provide critical services to children and families by keeping them together and reducing the trauma associated with foster care entry. We urge evaluators, program leaders, and policymakers to do the following:

• Monitor for program and federal guidance. Given their responsibility for ensuring fidelity to their program models, program purveyors are key partners in Family First Act implementation. Whenever possible, consult program purveyors about their plans and seek their guidance on the impact of program adaptations on fidelity, as recommended by the Children’s Bureau. Similarly, monitor the Clearinghouse and federal agencies for updated guidance and potential changes to Family First Act implementation timelines.
• Invest in documenting what is happening now. As program purveyors and implementers innovate and adapt to changing circumstances, it is important that they understand the timing and substance of programmatic changes. Process evaluations can help the child welfare field understand how programs are being adapted and implemented during the pandemic. Subsequent outcome evaluations can explore whether (and how) the adapted program helps children and families (i.e., improves outcomes). Program implementers may choose to continue delivering services remotely in the future (e.g., to reach families with transportation barriers and respond to families’ needs and preferences). By evaluating these programs now and sharing findings with practitioners and policymakers, the field can better understand the effectiveness of this approach.
• Critically examine and build the evidence for programs that are effective for children and families of color. As evaluators document how programs are affected by the pandemic, it will be important to assess whether these programs are effective for families disproportionately affected by the pandemic and by existing inequities. Ask critical questions to understand whether existing evidence speaks to the effectiveness of the program for children and families of color. Consider, for example, the racial and ethnic background of children and families in prior evaluations, whether the evaluations used culturally appropriate measures, whether the study provided contextual/historical reasons that may explain differences, and whether the study tested differential effectiveness for families based on their races and ethnicities. Child welfare leaders and policymakers should also prioritize the evaluation of culturally relevant prevention programs for the children and families who live in their jurisdictions. For example, it is important to continue building the evidence for culturally based prevention services for American Indian/Alaska Native families to fully understand whether services are effective in supporting these unique populations.
• Collaborate across agencies. Agencies are collaborating in new ways as they strive to keep children, families, and communities safe during the pandemic. There may be opportunities to leverage and sustain these strengthened relationships to implement prevention services. At the same time, collaborative partnerships may face increased stress due to shrinking budgets and increased competition for flexible resources.
• Partner closely with evaluators to understand the specific implications for each unique circumstance. This brief has outlined potential implications of COVID-19 on the evaluation provisions of the Family First Act. However, each program and evaluation is unique. State and community leaders, as well as program implementers, should work closely with their evaluation partners to explore ways in which the pandemic and recession may affect their ongoing work.

Conclusion

Prevention services, including those included in the Family First Act, can strengthen families’ protective factors to address those stressors which have been introduced or exacerbated by the pandemic. In addition to responding to families’ immediate needs, prevention services may result in future fiscal benefits by reducing the long-term costs associated with children’s entry into foster care. Implementing the prevention provisions of the Family First Act also presents an opportunity to promote race equity and address the disproportionate representation and disparate outcomes of families of color in the child welfare system.

Acknowledgments

We are thankful for our colleagues at Child Trends and in the child welfare field who provided insights and ideas during the development of this brief. We are particularly grateful for the thoughtful review and feedback from Jan DeCoursey, Esther Gross, Alyssa Liehr, Karin Malm, and Lauren Supplee at Child Trends; Meghann Dygert at APHSA; Nina Williams-Mbengue and Donna Wilson at NCSL; and Marla McDaniel at The Urban Institute.

Footnotes

[1] Other important provisions of the Family First Act are not discussed in the brief. For example, the Title IV-E prevention funds are linked to Title IV-E funding for certain congregate care settings for children in foster care. This provision both incentivizes and supports state investment in prevention while encouraging placement of children in family settings (foster and kinship), rather than in congregate care settings. All states are required to comply with the new funding limitations on congregate care by October 1, 2021, including those states that do not adopt a prevention plan. For a detailed summary of the law and updates on federal guidance and state implementation, see FamilyFirstAct.org.

[2] For more information on the evaluation requirements and evidence ratings, see Applying the Research and Evaluation Provisions of the Family First Prevention Services Act.

[3] Most recent data from the Children’s Bureau: https://www.acf.hhs.gov/cb/resource/title-iv-e-five-year-plan. Approved states: Arkansas; Washington, DC; Kansas; Kentucky; Maryland; North Dakota; Utah; Washington; West Virginia. Submitted states and tribes: Alaska; Aleut Community of St. Paul Island, Alaska; Colorado; Eastern Band of Cherokee Indians, Cherokee, North Carolina; Illinois; Iowa; Nebraska; Virginia.

[4] States may request a waiver of this requirement for well-supported programs.