Applying the Research and Evaluation Provisions of the Family First Prevention Services Act

Research BriefChild WelfareOct 24 2019

The federal Family First Prevention Services Act of 2018 (Family First Act) has changed the landscape of federal child welfare funding in significant ways to support children to remain safely in their homes and encourage family-based placement when foster care placement is necessary. With specific requirements around evidence-based programs and tools, the legislation expands opportunities to use research and data to drive decision making and direct funding to ensure that children and families receive the most effective services. This brief outlines these research and evaluation requirements and highlights next steps for state agencies, legislators, and researchers to achieve the Family First Act’s goals.

What is the Family First Act?

Passed in 2018, this landmark federal legislation advances key priorities, including:

  • Funding services for families to prevent foster care entry
  • Supporting kinship caregivers
  • Establishing criteria for appropriate use of residential treatment
  • Strengthening services for older youth

For more detail on these and other provisions of the Family First Act, see Children’s Defense Fund (2018).

Background

Title IV-E of the Social Security Act—the largest source of federal child welfare funding—previously reimbursed states for a portion of their eligible expenditures on child welfare services and activities. With the passage of the Family First Act, states have the option to use Title IV-E funds for services to prevent children from entering foster care. In the past, Title IV-E funding could only be used for prevention services through Child Welfare Waiver Demonstration projects approved by the U.S. Department of Health and Human Services. Now, under the Family First Act, children at imminent risk of entering foster care (“candidates for foster care,” as defined by each state), their parents or kin caregivers, and youth in foster care who are pregnant or parenting are eligible to receive time-limited, evidence-based prevention services to prevent foster care entry. By keeping children safely at home and preventing subsequent maltreatment, these services have the potential to greatly benefit children, families, and society.

When children must be removed from their homes, the Family First Act prioritizes the use of family-based settings, particularly with relatives. Evidence-based kinship navigator services, designed to connect relatives caring for children to information and supportive services, are funded through Title IV-E under the Family First Act. Further, the legislation establishes funding limitations on non-family placements. Reimbursement for the out-of-home placement needs of children will be limited to family-based settings, qualified residential treatment programs when indicated by a child assessment, and special treatment settings (e.g., for pregnant and parenting youth).

This brief represents our understanding of the research and evaluation provisions of the Family First Act, as of October 2019. We will continue to monitor the federal landscape for administrative guidance and/or legislative changes.

See the glossary for definitions of key terms.


Research and Evaluation Requirements in the Family First Act

The research and evaluation provisions of the Family First Act require that (1) prevention and kinship programs be evidence-based, (2) states rigorously evaluate and support implementation of prevention programs through data-driven continuous quality improvement efforts, and (3) assessment tools used to determine the appropriateness of residential treatment programs be evidence-based. This focus on evidence ensures that children and families receive services with demonstrated effectiveness, and is part of the larger movement toward evidence-based social policy (Supplee & Metz, 2015). The research and evaluation requirements of the Family First Act are summarized below for prevention services, kinship navigator services, and qualified residential treatment programs.

Prevention services

Keeping families together with services to support their safety, permanency, and well-being is a primary goal of the optional prevention services program of the Family First Act. Children who have been maltreated have a higher risk of experiencing a range of negative well-being outcomes, with significant direct and indirect societal costs,[i]  and children who are removed from their home may experience additional trauma that can compound the effects of maltreatment (Kolko et al., 2010). Trauma occurs when children experience or perceive one or more negative events or circumstances that cause them emotional pain and impair their ability to cope (Bartlett & Steber, 2019). Expanding federal Title IV-E reimbursement to include prevention services is therefore a critical step toward supporting child and family well-being.[1]

States choosing to implement the prevention program of the Family First Act may be reimbursed for these services:

  • Mental health prevention and treatment services
  • Substance abuse prevention and treatment services
  • In-home parent skill-based programs that incorporate parenting skills training, parent education, and/or individual/family counseling

Prevention services must be documented in a written prevention plan for each child. The Family First Act requires that organizations delivering prevention services use a trauma-informed approach that understands, recognizes, and responds to trauma within a treatment framework that includes intervention(s) designed to address trauma (U.S. Department of Health and Human Services [U.S. DHHS], 2018a).

Levels of evidence

Under the Family First Act, states can receive federal reimbursement for specific prevention programs that meet the criteria for promising, supported, or well-supported practice. At least 50 percent of state expenditures must be on well-supported programs, and the remainder can be spent on promising and/or supported programs.

The federally funded Title IV-E Prevention Services Clearinghouse (Prevention Services Clearinghouse) continuously and systematically reviews the available literature to assign evidence ratings to programs (see Figure 1 below) or to determine that programs do not currently meet the criteria. Under the Family First Act, programs that do not meet the criteria are not eligible for federal reimbursement. The Prevention Services Clearinghouse released its first set of program ratings in June 2019.

The Family First Act requires that studies be rated through an independent systematic review and determined to be well-designed and well-executed to contribute to the evidence base. When determining ratings, the Prevention Services Clearinghouse considers evidence from moderate- and high-quality comparison studies (U.S. DHHS, 2018b).[2] These are limited to randomized controlled trials (RCTs) and quasi-experimental designs (QEDs) that meet standards outlined in the Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures (Wilson et al., 2019).[3] In an RCT design, each unit has the same chance of being assigned to the intervention or control group, thereby creating equivalent groups that allow evaluators to determine cause and effect. Units of randomization may be children or families, or more expansive groupings such as offices or counties.[ii] QED studies rely on non-random assignment strategies to compare outcomes. For example, if rollout of a program’s implementation is staggered across multiple counties, the evaluator may compare outcomes for families in an early implementing county with outcomes for families in a similar county that has not yet implemented the program. Researchers may also use statistical methods to match children who participated in a program with similar children who did not participate in the program to compare outcomes.[iii]

The Prevention Services Clearinghouse casts a wide net in both the study population and outcomes considered in its review. Prevention programs are not required to be evaluated with a child welfare population; this means that the Clearinghouse’s review includes studies evaluating programs delivered through other child-, family-, and adult-serving systems (e.g., behavioral health or juvenile justice), regardless of whether the participants had current or past child welfare involvement (U.S. DHHS, 2018b).

Studies must examine one or more relevant target outcomes related to safety, permanency, and/or well-being. Safety and permanency encompass child welfare-specific outcomes such as substantiated maltreatment, placement stability, and reunification. Well-being outcomes may be for children (e.g., measures of behavioral, social, cognitive, and/or physical functioning) and adults (e.g., parenting, mental health, substance use, physical health, and/or economic/housing stability; Wilson et al., 2019).


Figure 1. Tiered Levels of Evidence for Prevention Services[4]

Download

Source: Wilson et al. (2019)


In recent decades, several clearinghouses have been established as repositories of evidence-based programs, including clearinghouses that focus primarily on interventions of interest to child welfare systems.[5] Although the Prevention Services Clearinghouse’s method of rating programs is informed by the methods that these other clearinghouses use, criteria and evidence ratings may differ across clearinghouses. For this reason, programs that are typically referred to as “evidence-based” may or may not satisfy all requirements of the Prevention Services Clearinghouse.

Programs with multiple variations or formal adaptations will be reviewed as separate programs by the Prevention Services Clearinghouse. For example, Multisystemic Therapy (MST) and Multisystemic Therapy – Child Abuse and Neglect (MST-CAN) were assigned separate evidence ratings as their manuals and model components differ. Program adaptations such as adding, subtracting, or significantly changing content; changing the format of service delivery; significantly changing content for cultural relevancy; and implementing the program with substantially different types of staff would all be considered new programs that require separate review by the Prevention Services Clearinghouse.

As the Prevention Services Clearinghouse continues to review interventions, the federal Children’s Bureau has released guidance that allows states to submit documentation of an independent systematic review concluding that a program not yet rated by the Prevention Services Clearinghouse qualifies as promising, supported, or well-supported. With approved documentation of an independent systematic review and all other requirements met, states can receive reimbursement (called “transitional payments”) for these services until the Prevention Services Clearinghouse completes its review of that program. However, the final evidence rating of these services will ultimately be determined by the Prevention Services Clearinghouse (U.S. DHHS, 2019a).

Tribal Title IV-E agencies are not required to adhere to the criteria for promising, supported, and well-supported prevention programs and must instead determine their own practice criteria for identifying prevention services that are relevant to their culture and context. These may include criteria such as longevity of the practice and community leader/elder approval (U.S. DHHS, 2018c).

Continuous quality improvement and evaluation requirements

For each promising, supported, or well-supported prevention program for which states are seeking federal reimbursement, the Family First Act requires states to outline (1) how they will implement the programs with fidelity using a continuous quality improvement (CQI) framework, and (2) how they will evaluate each program using a well-designed and rigorous process.

By implementing programs with fidelity, states will ensure that all key program elements associated with positive outcomes for children and families are maintained. Many program developers have created tools such as fidelity checklists to support supervisors and program administrators to regularly monitor these core elements. Used in combination with other program data, these measures feed into a cycle of continuous quality improvement through which states can strengthen prevention service delivery.

Beyond the fidelity requirements, states must develop and submit an evaluation plan for each prevention service. This requirement may be waived for well-supported interventions with documentation that the program has compelling evidence and the state will meet the CQI requirements. While the law does not define requirements for well-designed and rigorous evaluation strategies, states should consider the study quality standards established by the Prevention Services Clearinghouse to ensure that their evaluation activities will contribute to the evidence base. The Children’s Bureau has released a “tip sheet” for preparing evaluation plans. Meant to serve as a resource for states and their partners, the tip sheet outlines key components of evaluation plans and related reference materials (U.S. DHHS, 2019b). States can evaluate their prevention services or participate in a cross-site evaluation, if approved by the Administration for Children and Families (ACF).

Tribal Title IV-E agencies are afforded flexibility in their evaluation designs, with the recognition that some rigorous evaluation designs may not be feasible in the tribal context. Alternative evaluation strategies may include, for example, exploratory studies, qualitative studies, and community-based participatory research (U.S. DHHS, 2018c).

States must collect and report child-level data to the federal government. States are required to report the services that were provided to each child/family and the cost and duration of services, the placement status of each child 12 months after the prevention plan start date, whether the child entered foster care within two years of the prevention plan start date, whether the child is pregnant or parenting, and certain demographic characteristics. Some data elements are not required for pregnant or parenting youth (U.S. DHHS, 2019c).

The U.S. Department of Health and Human Services (HHS) may conduct evaluations and gather data to assess whether programs reduce foster care placements, increase kinship care, and/or improve child well-being. HHS may directly carry out this provision or enter into an agreement with external partner(s).

Kinship navigator services

When children cannot remain safely in their homes, placement with kin provides an opportunity to keep children connected to their families and culture. Kinship navigator programs support kin caregivers to learn about, find, and use programs and services for the children in their care and themselves. Kinship navigator programs use a variety of approaches to connect kin caregivers to public benefits and community resources (e.g., designated staff, peer-to-peer support, help lines). Kinship navigators also serve an important role in educating child welfare agency and community partners about the needs of kin caregivers (Casey Family Programs, 2018).

Levels of evidence

Under the Family First Act, states can receive federal reimbursement for up to 50 percent of their expenditures on kinship navigator programs that meet the criteria for promising, supported, or well-supported practice. The evidence requirements for promising, supported, and well-supported are the same as those for prevention services, with the exception of the target outcomes. Kinship navigator programs are reviewed for their effects on child safety, child permanency, child well-being, adult well-being, access to services, referral to services, and/or satisfaction with services. Beyond the evidence requirements, several other practice requirements must be met for reimbursement. For example, kinship navigator programs must coordinate with other state/local agencies; consult with kin, youth, and community stakeholders in designing and implementing programs; and establish information and referral systems, among other requirements (U.S. DHHS, 2018d). The Prevention Services Clearinghouse reviewed two kinship navigator programs in its first round of review, and neither program met the criteria.

Qualified residential treatment placements

Complementary to its focus on supporting families through prevention and kinship navigator services, the Family First Act directs federal funding to encourage agencies to place children who must be removed from their homes in families, not group settings, when appropriate. Exceptions are made for children assessed to have needs that necessitate treatment in a qualified residential treatment program, or children and youth who qualify for specified settings. This policy shift is consistent with the research evidence that children in family settings fare better across a variety of outcomes, compared to children in group settings (Annie E. Casey Foundation, 2015; Casey Family Programs, 2017). The Family First Act defines qualified residential treatment programs (QRTPs) as programs that are designed to address the needs of children with serious emotional or behavioral disorders or disturbances, using a trauma-informed treatment model. Specific requirements related to assessments, program staff, family engagement, discharge planning, aftercare, and accreditation are outlined in the law.

Evidence-based assessment tools

The Family First Act specifies several requirements for assessing the appropriateness of every placement of a child in a QRTP; these include using a validated, evidence-based assessment tool to determine the appropriate placement setting. The assessment tool must be administered by a trained professional or licensed clinician who is not employed by the state agency or affiliated with a placement setting. While this provision does not require further research on the assessment tools being used, it is an important element of the Family First Act and reinforces the focus on effectiveness.


Challenges and Next Steps

The Family First Act’s research and evaluation requirements ensure that children and families receive the most effective programs and services to meet their individual needs. The emphasis on evaluation, CQI, and evidence-based tools spurs child welfare stakeholders to work together to continue building the evidence base and monitoring program implementation and outcomes to ensure effectiveness.

Anticipating challenges

As with any significant change in policy and practice, states are likely to face challenges in implementing provisions of the Family First Act. Challenges for research and evaluation may include the following:

  • Identifying evidence-based programs. The evidence-base for child welfare programs lags behind that of other disciplines. While programs studied on other populations may be eligible for federal reimbursement, it remains to be seen how many of them will eventually qualify as promising, supported, and well-supported. The systematic review of interventions by the Prevention Services Clearinghouse is time-intensive, with only a small number of programs meeting these criteria to date. States can seek transitional funding for programs based on their own assessment of the evidence; however, this will require independent application of complex research standards and does not guarantee continued funding if the Prevention Services Clearinghouse reaches a different rating in their subsequent review.
  • Ensuring that programs produce desired outcomes. The Prevention Services Clearinghouse considers evidence from populations that do not have child welfare involvement (e.g., adults receiving substance abuse treatment or children receiving mental health services through the behavioral health system) in its reviews. It is possible that programs that are effective for other populations will not produce the same desired outcomes for children at risk of foster care entry and their families. The evaluation and continuous quality improvement requirements of the Family First Act will be essential for ensuring that such programs are indeed effective for families involved with the child welfare system.
  • Identifying culturally relevant, evidence-based programs. The existing evidence for programs may come from study samples that do not match the characteristics, needs, or experiences of children and families served in a local jurisdiction. When used with different populations, services may not work the same way or achieve the same results. Additionally, programs may not be designed to serve specific subgroups of children and families, and refinements to better align these services may result in formal adaptations that require a separate review by the Prevention Services Clearinghouse. Formal adaptations will require new research that will take time to conduct and be reviewed.
  • Building the evidence base for emerging programs. By requiring that programs be rated as promising, supported, or well-supported for states to receive federal reimbursement, the Family First Act incentivizes states to implement programs that already have some evidence of effectiveness. It will therefore be important to identify other funding sources to implement and continue building the evidence base for new and emerging programs and services that have not yet met the evidence requirements.
  • Aligning programs with families’ needs. At the family level, states may select promising, supported, or well-supported programs that ultimately do not meet the needs of individual families. By understanding the needs and characteristics of the families they serve, states can work to build a continuum of services that provides responsive, tailored supports.
  • Mobilizing support and financial resources for rigorous evaluations. Child welfare stakeholders may resist RCTs due to concerns about withholding services from some children and families; moreover, rigorous evaluations can be cost-prohibitive. While the Family First Act does specify that HHS may directly engage in or partner with other entities through grants, contracts, or interagency agreements to conduct evaluations, funding for this provision comes from the same allocation as the funding for the Prevention Services Clearinghouse, which limits availability of resources.

Next steps

For state agencies and legislators implementing the prevention, kinship navigator, and QRTP provisions of the Family First Act, key next steps related to research and evaluation include:

  • Assess the current landscape of services and alignment with child and family needs. Mapping existing programs and their corresponding evidence base will help states identify existing prevention and kinship navigator programs eligible for reimbursement. Concurrently, states can work to identify gaps in the service array by analyzing the needs of children identified as “candidates for foster care” and their families and reviewing programs’ effectiveness in addressing these needs. State legislators can support these efforts by allocating resources and requesting information on the availability and evidence base of existing prevention and kinship navigator services, as well as data on child and family needs.
  • Partner with researchers to assess opportunities for scaling up effective programs and building the evidence base for emerging programs. Building off the landscape analysis, states may engage research and program partners to plan next steps for each program, including identifying opportunities and resources needed to scale existing evidence-based programs and strengthening the evidence base where needed.
  • Invest in new and emerging programs and corresponding rigorous evaluations. Given that federal funding can be directed to programs rated promising or higher, states should seek other funding sources to implement and evaluate new and emerging programs. Opportunities for state legislators include appropriating funds, convening stakeholders in the philanthropic and research communities to develop partnerships, requesting regular updates on program implementation and fidelity, and requesting to see the data states submit to HHS.
  • Explore inter-state partnership opportunities for program training, implementation, and evaluation. Multiple states seeking to implement the same program may achieve economies of scale by jointly training providers, developing shared fidelity measures, and participating in cross-site evaluation, if approved by HHS. Partnering in these ways increases the potential for larger, more diverse study samples as well as more rigorous evaluation designs, and states can share the costs of the evaluation. To better understand these opportunities, state legislators may participate in site visits to relevant programs in surrounding states.
  • Review the evidence base for assessment tools used to determine appropriateness of residential treatment. By identifying currently used assessment tools and reviewing their evidence base, states can determine whether existing tools meet the evidence requirements or need to be replaced or further studied.

For researchers partnering with states to build the evidence base, key next steps include:

  • Design evaluation plans that fit local contexts. The Prevention Service Clearinghouse weighs evidence from both RCT and QED studies, which gives states and their evaluation partners more flexibility in study design if an RCT is not feasible. There may also be natural opportunities for researchers to use RCT designs as states expand services. A random assignment process ensures that all children and families have the same chance of receiving the services when program demand exceeds capacity, which often occurs with incremental expansion efforts.
  • Align evaluation plans with the Prevention Services Clearinghouse standards. States are required to develop well-designed and rigorous evaluation plans for prevention services. By aligning these evaluation plans with the Prevention Services Clearinghouse standards, researchers can ensure that their findings will contribute to the evidence base.
  • Identify what works and why. Through the combination of fidelity monitoring and rigorous evaluation, the child welfare community and state legislators will ultimately gain a deeper understanding of whether programs achieve their intended goals, and the mechanisms through which this occurs. This understanding can help to strengthen programs and identify new opportunities for innovation to better meet the needs of all children and families.

Conclusion

The Family First Act presents a significant opportunity to embrace the power of research to better serve children and families. It directs resources to what we know works best for children and families by supporting parents and kin caregivers to safely care for children in their homes and prioritizing family-based placements when children must enter the foster care system. The new requirements around evidence-based programs and tools hold great potential for improving the lives of children and families; however, significant work remains to ensure that states have the support they need to identify and implement evidence-based programs and evaluate their efforts in ways that continue to expand the evidence base. By deepening our collective understanding of what works and why, and channeling resources to proven interventions, the Family First Act contributes to ensuring that children and families receive high-quality, effective supports.


Glossary

Comparison study: A study that compares a group receiving an intervention with a group that receives no intervention, minimal intervention, or services as usual. Wilson et al. (2019) define two types of comparison studies as follows:

  • Randomized controlled trial (RCT): “A study in which units are assigned to intervention and control conditions via a random process (e.g., a lottery)” (p. 19).
  • Quasi-experimental design (QED): “A study in which units are assigned to the intervention and control conditions via a non-random process” (p. 19).

Formal adaptations: “Alternative manualized versions of the original program designed to address particular issues or populations” (Wilson et al., 2019, p. 4). Changes to a program’s content or approach (e.g., changing from group to individual treatment) are examples of formal adaptations.

Target outcomes:

  • For prevention services: Child safety, child permanency, child well-being, and/or adult well-being.
  • For kinship navigator services: Child safety, child permanency, child well-being, adult well-being, access to services, referral to services, and/or satisfaction with services (Wilson et al., 2019)

Usual care setting: A non-research setting where an existing provider delivers prevention or kinship navigator services. (Wilson et al., 2019)

Fidelity: Adherence to the specified components of a program.

Continuous quality improvement (CQI): An ongoing process of using data to identify and understand strengths and challenges and to test, implement, analyze, and strengthen solutions (National Child Welfare Resource Center for Organizational Improvement & Casey Family Programs, 2005).

Cross-site evaluation: An evaluation of the same program across multiple locations.


References

Annie E. Casey Foundation (2015). Every kid needs a family: Giving children in the child welfare system the best chance for success. Retrieved from https://www.aecf.org/m/resourcedoc/aecf-EveryKidNeedsAFamily-2015.pdf.

Bartlett, J.D., & Steber, K. (2019). How to implement trauma-informed care to build resilience to childhood trauma. Retrieved from https://www.childtrends.org/publications/how-to-implement-trauma-informed-care-to-build-resilience-to-childhood-trauma.

Casey Family Programs (2017). What are the outcomes for youth placed in congregate care settings? Retrieved from: https://caseyfamilypro-wpengine.netdna-ssl.com/media/SF_CC-Outcomes-Resource.pdf.

Casey Family Programs (2018). What are kinship navigator programs? Retrieved from:  https://caseyfamilypro-wpengine.netdna-ssl.com/media/SF_Kinship-navigator-programs.pdf.

Children’s Defense Fund (2018). The Family First Prevention Services Act: Historic reforms to the child welfare system will improve outcomes for vulnerable children. Retrieved from https://www.childrensdefense.org/wp-content/uploads/2018/08/family-first-detailed-summary.pdf.

Child Welfare Information Gateway (2017). Child maltreatment prevention: Past, present, and future. Washington, DC: U.S. Department of Health and Human Services, Children’s Bureau. Retrieved from https://www.childwelfare.gov/pubPDFs/cm_prevention.pdf.

Child Welfare Information Gateway (2019). Long-term consequences of child abuse and neglect. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau. Retrieved from https://www.childwelfare.gov/pubpdfs/long_term_consequences.pdf.

Kolko, D.J., Hurlburt, M.S., Zhang, J., Barth, R.P., Leslie, L.K., & Burns, B.J. (2010). Posttraumatic stress symptoms in children and adolescents referred for child welfare investigation. Child Maltreatment, 15(1), 48-63.

Moore, K.A. (2008). Quasi-experimental evaluations. Retrieved from https://cms.childtrends.org/wp-content/uploads/2008/01/Child_Trends-2008_01_16_Evaluation6.pdf.

Moore, K.A. & Metz, A. (2008). Random assignment evaluation studies: A guide for out-of-school time program practitioners. Retrieved from: https://cms.childtrends.org/wp-content/uploads/2008/01/Random-Assigment-Evaluations.pdf.

National Child Welfare Resource Center for Organizational Improvement & Casey Family Programs (2005). Using Continuous Quality Improvement to Improve Child Welfare Practice. Retrieved from: http://muskie.usm.maine.edu/helpkids/rcpdfs/CQIFramework.pdf.

Supplee, L. H. & Metz, A. (2015). Opportunities and challenges in evidence-based social policy. Social Policy Report, 28(4), 1-16.

U.S. Department of Health and Human Services. (2018a). ACYF-CB-PI-18-09: State requirements for electing Title IV-E prevention and family services and programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1809.pdf

U.S. Department of Health and Human Services. (2018b). Attachment C to ACYF-CB-PI-18-09: HHS initial practice criteria and first list of services and programs selected for review as part of the Title IV-E Prevention Services Clearinghouse. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1809.pdf

U.S. Department of Health and Human Services. (2018c). ACYF-CB-PI-18-10: Tribal Title-IV E agency requirements for electing Title IV-E prevention and family services and programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1810.pdf.

U.S. Department of Health and Human Services. (2018d). ACYF-CB-PI-18-05: Fiscal Year 2018 funding available for developing, enhancing or evaluating kinship navigator programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1805.pdf.

U.S. Department of Health and Human Services. (2019a). ACYF-CB-PI-16-06: Transitional payments for the Title IV-E prevention and family services programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1906.pdf.

U.S. Department of Health and Human Services. (2019b). ACYF-CB-IM-19-04: Evaluation plan development tip sheet. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/im1904.pdf.

U.S. Department of Health and Human Services (2019c). Technical bulletin #1: Title IV-E prevention program data elements. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/tb_1_iv_e_data_elements.pdf.

Wilson, S. J., Price, C. S., Kerns, S. E. U., Dastrup, S. D., & Brown, S. R. (2019). Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures, Version 1.0, OPRE Report # 2019-56, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. Retrieved from https://www.acf.hhs.gov/sites/default/files/opre/psc_handbook_
v1_final_508_compliant.pdf
.


Footnotes

[1] The services provided under the Family First Act can be categorized as tertiary prevention services, designed to support families already affected by child maltreatment. These interventions support child and family well-being by strengthening protective factors and reducing risk factors for maltreatment recurrence (Child Welfare Information Gateway, 2017). While states should consider a range of primary prevention efforts (i.e., services aimed at the entire population to prevent maltreatment before it happens) and secondary prevention efforts (i.e., services that are targeted at populations at higher risk for maltreatment), only tertiary prevention is covered under the Family First Act.

[2] The Prevention Services Clearinghouse considers a range of factors when determining study design quality, further outlined throughout the Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures (Wilson et al., 2019). Examples of these factors include, but are not limited to, the similarity between the treatment and control groups at the start of treatment, and whether participants dropping out of the study could bias results.

[3] For a program to qualify as promising, supported, or well-supported, the Family First Act requires evidence of significant positive effects in well-designed and well-executed stud(ies). The Family First Act explicitly requires a QED or RCT study design for programs to be rated as supported or well-supported. According to the Act, promising programs may be evaluated using some form of control, with examples including an untreated group, placebo group, or waitlist study (section 471(e)(4)(C) of the Act). The Prevention Services Clearinghouse has interpreted the language of the Act to require an RCT or QED for all relevant research studies (Wilson et al., 2019).

[4] The Family First Act specifies that well-supported programs be evaluated in at least two studies (section 471(e)(4)(C)(v)). Because researchers often publish more than one analysis of the same sample, the Prevention Services Clearinghouse specifies that findings must come from two or more non-overlapping samples.

[5] Examples of related clearinghouses include the Department of Education’s What Works Clearinghouse, US DHHS’s Home Visiting Evidence of Effectiveness (HomVEE), Blueprints for Healthy Youth Development, and the California Evidence-Based Clearinghouse for Child Welfare.

[i] For a summary, see: Child Welfare Information Gateway (2019).

[ii] For more information on the design and implementation of randomized controlled trials, see Moore and Katz (2008).

[iii] For more information on the design and implementation of quasi-experimental evaluations, see Moore (2008).

Newsletters