Strategies to Build Evidence for Kinship Navigator Programs Under the Family First Act

Kinship care has deep, historical roots in Black and Indigenous communities. For example, after white slave owners separated families by selling children’s enslaved parents, relatives often cared for the children; in Indigenous communities, many tribal members consider themselves part of an extended family group who support each other in times of need. Currently, relatives continue to provide care when many Black and Indigenous children cannot live with their parents. For example, in one state, Black children in child welfare custody are 1.5 times more likely to be placed with kin compared to children of other races. Indigenous children are more likely than any other racial or ethnic group to live with kinship caregivers within or outside foster care.

Unlicensed, informal kinship care providers often go without the financial supports and oversight that agencies provide to licensed kinship care providers. Thus, any racial/ethnic disparities in the use of unlicensed or informal kinship care[4] could result in inequitable access to supports and, in turn, to inequitable child and family outcomes. However, because kinship navigator programs can serve not only formal, licensed kinship caregivers but also informal, unlicensed caregivers, they have the potential to help engage communities who may traditionally not seek services and to promote increased access to services regardless of the circumstances of kinship care.

While the federal government has demonstrated support for kinship navigator programs through the Family First Act, as of February 2021, no kinship navigator programs have met the Family First Act’s threshold for evidence as articulated by the Title IV-E Prevention Services Clearinghouse (Clearinghouse). Kinship navigator programs must be rated at least “promising,” meaning that at least one study meets the Clearinghouse’s design standards and demonstrates a positive effect on a relevant outcome. The evidence requirements for kinship navigator programs have been waived through September 30, 2021 during the COVID-19 public health emergency through the Consolidated Appropriations Act of 2021, allowing states to draw down federal funding to continue serving kinship families and evaluating their programs.

The limited evidence base and the challenges involved in developing proof of effectiveness for child welfare interventions are not unique to kinship navigator programs. Evaluations of child welfare interventions often fail to identify program impacts. However, limitations in many studies make it difficult to determine if an impact was not detected because of a study’s limitations or because the program was no more effective than services as usual. Below, we identify common challenges agency leaders and program evaluators face in evaluating kinship navigator programs, and we suggest potential solutions that align with the Clearinghouse requirements to address those challenges.

---

Common Challenges in Evaluating Kinship Navigator Programs

Defining the program models

Kinship navigator programs are usually composed of several components (see text box “What are Kinship Navigator Programs” above). To be considered by the Clearinghouse, the program must have a manual identifying its components and how the components should be implemented. This information helps measure the level of fidelity attained in practice to contextualize evaluation findings. A manual is also important for replicating effective program models in other jurisdictions.

However, manualizing a kinship navigator program can be difficult, since components have different requirements, lengths of service, intensity of service, and ranges of options available to recipients. Implementing each component can also differ depending on the need of the kinship caregiver being served. Some kinship caregivers may need different components of the program—or different doses of a particular component—than others. Further, most kinship navigator programs involve voluntary, open-ended services allowing for ebbs and flows of caregiving and recurring needs, resulting in varying length and intensity of services. Identifying and addressing these variations in caregiver need may require staff time and effort but can increase the effectiveness of the program overall. In addition, tailoring a statewide program to serve rural and urban communities with differing needs and resources requires flexibility. However, the customizability of the program model can make it difficult to design an evaluation, identify appropriate outcomes and metrics, and assess if the program model is implemented with fidelity.

Given the voluntary nature of services, enrollment and engagement among eligible families may be a problem. Families may face a language barrier when information and referral services are offered only in English, and/or when there are no kinship navigators who speak languages other than English. Families of color, including Black and Indigenous families, may be particularly distrustful of health care services. Some caregivers may be reluctant to engage with social services, especially if the service is offered by the public child welfare agency, which they may distrust. There is also a demonstrated long history of racial bias in child welfare services that can lead to distrust particularly among families of color. This distrust can hinder equitable access of, and engagement with, services such as a kinship caregiver program. If many families are unwilling to engage with services, then there may be no impact for an intervention to detect, or the intervention may improve outcomes for white families, but not for families of color. Indeed, in the domain of mental health care, research indicates that families of color typically have less access to and are less likely to obtain the care they need.

Selecting a comparison group

For kinship navigator programs to reach the level of evidence needed to be eligible for ongoing federal reimbursement under the Family First Act,[5] an evaluation must compare two similar groups, one of which (the treatment group) receives the service, and the other of which (the comparison or control group[6]) does not. However, it can be challenging to ensure that the characteristics of the treatment and comparison groups do not differ systematically. Differences in the study groups weaken the level of evidence generated, because it can be difficult to determine if outcomes are a result of the program or are due to differences in the characteristics or contexts of the caregiver at the outset that may also influence the outcomes of interest. The easiest way to ensure comparability of the groups is to randomly assign participants to the study groups. If each study participant has an equal likelihood of receiving or not receiving intervention services, then, with a sufficiently large sample size, the two groups will be statistically similar.

However, some agency leaders and service administrators worry that a rigorous study will involve withholding needed services from caregivers in the comparison group, which they believe is unethical. Program implementers and evaluators can often resolve ethical and equity concerns through open planning discussions and by ensuring that both groups of caregivers receive some type of services and supports. A limitation of this approach is it may make it more difficult to identify positive program impacts because the comparison group may also experience improvements; further, the Clearinghouse only reviews studies that compare outcomes for kinship caregivers who receive services (treatment) through a  kinship navigator program with outcomes for kinship caregivers who received no or minimal treatment (e.g., a basic referral service or handouts) or treatment as usual (services typically available for families).[7]

It can also be challenging to identify kinship caregivers who might serve as a comparison group. When a kinship navigator program is universally available in a geographic region, some families will be involved in the child welfare system and likely receive a different level of services and support than families who are not involved in the child welfare system. Therefore, it would be inappropriate to compare these two groups of families. It is possible to use child welfare administrative data to identify the universe of child welfare involved kinship caregivers and then compare those who receive kinship navigator services with those who do not. This approach overlooks the fact that families that seek and engage in services may differ systematically from those who do not; thus, in comparing outcomes for these two groups, it would be difficult to determine if differences are due to the intervention or to pre-existing differences in the groups. Another drawback of using child welfare administrative data to identify the comparison group is that it omits informal kinship caregivers for whom there is no child welfare data.

Determining sample size

To determine if any observed differences in outcomes between the kinship navigator program group and a comparison group can be attributed to the program rather than to chance (assuming other factors related to outcomes have been controlled), a sufficient number of families must participate in the study.[10] Further, the smaller the anticipated program impact, the larger the sample size that is needed to detect differences due to the program. Kinship caregivers may make up a small share of the overall population, which can make it difficult to recruit enough participants. This is especially challenging in small jurisdictions. It may take time to advertise the program and become known as a useful resource.

Selecting appropriate reliable and valid measures

The Clearinghouse considers a wide range of outcomes for kinship navigator programs, including child safety, permanency, and well-being; adult well-being; and access to, referral to, and/or satisfaction with services. Implementers of kinship navigator programs may be unsure which outcomes they can realistically expect to impact.

The availability of data (e.g., administrative data from the child welfare system) sometimes drives the outcomes and measures used in a study. While using available data can help minimize the burden for staff and families participating in an evaluation, available data may not align closely with the outcomes targeted by the program. Further, when kinship caregivers and other stakeholders are not included in the design of a program’s evaluation plan, the outcomes selected may be less salient to the families targeted by these services.

Another challenge is that existing measures may not have been validated for the populations in the study. If that were the case, the outcomes may not accurately reflect their experience, making the reliability of evaluation results suspect. For example, some outcome measures have proven reliable and valid for parents generally but not for kinship caregivers, or measures may not have been developed with individuals or families that reflect the racial/ethnic background of families served by kinship navigator programs. Prior research has often tested measures with white families, but many kinship caregivers are Black, Indigenous, or persons of color.

Collecting data

Once outcome measures are identified for the evaluation, plans must be developed regarding when and how to gather this information. As discussed earlier, many services delivered by kinship navigator programs are open-ended, making it difficult to determine the “end” of treatment. This affects when outcomes should be assessed, especially if families can continue to access services as needed. According to the Clearinghouse, the study must measure outcomes at the end of treatment but does not need to follow participants after the end of services to earn a rating of “promising.”

A common challenge is engaging participants in the comparison group in data collection due to a program’s lack of or limited contact with them. Similarly, some participants in the treatment group may also need and receive less direct engagement with program staff (e.g., when receiving a one-time service), which may make it more difficult to collect data on their outcomes.

Evaluators often turn to administrative data from the child welfare and other family-serving systems to ensure data availability for the treatment and the comparison groups. However, as described above, these systems may not include data that addresses all outcomes of interest or data for both study groups. In addition, evaluators may face challenges accessing administrative data, including negotiating data sharing agreements and partnering with child welfare agency staff, TANF staff, or aging services staff, who face competing priorities for their time, to help with obtaining and contextualizing the data.

With administrative or other forms of data, such as survey data, evaluators may face data quality challenges when they begin analysis. Common challenges can include missing data, data entry errors, and different interpretations of the meaning of data elements among those entering the information.

---

Conclusion

Kinship caregivers and their families deserve services that effectively meet their needs. For states seeking to continue and expand kinship navigator programs, federal Title IV-E funding is a valuable resource, particularly as states face pandemic-related budget shortfalls. However, after the kinship navigator provisions from the Consolidated Appropriations Act of 2021 end, federal funding will depend on demonstrating that a kinship program is at least “promising” when it comes to improving outcomes. Kinship navigator program evaluators will need to address the challenges described above with program administrators before determining rigorous, feasible designs for evaluating their programs.

Several kinship navigator program evaluations are currently underway, and many stakeholders are involved in strengthening the delivery and evaluation of kinship navigator programs. This ongoing work highlights the need for continued conversations across stakeholders and jurisdictions to identify and respond to evidence-building challenges as the field strives to best serve kinship families. Engaging the community, kinship navigators, kinship caregivers, data providers, and youth with lived experience in these conversations and throughout the process of designing and conducting evaluative studies will help ensure the designs yield findings that are relevant, actionable, and promote equitable outcomes for children and families. Open and transparent communication throughout the evaluation process will also keep stakeholders informed about how evaluation findings are used to inform next steps and refine programming. Evaluation partners, such as Child Trends, can help facilitate this process.

---

Acknowledgements

We are grateful to our colleagues at Child Trends and in the child welfare field who provided insights and feedback during the development of this brief, including Elizabeth Jordan and Karin Malm, at Child Trends; Ana Beltran and Jaia Lent at Generations United; and Lynn Urvina, Kinship Program Manager at Family Education and Support Services, Tumwater WA.

---

References

[1] The term “kin” refers to relatives and to fictive kin; that is, non-related adults who have pre-existing relationships with children. The official data on children in formal foster care who are living with kin exclude fictive kin.

[2] Kinship care typically falls into one of two types: formal kinship care (also known as relative foster care), which occurs when children are in custody of a public child welfare agency (typically because of abuse or neglect); and informal kinship care, which is arranged between parent and caregiver and may or may not involve child welfare but does not include state custody. Family Court orders that assign custody to a caregiver without child welfare agency involvement are considered informal placements.

[3] In a survey of kinship caregivers during the COVID-19 pandemic, 38 percent reported being unable to pay or concerned about paying their mortgage or rent; 43 percent expressed fear over leaving their home to obtain food; and 30 percent had no caregiving plan for the children in their care if they died).

[4] Practices around informal and formal kinship care vary across jurisdictions, and data are not always available for informal kinship care placements. Child Trends analyzed administrative data from two states and found that Black children in one state were more likely to be placed in formal kinship care compared to informal kinship care (15%, compared to 7%). Referral data were not available in the second state, precluding analysis of racial/ethnic disparities in placement in formal versus informal kinship care.

[5] The Consolidated Appropriations Act of 2021 temporarily waived the evidence requirements for kinship navigator programs during the COVID-19 public health emergency.

[6] When referring to the group not receiving the intervention, we use “comparison group” throughout this brief for simplicity.

[7] The Title IV-E Clearinghouse defines “no or minimal intervention” as receiving handouts, referrals to available services, or similar nominal interventions and defines “treatment as usual” as the usual or typical services available for that population in the study.

[8] “TANF child-only grants were designed to consider only the needs and income of the child. A child’s income might include child support payments or a public benefit like Supplemental Security Income (SSI). Because most children have limited income, most relative caregivers can receive a child-only grant on behalf of the children in their care.”

[9] This waitlist approach could be used with an RCT if families are randomly assigned to receive services right away or be waitlisted. This works only if there is not enough capacity to serve all eligible families right away.

[10] This is referred to as the power to tell if the findings are an actual effect of the intervention or just occurring by chance.

[11] For example, one patient health navigator study used a child self-rated health survey at 6-months after completing the intervention. Other outcomes included quality of life, caregiver self-rated health, and satisfaction with healthcare.