Program implementers should engage early and often with evaluators to improve interventions

Pulse is a web-based mobile health app designed for Black and Latinx women ages 18 to 20, intended to help them choose effective contraception that aligns with their needs, seek reproductive health services, and, ultimately, prevent unplanned pregnancies. One key challenge that our partners faced while recruiting participants online was ineligible participants joining the study. Since participants were recruited online and all enrollment was done without any personal interaction with study staff, ineligible participants were sometimes able to enroll either by completing the eligibility screener multiple times until they got in (scammers) or by creating more than one account (duplicates). To ensure that only people who met our recruitment criteria enrolled in the study, we developed detailed procedures to identify and remove scammers and duplicate accounts from the sample. To improve this process for further evaluations, our team developed machine learning techniques to automate the process.

Improving enrollment and attendance in a community-based program

Manhood 2.0 is a TPP program for young men ages 16 to 22 implemented in Washington, DC. The eight-session program encourages young men to discuss gender norms, masculinity, and fatherhood as a gateway to broader discussion about contraceptive use, violence, and teen pregnancy prevention. One key challenge from our pilot of Manhood 2.0 was high attrition rates between enrollment and attending the first session. Initially, our implementation partner requested that we enroll participants, conduct the baseline survey, and randomize participants before the start of the program, and then ask participants to return for the first session at a later date. However, it was difficult to get participants to attend an after-school community-based program with which they had yet to engage, so we added an additional hour of curricula to include a “welcome session.” Eligible participants would complete the baseline survey, immediately be randomized, and participate in an introductory activity with their treatment or control group cohort to build interest in the program and a level of familiarity with facilitators. This and other approaches—such as providing food, allowing participants to bring friends, and offering multiple randomization days and multi-location implementation—increased the percentage of participants attending at least one program session from 21 percent during the pilot to 89 percent in the main study. In addition, 61 percent attended at least three quarters of the eight program sessions, a strong attendance rate for a community-based program for young men.

Child Trends’ flexible partnerships helped us respond to challenges and allowed us to adapt and improve TPP program implementation, thereby improving program evaluation. Other organizations tasked with implementing similar programs should consider early evaluator engagement and strong evaluator/program partnerships to maximize evaluation success.