Evaluating Young Women’s Engagement with an Online Reproductive Health App

One challenge to implementing tech-only interventions is achieving adequate dosage among participants. Ideally, each Pulse participant would visit every activity within all six sections (ranging from three to 11 activities per section) and would frequently visit the app throughout the six-week intervention period. However, the study team acknowledged the dosage limitations of an online program and set the threshold for engagement as completing at least one activity within each of the six sections. The study team agreed that this threshold ensured that participants had navigated and interacted with at least several components of the app.

The study team classified participants into three categories of app usage:

• Non-users: Participants who never registered with the app
• Limited users: Participants who registered with the app but did not complete at least one activity in all six sections
• Engaged users: Participants who completed at least one activity in all six sections

---

Methods

Analyses in this brief focus on the 1,166 young women who participated in the Pulse study (the intervention group). To better understand the characteristics of the women who participated in Pulse, we compared the baseline characteristics of black and Latinx Pulse participants (86 percent of the sample) against a nationally representative sample with a similar age and racial/ethnic makeup drawn from the 2015-2017 National Survey of Family Growth (NSFG). We tested for significant differences by calculating the means and standard deviations for the measures of interest from each dataset (including applying appropriate weights from the NSFG) and conducting t-tests.

To better understand the characteristics of Pulse participants who engaged with the app, we examined whether different baseline characteristics were associated with being an engaged user, versus a non-user or limited user. We combined non-use and limited use for this analysis because researchers are mainly interested in understanding what distinguishes engaged users from other participants, rather than distinguishing between non-users and limited users. Additionally, there were few non-users in the sample, making it difficult to detect statistically significant differences for this group. To describe the relationship(s) between level of app usage (binary: engaged users versus non-users and limited users combined) and participation levels (app visits and section clicks) of Pulse participants (continuous variables), we conducted two-sample t-tests by group (engaged users versus non-users and limited users combined). To describe the relationship(s) between level of app usage and background characteristics (binary and categorical), the team performed chi-squared tests and logistic regressions. All analyses were conducted using Stata 13.1. In our findings we report differences that were significant at p < 0.05.

---

Findings

Characteristics of Pulse participants

Compared to a nationally representative sample with similar demographics (ages 18 to 20, primarily black and Latinx), Pulse participants had higher levels of educational attainment (65 percent of Pulse participants had some college or technical school education compared to 23 percent nationally) and current school enrollment (78 percent of Pulse participants were currently enrolled in school compared to 69 percent nationally). Pulse participants were also less likely to have ever been pregnant (12 percent of Pulse participants had ever been pregnant compared to 21 percent nationally).

Participation levels

In total, 8 percent (95 participants) of the 1,166 participants were categorized as non-users, 67 percent (785 participants) were categorized as limited users, and 25 percent (286 participants) were categorized as engaged users.

Not surprisingly, engaged users visited the app more frequently than limited users, an average of 4.4 times over the course of the six weeks, versus 1.7 times for limited users. Engaged users also had a significantly higher average number of section visits within the app (six sections for engaged users versus three for limited users). Additionally, engaged users had an average of 56 app clicks (number of clicks across sections and activities) versus limited users who had an average of 16 app clicks.

Engaged users were more involved in the study overall. They had higher response rates to follow-up evaluation surveys: 97 percent of engaged users completed the follow-up survey compared to 84 percent of limited users and 62 percent of non-users. As part of enrollment in the study, participants were automatically opted in to receive study-related text messages, which included program content as well as reminders to visit the app; however, they could opt out of receiving these text messages at any time. Engaged users were less likely to opt out of receiving these text messages than limited users (7 percent versus 13 percent). Engaged users were also less likely to experience text message bounce back (i.e., message delivery failure): 16 percent of engaged users versus 21 percent of limited users and 34 percent of non-users.

Participant characteristics linked to usage

Education, employment, and parenthood were associated with participants’ levels of engagement with the app. Figure 2 shows the percentage of participants with certain characteristics who were engaged users versus limited users and non-users. The findings (and statistical testing) focus on engaged users versus non-users and limited users combined. Patterns suggest similar findings for non-users and limited users (data not shown).

Participants with at least some college or technical education, participants who were not currently working, and participants without a child were more likely to be engaged users. For example, participants who did not have a child were more than twice as likely as those with a child to be engaged users (26 percent versus 11 percent; Figure 2).

Participants’ baseline knowledge of reproductive health and their attitudes about birth control were also associated with levels of engagement with the app. Figure 3 shows the level of participants’ knowledge across four true/false sexual and reproductive health questions (low knowledge = 0 correct answers, moderate knowledge = 1 or 2 correct answers, high knowledge = 3 or 4 correct answers) and participants’ attitudes about birth control (those with positive attitudes disagreed with the statement “Birth control is too much of a hassle to use”).

Participants with high levels of sexual and reproductive health knowledge were the most likely to be engaged users (30 percent compared with 26 percent for moderate knowledge and 17 percent for low knowledge). In addition, participants with low knowledge were more than twice as likely to be non-users than those with high knowledge (data not shown). Participants with more positive attitudes about birth control (who disagreed that birth control is a hassle to use) were more likely to be engaged users than those who had more negative views of birth control.

On the baseline survey, participants were asked, “Where do you get most of your information about health?” Those who reported receiving sexual and reproductive health information primarily from online sources (versus family, friends, and school; or a health care provider) were more likely to be engaged users (29 percent of those who received most information online were engaged users, compared to 19 to 21 percent of those who reported other sources of information; see Figure 4).

---

Discussion

The Pulse program was designed to help users prevent unplanned pregnancies by choosing effective birth control and seeking reproductive health services. The program was designed for young black and Latinx women ages 18 to 20 who were not currently pregnant or seeking pregnancy. The study team successfully reached this target population, although we found that Pulse participants were not representative of black and Latinx women of similar age on a few key demographic characteristics. Compared to black and Latinx women in a nationally representative survey conducted at a similar point in time, Pulse participants were more highly educated, more likely to be in school, and less likely to have experienced a pregnancy. The fact that Pulse was most successful in connecting with more educated and less at-risk young adults could be due to the process of recruiting online through social media and the requirement that Pulse participants have daily access to a smartphone. Studies of social media users have found that a higher percentage of college-educated adults use Facebook and Instagram (the sites used for recruitment) than adults with a high school degree or less.[11] Other studies have also found that recruitment through Facebook yields a disproportionately large number of youth with higher education.[12],[13] More specific to Pulse, the demographics of participants could also reflect the appeal of a sexual health app to young women who are more educated.

Among Pulse participants, only one-quarter were considered engaged users, visiting all six sections of the app at least once and completing one activity in each section during the six-week program period.  This metric was chosen to ensure that participants received a variety of program content. Therefore, most participants did not receive a wide range of program content. These findings highlight both the strengths and limitations of online programming, which allows implementers to reach many racially, ethnically, and geographically diverse young adults—who do not generally receive programming—with less effort and expense than traditional school- or community-based teen pregnancy prevention approaches require.[12],[14] However, many youth may not receive the online program content as desired. While the populations that other approaches reach may be younger and less diverse, those youth receive a larger amount of program content.

Additionally, while overall response rates were high for an online study (86%), the follow-up survey rates were highest among engaged users, compared to rates for limited users and non-users. This has implications for conducting high-quality program evaluations, since high response rates across participants are required to detect statistically significant differences between treatment and control groups and to ensure that findings are not biased.

Overall, the young black and Latinx women who likely received the most program content through Pulse (through original enrollment and then engagement with the app) were women who may be less at risk of unintended pregnancy than other young black and Latinx women because of their higher levels of education, more likely enrollment in school, greater sexual and reproductive health knowledge, and more positive attitudes about birth control. Additionally, those who already sought reproductive health information online were more likely to engage in the app. Further, those who were working full-time or who had a child were less likely to have high levels of engagement in the Pulse app, highlighting their lower time availability for app-based content. Findings suggest that while online pregnancy prevention programs can connect with diverse young adult populations that would be hard to reach through school- or community-based programming, programs that are conducted entirely online through social media recruitment may not target at-risk populations as successfully. Future studies should examine ways to target populations to focus on racially and ethnically diverse young adults who have lower educational attainment and lower socioeconomic status.

---

This publication was made possible by Grant Number TP2AH000038 from the Office of
Population Affairs (OPA), U.S. Department of Health and Human Services (HHS). Its contents
are solely the responsibility of the authors and do not necessarily represent the official views of
the OPA or HHS.

---

References

[1] Finer, L. B., & Zolna, M. R. (2016). Declines in unintended pregnancy in the United States, 2008–2011. New England Journal of Medicine, 374(9), 843-852. doi:doi:10.1056/NEJMsa1506575

[2] Juras, R., Tanner-Smith, E., Kelsey, M., Lipsey, M., & Layzer, J. (2019). Adolescent Pregnancy Prevention: Meta-Analysis of Federally Funded Program Evaluations. Am J Public Health, 109(4), e1-e8. doi:10.2105/ajph.2018.304925

[3] Lugo-Gil, J., Lee, A., Vohra, D., Harding, J., Ochoa, L., & Goesling, B. (2018). Updated findings from the HHS Teen Pregnancy Prevention Evidence Review: August 2015 through October 2016. Retrieved from Washington, DC: https://tppevidencereview.aspe.hhs.gov/pdfs/Summary_of_findings_2016-2017.pdf

[4] Martin, J. A., Hamilton, B. E., Osterman, M. J. K., Driscoll, A. K., & Drake, P. (2018). Births: Final Data for 2017. Natl Vital Stat Rep, 67(8), 1-50.

[5] Antonishak, J., Kaye, K., & Swiader, L. (2015). Impact of an Online Birth Control Support Network on Unintended Pregnancy. Social Marketing Quarterly, 21(1), 23-36. doi:10.1177/1524500414566698

[6] Bull, S., Devine, S., Schmiege, S. J., Pickard, L., Campbell, J., & Shlay, J. C. (2016). Text Messaging, Teen Outreach Program, and Sexual Health Behavior: A Cluster Randomized Trial. American Journal of Public Health, 106(S1), S117-S124. doi:10.2105/AJPH.2016.303363

[7] Guse, K., Levine, D., Martins, S., Lira, A., Gaarde, J., Westmorland, W., & Gilliam, M. (2012). Interventions Using New Digital Media to Improve Adolescent Sexual Health: A Systematic Review. Journal of Adolescent Health, 51(6), 535-543. doi:10.1016/j.jadohealth.2012.03.014

[8] Kissinger, P., Schmidt, N., Green, J., Latimer, J., Madkour, A., Clum, G., & Johnson, C. (2015). Evaluation of BUtiful: An Internet pregnancy prevention for older teenage girls in New Orleans, Louisiana. Retrieved from New Orleans, LA: https://www.hhs.gov/ash/oah/sites/default/files/ash/oah/oah-initiatives/evaluation/grantee-led-evaluation/reports/tulane-final-report.pdf

[9] Steinberg, A., Griffin-Tomas, M., Abu-Odeh, D., & Whitten, A. (2018). Evaluation of a Mobile Phone App for Providing Adolescents With Sexual and Reproductive Health Information, New York City, 2013-2016. Public Health Rep, 133(3), 234-239. doi:10.1177/0033354918769289

[10] Widman, L., Nesi, J., Kamke, K., Choukas-Bradley, S., & Stewart, J. L. (2018). Technology-Based Interventions to Reduce Sexually Transmitted Infections and Unintended Pregnancy Among Youth. Journal of Adolescent Health, 62(6), 651-660. doi:10.1016/j.jadohealth.2018.02.007

[11] Pew Research Center. (2018). Social Media Fact Sheet. Retrieved from https://www.pewinternet.org/fact-sheet/social-media/

[12] Fenner, Y., Garland, S. M., Moore, E. E., Jayasinghe, Y., Fletcher, A., Tabrizi, S. N., . . . Wark, J. D. (2012). Web-Based Recruiting for Health Research Using a Social Networking Site: An Exploratory Study. J Med Internet Res, 14(1), e20. doi:10.2196/jmir.1978

[13] Loxton, D., Powers, J., Anderson, A. E., Townsend, N., Harris, M. L., Tuckerman, R., . . . Byles, J. (2015). Online and Offline Recruitment of Young Women for a Longitudinal Health Survey: Findings From the Australian Longitudinal Study on Women’s Health 1989-95 Cohort. J Med Internet Res, 17(5), e109. doi:10.2196/jmir.4261

[14] Thornton, L., Batterham, P. J., Fassnacht, D. B., Kay-Lambkin, F., Calear, A. L., & Hunt, S. (2016). Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interventions, 4, 72-81. doi:https://doi.org/10.1016/j.invent.2016.02.001