Evaluating Young Women’s Engagement with an Online Reproductive Health App

Research BriefHealthJul 31 2019

Older teens and young adults have the highest rates of unplanned pregnancy in the United States,[1] yet the majority of pregnancy prevention programs are designed for and implemented with younger teens.[2],[3],[4] Technology-based interventions offer an innovative approach to providing pregnancy prevention programming for hard-to-reach populations, including older teens who are no longer in school (either graduated or dropped out), and for racial and ethnic minority populations, such as Latinx and black women, that experience birth rates twice as high as rates for white teens.[1] As the use of mobile devices has become nearly universal, new pregnancy prevention programs have been designed with technology-based service delivery methods, such as mobile apps and text messaging, to better engage hard-to-reach populations.[5],[6],[7],[8],[9],[10]

Child Trends is currently conducting a randomized controlled trial to evaluate the impact of one such technology-based pregnancy prevention program, Pulse. Forthcoming results of this evaluation will focus on the potential impact Pulse has on participants’ behaviors, attitudes, knowledge, and intentions related to reproductive health. In this research brief, we highlight the characteristics of women who participated in the Pulse program, their differing levels of engagement with the mobile app, and the characteristics of the participants who engaged with the app versus those who had little or no engagement. Results provide important lessons about the types of young women who can be effectively reached through a solely tech-based pregnancy prevention program.


Key findings

  • Pulse participants were self-identified females ages 18 to 20 who were not pregnant or seeking pregnancy. The majority were black (42 percent) or Latinx (44 percent). Compared to a nationally representative sample with similar demographics, at baseline, Pulse participants had:
    • Higher levels of educational attainment
    • Higher levels of current school enrollment
    • Lower levels of reported pregnancy
  • As a part of the evaluation design, participants were considered engaged users if they visited all six sections of the app and visited an activity within each section. Only 25 percent of Pulse participants were engaged users. Eight percent of participants were non-users and 67 percent were limited users.
  • Engaged app users had higher response rates to follow-up evaluation surveys and were less likely to opt out of receiving program-related text messages.
  • Participant baseline characteristics were associated with app engagement:
    • Participants with higher levels of education, more knowledge about sexual and reproductive health, and positive attitudes about birth control were more likely to be engaged users.
    • Participants working full-time and those with a child were less likely to be engaged users.
    • Participants who reported receiving most of their sexual and reproductive health information online were more likely to be engaged users.

Background

The study was conducted between November 2016 and April 2019 with continuous enrollment. Participants began their intervention window after being randomized. Each participant received the intervention for six weeks. During the six-week period, participants had unlimited access to Pulse and received text messages with app-related content. The program and evaluation were fully technology-based, meaning all recruitment, enrollment, intervention content, and pre- and post-survey participation took place online, primarily on smartphones. Program implementers and evaluators had no in-person contact with participants. To encourage participants to use the app, the study team developed targeted text message reminders that were sent to participants who never registered with the app, never logged in, or who did not meet the usage requirement for adequate dosage.

Download

One challenge to implementing tech-only interventions is achieving adequate dosage among participants. Ideally, each Pulse participant would visit every activity within all six sections (ranging from three to 11 activities per section) and would frequently visit the app throughout the six-week intervention period. However, the study team acknowledged the dosage limitations of an online program and set the threshold for engagement as completing at least one activity within each of the six sections. The study team agreed that this threshold ensured that participants had navigated and interacted with at least several components of the app.

The study team classified participants into three categories of app usage:

  • Non-users: Participants who never registered with the app
  • Limited users: Participants who registered with the app but did not complete at least one activity in all six sections
  • Engaged users: Participants who completed at least one activity in all six sections

Methods

Analyses in this brief focus on the 1,166 young women who participated in the Pulse study (the intervention group). To better understand the characteristics of the women who participated in Pulse, we compared the baseline characteristics of black and Latinx Pulse participants (86 percent of the sample) against a nationally representative sample with a similar age and racial/ethnic makeup drawn from the 2015-2017 National Survey of Family Growth (NSFG). We tested for significant differences by calculating the means and standard deviations for the measures of interest from each dataset (including applying appropriate weights from the NSFG) and conducting t-tests.

To better understand the characteristics of Pulse participants who engaged with the app, we examined whether different baseline characteristics were associated with being an engaged user, versus a non-user or limited user. We combined non-use and limited use for this analysis because researchers are mainly interested in understanding what distinguishes engaged users from other participants, rather than distinguishing between non-users and limited users. Additionally, there were few non-users in the sample, making it difficult to detect statistically significant differences for this group. To describe the relationship(s) between level of app usage (binary: engaged users versus non-users and limited users combined) and participation levels (app visits and section clicks) of Pulse participants (continuous variables), we conducted two-sample t-tests by group (engaged users versus non-users and limited users combined). To describe the relationship(s) between level of app usage and background characteristics (binary and categorical), the team performed chi-squared tests and logistic regressions. All analyses were conducted using Stata 13.1. In our findings we report differences that were significant at p < 0.05.


Findings

Characteristics of Pulse participants

Compared to a nationally representative sample with similar demographics (ages 18 to 20, primarily black and Latinx), Pulse participants had higher levels of educational attainment (65 percent of Pulse participants had some college or technical school education compared to 23 percent nationally) and current school enrollment (78 percent of Pulse participants were currently enrolled in school compared to 69 percent nationally). Pulse participants were also less likely to have ever been pregnant (12 percent of Pulse participants had ever been pregnant compared to 21 percent nationally).

Participation levels

In total, 8 percent (95 participants) of the 1,166 participants were categorized as non-users, 67 percent (785 participants) were categorized as limited users, and 25 percent (286 participants) were categorized as engaged users.

Participant app usage

Not surprisingly, engaged users visited the app more frequently than limited users, an average of 4.4 times over the course of the six weeks, versus 1.7 times for limited users. Engaged users also had a significantly higher average number of section visits within the app (six sections for engaged users versus three for limited users). Additionally, engaged users had an average of 56 app clicks (number of clicks across sections and activities) versus limited users who had an average of 16 app clicks.

Engaged users were more involved in the study overall. They had higher response rates to follow-up evaluation surveys: 97 percent of engaged users completed the follow-up survey compared to 84 percent of limited users and 62 percent of non-users. As part of enrollment in the study, participants were automatically opted in to receive study-related text messages, which included program content as well as reminders to visit the app; however, they could opt out of receiving these text messages at any time. Engaged users were less likely to opt out of receiving these text messages than limited users (7 percent versus 13 percent). Engaged users were also less likely to experience text message bounce back (i.e., message delivery failure): 16 percent of engaged users versus 21 percent of limited users and 34 percent of non-users.

Participant characteristics linked to usage

Education, employment, and parenthood were associated with participants’ levels of engagement with the app. Figure 2 shows the percentage of participants with certain characteristics who were engaged users versus limited users and non-users. The findings (and statistical testing) focus on engaged users versus non-users and limited users combined. Patterns suggest similar findings for non-users and limited users (data not shown).

Participants with at least some college or technical education, participants who were not currently working, and participants without a child were more likely to be engaged users. For example, participants who did not have a child were more than twice as likely as those with a child to be engaged users (26 percent versus 11 percent; Figure 2).

App usage by education, employment, and parenthood

Participants’ baseline knowledge of reproductive health and their attitudes about birth control were also associated with levels of engagement with the app. Figure 3 shows the level of participants’ knowledge across four true/false sexual and reproductive health questions (low knowledge = 0 correct answers, moderate knowledge = 1 or 2 correct answers, high knowledge = 3 or 4 correct answers) and participants’ attitudes about birth control (those with positive attitudes disagreed with the statement “Birth control is too much of a hassle to use”).

Participants with high levels of sexual and reproductive health knowledge were the most likely to be engaged users (30 percent compared with 26 percent for moderate knowledge and 17 percent for low knowledge). In addition, participants with low knowledge were more than twice as likely to be non-users than those with high knowledge (data not shown). Participants with more positive attitudes about birth control (who disagreed that birth control is a hassle to use) were more likely to be engaged users than those who had more negative views of birth control.

App usage by knowledge and attitudes

On the baseline survey, participants were asked, “Where do you get most of your information about health?” Those who reported receiving sexual and reproductive health information primarily from online sources (versus family, friends, and school; or a health care provider) were more likely to be engaged users (29 percent of those who received most information online were engaged users, compared to 19 to 21 percent of those who reported other sources of information; see Figure 4).

App usage by source of sexual and reproductive health information

Discussion

The Pulse program was designed to help users prevent unplanned pregnancies by choosing effective birth control and seeking reproductive health services. The program was designed for young black and Latinx women ages 18 to 20 who were not currently pregnant or seeking pregnancy. The study team successfully reached this target population, although we found that Pulse participants were not representative of black and Latinx women of similar age on a few key demographic characteristics. Compared to black and Latinx women in a nationally representative survey conducted at a similar point in time, Pulse participants were more highly educated, more likely to be in school, and less likely to have experienced a pregnancy. The fact that Pulse was most successful in connecting with more educated and less at-risk young adults could be due to the process of recruiting online through social media and the requirement that Pulse participants have daily access to a smartphone. Studies of social media users have found that a higher percentage of college-educated adults use Facebook and Instagram (the sites used for recruitment) than adults with a high school degree or less.[11] Other studies have also found that recruitment through Facebook yields a disproportionately large number of youth with higher education.[12],[13] More specific to Pulse, the demographics of participants could also reflect the appeal of a sexual health app to young women who are more educated.

Among Pulse participants, only one-quarter were considered engaged users, visiting all six sections of the app at least once and completing one activity in each section during the six-week program period.  This metric was chosen to ensure that participants received a variety of program content. Therefore, most participants did not receive a wide range of program content. These findings highlight both the strengths and limitations of online programming, which allows implementers to reach many racially, ethnically, and geographically diverse young adults—who do not generally receive programming—with less effort and expense than traditional school- or community-based teen pregnancy prevention approaches require.[12],[14] However, many youth may not receive the online program content as desired. While the populations that other approaches reach may be younger and less diverse, those youth receive a larger amount of program content.

Additionally, while overall response rates were high for an online study (86%), the follow-up survey rates were highest among engaged users, compared to rates for limited users and non-users. This has implications for conducting high-quality program evaluations, since high response rates across participants are required to detect statistically significant differences between treatment and control groups and to ensure that findings are not biased.

Overall, the young black and Latinx women who likely received the most program content through Pulse (through original enrollment and then engagement with the app) were women who may be less at risk of unintended pregnancy than other young black and Latinx women because of their higher levels of education, more likely enrollment in school, greater sexual and reproductive health knowledge, and more positive attitudes about birth control. Additionally, those who already sought reproductive health information online were more likely to engage in the app. Further, those who were working full-time or who had a child were less likely to have high levels of engagement in the Pulse app, highlighting their lower time availability for app-based content. Findings suggest that while online pregnancy prevention programs can connect with diverse young adult populations that would be hard to reach through school- or community-based programming, programs that are conducted entirely online through social media recruitment may not target at-risk populations as successfully. Future studies should examine ways to target populations to focus on racially and ethnically diverse young adults who have lower educational attainment and lower socioeconomic status.


This publication was made possible by Grant Number TP2AH000038 from the Office of
Population Affairs (OPA), U.S. Department of Health and Human Services (HHS). Its contents
are solely the responsibility of the authors and do not necessarily represent the official views of
the OPA or HHS.


References

[1] Finer, L. B., & Zolna, M. R. (2016). Declines in unintended pregnancy in the United States, 2008–2011. New England Journal of Medicine, 374(9), 843-852. doi:doi:10.1056/NEJMsa1506575

[2] Juras, R., Tanner-Smith, E., Kelsey, M., Lipsey, M., & Layzer, J. (2019). Adolescent Pregnancy Prevention: Meta-Analysis of Federally Funded Program Evaluations. Am J Public Health, 109(4), e1-e8. doi:10.2105/ajph.2018.304925

[3] Lugo-Gil, J., Lee, A., Vohra, D., Harding, J., Ochoa, L., & Goesling, B. (2018). Updated findings from the HHS Teen Pregnancy Prevention Evidence Review: August 2015 through October 2016. Retrieved from Washington, DC: https://tppevidencereview.aspe.hhs.gov/pdfs/Summary_of_findings_2016-2017.pdf

[4] Martin, J. A., Hamilton, B. E., Osterman, M. J. K., Driscoll, A. K., & Drake, P. (2018). Births: Final Data for 2017. Natl Vital Stat Rep, 67(8), 1-50.

[5] Antonishak, J., Kaye, K., & Swiader, L. (2015). Impact of an Online Birth Control Support Network on Unintended Pregnancy. Social Marketing Quarterly, 21(1), 23-36. doi:10.1177/1524500414566698

[6] Bull, S., Devine, S., Schmiege, S. J., Pickard, L., Campbell, J., & Shlay, J. C. (2016). Text Messaging, Teen Outreach Program, and Sexual Health Behavior: A Cluster Randomized Trial. American Journal of Public Health, 106(S1), S117-S124. doi:10.2105/AJPH.2016.303363

[7] Guse, K., Levine, D., Martins, S., Lira, A., Gaarde, J., Westmorland, W., & Gilliam, M. (2012). Interventions Using New Digital Media to Improve Adolescent Sexual Health: A Systematic Review. Journal of Adolescent Health, 51(6), 535-543. doi:10.1016/j.jadohealth.2012.03.014

[8] Kissinger, P., Schmidt, N., Green, J., Latimer, J., Madkour, A., Clum, G., & Johnson, C. (2015). Evaluation of BUtiful: An Internet pregnancy prevention for older teenage girls in New Orleans, Louisiana. Retrieved from New Orleans, LA: https://www.hhs.gov/ash/oah/sites/default/files/ash/oah/oah-initiatives/evaluation/grantee-led-evaluation/reports/tulane-final-report.pdf

[9] Steinberg, A., Griffin-Tomas, M., Abu-Odeh, D., & Whitten, A. (2018). Evaluation of a Mobile Phone App for Providing Adolescents With Sexual and Reproductive Health Information, New York City, 2013-2016. Public Health Rep, 133(3), 234-239. doi:10.1177/0033354918769289

[10] Widman, L., Nesi, J., Kamke, K., Choukas-Bradley, S., & Stewart, J. L. (2018). Technology-Based Interventions to Reduce Sexually Transmitted Infections and Unintended Pregnancy Among Youth. Journal of Adolescent Health, 62(6), 651-660. doi:10.1016/j.jadohealth.2018.02.007

[11] Pew Research Center. (2018). Social Media Fact Sheet. Retrieved from https://www.pewinternet.org/fact-sheet/social-media/

[12] Fenner, Y., Garland, S. M., Moore, E. E., Jayasinghe, Y., Fletcher, A., Tabrizi, S. N., . . . Wark, J. D. (2012). Web-Based Recruiting for Health Research Using a Social Networking Site: An Exploratory Study. J Med Internet Res, 14(1), e20. doi:10.2196/jmir.1978

[13] Loxton, D., Powers, J., Anderson, A. E., Townsend, N., Harris, M. L., Tuckerman, R., . . . Byles, J. (2015). Online and Offline Recruitment of Young Women for a Longitudinal Health Survey: Findings From the Australian Longitudinal Study on Women’s Health 1989-95 Cohort. J Med Internet Res, 17(5), e109. doi:10.2196/jmir.4261

[14] Thornton, L., Batterham, P. J., Fassnacht, D. B., Kay-Lambkin, F., Calear, A. L., & Hunt, S. (2016). Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interventions, 4, 72-81. doi:https://doi.org/10.1016/j.invent.2016.02.001

Newsletters