Evaluation, Step by Step: The Case of Raising A Reader

For years, researchers have noted that most of the interventions communities offer children and families—whether mentoring programs, after-school arts and crafts, midnight basketball, tutoring, or teen centers—have little hard evidence behind them.  With few exceptions, these activities haven’t undergone the kinds of rigorous tests we expect from many other things we rely on—medicines, cars, computers—to do what they’re promoted to do. This makes little sense, both because the appropriate research tools are available, and because children deserve better than a-wing-and-a-prayer. A recent Child Trends research report details the trajectory of a promising, low-cost, early education and parent engagement program, Raising A Reader (RAR), that has built an evidence base and is continuing to use research to refine its model.

Youth development scientists tell us we’re building, brick by brick, this evidence base, primarily through carefully designed evaluation studies. Experienced evaluators understand that evaluation is more often a developmental, bootstrapping process: it starts with a program’s becoming committed to ongoing learning through use of data, evolves through small “plan-do-study-adjust” feedback loops that result in successive improvements in how the program is carried out, and develops into a grounded theory of change with testable propositions about how particular activities lead to short-, mid- , and long-term results for participants. Only then does it make sense to begin to answer the question, “does it work?” Nevertheless, there is often pressure to demonstrate evidence of effectiveness before programs have reached this level of maturity.

Raising A Reader (RAR) is an early-literacy promotion program that encourages shared book-reading between children and parents, especially those parents less likely to choose this activity on their own. RAR works with families with children birth to eight years of age. A series of parent training workshops give participants an enhanced understanding of their role, and particularly of the value of encouraging literacy at home by developing a routine of shared reading between parent (or sibling) and child. Each week, classroom teachers send children home with a colorful bag of developmentally appropriate books. RAR also connects children and their families to local libraries, to encourage a lifelong habit of book borrowing. RAR reaches more than 100,000 children annually, across 32 states.

We hear frequently that parents are a child’s first teachers, and a great deal of research has established that what parents do outweighs nearly any other single influence on children. However, good parenting is neither inborn, nor limited to those with “the right” temperament; it can effectively be learned by all parents.

RAR has many of the challenges shared by other child- and family-serving programs: serving diverse populations, maintaining family engagement over time, identifying which program elements are optional and which are essential. Child Trends, and other external evaluators, have worked with RAR in recent years to help them improve their knowledge of what’s working, and refine their programming accordingly. The recent Child Trends research report on this work is an instructive example of how a number of quality-improvement steps prepare the groundwork for a more comprehensive evaluation.

A number of positive findings have already emerged from earlier evaluations. Parents who complete RAR are more likely to share books with their children on a regular basis. And RAR parents report their children are more engaged in shared literacy activities. The Child Trends report concludes that:

• RAR is grounded in a robust body of research that highlights the importance of family involvement in promoting children’s literacy;
• RAR, through numerous small-scale evaluations in a wide variety of sites across the country, has established strong emerging evidence for the effectiveness of its program model; and,
• Building on those previous findings, RAR is preparing for a large-scale, multi-site, impact evaluation, using random assignment, in order to inform its expansion goals.

Evaluation is an ongoing enterprise, not a line in the sand. Some promoters refer to “proven” programs—but science, unlike mathematics, doesn’t deal in proofs; rather, it makes successive attempts to refine and generalize knowledge. The program that worked in Milwaukee did not work in El Paso—the discrepancy becomes the spur to further investigation. Judgment about effectiveness will always be contingent, subject to revision as we gather more information.