Getting Off The Gold Standard
Experts gathered at the Aspen ThinkXChange Design Lab "Counting Change" to discuss metrics for two-generation approaches.
The following blog was originally posted at the Urban Institute’s MetroTrends Blog on October 23, 2012.
Last week, I attended the Aspen ThinkXChange conference, which brought together a diverse group from the Ascend network. Ascend brings together leaders from the worlds of research, philanthropy, media, social services, and government to focus on developing two-generation strategies to promote family economic security. The theme for this year’s conference was using two-generation approaches in innovative ways to help improve the life chances of low-income families. The goal of these approaches is addressing the needs of whole families at the same time, helping adults move toward economic independence while helping their children succeed in school. I was invited because of our work on the HOST Demonstration, which is developing intensive two-generation case management models in public housing to mitigate the consequences of living with trauma and disadvantage and to help bring about better outcomes for both adults and children.
At the ThinkXChange, I participated in a session discussing how we should evaluate two-generation programs like HOST to learn what strategies work—not an easy question, given the complexity and multi-faceted nature of these interventions. Policymakers and scholars often describe the random controlled trial as the “gold standard” for evaluation and insist on seeing results from this type of study before they consider a program or intervention model to be “evidence-based.” But I would argue that this standard makes no sense for evaluating place-based two-generation strategies.
First, these approaches do not lend themselves to “cookie-cutter” models that can be neatly replicated in different settings. Specific programs need to be tailored to meet the needs of very different types of populations—what works for single mothers coming out of prison may not be the right set of services and supports for teen parents working toward secondary education at a community college. Second, many of these interventions, such as HOST or Promise Neighborhoods, are place-based; a study design that calls for randomly assigning whole neighborhoods to different conditions requires both enormous resources and a leap of faith that the communities do not differ in important ways that could undermine the experimental design. Finally, the field of two-generation approaches is still young; most models are not refined enough to justify the kind of investment that a random controlled trial would require.
Acknowledging these challenges does not imply that we cannot conduct rigorous research on two-generation strategies that will yield important evidence about what works. It just means we need to be creative and consciously choose evaluation designs that take the complexity and messiness of the interventions and their community settings into account. For example, the HOST evaluation combines administrative interviews, focus groups, and rigorous collection of program data from participating sites. We use this information to not only monitor progress, but also to provide regular feedback and lessons to the sites so they can continue to refine their efforts. Each site’s program is slightly different, but all start with building a logic model (which identifies and links program goals, activities, and outcomes) that uses a common framework and set of indicators for both adults and children. We track outcomes over time through surveys and program data and are collecting as much administrative data as possible so we can triangulate the information from the survey and field work and help create benchmarks for participating families. Finally, our survey uses items from national surveys or studies of comparable populations as another means of creating benchmarks.
We are not rejecting the “gold standard.” As HOST progresses, we hope to, at minimum, add a matched comparison group at one site. But in the meantime, we are learning an enormous amount of valuable information about what it takes to effectively use public housing as a platform for implementing two-generation strategies—information that is timely, policy-relevant, and helpful to inform this growing and important field.
Related Posts
