From soda bans to bike lanes: Which 'natural experiments' really reduce obesity?

To answer that question, many public health researchers take advantage of "natural experiments"--looking at people's calorie consumption or physical activity levels, either comparing before and after a policy or environmental change, or comparing against a similar group of people not affected by that change. But not all natural experiments are created equal.

"Rigorous science is needed to evaluate these natural or quasi-experiments," said Amy Auchincloss, PhD, an associate professor at the Drexel University School of Public Health who was a member of a research team that authored a new study published online ahead of print in Obesity Reviews: "Impact of policy and built environment changes on obesity-related outcomes: a systematic review of naturally occurring experiments." The review was led by Stephanie Mayne, a doctoral student supervised by Auchincloss, and also co-authored by Yvonne Michael, ScD, an associate professor and associate dean for academic and faculty affairs in the Drexel University School of Public Health.

The Drexel team reviewed the state of the science on this topic, evaluating the results and methods of all previous such studies published in the medical literature, in particular:

1. Which policies and built environment changes have been evaluated via natural- or quasi-experiments and what are the results from these studies?
2. Are there issues of concern with the studies' design, including methods of assessment?
3. What are the limitations of these studies and areas where additional science is needed?

This is the first review that has examined the use of natural- or quasi-experiments to evaluate the efficacy of policy and built environment changes on obesity-related outcomes (body mass index, diet or physical activity). The review included PubMed (Medline) articles published 2005-2013; 1,175 abstracts and 115 papers were reviewed and ultimately 37 studies were included in the review.

The review identified certain types of interventions that are more successful than others in improving obesity-related outcomes, and identified areas where more research is needed to draw conclusions about obesity-related outcomes:

Diet & Food Policy Changes

Changes with strong impacts were ones that improved the nutritional quality of foods:

• Trans-fat bans
• Sugary food and beverage availability limits
• Higher-fat food availability limits

Changes that had smaller or no impacts in the research to date included:

• Nutritional information requirements
• Supermarkets built in underserved areas

Physical Activity Focused Changes

Changes with stronger impacts included:

• Active transportation infrastructure improvements
• Changes studied after longer-term follow-up periods

More research is needed to look at physical activity effects (not just use of amenities) for built environment changes including:

• Park improvements
• Trails
• Active transportation infrastructure

The researchers noted that a common shortcoming in many studies is that they only measured process outcomes such as food purchases or use of bike/transit infrastructure, rather than measuring the desired health outcomes, such as weight loss.

"Research suggests that people will use new amenities like bike shares, and limit purchases of unhealthy foods in specific contexts like schools," said Mayne. "But it is less clear whether these changes translate into overall improvements in diet and physical activity."

Likewise, only a few studies directly assessed impacts on BMI or weight; thus, the authors concluded that evidence is lacking on whether environmental and policy modifications are successful in maintaining healthy weight or reducing excess weight.

A key value of a natural experiment is that it can narrowly focus on the direct impact of a change in policy or infrastructure on an affected population--making natural experiments an important way to check on what kinds of public policies and investments make real-world impacts on health, and to what degree. The authors concluded that more natural experiments are needed to strengthen the evidence base about obesity-related policies and interventions. They also recommend more natural experiments to explore whether the timing of a change or repeated exposure to the changed condition enhances or reduces impacts on obesity-related outcomes.

The authors generally found stronger results in studies that had longer follow-up periods after a policy change or other intervention.