How well do vaccines work? New technique offers greater insight

“Synthetic controls” give health researchers a better statistical tool.

Roheeni Saxena – Feb 8, 2017 4:55 pm | 45

To determine if new vaccines are effective, researchers often closely monitor trends in disease rates for a city or community. However, these observations can be confounded by changes in the health or behavior of the population, so a better “control” comparison is needed. One social science comparison technique called “the synthetic control method” presents a unique way to address this problem: combine the information from several possible control communities to create a superior aggregated control built from many possible controls.

Using this method, authors of a recent paper in PNAS found that vaccines for pneumonia significantly reduce pneumonia disease incidence. This paper works on two levels: it provides compelling evidence that vaccinations against pneumonia are effective in specific populations, and it shows that this social science technique can be effectively used to study human health.

Previous studies that attempted to examine the effects of pneumonia vaccines started with a comparison between the pre-vaccination rates of pneumonia and another infection. These were compared to post-vaccination differences in the rates of the same two diseases—a technique called difference-in-difference. If the ratio of the infections changes in the appropriate direction, it’s a sign the vaccine is effective.

Historically, the difference-in-difference technique is needed because intervention studies such as a vaccine implementation trial do not necessarily have a randomized control aspect. Controls allow scientists to see the effect of an intervention by comparing a community that received the intervention to a “control” community that did not. But if a vaccine is known to be effective, it is often difficult to ethically justify denying the vaccine to a control group. That makes it really hard to see just how effective the vaccine is.

Difference-in-difference provides some indication of a vaccine’s potency. However, the authors of the new paper say that there hasn’t been a standardized methodology for selecting comparison diseases, so this method can’t give consistent estimates of vaccine effects.

“Synthetic controls” are a comparison method typically used for website analytics and economics research. The technique involves combining various measurements of different “control” populations at different time points. Each contributing control comparison is then weighted according to how closely it is associated with the outcome of interest. In this case, if some other infection consistently varies with pneumonia rates, it would receive a heavier weight. This process results in a “synthetic control”—that is, a composite of several individual control measures.

Synthetic controls have been applied successfully to some public health studies, including assessing how tobacco control programs effect smoking rates, so there is precedent here.

The authors of the new paper used data from electronic healthcare databases in Brazil, Chile, Ecuador, Mexico, and the US to examine the effects of the pneumonia vaccine and constructed synthetic controls from these same databases. Their analysis showed that the introduction of the pneumonia vaccine resulted in significant declines in hospitalizations for pneumonia among children, but it did not make a difference in hospitalizations among 65-79 year olds. That’s a significant finding, since it can tell us how to most efficiently deploy public health efforts.

The authors then validated their synthetic control by comparing the results to those produced using other approaches. While other models yielded similar results, the synthetic control was more sensitive, able to pick up an effect in children that some of the validation models rated as statistically insignificant.

This provides compelling evidence that the synthetic control technique can be used by public health researchers and that it avoids some types of bias and confounding. Despite the complex math involved in this type of study, the authors did their calculations using a publicly available package for the open-source statistical software R. They even released R programming code for implementing synthetic controls using administrative hospitalization data.

Since use of synthetic controls in public health is a new application of this method, replication of these findings is definitely needed before this technique can be more widely accepted. Having access to the software should help.

PNAS , 2017. DOI: 10.1073/pnas.1612833114 (About DOIs)