Earlier this year, the U.S. Department of Health and Human Services (HHS) stopped recommending that all children get an annual influenza vaccine. Instead, it made the shot a matter of “shared clinical decision-making”—something for parents and a doctor to weigh case by case—citing, among other reasons, a lack of randomized controlled trials proving the vaccine’s efficacy in children, including the very young
After a lawsuit from public health organizations, a federal court blocked it, leaving the previous recommendation in place. But the case is ongoing, and earlier this year President Donald Trump issued an executive order directing the government to treat a recent HHS assessment as a “guiding resource” and to revisit the childhood vaccine schedule. That assessment argues that recent evidence behind annual flu shots for children, much of it from observational studies rather than randomized trials, is thin. While we believe that the evidence base, which in fact includes many clinical trials, is stronger than suggested by that assessment, it is correct to note that observational studies of annual flu shots suffer from various statistical biases.
In a new study published this summer, we showed one way that vaccine efficacy can be reliably measured each season using data we already collect, minimizing statistical bias without running a randomized trial at all. In principle, it’s a measurement that could be repeated every year
Randomized trials have earned their reputation as the “gold standard” because they address a real statistical problem. When we simply compare children who got the flu shot with those who didn’t, for example, the two groups can differ in ways that have nothing to do with the vaccine, like how cautious their parents are or how often they visit a doctor. A randomized trial solves this by assigning the shot purely by chance, so the only way the groups differ is in whether they got a flu shot.
But randomized trials are not the only place to find randomized data. Sometimes the world randomizes people for us by accident.
Young children tend to have their annual checkup around their birthday, and that visit is a convenient moment for a flu shot if the vaccine happens to be available in the pediatrician’s office. Children with fall birthdays, who tend to see their pediatrician in the fall, just as the season’s vaccine arrives, can get it then and there. Children with summer birthdays have to make a separate trip, which many families never get around to doing. And birth month is essentially random when it comes to the flu; there is no biological reason a child born in October should need a flu shot more than a child born in June.
This lottery has real consequences. In a prior study, we showed that among children aged two to five, those with fall birthdays are more likely to be vaccinated, less likely to be diagnosed with the flu, and less likely to have a family member catch it than children with summer birthdays
For this new study, we took advantage of this same randomization, but this time we used it to estimate how effective the vaccine was in each of five recent flu seasons, tracking vaccination and influenza rates among two- to five-year-olds with fall vs. summer birthdays.
In a given flu season, if children with fall birthdays were vaccinated more than children with summer birthdays, but didn’t get the flu less often, it would suggest that the flu shot wasn’t very effective in that season. This could happen if the strains of influenza that the flu shot protected against didn’t end up circulating that season. Alternatively, in a season where children with fall birthdays got vaccinated more and also avoided more flu than the summer-born children, it would tell us the vaccine was effective in that season.
In every season we examined, the vaccine clearly worked as intended: for every 100 children vaccinated because of the timing of their birthday—the children whose shot hinged on that convenient scheduling—there were between 9 and 14 fewer diagnosed cases of influenza, depending on the season.
Advertisement
If children with fall and summer birthdays are truly comparable—as they would be in a randomized trial—we shouldn’t see differences in conditions the flu shot doesn’t prevent. So, we compared their rates of non-influenza infections, like stomach viruses and common colds, and found no difference. That result suggested our flu findings weren’t the result of one group simply seeing the doctor more often, or being more health-conscious than the other
As helpful as accidental randomization can be, producing what are called “natural experiments” like this one, true randomized controlled trials remain the most rigorous form of evidence. But it is simply not feasible to run a trial to settle every question medicine and public health face each year. Trials are slow, expensive, and logistically challenging. And when it comes to research on existing treatments, they can be unethical, since researchers cannot withhold treatments believed to be effective just to keep proving the point.
Advertisement
If the federal government’s concern is that we lack fresh randomized evidence that a long-established treatment is effective, the solution isn’t to stop the treatment and wait for a trial that may never come. Using the enormous quantity of data the health care system already generates—that is largely sitting idle and unexamined—to its fullest potential is an excellent alternative. Birthdays handed us a natural experiment that, unlike a randomized trial, didn’t require enrolling thousands of patients, spending millions of dollars, or waiting years to complete.
With a bit of creativity and rigorous statistical methods, that evidence can be drawn from the data we already have in the form of natural experiments. The efficacy of flu shots in children is just one of thousands of questions we could answer this way—no new trial required
