Nutrition is a complex field regularly cursed by provocative media headlines that often precede an overly simplistic summary. If you are a savvy consumer of nutrition news, it may be wise to look beyond the catchy headlines to assess the quality of the reported study, evaluating the latest findings in the context of existing evidence on the topic. But what approach do you take when the study itself, by design, is supposed to take existing evidence into account?
“Meta-analyses have emerged as one of the most influential types of research in the biomedical literature, directly influencing health care guidelines and international policies, from the National Academy of Medicine guidelines and the United States Department of Agriculture, up to the Organization for the World Economy’s global economic burden of health forecasts. Cooperation and development,” said Harvard epidemiologist Chan. Dr. Eric Feigl-Ding. “However, because of their power to guide public policy, there are often perverse incentives to distort meta-analysis methods in order to bias their conclusions. »
Feigl-Ding and colleagues at Harvard Chan School of Public Health and George Washington University explored the misuse of meta-analysis in nutrition research in the September 2017 issue of JAMA(2) examining a range of challenges related to combining multiple studies and discussing the impacts when results are the product of flawed methods.
Misuse and pitfalls of meta-analysis
In general, problems arise with meta-analyses when there is too much variation in the studies included: different types of studies, different study methods, different samples of people. Another problem is selection bias, that is, if some studies are intentionally omitted (or included) in the analysis, which can easily cause its conclusion to vary in a certain direction. Having too many differences in a meta-analysis leads to greater variation in the results, which can distort the conclusion or dilute an otherwise meaningful result.
According to the authors, applying meta-analysis to nutritional science adds a unique set of challenges. For example, unlike drug trials where interventions are often comparable, nutrition intervention trials regularly differ in methodology. In observational studies, populations vary widely in their dietary habits (and reporting of those habits), making it difficult to quantify the consumption of many foods and nutrients. Researchers can also use a variety of methods to assess and report eating behaviors.
Another major challenge in nutrition research is measuring the effects of specific foods and nutrients in the context of the rest of the diet. Since “the effects of a given dietary exposure depend on what that exposure is compared to,” the authors note that meta-analyses should ideally include an adequate number of studies focused on a single “comparator” (i.e. i.e. when looking at heart disease risk, all included studies compared red meat consumption to a plant-based food, rather than mixed studies comparing red meat to other meats).
It’s also important to recognize that eating less of one type of food often means eating more of it. If a number of study participants reduce their intake of saturated fats, are they eating more foods containing unsaturated fats, more complex carbohydrates like vegetables and whole grains, or refined carbohydrates and simple sugars? Because of this complexity, the authors say that understanding the effects of specific food substitutions can “lead to more robust and informative results than focusing on the effects of one nutrient or food alone relative to everything else of the diet”.
A need for improvement
So, are meta-analyses useful for advancing nutrition research, or do they do more harm than good? According to Dr. Neal Barnard, assistant professor of medicine at George Washington University and co-author of the article, it depends on the use and quality of the study method:
In theory, combining studies provides access to a larger population, so meta-analyses should have more power than any single study. However, some meta-analyses combine studies with very different populations and methods, and this variability reduces their power, so the true effects may go unnoticed. Suddenly, eating saturated fats or being overweight no longer seems so dangerous. But this is just an artifact of a faulty method.
Given that meta-analyses receive (and will likely continue to receive) widespread media attention and have the power to influence health policy and change clinical practice, the article offers suggestions on how to ensure that meta-analyses meta-analyses are of good quality before their publication:
- Implement a review process by editors with expertise in the topic as well as in designing robust meta-analyses.
- Require meta-analysis authors to confirm with the authors of the original studies that their data were accurately represented.
- Require meta-analysis authors to share their methods and summary data.
- Include original primary data rather than just published summary data.
- Create a registry that actively monitors the conflicts of interest of all published researchers.
The paper’s co-authors and their colleagues are also working on additional guidelines for conducting meta-analyses, as well as digital tools to help address some of the limitations. Additionally, special training of journalists may be needed to better translate the results of meta-analyses and more accurately communicate health messages to the general public.
Dr Walter Willettco-author and professor of epidemiology and nutrition at the Harvard Chan School of Public Health, adds that “meta-analyses will continue to play a valuable role in synthesizing a growing body of evidence in almost every field, and For this reason, it is essential that we improve the conduct and review of these summaries.
Related
Systematic review and meta-analyses (SRMA) have gained popularity in science, including diabetes research. Although well-conducted SRMAs are an indispensable tool for informing evidence-based medicine, the proliferation of SRMAs has led to many analyzes of questionable quality and misleading conclusions. The aim of this article is to provide up-to-date knowledge and a comprehensive understanding of the strengths and limitations of SRMA.
The references
- Lesser, LI, Ebbeling, CB, Goozner, M., Wypij, D. and Ludwig, DS (2007). Relationship between funding source and conclusion among nutrition-related scientific articles. PLoS Medicine, 4(1), e5.
- Barnard ND, Willett, WC, Ding, EL The misuse of meta-analysis in nutrition research. Published online September 18, 2017. doi:10.1001/jama.2017.12083
