The Misleading Nature Of Media-Hyped Research

Aaron Carroll is an American pediatrician and professor of pediatrics at Indiana University School of Medicine, as well as the Vice Chair for Health Policy and Outcomes Research and the Director of the Center for Health Policy and Professionalism Research. He favored the New York Times with an unusually clear and unbiased explanation of why so much “consensus” research used to panic the public is dubious, and mirabile dictu, they published it. For some reason, however, it ended up inside the Times Business section, despite Times having a perfect forum for it, its weekly Science insert.

I’m going to apply Hanlon’s Razor and attribute this to lunk-headedness rather than sinister instincts, even though Carroll’s observations clarify much of what’s wrong with “climate science.” Professor Carroll’s specific complaint involves the myths, as he calls them, declaring that diet soda is deadly, but his points apply to other scientific research and public opinion manipulation as well. Among them:

The public’s fear of “chemicals”

“Everything is a chemical,” Carroll writes, “including dihydrogen monoxide (that’s another way of saying water). These are just words we use to describe ingredients. Some ingredients occur naturally, and some are coaxed into existence. That doesn’t inherently make one better than another.”

[As an aside, the same kind of intentional confusion occurs regarding the term “drugs.” I saw a TV ad last might for melatonin tablets that repeated over and over that the pills were “100% drug free.” Melatonin is a hormone, and hormones are drugs, defined as any substance “that causes a change in an organism’s physiology or psychology when consumed.” Ah, but chemicals and drugs are scary.]

“Soda is an easy target”

Like energy companies and guns, to name two other examples.

Carroll observes, “It’s true that no one “needs” soda…but there are many things we eat and drink that we don’t “need.” We don’t need ice cream or pie, but for a lot of people, life would be less enjoyable without those things.”

In fact, we do need fossil fuels.

“Scientists need to publish to keep their jobs”

I’m a professor on the research tenure track, and I’m here to tell you that the coin of the realm is grants and papers. You need funding to survive, and you need to publish to get funding.

“As a junior faculty member, or even as a doctoral student or postdoctoral fellow, you need to publish research, Carroll writes. “Often, the easiest step is to take a large data set and publish an analysis from it showing a correlation between some factor and some outcome…That’s how we hear year after year that everyone is dehydrated and we need to drink more water. It’s how we hear that coffee is affecting health in this way or that. It’s how we wind up with a lot of nutritional studies that find associations in one way or another. As long as the culture of science demands output as the measure of success, these studies will appear. And given that the news media also needs to publish to survive …we’ll continue to read stories about how diet soda will kill us.”

The news media is excessively impressed by prestigious institutions

Carroll: “To do the kinds of analyses described here, you need large data sets that researchers can pore over. Building the data set is the hardest part of the work….Because of this, a few universities produce a disproportionate amount of the research on these topics. They also tend to be the universities with the most resources and the most recognizable names….They also get more media attention because of having access to more researchers, prestige and funding. If the research is coming out of a super-respected institution, it must be important.”

The limitations of observational studies

No matter how many times you stress the difference between correlation and causation, people still look at “increased risk” and determine that the risk is causing the bad outcome. …With respect to diet sodas, it’s plausible that the people who tend to drink them also tend to be worried about their weight or health; it could be a recent heart attack or other health setback that is causing the consumption rather than the other way around. But you shouldn’t assume that diet sodas cause better health either; it could be that more health-conscious people avoid added sugars….Dr. John Ioannidis wrote in a seminal editorial: “Individuals consume thousands of chemicals in millions of possible daily combinations. For instance, there are more than 250,000 different foods and even more potentially edible items, with 300,000 edible plants alone.”

And yet, he added, “much of the literature silently assumes disease risk” is governed by the “most abundant substances; for example, carbohydrates or fats.” We don’t know what else is at play, and using observational studies, we never will”…

Moreover, too many reports still focus only on the relative risk and not on the absolute risk. If a risk increases by 10 percent, for example, that sounds bad. But if the baseline risk is 0.1 percent, that 10 percent increase winds up moving the baseline to only 0.11 percent.”

The point is not that research regarding diet soda is identical to, for example, climate change research, but that most research is more complicated and less certain in its conclusions than the news media reports or the public understands. A healthy degree of skepticism is warranted, not just because of researcher bias, but because of research methodology as well.

Carroll concludes, “It would probably be a public service if we stopped repeating a lot of this research — and stopped reporting on it breathlessly. If that’s impossible, the best people can do is stop paying so much attention.”

This is a valuable message, as are all of his observations. The article is exactly the kind of information the layman public requires to put all the scientific hype into perspective. Burying it in the New York Times Business section is like putting the World Series game report in the Food section.


20 thoughts on “The Misleading Nature Of Media-Hyped Research

  1. You’re more charitable than I am. Burying stories that are in disagreement with progressive cant or flattery to their political or social issue opponents is a tool of the so-called resistance to control the narrative. It would not surprise me if the Times recognize the climate change applications when it comes to questioning constant academic and scientific articles and decided to be safe by burying it where nobody would expect to see it.

  2. I read the food section far more often than the editorial. A good recipe is often the best thing on a particular day. A skeptical eye can’t make the cake rise. Climate panic is the same as flat-Earth theories to me.

  3. Great article. Great post.

    If only some would engage with critical thinking and a tinge of common sense most baiting misinforming headlines would be ignored.

    Great find! Thanks.

  4. I am glad this got printed it as I wrote something similar to this years ago.

    Grant writers know what appeals to reviewers or funding sources. What bugs me to no end is when industry sponsored research is called dubious if it benefits the industry while government funded research is considered to be bias free. The government reviewers are selected from scientists who are themselves compromised because they were selected BECAUSE they have staked their reputations on a given set of facts that if challenged undermines their research. If one presents a grant app that postulates something that if proven negates their theories that grant app will never get past the first cut.

    More insidious are reviewers whose research resulted in patents and royalties to them. Any novel approach that could prove to be a more effective treatment than theirs will result in what I call the Galileo treatment; the applicant’s scientific credentials are called into question.

    There is a good reason why so many climate scientists agree. They need to keep getting paid.

    • GREAT post Chris, but at the end of the day “the Galileo treatment” (IMO) wasn’t so awful bad; it was just house arrest…and Fire-n-Brimstone humiliation/condemnation.

      Turns out the treatment of his pal/contemporary, Giordano Bruno was less than good, am I right?

        • ”Being tried for heresy then was a pretty big deal.”

          Didn’t intend to diminish it; then, as now, you don’t buck settled science without paying a price, am I right?

          I chuckle when people claim how strong and influential they believe The Church is today; heck, at least they don’t burn you’s at the stake in a pitch-soaked shirt.

  5. Content of the post aside, I just want to say that deep in our hearts all scientists- from the highest Nobel Laureate to the lowest lab monkey- picture ourselves just like the image for this post.

  6. The author is being too kind. Much of the abuse has to do with the biology and medical fields use of ‘p-values’. Basically, if a test shows that there is less than a 5% chance that the data can be explained by random chance, it can be published. I can often tell these types of stories by the headlines. “Hot Chocolate Found to Increase Risk of Throat Cancer” news at 11. Why? Well, my suspicion is that they sampled people who drank hot beverages and didn’t get a good enough p-value. Then they looked at the coffee drinkers…not a good enough p-value. Then the hot tea drinkers…not a good enough p-value. Then the hot chocolate drinkers…aha! A good enough p-value! Publish it. Remember, the threshold is only a 1 in 20 chance. If you look at your data 20 different ways, you are likely to find one of those ways has a good enough p-value.

    • And when you start to work with extremely small probabilities, the P-value can be extremely inflated- when something increases from a one-in-a-billion chance to a 3-in-a-billion chance, the P-value will often indicate a significant change even though your average person wouldn’t consider that noteworthy. That’s misleading enough even if reported accurately, but a situation like that will often be reported as “Hot chocolate triples the risk of throat cancer!” instead of the more accurate “hot chocolate increases the risk of throat cancer by 0.000000002 percentage points!”

      A word in defense of science, though: In a study like you’re talking about, where you look at a huge raft of different conditions and see if any of them look significant, it’s not generally the case that the purpose of the test is to pretend they were looking for the risk of one of them all along. It’s much more efficient to test 50 things at once, see if any of them look significant, and THEN go back and do a study focused on that one thing to see if the findings hold up. Doubtless some scientists are content to mislead, but an awful lot of us are frustrated when the press holds up an interesting prospective study that suggests further lines of testing as “proof that hot chocolate causes throat cancer.”

      • I was talking about the practice of ‘p-hacking’ where people go through existing data to try to find a sub-group with a good enough p-value to publish. If you check enough sub-groups, you are almost guaranteed to find something publishable. Although it should be obvious that this is unethical, there are researchers who consider it a legitimate exercise.

  7. A number of years ago, I ran across a story headlined rather vociferously that researchers had found a positive point 4 correlation between the number of hot dogs a child consumed and the incidence of childhood leukemia. A6t no point did the story mention that point 4 is called a “universal correlation”. That is to say that an observational study will likely provide a point 4 correlation between ANYTHING and anything else…rainfall in Bombay and crime rate in New York City, positive point 4. I don’t know who I trust least, the researchers or the media reporting this garbage.

  8. Perusing Facebook, it seems like a lot of articles contain “science” in the headline. “Metallica is better than Slayer-and science tells us why.” I could come up with all kinds of goofy headlines (I mean, cuz, we all know Slayer is better, right?), but the point is made. If you have not noticed this, watch for it.

    I think “science” has become the new fallacy. If you can say “science says this,” your argument is complete; anyone who disagrees with you is a “denier,” a Luddite, and probably a Republican. it really is annoying because the uttering of the word signals that there is no need for further thought.

    (Brief digression, if the previous comment was not. Mentioning dihydrogen monoxide was amusing. Some guy did a petition on some college campus to ban dihydrogen monoxide and got lots of signatures. he may have even complained that it was in the water on campus. Apart from being reminded of that, I was reminded of some stupid thing on Facebook that said margarine (or something) was one atom different from being plastic. That might be true, but what am I supposed to gather from that? Water is one atom away from hydrogen peroxide, but that one atom makes a all the difference with respect to drinkability.)


    • Regarding “margarine is only one atom away from being plastic,” the best breakdown I ever heard of that was:

      Water is one atom away from toxic hydrogen peroxide.
      Table salt is zero atoms away from poison chlorine gas and explosive sodium.
      A farm is one letter away from a fart.

      Who cares?

  9. Is this a great commentariat or what? Here’s a statistic I’d like to know: the number of journalists who have taken a course in statistics.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.