Can You Spot What’s Misleading And Incompetent About This “Study”?

Of course you can!

So why couldn’t the researchers?

An online survey  asked consumers to order virtual meals after randomly looking over menus that either had some form of climate labeling or none at all. 23.5% more of those who ordered from a menu that noted  “the least green” choices made a “sustainable” meal choice than those who ordered from menus without such information.

More than 5,000 adults 18 and older participated in the test in March and April of this year. They were told to imagine that they were at a restaurant ordering dinner. Subjects were randomly assigned to view only one of three menus on which every food option was identified by a photo that could be clicked when placing an order. One menu featured standard, climate neutral codes below each meal photo. Another featured red labels stating “high climate impact” under meals that included beef. A third menu featured green labels stating “low climate impact” under those meals that did not include beef.

The researchers concluded that both the high and low climate impact menu labels were effective at encouraging more sustainable food selections compared to the control.

No, they were not. The study literally proves nothing about actual food selection, because the study involved theoretical food choices only in which the real life consequences most central to a restaurant experience were conspicuously missing. The subjects picked food knowing that 1) they didn’t have to pay for it and, most importantly, 2) they didn’t have to actually eat it. There was no downside whatsoever to choosing the “sustainable” food. Of course more people decided to be “environmentally responsible.”

Hell, I’ll choose a plant-based burger hypothetically all day long—just so I know I’ll never have to eat the real thing.

Studies like this are constructed to support agendas, and then are cited by journalists and policy-makers to justify dubious laws and regulations. This is an excellent example of why the public increasingly doesn’t trust “science” and research. Scientist and researchers are too often untrustworthy.


20 thoughts on “Can You Spot What’s Misleading And Incompetent About This “Study”?

  1. In all honesty, I’ve had some plant-based burgers I actually enjoyed. No, they didn’t taste like a proper hamburger, but as a faux hamburger goes, they were fine.

    As long as you understand you’re in for something different, that is. Calling such things “hamburgers” is an offense against reason. And taste. But that doesn’t mean they don’t taste good, just not like a hamburger.

  2. If you measure sustainability by using metrics only found in the choice that you wish to diminish then your study is flawed from the outset.
    Beef production metrics regarding carbon footprint when compared to vegetable based products will always tend to favor vegetable based goods when you fail to factor in the carbon associated with growing and harvesting the vegetable matter. Pounds of protein per ton of carbon released in production is the only way to gauge one versus another but if you must also consider the production footprint required. If it takes 1000 acres to obtain the equivalent yield of vegetable protein from 100 acres for beef protein then you must commit 10 x the land to create sufficient food. One thing is certain, land for agriculture will not grow because they are not making any more of it. Sustainability means more than just minimizing carbon. Land must lie fallow to keep it productive and cattle fed on grass lands helps preserve the productive capacity of the land.
    The other day someone here explained how the flavoring is produced which involves some rather elaborate synthetic processing. Ironically, those most concerned about sustainability are also worried about genetically modified foods yet seem unfazed by the complex processing of this product to obtain a meat flavor.
    I have absolutely no technical expertise in this area but I do know you cannot apply metrics for one that cannot be used by another to determine which one is “better”.

  3. I think studies and reports like this are great. It gives you people/publications/researchers that you can cross off your ‘trusted’ list without hesitation!

  4. The plant-based burgers, etc. are just a transition phase on our way to their real goal of having us eat bug-based foods. I’m considering a career as a “meat bootlegger” once they outlaw meat. “When meat is outlawed, only outlaws will eat meat!”

  5. How did the study decide who should participate? Solicit people visiting enlightened websites? Salon? Vox? The Atlantic?

  6. Hell, I’ll choose a plant-based burger hypothetically all day long—just so I know I’ll never have to eat the real thing.

    There’s a related story about two diggers in the trenches. One had just heard the socialist dictum about “from each according to his abilities, to each according to his needs”, so he asked his mate a few questions inspired by that:-

    Digger 1: “If you had two hundred pounds, would you give me one hundred?”
    Digger 2: “Yes, of course.”
    Digger 1: “And if you had two horses, would you give me one?”
    Digger 2: “Yes, of course.”
    Digger 1: “And if you had two shirts, would you give me one?”
    Digger 2: “Hell, no, I’ve got two shirts.”

        • It’s why capitalism is the only viable economic system. Capitalism relies on the dominant human characteristic: self-interest. There’s simply nothing better to rely upon.

          • It’s why capitalism is the only viable economic system. Capitalism relies on the dominant human characteristic: self-interest. There’s simply nothing better to rely upon.

            And that is an example of a certain kind of fallacy I recently pointed out in relation to the reasoning of some anarchists. They (rightly) point out the defects of various forms of government, both those on offer and hypothetical ones (though perhaps not all of those), and then (wrongly) infer that that on its own is enough to make non-government workable. Well, it might separately be workable for yet other reasons, but that approach neither explores what else might be needed for that nor shows that there are realistic ways to make it work. And it is quite possible that neither government nor anarchism would work, at any rate among the deliverable forms of either.

            Similarly, you are reasoning from the defects of socialism and similar that capitalism must work. You, too, are (wrongly) inferring that capitalism must work because you have (rightly) seen that the alternatives do not (at any rate among those you have looked at). Remember, too, that socialists often reason the same way from the defects of “actually existing capitalism”.

            However, both instances of the reasoning ignore the possibilities that neither kind of alternative could ever really work, or that you have overlooked possible solutions of both kinds (“socialism has never really been tried”/”capitalism has never really been tried” – true even though only relevant to that sort of wider search, not to criticising what is on offer, i.e. to criticising “actually existing socialism”/”actually existing capitalism”). In fact, that sort of “if not this then that” reasoning is only ever sound on the back of some sort of existence proof that there is one and only one right answer, combined with a demonstration that the other alternatives do not pass muster (this may or may not need an exhaustive search of the space of potential solutions). But maybe nothing truly works, and the least worst available is something that limps along for a while.

  7. What we have here is the Streetlight Effect, named for the old joke:

    A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, “this is where the light is”.

    The effect in sciences describes when people want to know about a subject that would be difficult to study, whether because of money, time, population, or ethics. So instead they design a study that’s easy but doesn’t actually tell them what they wanted to know.

    A good example is that obviously people would like to know if children playing violent video games has an effect on their attitudes towards real violence as adults. So there’s a large body of research where scientists make adults play a violent video game and then give then a survey afterwards, or a few days later. This obviously tells us nothing about children with years of repeated exposure, but it’s hard to get parents to let you experiment on their kids, and even harder to run a study for ten years, so… We know a lot about adults response to violent video games next week! It’s where the light is!

    It’s common in all sciences, but in medicine it’s more likely to show up in sample sizes and population (it’s easy to get smaller samples, and get them from populations of adults who will do a medical study for a small payment. It’s much harder to do a study on children or elderly people, or on hundreds of people, so they rarely do that.) In the social sciences it’s more likely to be a slight of hand like this one, where some gap is covered up by “lets just pretend that behavior in this situation is the same as behavior in a totally different one.”

    Then the media obviously doesn’t help, because even when researches specifically note how limited their study was, the headline writers at news outlets don’t.

    I have a whole rant on how understanding studies and statistical analysis should be the most important thing being taught to everyone in high school today, and this is a small part of the reason why. This is a well known way of cheating — it has a name! — and it sucks up grant money and spits out uninformed people who think they have “science” on their side.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.