Friday, May 10, 2013

About That Wine Experiment



There’s a story making the rounds again, concerning wine and wine professionals. Chances are you’ve heard it before. Told the shortest way, it’s this: Even so-called ‘wine experts’ can’t tell the difference between red and white wine by taste.

A fuller version is: “An experimenter gave a group of wine experts a red wine and a white wine and had them rate the wines. What they didn’t know is that the ‘red’ wine was the same white wine that was in the other glass, just dyed red. None of them could tell the difference, and they described the red wine as ‘jammy’ and so forth.”

The purported lesson is that wine rankings are complete bunk. (or, if the writer is being more generous, that we are fooled by our expectations and perception) It’s been featured in all kinds of prominent media, like The Atlantic, The New Yorker, and it’s hurling through cyberspace right now because it’s included in an excerpt from David McRaney’s new book. It’s been a popular topic for Jonah Lehrer, who has included it in pieces written in 2007, 2011, 2012, and his book, [no big surprise from that serial self-plagiarizer, I guess?] which were popularly linked to.

The problem, simply, is that the study never showed that.

The study, conducted by Frédéric Brochet in 2001, was part of a larger dissertation on perception and wine tasting, which largely consisted of lexical analysis of published wine reviews using computer software. His work only garnered attention after he submitted it to a wine industry research competition where it won a runner-up prize and the ‘dyed wine’ experiment got picked up by the press.

There are two levels to the inaccuracy of the popular story. The first is that several of the details that have been routinely reported are simply incorrect, having been copied from one article to another. So let’s break them down.

-The most important is that the subjects in this experiment were not, in fact ‘wine experts’. They were undergraduate enology students. They are probably more knowledgeable about wine than the average person, but they were not in any way ‘experts’, or even ‘professionals’.

- It’s simply not true that “Every single one, all 54, could not tell it was white.” as is frequently stated. Even Brochet doesn’t claim that, saying “About 2 or 3 per cent of people detect the white wine flavour”, and the paper that is frequently cited shows that indeed some people gave ‘white wine’ descriptions to the dyed-red wine.

But even more important than those errors, the study never demonstrated that people can’t tell red wine from white.

So let’s actually look at what the study did. As near as I can tell, the TASTING study has never been published in a peer reviewed journals (which itself makes me suspicious of the findings) but the olfactory (smell) component of the study was published in 2001, allowing us to look into the actual procedures. The undergraduate subjects came into the lab one week and were given a glass of a red wine and a glass of a white wine. (both Bordeaux, but the experimental details do not include any label or vintage, so we are unable to judge them) They were supplied with a list of potential descriptive words, and told to make a list of words and phrases that best described each wine, either from the supplied list or in their own words. The following week they return to the lab for another session. They were presented with two glasses, one containing white wine, and the other containing the same wine dyed red. They were then given the list of descriptors that they had used to describe the wines from the previous week, and asked to choose which of the wines in front of them best represented each descriptor. It was a forced-choice setup.

So they were given two identical white wines, different only in color, and asked to assign them descriptors from both red and white wines. With no other difference between them, and being required to assign the red descriptors to one of the wines, most people assigned them to the red-colored wine instead of randomly. This may show that people have a preconceived notion of what red wines should smell like, but it doesn’t come close to showing that people can’t tell red from white once you change the color. A much simpler design would have been to run the second week exactly like the first, seeing what descriptors people would have assigned to the dyed wine without any prompting. That is NOT what this study did, even though that is the popular understanding.

So who’s to blame for this urban legend? Are we to blame Brochet, who was making a name for himself and clearly believes that the wine industry is full of itself? Do we blame Lehrer and his ilk for continually plugging a story that they never bothered to do much research on? Do we blame all of the bloggers that simply repeat the story and link to the same text? The editors and publishers who allow it to go into books without double-checking it? We blame all of them, and I think we blame our current science as pop-culture mindset behind Lehrer, McRaney, and Malcolm Gladwell, where a cute, pithy anecdote to make a point travels farther and has more influence than a properly reported study. I don’t think any of them are bad people, or that they are intentionally misleading readers, but they are intelligent, educated people who hear a story and spread it around, sometimes without checking its veracity. And I think we can expect more from them.

Look, I understand why the story has such strong legs. We love a story that shows that ‘experts’ have no idea what they’re talking about, that shows that they can be easily fooled, and that deflates the perceived pretentiousness of something like wine. It’s a good story. But it isn’t true. So let’s try to set the record straight. Let’s go all Snopes on this story’s ass. Put the link to this any time you see someone claim that crap about ‘even experts can’t tell red wine from white’.



12 comments:

  1. "The undergraduate subjects came into the lab one week and were given a glass of a red wine and a glass of a white wine...both Bordeaux..."

    uh so they were given wine made from the same kind of grapes and they...weren't able to tell the wines apart...

    :|

    ReplyDelete
    Replies
    1. Bordeaux is a region. Not a grape. Several varieties of both white and red are grown in the area

      Delete
  2. Actually, white bordeaux is made from completely different grapes to the red stuff. It's just from the same place.

    ReplyDelete
    Replies
    1. The red wine was white wine dyed red... Did you even read it?

      Delete
    2. They were replying to a different comment by halojones-fan to clarify about the wines.

      halojones-fan "'The undergraduate subjects came into the lab one week and were given a glass of a red wine and a glass of a white wine...both Bordeaux...'

      uh so they were given wine made from the same kind of grapes and they...weren't able to tell the wines apart..."

      They probably won't see your comment as they provided their clarification in 2015.

      Delete
  3. All of that is fine, but it's ignoring Hodgson's test, where a panel of NOT random volunteers or even wine students, but highly respected experts in the field, judged the exact same wine with an average spread of +/- four points? One in ten managed to keep their scores within plus or minus two, but that means that several were even WORSE than plus or minus four.

    Oh, and lest this be regarded as a one-off fluke, they've done the same test for several years now and the results are consistently inconsistent.

    It's not as spectacular as "can't tell red from white", but if even experts can't give two sips from the same bottle the same score within a point or two, that throws the results of all wine competitions where the difference between a "gold" and "bronze" medal is only a few points into question.

    ReplyDelete
  4. Dude, you're being completely biased. First off, you created this blog for the sole purpose of trying to discredit this one experiment based on wine tasting.

    Second, you're lying about the way the test was conducted and about the results as well. If you read the actual paper, the information is clearly stated and it's compatible with what was spread by the midia.

    Also, you mention that enology undergraduates are not wine experts. How can you say that? Most students in this field are wine ethusiasts and drink a lot of it. Many of the come from families with a tradition of wine tasting. They will work on this field one day...

    But i know you're saying that because you believe they don't understand about wine because they're too young, right? You're a age snob as well, ain't ya?

    ReplyDelete
    Replies
    1. Being an ethusiasts and drinking a lot of wine doesn't make you any more a wine expert than playing Pick-up games on the weekends. There wouldn't be Wine Schools if you could do that, and what the fuck does being young have to do with this. And old ass dinosaur wine student is just as green as an infant.

      Delete
  5. Out of curiosity, I read the published study. The conclusion of Morrot's "The Color of Odor" study is about how our sense of vision acts as a force of suggestion on our olfactory senses. We see red wine, so we expect to smell red wine, so we are biased when we judge the scent (even for wine enthusiasts). The "wine experts are a scam" commentary is click-bait BS. If he wanted to say people can't tell the difference between red and white, a simple blind tasting (or smelling) would do that quite decisively. He didn't design the experiment that way because that's not what he was trying to study.

    So no, he didn't prove that all wine tastes the same, or that wine experts are a sham, but he was never trying to do that in the first place. In short, I believe his conclusion that vision affects our olfactory perception of wine WAS shown and it WAS good science but he did not prove this imaginary conclusion that everyone keeps fabricating when they circulate this story.

    I realize that you probably sensationalized your post a little, but you seem ready to trash this guy's dissertation based on a paper I think you misunderstood.

    ReplyDelete
  6. Mat,

    The forced choice issue was even more distorting than you described. The study (page 4) says: ""The subjects were asked to indicate for each descriptor of their list which of the two wines most intensely presented the character of this descriptor. " Given that the subjects saw the color of the wine, and were not told to judge only on the odor, the fake red wine was in fact the correct choice for the red descriptors - the odor did not match at all, but the color implied by the descriptor did, so the red colored wine in actually fact "most intensely presented the character of the descriptor."

    If the published description of the test is accurate, then the data do not support any of the stated conclusions, because the subjects objectively gave correct and accurate responses.

    It is interesting to note that the bar-charts in figure 3 to show that the sham samples were actually juged differently than real red wine. In session 1, roughly 1.3 times as many red descriptors were used as were white descriptors, while in session two, fewer red than white descriptors were used. This means that the subjects did perceive that session 2 was different that session 1. Oddly, the authors do not comment on this aspect of their data which contradicts their conclusion.

    Of course, the typical higher acidity and higher alcohol content of red would be missing in this scent and color only test, so stating that this study had any relevance to actual wine tasting would simply be bad journalism. Bad journalism about bad science. What a sad waste of nice French wines! But if people believe such reporting, it means there will be more nice wine for your and I....

    ReplyDelete
  7. This comment has been removed by a blog administrator.

    ReplyDelete
  8. This comment has been removed by a blog administrator.

    ReplyDelete