Do Wine Judges and Critics Lack Consistency?

By Julianna Hayes
If you’re the kind of wine consumer who relies on critical reviews or gold medals to decide what to drink, consider this: apparently, so-called experts like me can’t always tell when they’re sipping the same wine over and over again.

A four-year study published last week in the Journal of Wine Economics revealed that only 10 per cent of judges were able to consistently give the same rating, or something very close, to the identical wine sampled multiple times in a large blind tasting.

Even more unsettling is the fact that another 10 per cent of judges at the California State Fair gave the very same wine far different ratings, ranging from deserving of a gold medal to meriting no medal at all.

Now you might think that the small percentage who gave consistent scores would be considered “super judges” and would be the “go-to” people for future competitions and ratings, but you’d be wrong. The study also found that these judges didn’t maintain their consistency from year to year.

Yikes.

In the wine-soaked movie, Sideways, the character Maya spoke about wine being a “living thing” that is constantly changing. “I love how wine continues to evolve, how every time I open a bottle it's going to taste different than if I had opened it on any other day. Because a bottle of wine is actually alive.”

Though Maya may have been a fictional person, what she said is, in fact, true. And it accounts for how judges in one competition could assess a wine differently months later in another. But it can’t easily explain away how a wine can be so profoundly different to the same taster on the same day.

"Consumers should have a healthy skepticism about the medals awarded to wines from the various competitions," said Roberts Hodgson, a retired Humboldt State professor, who conducted the study.

He also said he doesn't have any more faith in the 100-point-scale ratings of wines in magazines, newspapers and newsletters, like the ones published in this column.

I have always said consumers shouldn’t be distracted by shiny hardware or allow reviews to dictate their buying patterns. That’s because I want them to have the confidence to trust their own palates and decide for themselves what they like. But I realize the findings of this study are troublesome.

There are, however, some explanations. Despite its glamorous image, if you’ve ever had the opportunity to observe a wine judging, or better yet, serve as a “ghost judge,” you’d understand how gruelling an ordeal it is. It’s not uncommon for a taster to sample in excess of 100 wines in a sitting. Over time our once fresh palates will become weary and battered, even though we’re spitting and cleansing with water regularly. I leave the judges’ chambers with teeth befitting a ghoul and a tongue that tastes like my gym socks.

I’ve noticed throughout the exercise that my taste buds start to get numb, but my sense of smell gets more heightened. Thus I suspect if a wine I tasted earlier in the competition appeared again before me later in the day, I might notice less on the palate but more in the nose and those differences may or may not do the wine justice.

And like how food will change how a wine tastes, sampling other wines will affect our perspective of a given wine’s characteristics.

Consumers may be alarmed by these factors but they should consider a few things before dismissing the worthiness of all those gold medal wines they’ve invested in over the years. First of all, no wine wins or loses on the say so of a single judge. There has to be a majority consensus of the panel – usually consisting of a half dozen or more judges – for a bottle to score gold.

The second consideration is that most wines aren’t awarded the top prizes based on just one taste. Typically, wines that show well get pushed forward for a second tasting, usually the next day when the judges’ palates will be refreshed and they’ll also be trying a new bottle.

Is the situation ideal? No, but wine, and one’s perspective of it, is subjective – I’ve never pretended otherwise.

The study has prompted California State Fair officials to consider changes in the way they operate future wine competitions.

They plan to reduce the number of wines sampled per day -- currently is 150 or more -- to around 75 to in hopes of avoiding palate fatigue. They also want to start weeding out judges who demonstrate a lack of consistency year after year.

Hodgson, who taught oceanography and statistics at the university, now owns Fieldbrook Winery in Humboldt County. He designed the study because he didn't understand why "we would have wines that we sent off and would get gold medals in some competitions and in others would get poop. It seemed like a gold medal was just a matter of luck."

But the study’s findings have created a bit of a quandary for Hodgson, who has been using medals to sell his wines.

“And now I have written this paper saying the wine competition system that awards those medals isn't perfect."


Wine Notes

Township 7 2006 Chardonnay
Aromas: Butter, orange rind, vanilla, tropical fruit, peaches, lemon oil, some mineral
Flavours: Fresh tropical fruit, citrus rind, butter, spice, caramel
Body and Finish: Bright ripe fruit entry, soft and round with a layer of butter on the palate, clean finish
Overall Impression: Consistently well made year after year. Definitely for those who like some oak that is not overdone
Would I Buy It? Yes
Cellaring Potential: Drink now
Score: 88
Price: $20
Availability: BC LDBS, VQA shops, private retailers


Mission Hill Five Vineyards 2006 Cabernet Merlot
Aromas: Chocolate covered cherries, cassis, menthol, earth, pepper, coffee bean
Flavours: Bright, jammy fruit, pepper, spice, black cherry, dusty chocolate, herbaceous notes
Body and Finish: Bold entry with nice weighty texture on the palate, moderate tannins, slightly spicy finish
Overall Impression: Budget-friendly wine is stylish and shows good depth
Would I Buy It? Once in a while
Cellaring Potential: Drinkable now, cellar up to five years
Score: 88
Price: $19
Availability: BC LDBS, VQA shops, private retailers


Zero Balance 2007 Project
Aromas: Quite aromatic with peach, nectarine, marmalade, floral and spicy notes
Flavours: Peach, apricot, nectarine, spice, some mineral
Body and Finish: Racy entry with luscious fruit and good mid-palate acidity
Overall Impression: A bright, easy to drink wine with some residual sugar making it a good choice for Asian fare – Another Holman Lang enterprise on the Naramata Bench
Would I Buy It? Occasionally
Cellaring Potential: Drink now
Score: 86
Price: $18
Availability: BC LDBS, VQA shops, private retailers