August 25th, 2008
Lies, Damn Lies and Surveys
Hilarity. An old High School friend forwarded me a bit of news today about how Wine Spectator recently awarded a coveted “Wine Spectactor Award of Excellence” to a restaurant that doesn’t exist. Apparently, as part of the research for an academic paper he is working on, wine writer Robin Goldstein…
…submitted an application for a Wine Spectator Award of Excellence. I named the restaurant “Osteria L’Intrepido” (a play on the name of a restaurant guide series that I founded, Fearless Critic). I submitted the fee ($250), a cover letter, a copy of the restaurant’s menu (a fun amalgamation of somewhat bumbling nouvelle-Italian recipes), and a wine list.
Goldstein posted the whole sorry tale — including the wine list — about how Osteria L’Intrepido came to be named one of the world’s best wine restaurants. Funny stuff — but don’t just dismiss it as something limited to the world of food and drink. Such “top ten” and “best of” lists are all the rage, but far too often they provide few insights or critical assessments, making them little more than marketing tools for the magazines who publish them.
One of my least enjoyable tasks at Johnson & Johnson is to assess the numerous survey requests the company receives to determine whether participating will be worth the time spent completing the application.
At times, this has been pretty time consuming. It seems like almost every magazine publishes a “best places to work for”-type survey to provide potential job hunters with a list of what the publication deems to be THE BEST company for recent graduates, moms, scientists, engineers, chemists, liberal arts majors, etc., etc., etc…
Some do try to provide a critical evaluation. In these cases, the survey administrators ask hundreds of different companies to complete carefully constructed questionnaires that are then assessed and ranked by a team of experts who compile the list.
But frequently I’ve found others that don’t take the task seriously at all. In these cases, a simple phone call will reveal little or no scientific rigor involved in the survey design or execution and very few participants. It appears that simply showing up will get you on the list.
Rather than aiding and abetting the perpetuation of such bunk, I usually tell such survey organizers thanks but no thanks. The pity is, however, that their findings are rarely challenged, allowing such lists to not only survive but proliferate.