For the original product evaluation scheme, the overall rankings of each product variation are found by computing its average over all test sub- jects. Comment on whether you think that this is a reasonable ap- proach. What other approaches might you take?

A few months later, you are again approached by the same marketing director
as in Exercise 3. This time, he has devised a better approach to measure the
extent to which a customer prefers one product over other, similar products.
He explains, “When we develop new products, we typically create several
variations and evaluate which one customers prefer. Our standard procedure
is to give our test subjects all of the product variations at one time and then

7
ask them to rank the product variations in order of preference. However, our
test subjects are very indecisive, especially when there are more than two
products. As a result, testing takes forever. I suggested that we perform
the comparisons in pairs and then use these comparisons to get the rankings.
Thus, if we have three product variations, we have the customers compare
variations 1 and 2, then 2 and 3, and finally 3 and 1. Our testing time with
my new procedure is a third of what it was for the old procedure, but the
employees conducting the tests complain that they cannot come up with a
consistent ranking from the results. And my boss wants the latest product
evaluations, yesterday. I should also mention that he was the person who
came up with the old product evaluation approach. Can you help me?”

First, there is the issue that the scale is likely not an interval or ratio
scale. Nonetheless, for practical purposes, an average may be good
enough. A more important concern is that a few extreme ratings might
result in an overall rating that is misleading. Thus, the median or a
trimmed mean (see Chapter 3) might be a better choice.