Is there a way to fix the marketing director’s approach? More generally, what can you say about trying to create an ordinal measurement scale based on pairwise comparisons?

A few months later, you are again approached by the same marketing director
as in Exercise 3. This time, he has devised a better approach to measure the
extent to which a customer prefers one product over other, similar products.
He explains, “When we develop new products, we typically create several
variations and evaluate which one customers prefer. Our standard procedure
is to give our test subjects all of the product variations at one time and then
ask them to rank the product variations in order of preference. However, our
test subjects are very indecisive, especially when there are more than two
products. As a result, testing takes forever. I suggested that we perform
the comparisons in pairs and then use these comparisons to get the rankings.
Thus, if we have three product variations, we have the customers compare
variations 1 and 2, then 2 and 3, and finally 3 and 1. Our testing time with
my new procedure is a third of what it was for the old procedure, but the
employees conducting the tests complain that they cannot come up with a
consistent ranking from the results. And my boss wants the latest product
evaluations, yesterday. I should also mention that he was the person who
came up with the old product evaluation approach. Can you help me?”

One solution: For three items, do only the first two comparisons. A

more general solution: Put the choice to the customer as one of order-
ing the product, but still only allow pairwise comparisons. In general,
creating an ordinal measurement scale based on pairwise comparison is
difficult because of possible inconsistencies.