MaxDiff Analysis (Best-Worst Scaling)

How MaxDiff works

MaxDiff, short for maximum difference scaling and also called best-worst scaling (BWS), measures the relative importance or appeal of a list of items. Instead of asking people to rate each item on a scale, a MaxDiff survey shows small sets of items, typically four or five at a time, and asks the respondent to pick the best (most important, most appealing, most likely to make them buy) and the worst (least important) item in each set. Each respondent sees many sets, built from an experimental design so that every item appears about equally often and next to many different partners.

The method was developed by Jordan Louviere in the late 1980s and has become one of the most widely used techniques in marketing research because it fixes the main weakness of rating scales. When people rate items one at a time, they tend to call almost everything “very important”, use the scale differently from one another (some never use the top box, others always do), and give answers that do not discriminate. A forced choice between items removes these problems: respondents have to trade items off against each other, there is no scale to interpret, and a choice of “best” and “worst” is easy and quick to make even on a phone.

Every best-worst answer contains a lot of information. With four items A, B, C and D, choosing A as best and D as worst tells you that A beats B, C and D, and that B and C both beat D: five of the six possible pairwise comparisons from a single click.

What this MaxDiff calculator estimates

The tool reports two families of results, so you can see that they agree.

• Counting analysis. For each item it counts how many times it was shown, chosen best and chosen worst. The B−W score is (times best − times worst) ÷ times shown, which runs from −1 (always worst) to +1 (always best). The tool also computes B−W scores for each respondent separately and averages them, gives their standard deviation (how much people disagree), and shows the share of respondents for whom each item is the top item.
• Multinomial logit utilities. The best and worst choices are analysed together with a sequential best-worst conditional logit model, the same model that Sawtooth Software and the R package support.BWS use for aggregate estimates. A best choice is modelled as a choice among the items shown with utilities u; a worst choice as a choice among the same items with utilities −u. The model is estimated by maximum likelihood, utilities are zero-centred so they sum to zero, and standard errors come from the information matrix.
• Shares of preference. Utilities are on a log-odds scale that is hard to explain to clients, so the tool converts them to shares of preference: e^u ÷ Σ e^u. Shares add to 100% and have a direct meaning: the probability that an item would be chosen as best if all items were shown together. Ratios of shares are ratios of choice probabilities, so an item with a 20% share is twice as preferred as one with 10%.

Worked example with the built-in data

The example data set simulates 120 bank customers who each completed nine MaxDiff tasks with four of twelve banking features per task (1,080 choice sets, 4,320 rows). The model is highly significant (LR χ²(11) = 725.8, p < .001). Strong security comes first with a share of preference of about 18%, followed by Low monthly fee (about 15%) and Mobile app quality (about 13%). Ethical investments is last with about 2%, so in a head-to-head choice security would be chosen around eight to nine times as often. The B−W count scores rank the items in almost exactly the same order, which is what you expect with a balanced design and shows the result is robust to the method of analysis. The individual-level standard deviations of about .4 show that preferences differ quite a lot from person to person, a sign that segmentation could be worthwhile.

Designing a good MaxDiff study

• Number of items. MaxDiff works best with roughly 8 to 30 items. With fewer than six items a simple ranking question is enough; with more than about 30 the survey gets long, and sparse or express designs are needed.
• Items per set. Show four or five items per set; three is too few to be efficient and six or more make the choice tiring. A common rule is to show no more than half of the items in one set.
• Number of sets. Aim for each item to appear at least three times per respondent, so the number of sets ≈ 3 × items ÷ items per set. Twelve items in sets of four gives nine sets, as in the example.
• Balance. Use a balanced incomplete block design or a randomised near-balanced design so that each item appears equally often, in different positions, and with different partners. Most survey platforms (Qualtrics, Sawtooth, Conjointly, SurveyMonkey add-ons) generate such designs and export the data.
• Item wording. Items must be comparable, of similar length and on the same level of abstraction. Avoid double-barrelled items (“fast and friendly service”) and items that are logically nested.
• Sample size. For stable aggregate scores, 150 to 300 respondents is usually plenty; if you want to compare segments, plan about 100 per segment.

Preparing your data

Most platforms export MaxDiff in a wide layout with one row per respondent. Reshape it to the long layout this tool uses: one row per item shown in each task, with columns for the respondent ID, the task number, the item name, and a response coded 1 for best, −1 for worst and 0 for the other items in the set. If your export uses other codes (for example “B”, “W” and blank), simply choose those values in the “best” and “worst” drop-downs. Sets that do not contain exactly one best and one worst answer, or that repeat an item, are excluded automatically and reported in a notice.

How to interpret and use the results

Lead with the shares of preference: they are intuitive, sum to 100% and are comparable across items. Report the ranking, then highlight large gaps rather than small differences between neighbouring items; items whose utilities differ by less than about two standard errors are not clearly different. Use the count table as a check and to explain the method to non-technical readers. The individual B−W scores and “ranked first by” column show whether the population is homogeneous or whether different groups want different things. If the spread is large, export the individual scores and run a cluster analysis or a latent class analysis to find segments.

Typical applications are prioritising product features for a roadmap, choosing which benefits to lead with in advertising, testing claims, names and packaging messages, ranking reasons for churn or purchase, and prioritising public services or policy options. Because MaxDiff gives ratio-scaled preferences, it is also a good input to TURF analysis when you need to pick a small set of items that appeals to as many people as possible.

Common mistakes to avoid

• Reading utilities as absolute importance. MaxDiff is relative: it tells you which items matter more than others in the list, not whether any of them matter at all. Anchored MaxDiff adds a question about absolute importance if you need it.
• Comparing utilities across separate studies with different item lists. The scale depends on the items included; compare shares only within the same list.
• Treating tiny differences as real. Look at the standard errors and the count scores before claiming that item 5 beats item 6.
• Using an unbalanced design, for example one where some items appear far more often than others; the logit model adjusts for this, but precision suffers.

MaxDiff compared with other methods

Compared with rating scales, MaxDiff discriminates far better and is immune to scale-use bias. Compared with ranking all items, it is much easier for respondents when there are more than six or seven items. Compared with conjoint analysis, MaxDiff evaluates single items rather than combinations of attribute levels, so it cannot estimate trade-offs between, say, price and features; use the choice-based conjoint tool for that. Hierarchical Bayes estimation, available in commercial software, gives individual-level utilities; the aggregate logit and individual count scores here answer most business questions and match commercial aggregate results.

Key references: Louviere, Flynn and Marley (2015), Best-Worst Scaling: Theory, Methods and Applications; Finn and Louviere (1992), Journal of Public Policy & Marketing.