Check the reliability of your scales: Lesson 4 of Data Analysis for Your Thesis

A scale score such as “stress” is the average of several questions that are supposed to measure the same thing. Before you average them, you need evidence that they hang together. Internal consistency reliability asks whether people who score high on one item tend to score high on the others. The most widely reported index is Cronbach’s alpha.

What alpha means

Alpha ranges from 0 to 1 (it can even be negative when something is wrong). It rises when the items correlate strongly with each other and when there are more items. Common guidelines are: .90 and above excellent, .80 to .90 good, .70 to .80 acceptable, .60 to .70 questionable and below .60 poor. Very high values (above .95) are not always good news: they can mean the items are near-duplicates that ask the same question in slightly different words.

A deliberate mistake first

Run alpha on the four stress items exactly as they were collected, with ST3 not yet reversed.

Try it in the tool

Alpha before reverse-coding (the mistake)

Items: ST1, ST2, ST3 (not reversed) and ST4.
Alpha is −.07 and ST3 has a corrected item-total correlation of −.70.
The tool tells you that removing ST3 would raise alpha: the classic sign of an item that still needs reverse-coding.

Open alpha with the original ST3 ↗

The result is alarming: α = −.07, which the tool labels unacceptable. The item-total table explains why. ST1, ST2 and ST4 each correlate positively with the rest of the scale (around .40), but ST3 correlates −.70: the more stressed the student, the lower their agreement with “I feel in control of my studies”. ST3 is a perfectly good item; it just points the other way. A negative item-total correlation is almost always a reverse-worded item that has not been recoded, or a data-entry error.

The correct analysis

Try it in the tool

Alpha for the stress scale with ST3 reversed

Items: ST1, ST2, ST3_r and ST4.
Alpha is .848 (95% CI [.81, .88]), and every item-total correlation is above .67.
Then select SS1 to SS4 to check the support scale: α = .808.

Open alpha for the stress scale ↗

With ST3_r in place of ST3, alpha jumps to .848, 95% CI [.81, .88], which is good. All corrected item-total correlations are between .67 and .70, and the “alpha if item deleted” column shows that no item would raise alpha if removed. The social support scale gives α = .808, 95% CI [.77, .85], also good, with item-total correlations from .58 to .66.

Reading the item-total table

Corrected item-total correlation: the correlation between an item and the sum of the other items. Values below .30 suggest the item measures something different.
Alpha if item deleted: what alpha would be without that item. If it is clearly higher than the current alpha, the item weakens the scale. Consider removing it only if there is also a substantive reason, such as an ambiguous wording, and report the decision.
Mean inter-item correlation: the average correlation between items, ideally between .20 and .50 for broad constructs. Here it is .58 for stress, a sign of a tightly focused scale.

Alpha is not the whole story

Alpha assumes that every item contributes equally to the construct (tau-equivalence). When items have different loadings, McDonald’s omega is a better estimate, and the reliability and validity tool reports both, along with composite reliability and average variance extracted from a confirmatory factor analysis. Alpha also says nothing about validity: four items can be consistent with each other and still measure the wrong thing. If your scale comes from published research, cite the original validation and report alpha in your sample as well. If you built a new scale, an exploratory factor analysis should come first.

When there are only two items

For two-item scales, alpha is a poor index; report the Spearman-Brown coefficient (available in the split-half reliability tool) or the inter-item correlation instead.

Reporting reliability

Perceived stress was measured with four items (e.g., “I feel overwhelmed by my coursework”) on a five-point scale from 1 (strongly disagree) to 5 (strongly agree). Item 3 was reverse-coded. Internal consistency was good, Cronbach’s α = .85, 95% CI [.81, .88]. Social support was measured with four items (α = .81, 95% CI [.77, .85]).

Report reliability in the methods chapter, under each measure, for your own sample, even when the original authors reported it for theirs.

Key takeaways

Compute alpha for every multi-item scale, in your own data, before computing scale scores.
A negative item-total correlation almost always means a reverse-worded item was not recoded.
Aim for α ≥ .70; remove items only with a substantive reason and report the decision.
Reliability is not validity, and omega is preferable when items have unequal loadings.