Plan your analysis before you collect data

Most problems in a results chapter start long before the analysis. A vague research question leads to a questionnaire that measures the wrong things, a sample that is too small to detect anything, and a supervisor asking “why did you use this test?”. Ten minutes of planning saves weeks later. In this lesson you plan the analysis for the study that runs through the whole course.

The course study

Imagine a master’s student studying wellbeing and academic performance at a university with 12,000 students. She surveys 240 students and records gender, age, faculty (Arts, Business, Engineering or Science), weekly study hours, average nightly sleep, a four-item stress scale (ST1 to ST4), a four-item social support scale (SS1 to SS4), whether the student has used the counselling service, and the student’s midterm and final exam scores. Her five research questions are:

1. Do female and male students differ in perceived stress?
2. Does stress differ between faculties?
3. Is gender associated with using the counselling service?
4. Which factors predict final exam performance?
5. Does social support improve performance by reducing stress?

From question to hypothesis

A hypothesis is a testable statement about the population. Each one names the variables, the expected direction and, implicitly, the test. For question 1: H1: Female students report higher perceived stress than male students. The statistical null hypothesis is the opposite, “there is no difference in mean stress between female and male students”, and the test tells you how surprising your data would be if the null were true. Write one hypothesis per question; it makes the choice of test almost automatic.

Know your variables

The test you need depends on the measurement level of each variable and on its role (predictor or outcome).

The mean of several Likert items, such as the stress score, is conventionally treated as a continuous scale variable. A single Likert item is better treated as ordinal.

Choosing the test

With the variables classified, match each hypothesis to a test. The table below covers the questions in this course and most thesis designs. The “Which test?” finder on the homepage asks the same questions interactively.

So the plan is: an independent t-test for RQ1, a one-way ANOVA for RQ2, a chi-square test for RQ3, multiple regression for RQ4 and a mediation model for RQ5. Write this plan into your methods chapter before you see the data; it protects you from the temptation to try test after test until something is significant.

How many participants do you need?

A study that is too small has low statistical power: even if the effect exists, the test will probably miss it. A power analysis asks four linked quantities: the expected effect size, the significance level α (usually .05), the desired power (usually .80) and the sample size. Fix three and you get the fourth. For a t-test, a “medium” effect is Cohen’s d = 0.5. Base the expected effect on earlier studies whenever you can; Cohen’s small, medium and large labels are a last resort.

For a medium difference between two groups you need 128 students, 64 per group. For the ANOVA with four faculties and a medium effect (f = 0.25), the same tool gives 180 in total, and for a regression with four predictors and f² = 0.15 it gives 85. The study’s 240 students cover all of these. If you expect small effects (d = 0.2), you would need almost 800 students, which is why it is worth checking before you start.

Sample size for describing a population

If your aim is to estimate a percentage, such as the share of students who use counselling, the question is how precise the estimate must be. The survey sample size calculator uses the margin of error instead of an effect size.

With a population of 12,000 and a ±5% margin of error at 95% confidence, you need 373 completed responses. Our student has 240, which gives a margin of error of about ±6.3%, perfectly adequate for testing relationships but worth mentioning as a limitation if she reports percentages.