Learning Outcomes
By the end of this session, students will be able to:
 Interpret pvalues
 Describe type I and type II errors
 Calculate and interpret ttests for a difference in means
You can download a copy of the slides here: 1.5: Hypothesis Testing
Video A1.5a – Estimation and Hypothesis Testing (10 minutes)
Video A1.5b – Type I and Type II Errors (2 minutes)
Video A1.5c – Hypothesis Testing & Confidence Intervals (7 minutes)
Video A1.5d – Ttests (7 minutes)
A1.5 PRACTICAL: Stata
A ttest compares the mean of a sample against another value or sample mean. Prior to performing a ttest we should think about whether the data are continuous, have been randomly sampled from a population, the variance is approximately equal and the distribution is approximately normal.
To test whether the variability of the data in each group is similar:
robvar chol, by(prior_t2dm)
If pvalues are >0.05, then we can assume equal variances and continue with the ttest.
If variances are not equal, we can still perform a ttest, but a slightly different version called a Welch ttest.
To check the distribution of the data:
histogram chol
We can also check whether both groups follow a normal distribution:
histogram chol, by(prior_t2dm) normal
There are a number of different ttests you can perform:
 A one sample ttest
 A two sample ttest
 A paired ttest
One sample ttest
A onesample ttest is used to compare the sample mean to a specific value (or hypothesised population mean).
For example, does the mean cholesterol level in this sample differ from the national average of 5.55?
ttest chol == 5.55
In the output from Stata, you should be able to see the sample size, the mean, standard error, standard deviation and 95% confidence intervals for cholesterol. At the bottom of the output, in the centre, should be able to see a pvalue for the result ‘Pr(T . t)’. This refers to whether the mean cholesterol level in this sample is equal to the national mean of 5.55. The results indicate that the sample mean of 5.51 is significantly different from the national average (p=0.009).
Twosample ttest
A twosampled ttest is used to compare means between two groups. Check how your variables are categorised – you may need to create a new, binary variable in order to perform the ttest.
Example 1:
A systolic blood pressure of over 130 is defined as high. If we want to test whether cholesterol levels differ if an individual has high systolic blood pressure or not, we can use the following code:
 Create a new binary variable for systolic blood pressure
gen h_sbp=1 if sbp>130 & sbp~=.
replace h_sbp=0 if h_sbp==.
label var h_sbp “high SBP”
label values h_sbp h_sbp
label define h_sbp 1 “High SBP” 0 “Not high SBP”
tab h_sbp, m
 Perform the ttest using new variable
ttest chol, by (h_sbp)
Example 2:
Does the mean total cholesterol level differ between people with and without type 2 diabetes?
ttest chol, by (prior_t2dm)
If you can’t assume equal variances, you can use either of the following commands:
ttest chol, by (prior_t2dm) welch
ttest chol, by (prior_t2dm) unequal
In the output from Stata, you should be able to see the sample size, the mean, standard error, standard deviation and 95% confidence intervals for cholesterol level in each group (diabetes: yes or no). At the bottom of the output, in the centre, should be able to see a pvalue for the result ‘Pr(T > t) ‘ for the twotailed pvalue for the ttest. If the pvalue for ‘Pr(T > t)’ is >0.05, there is no significant difference between the means of both groups. However, if the pvalue is <0.05, then we infer that there is a statistically significant difference between the mean cholesterol levels in people with and without type 2 diabetes.
 See if you can perform a twosample ttest to compare cholesterol levels between two other groups.
To perform a ttest for each level of a categorical variable you can use the command ‘by’. To do this sort your data by the categorical variable and then perform the ttest:
sort death
by death: ttest chol, by (prior_t2dm)
The above command should perform two ttests: One comparing mean cholesterol in those with and without type 2 diabetes, but only in those who died. The other comparing mean cholesterol in those with and without type 2 diabetes, but only in those who did not die.
Paired ttest
A paired ttest compares the means between two groups when there are paired observations. This will be covered in Module B2.5
Question A1.5a: Recode BMI into a binary variable so that one group has a BMI below 25, and the other group has a BMI of 25 and above. Perform a ttest to compare the mean SBP in those with BMI<25 and those with BMI≥. Answer the questions:

 What is the mean SBP where BMI <25 () ?
 What is the mean SBP where BMI ≥25 ()?
 What is the mean difference ()?
 What is the test statistic t ?
 What is 95% CI for the mean difference?
 What is the pvalue for this ttest and what does it mean?
Question A1.5b: If a clinician has decided that a difference of at least 5 mmHg is considered a clinically worthwhile difference in blood pressure with regard to morbidity associated with high blood pressure, do you consider the result in A1.5a to be clinically significant?
Answer
Answer A1.5a:
Code looks similar to:
gen bmi_2=1 if bmi<25
replace bmi_2=0 if bmi_2==.
drop if bmi==.
ttest sbp, by(bmi_2)

 129.49
 131.65
 2.16
 4.01
 1.11 – 3.22
 P<0.001 which means there is a significant difference in the mean SBP measures in people who are overweight and those who are not
Answer A1.5b: Whilst there is a statistically significant difference between the 2 mean SBP measures, with a mean difference of 2.16 mmHg, the result is not clinically significant.
A1.5 PRACTICAL: R
A ttest is an inferential statistic that can be calculated in order to determine the relationship and significance of the difference between the mean values of two separate population variables. There are several assumptions that we take to be true when performing a ttest:
 The data are continuous
 The sample data have been randomly sampled from a population
 There is homogeneity of variance (i.e. the variability of the data in each group is similar)
 The distribution is approximately normal
A onesample ttest (or student’s ttest) is used on a continuous measurement to decide whether a population mean is equal to a specific value — i.e. is the systolic blood pressure of our population 120, or not. In R, to complete a onesample ttest, the syntax is:
t.test(x, conf.level=0.95 , …[options] )
For a twosample ttest (independent samples ttest), the following code should be used:
t.test(x[group==1], x[group==2] , conf.level=0.95 , …[options] )
For a twosample test, x specifies the variable on which to perform the ttest, while the group variable defines the groups to be compared. An alternative formulation for the two sample t.test is:
t.test(x~group, data=)
See ?t.test for details of the other options within the native R help files.
Question A1.5a: Recode BMI into a binary variable so that one group has a BMI below 25, and the other group has a BMI of 25 and above. Perform a ttest to compare the mean SBP in those with BMI<25 and those with BMI≥. Answer the questions:
 What is the mean SBP where BMI <25 ()?
 What is the mean SBP where BMI ≥25 ()?
 What is the mean difference ()?
 What is the test statistic t ?
 What is 95% CI for the mean difference?
 What is the pvalue for this ttest and what does it mean?
Question A1.5b: If a clinician has decided that a difference of at least 5 mmHg is considered a clinically worthwhile difference in blood pressure with regard to morbidity associated with high blood pressure, do you consider the result in A1.5a to be clinically significant?
Answer
whitehall.data$bmibinary < NA
whitehall.data$bmibinary[whitehall.data$bmi <25] = 1
whitehall.data$bmibinary[whitehall.data$bmi >= 25] = 2
table (whitehall.data$bmibinary)
t.test(sbp~bmibinary, data=whitehall.data)
Welch Two Sample ttest
data: sbp by bmibinary
t = 4.0068, df = 3959, pvalue = 6.267e05
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
3.219796 – 1.104094
sample estimates:
mean in group 1 mean in group 2
129.4905 131.6524
 129.49mmHg
 131.65mmHg
 Using basic arithmetic in R:131.6524129.4905[1] 2.1619
 The tstatistic is the ratio of the departure of the estimated value of a parameter from its hypothesized value to its standard error. It can be utilised to tell us about whether we should support or reject the null hypothesis.t = 4.0068
 95 percent confidence interval: 3.22, 1.10That is to say, we are 95% confident that the difference in the mean systolic blood pressure between those with BMI <25 and BMI 25 lies between 1.10 and 3.22 mmHg.
 The pvalue is 6.267e05 = 0.00006267. Remember that pvalues provide us with the probability of getting the mean difference we saw here, or one greater, if the null hypothesis was actually true. In this case, that probability is extremely low.Here, our pvalue tells us that there is very strong evidence against the null hypothesis (which would be that there is no difference in SBP between our two BMI groups).Therefore, we can say that there is a significant difference in the mean SBP measures in people who are overweight and those who are not.
Answer A1.5b: No. Note here the difference between clinical and statistical significance. Whilst there is a statistically significant difference between the 2 mean SBP measures, with a mean difference of 2.16 mmHg, the result is not clinically significant.
A1.5 PRACTICAL: SPSS
 A ttest compares the mean of a sample against another value or sample mean. Prior to performing a ttest we should think about whether the data are continuous, have been randomly sampled from a population, the variance is approximately equal across groups and the distribution is approximately normal.There are a number of different ttests you can perform:
 A one sample ttest
 A two sample ttest
 A paired ttest (covered in Module B2)
One sample ttest
A onesample ttest is used to compare the sample mean to a specific value (or hypothesised population mean).
For example, does the mean cholesterol level in this sample differ from the national average of 5.55?
Run the OneSample T Test we used in practical A1.4, but this time put cholesterol (chol) in the test Variable List and add 5.55 in the ‘Test Value’ box.
Answer
In the output window you will see one table which contains the sample size (N), the mean, standard error, and standard deviation for cholesterol.
Then a second table which contains the ttest statistic(t), degrees of freedom (df) and then the significance (p). This refers to whether the mean cholesterol level in this sample is equal to the national mean of 5.55. The results indicate that the sample mean of 5.51 is significantly different from the national average (p=0.009).
We always use the twosided (two tailed) p value if SPSS offers both.
Twosample ttest
A twosampled ttest is used to compare means between two groups. In SPSS this is referred to as an Independent Samples T Test.
Example 1:
A systolic blood pressure of over 130 is defined as high. We want to test whether cholesterol levels differ if an individual has high systolic blood pressure or not. In SPSS we can use a continuous variable to define groups within the ttest, if we know what our cut off value is.
Select
Analyze >> Compare Means and Proportions >> Independent Samples T Test.
Move cholesterol (chol) into the Test Variable(s) list and the blood pressure (sbp) into the Grouping Variable box. You then need to click on ‘Define Groups’, select ‘Cut point’ and input the lower bound of the top group (i.e. 131 in this case) as the cut point. Press Continue and then OK to run the test.
Answer
The SPSS output will look like this
One of the assumptions of the ttest is that the two groups have roughly equal variance. SPSS automatically does the Levene’s test for Equality of Variances (more on this in Module B3). If the Sig. (P value) for Levene’s is <0.05 then the two groups do not have equal variances, if P ≥ 0,05 then the two groups do have equal variances. SPSS automatically runs two versions of the ttest, one for equal variances, and one for unequal variances (equal variances not assumed). Use whichever row is appropriate for your data based on the outcome of the Levene’s test. In this case we would use ‘equal variances assumed’ on the top row.
Example 2:
Does the mean total cholesterol level differ between people with and without type 2 diabetes?
To answer this run the independent samples ttest again, but this time use the binary variable prior_t2dm as the grouping variable. When you get to the point of defining groups select the top option of ‘Use specified values’ and input the numerical codes which have been used to assign groups, in this case 0 and 1.
Answer
The SPSS output will look like this
Example 3:
Perform a ttest to compare the mean SBP in those with BMI<25 and those with BMI≥25.
Answer the questions:


 What is the mean SBP where BMI <25?
 What is the mean SBP where BMI ≥25 ?
 What is the mean difference?
 What is the test statistic t ?
 What is 95% CI for the mean difference?
 What is the pvalue for this ttest and what does it mean?
 If a clinician has decided that a difference of at least 5 mmHg is considered a clinically worthwhile difference in blood pressure with regard to morbidity associated with high blood pressure, do you consider the result in this task to be clinically significant?

Answer
The SPSS output will look like this
 129.49
 131.65
 2.16
 4.01
 1.11 – 3.22
 P<0.001 which means there is a significant difference in the mean SBP measures in people who are overweight and those who are not
 Whilst there is a statistically significant difference between the two mean SBP measures, with a mean difference of 2.16 mmHg, the result is not clinically significant.
Thank you for the excellent presentations, notes and practice questions for a not too easy concept.