B2.2 Correcting For Multiple Comparisons

False Discovery Rate

To use this method, we must first assign a value for q. Whereas α is the proportion of false positive tests we are willing to accept across the whole sample typically 0.05 (5%), q is the proportion of false positives (false ‘discoveries’) that we are willing to accept within the significant results. In this example we will set q to 0.05 (5%) as well.

Here we are going to apply the Benjamini-Hochberg (BH) method to the data from the Fishers LSD test in the previous practical in section B2.1

Firstly, take the P values for each comparison of pairs and put them in ascending numerical order. Then assign a rank number in that order (smallest P value is rank 1, next smallest rank 2 and so on).

Calculate the BH critical value for each P value, using the formula q(i/m), where:

i = the individual p-value’s rank,

m = total number of tests,

q= the false discovery rate you have selected (0.05)

P values which are lower than our BH critical values are considered true ‘discoveries’. The first P value which is higher than the BH critical value and all significant P values below that (below in terms of the table, higher rank numbers) would be considered false ‘discoveries’. P values of ≥0.05 are not treated any differently to normal.

From this we can conclude that all three of our statistically significant differences from out pairwise comparisons are in true ‘discoveries’, and none of them should be discounted.

With small numbers of comparisons like this it is easy to use this method by hand, however where the FDR approach is most useful is when we are making very large numbers (hundreds) of comparisons.

—-

To compute this in R use the code

BH(u, alpha = 0.05)

where u is a vector containing all of the P values, and alpha is in fact your specified value for q

—-

To compute this is Stata you need to download the package smileplot which contains the programme multploc. This will allow you to perform the BH test. More details can be seen on this here- https://www.rogernewsonresources.org.uk/papers/multproc.pdf

—-

In SPSS there is a function to do this test within one of the extension bundles, but you can create a variable which contains all of your P values and then conduct a calculation for BH critical values using the ‘compute variable’ function (see A1.6) then make a visual assessment.

—-

There are also online FDR calculator you can use like this one https://tools.carbocation.com/FDR.

Try analysing the data from our example with at least one other method to check your results.

In Practical B2.1, we ran the Fisher’s LSD test to compare systolic blood pressure (SBP) between each possible pairing of groups in the BMI group categorical variable. Now we know a little be more about post hoc tests and correcting for multiple comparisons, we are going to go back and conduct the Bonferroni post hoc and Tukey’s HSD post hoc on the same data.   

The post-estimation command ‘pwcompare’  with the ‘mcompare(method)’ option specifies the method for computing p-values and confidence intervals that account for multiple comparisons.  

In the Fisher’s LSD example we used ‘mcompare(noadjust)’, meaning there is no adjustment for multiple comparisons.  

To run the same test with Bonferroni post hoc replace this with ‘mcompare(bonferroni)’. 

To run the Tukey’s post hoc replace this with ‘mcompare(tukey)’  

See the output here: 

quietly: anova sbp bmi_grp4 
pwcompare bmi_grp4, mcompare(bonferroni) effects 

pwcompare bmi_grp4, mcompare(tukey) effects 

Consider the different post hoc tests and the results for each comparison in them. Is one method producing higher or lower p values than the others? Do any previously significant findings become non-significant after correction for multiple comparisons?