Some practical considerations in arguing for the null

Learning about Bayes Factors can be a very liberating experience (as pointed out by Evie Vergauwe at the ESCOP session about Bayesian analyses and subsequently by Candice Morey in her blog). The main attraction of Bayes Factors is that you can use them to argue for the null hypothesis, which you can’t do with frequentist statistics. This means that you can go back to your file drawer, and make some conclusions based on all of those non-significant results that have probably accumulated over the years (even if the conclusion might be that you can’t draw any conclusions given this data).

When I learned about Bayes Factors just before my thesis submission, I thought: “Sweet! Now I can interpret all of my non-significant results!” I reported Bayes Factors in addition to most of the frequentist analyses, which indeed allowed me to draw some conclusions that would otherwise have been impossible: I provided some evidence for null effects, and showed that for some unexpected statistically significant effects, the Bayes Factor provided equivocal evidence, suggesting that they should not be taken too seriously. Soon I realised, however, that there is more to arguing for a null hypothesis than just calculating a Bayes Factor.

Arguing for the null comes with some challenges that are less relevant if the experiment is simply designed to maximise the chances of getting a significant p-value. Therefore, it is not always meaningful to add Bayes Factors to an analysis ad hoc, if the p-value did not come out as significant. When designing a study with the a priori intention of using Bayes Factor analyses, it is important to ask the question: “Does my design maximise my chances of drawing a meaningful conclusion, regardless of whether I get evidence for H₁ or H₀?” This is a switch in the mind set of the traditional question, “Does my design maximise my chances of getting a significant p-value?” The latter question is, of course, problematic, because often, when such an experiment yields a non-significant p-value, it can only be discarded. This means that a carefully designed experiment addressing theoretically interesting and practically important questions may well turn into nothing more than a waste of the researcher’s resources and an impediment to scientific progress.

Here, I discuss some practical considerations that I came across in some attempts at arguing for the null. I hope this will be helpful to others who are figuring out how to get the most out of Bayes Factors. If anyone has any further suggestions, I would be very happy to get some feedback!

Sample sizes

Having a large sample size is always important, because small samples are prone to be influenced by extreme chance events. Bayes Factor analyses are somewhat less affected by this problem than frequentist statistics: a frequentist analysis will give you a p-value no matter what, while the Bayes Factor also tells you how confident you should be in the result. If the sample is small, Bayes Factor values are likely to hover around the value of 1, thus providing very little evidence about whether the data is more compatible with H₀ or H₁. Such an equivocal value tells you that you need more data if you want to draw any conclusions.

However, small-sampled studies should always be taken with a grain of salt, as the law of large numbers still applies. If the effect is small, a population group difference may yield identical means in a small sample. Similarly, a numerical difference between two conditions in the absence of a population effect is more likely in a small than large sample. Therefore, bigger is always better when it comes to sample size.

Think about the expected effect size

The key point behind Bayes Factor analyses is that they compare the degree to which the data is consistent with a null hypothesis compared to a pre-specified alternative hypothesis. This is good, because it forces the researcher to be specific about the kind of effect she expects. As a drawback, it allows critics to argue that the evidence for the null is due to the use of an unrealistically high prior for the H₁ effect size, and that with a smaller prior the Bayes Factor would probably provide evidence for H₁. This issue makes it theoretically impossible to be 100% confident that there is no effect. However, some consideration of the minimum effect size that would be of interest would allow the researcher to use this information in the construction of the H₁ prior, and to argue that if an effect exists, it is likely to be even smaller than this minimum.

Avoid confounds, if possible!

Confounding variables are a bigger problem when arguing for H₀ than when arguing for H₁. For example, I would really like to know whether statistical learning ability is associated with the ease with which children learn print-to-speech correspondences (as measured, for example, by Grapheme-Phoneme Correspondence, or GPC knowledge). One way to go might be to recruit a large number of children, and measure them on an extensive battery of tests. One could even go all out and design a longitudinal study, to see if statistical learning ability in pre-school predicts GPC knowledge after the onset of reading instruction. We know from previous research that statistical learning ability is correlated with vocabulary knowledge and phonological awareness (Spencer, Kaschak, Jones, & Lonigan), which, in turn, are well-known correlates of reading ability and GPC knowledge. So we want to test vocabulary knowledge, phonological awareness, and word reading skills as well, to be sure that a correlation between statistical learning ability and GPC knowledge is not due to these confounding variables. We also want to test other potentially correlated participant-level factors, such as age, intelligence, attention, etc. We expect all of the variables that we measure to correlate with each other (because this is generally the case with developmental data), so the only meaningful result will be the partial regression coefficient of statistical learning ability on GPC knowledge.

If the partial regression coefficient is significant (or, even better, if the Bayes Factor provides evidence for a model including statistical learning ability as well as all the covariates compared to a model including the covariates only), I can conclude that statistical learning ability is correlated with GPC knowledge over and above the covariates.

If the regression coefficient is not significant, and even if we get evidence for the base model including the covariates only over the alternative model, drawing conclusions is trickier. It is possible that statistical learning ability does not have a direct effect on GPC knowledge, but that it affects vocabulary knowledge, which in turn affects phonological awareness, which in turn affects GPC knowledge. Perhaps one could even strengthen a case for such a potential causal pathway with one of ‘em fancy Structural Equation Models. Given the inter-correlated nature of all my independent variables, however, there would be a large amount of possible mediators, and it is likely that even sophisticated statistical analyses cannot give me much useful information. For example, the relationship between statistical learning ability and GPC knowledge may disappear once we take into account phonological awareness. But then, the relationship between statistical learning ability and phonological awareness may also disappear after we take into account GPC knowledge. Thus, we won’t be able to conclude that one mediates the other, because both causal pathways are equally plausible – as are other causal pathways, such as the possibility that all three variables are affected by yet another confound, such as the child’s attentional capacity.

In short, such large-scale experiments with inter-correlated variables is an example of a design that does not maximise the researcher’s chances of being able to draw meaningful conclusions, regardless of the outcome. This is unfortunate, because such large-scale studies are often very time-consuming (for the researcher and participants) and expensive to run. They could still be useful for exploratory purposes, but they are not the best way to answer questions about the relationship between two variables.

Transparency

To convince an audience of a null result[1], it might be especially important to be as transparent as possible: if the experiment is a failure-to-replicate, desperate authors of the original study may clutch at straws and accuse the replicator of creatively excluding outliers, choosing the wrong priors, or being generally incompetent. Such claims can be mostly counteracted by providing the raw data and the analysis scripts – and, even better, by pre-registering studies, and in an ideal-case scenario, getting the original authors’ approval of the experimental design and analysis plan before data is collected.

If the data and analyses scripts are available, and if original-authors-as-reviewers want to make claims about flaws in the analyses, they can (and should) show (1) how and why the replicator’s analysis are problematic, and (2) that there is evidence for their original effect (or equivocal evidence) once the analyses are done correctly.

Conclusion

In summary, I argue that it is important to design an experiment in such a way that maximises the chance of being able to draw meaningful results, regardless of whether H₁ or H₀ is supported. Reading through what I have written so far, it strikes me that all of the issues described above apply to all experimental designs, really. However, because some researchers continue to be more easily convinced by relatively shaky evidence for an H₁ than by relatively strong evidence for H₀,[2] it is especially important to make sure to maximise the strength of an experiment when there is a real possibility that H₀ will be supported. I listed four considerations which might help a researcher to design a strong experiment: (1) Large sample sizes, (2) a careful consideration of the expected effect size, (3) avoiding confounding variables, and (4) being as transparent as possible by making the raw data available and by providing full information about all analyses.

Reference

Spencer, M., Kaschak, M. P., Jones, J. L., & Lonigan, C. J. Statistical learning is related to early literacy-related skills. Reading and Writing, 1-24.

---

[1] Admittedly, I have not (yet) succeeded in doing this.

[2] Based on my subjective experience.