How big can a dyslexic deficit be?

TL;DR: Thinking about effect size can help to plan experiments and evaluate existing studies. Bigger is not always better!

In grant applications, requests for ethics permission, article introductions or pre-registered reports it is often expected that the researcher provides an estimate of the expected effect size. This should then be used to determine the sample size that the researcher aims to recruit. At first glance, it may seem counter-intuitive that one has to specify an effect size before one collects the data – after all, the aim of (most) experiments is to establish what this effect size could be, and/or whether it is different from zero. At second glance, specifying an expected effect size may be one of the most important issues to consider in the planning of an experiment. Underpowered studies reduce the researcher’s chance to find an effect, even if the effect exists in the population – running an experiment with 30% power gives one a worse chance to find the effect than tossing a coin to determine the experiment’s outcome. Significant effects in underpowered studies overestimate the true effect size, because the z-score cut-off that is needed to reach significance is larger than the population effect (Gelman & Carlin, 2014; Gelman & Weakliem, 2009; Schmidt, 1992).

So, given that it is important to have an idea of the effect before running the study, what is the best way to come up with a reasonable a priori effect size estimate? One possibility is to run a pilot test; another is to specify the smallest effect size that would be of (theoretical and/or practical) significance; the third would be to consider everything we know about effects that are kind of like the one we are interested in, and to use that to guide our expectations. All three have both benefits and drawbacks – my aim is not to discuss these. My aim here is to focus on the third possibility: knowing how big the most stable effects in your research are can be used to constrain your expectations – it is unlikely that your “new” effect, a high-hanging fruit, is bigger than the low-hanging fruit that has been plucked decades ago. Specifically, I will try to provide an effect size estimate which could be used by researchers seeking to find potential deficits underlying developmental dyslexia, and discuss the limitations of this approach in this particular instance.

Case study: Developmental dyslexia

A large proportion of the literature on developmental dyslexia focuses on finding potentially causal deficits. Such studies generally test dyslexic and control participants on a task which is not directly related to reading, such as visual attention, implicit learning, balance, etc., etc. If dyslexic participants perform significantly worse than control participants, it is suggested that there is the possibility that the task might tap an underlying deficit, which causes dyslexia, can be used as an early marker of dyslexia, and can be treated to improve reading skills. The abundance of such studies has lead to an abundance of theories of developmental dyslexia, which has lead to – well – a bit of a mess. I will not go into details here; the interested reader can scan the titles of papers from any reading- or dyslexia-related journal for some examples.

A problem arises now that we know that many published studies are Type-I errors (that find a significant group difference in their sample that is absent in the population; see Ioannidis, 2005; Open Science Collaboration, 2015). Sifting through the vast and often contradictory literature to find out whether there is a sound case for a dyslexic deficit on each of the tasks that have been studied to date would be a mammoth task. Yet, this would be an important step for an integrative theory of dyslexia. Making sense of the existing literature would not only constrain theory, but also prevent researchers from wasting resources on treatment studies which are not likely to yield fruitful results.

Both to evaluate the existing literature, and to plan future studies, it would be helpful to have an idea of how big an effect can be reasonably expected. To this end, I decided to calculate an effect size estimate of the phonological awareness deficit. The phonological awareness deficit relates to the consistent finding that participants at dyslexia perform more poorly than controls on tasks involving the manipulation of phonemes and other sublexical spoken units (Melby-Lervåg, Lyster, & Hulme, 2012; Snowling, 2000; for a discussion about potential causality see Castles & Coltheart, 2004).  

Methods

Studies which are not about phonological awareness often include phonological awareness tasks in their test battery when comparing dyslexic participants to controls. In these instances, phonological awareness is not the variable of interest, therefore there is little reason to suspect that there is any p-hacking (e.g., including covariates, removing outliers, or adding participants until the group difference becomes significant). For this reason, the effect size estimate is based only on studies where phonological awareness was not the critical experimental task. To this end, I looked through all papers that are lying around on my desk on the subject of statistical learning and dyslexia, and on magnocellular/dorsal functioning in dyslexia. I included, in the analysis, all papers which provided a table with group means and standard deviations on tasks of phonological awareness. This resulted in 15 papers, 6 testing children and 9 testing adults, from 5 different languages. Mostly, the phonological awareness measures were spoonerism tasks. A full list of studies with the tasks, participant characteristics, and full references to the papers can be found in the dropbox folder linked below.

I generated a forest plot with the metafor package for R (Viechtbauer, 2010). I used a random-effect model, and the Sidik-Jonkman method (Sidik & Jonkman, 2005; see Edit below). The R analysis script, and the datafile (including full references to all papers that were included) can be found here: osf.io/69a2p

Results

The results are shown in the figure below. All studies show a negative effect (i.e., worse performance on phonological awareness tasks for dyslexic compared to control groups). The effect size estimate is d = 1.24, with a relatively narrow confidence interval, 95% CI [0.95, 1.54].

Conclusions and limitations

When comparing dyslexic participants and controls on phonological awareness tasks, one can expect a large effect size of around d = 1.2. A phonological awareness deficit is the most established phenomenon in dyslexia research; given that all other effects (as far as I know) are contentious, and have often been successfully replicated by some labs, but not by others, it is likely that a group difference on any other task (which is not directly related to reading) will be smaller. For researchers, peer reviewers, and editors, the obtained effect size of a new experiment can be used as a red flag: if a study examines whether or not there is a group difference in, say, chewing speed between dyslexic and control groups*, and obtains an effect size of d > 1, one should raise an eyebrow. This would be a clear indicator that one has obtained a Magnitude error (Gelman & Carlin, 2014). Obtaining several effect sizes of this magnitude in a multi-experiment study could even be an indicator of p-hacking or worse (Schimmack, 2012).

The are two limitations of the current blog post. The first is methodological, the second is theoretical. First, I have deliberately chosen to include studies where phonological awareness was not of primary interest. This means that the original authors would not have had an incentive to make this effect look bigger than it actually is. However, there is a potential alternative source of bias: it is possible that (at least some) researchers use phonological awareness as a manipulation check: given that this effect is well-established, a failure to find it in one’s sample could be taken to suggest that the research assistant who collected the data screwed up. This could lead to a file-drawer problem, where researchers discard all datasets where the group difference in phonological awareness was not significant. It is also plausible that researchers would simply not report having tested this variable if it did not yield a significant difference, as the critical comparison in all papers was on other cognitive tasks. If other studies have obtained non-significant differences in phonological awareness but did not report these, the current analysis presents a hugely inflated effect size estimate.

The second issue relates less to the validity of the effect size, and more to its utility: the effect size estimate is really, really big. A researcher paying attention to effect sizes will be sceptical, anyway, if they see an effect size this big. Thus, in this case, the effect size of the most stable effect in the area cannot be used to constrain our expectation of potential effect sizes of other experiments. To guide the expectation of an effect size for future studies, one might therefore want to turn to alternative approaches. Pilot testing could be useful, but its limitation is that it is often difficult to recruit enough participants to get a meaningful pilot study plus a well-powered full experiment. At this stage, it would also be difficult to define a minimum effect size that would be of interest. This could change, however, if we develop models of reading and dyslexia that make quantitative predictions. (This is unlikely to happen anytime soon, though.) Currently, the most useful approach to determining whether or not there is an effect, given limited resources and no effect size estimate, seems to be optional stopping. This can be done if it is planned in advance; in the frequentist framework, alpha-levels need to be adjusted a priori (Lakens, 2014; for Bayesian approaches see Rouder, 2014; Schönbrodt, 2015).  

Despite the limitations of the current analysis, I hope the readers of this blog post will be encouraged to consider issues in estimating effect sizes, and will find some useful references below.  

References

Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

Gelman, A., & Carlin, J. (2014). Beyond Power Calculations Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641-651.

Gelman, A., & Weakliem, D. (2009). Of beauty, sex and power: Too little attention has been paid to the statistical challenges in estimating small effects. American Scientist, 97(4), 310-316.

Hartung, J., & Makambi, K. H. (2003). Reducing the number of unjustified significant results in meta-analysis. Communications in Statistics-Simulation and Computation, 32(4), 1179-1190.

IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014). The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC medical research methodology, 14(1), 1.

Ioannidis, J. P. A. (2005). Why most published research findings are false. Plos Medicine, 2(8), 696-701. doi:10.1371/journal.pmed.0020124

Lakens, D. (2014). Performing high‐powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710.

Melby-Lervåg, M., Lyster, S.-A. H., & Hulme, C. (2012). Phonological skills and their role in learning to read: a meta-analytic review. Psychological Bulletin, 138(2), 322.

Rouder, J. N. (2014). Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review, 21(2), 301-308.

Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17(4), 551.

Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173.

Schönbrodt, F. D. (2015). Sequential Hypothesis Testing with Bayes Factors: Efficiently Testing Mean Differences. Available at SSRN 2604513.

Sidik, K., & Jonkman, J. N. (2005). Simple heterogeneity variance estimation for meta‐analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics), 54(2), 367-384.

Snowling, M. J. (2000). Dyslexia. Malden, MA: Blackwell Publishers.

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1-48.  Retrieved from http://www.jstatsoft.org/v36/i03/.

*************************************************

* It’s not easy coming up with a counterintuitive-sounding “dyslexic deficit” which hasn’t already been proposed!

*************************************************
Edit 5.4.16 #1: Added a new link to data (to OSF rather than dropbox).
Edit 5.4.16 #2: I used the "SJ" method for ES estimation, which I thought referred to the one recommended by IntHout et al., referenced above. Apparently, it doesn't. Using, instead, the REML method slightly reduces the effect size, to d = 1.2, 95% CI [1.0, 1.4]. The estimate is similarly large to the one described in the text, therefore it does not change the conclusions of the analyses. Thanks to Robert Ross for pointing this out!