Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review

The reasons why students may be struggling laid out in the previous section are multifaceted, and the theoretical perspectives underlying the included interventions are broad. Nevertheless, three superordinate components are characteristic for the majority of the included programmes:

We emphasise that the following presentation of these three theoretical perspectives is not exhaustive, and, although components are presented as demarcated, they contain some conceptual overlap.

Social learning theory has its origins in social and personality psychology, and was initially developed by psychologist Julian Rotter and further developed especially by Albert Bandura, (1977, 1986). From the perspective of social learning theory, behaviour and skills are primarily learned by observing and imitating the actions of others, and behaviour is in turn regulated by the recognition of those actions by others (reinforcement), or discouraged by lack of recognition or sanctions (punishment). According to social learning theory, creating the right social context for the student can therefore stimulate more productive behaviour through social modelling and reinforcement of certain behaviours that can lead to higher academic achievement.

Cognitive developmental theory is not one particular theory, but rather a myriad of theories about human development that focus on how cognitive functions such as language skills, comprehension, memory and problem‐solving skills enable students to think, act and learn in their social environment. Some theories emphasise a concept of intelligence where children gradually come to acquire, construct, and use cognitive functions as the child naturally matures with age (e.g., Piaget, 2001; Perry, 1999). Other theories hold a more socio‐cultural view of cognitive development and use a more culturally distinct and individualised concept of intelligence that to a greater extent includes social interaction and individual experience as the basis for cognitive development. Examples include the theories of Robert Sternberg (2009) and Howard Gardner (1999).

Pedagogical theory draws on the different disciplines in psychology and social theory such as cognitivism, social‐interactional theory and socio‐cultural theory of learning and development. There is not one uniform pedagogical model, but examples of contemporary models in mainstream pedagogy are concepts such as Scaffolding (Bruner, 2006) and the Zone of Proximal Development (Vygotsky, 1978), which originated in developmental and educational psychology. These notions hold that learning and development emerge through practical activity and interaction. Acquisition of new knowledge is therefore considered to be dependent on social experience and previous learning, as well as the availability and type of instruction. Accordingly, school interventions require educators to interact and organise the learning environment for the student in certain ways to fit the individual student's needs and potentials for development.

School interventions affect academic achievement by changing the methods by which instruction is given (instructional methods) and by targeting certain content (the content domain), and many combine several intervention components as well as theoretical perspectives. Examples of instructional methods covered in earlier reviews are tutoring, coaching of personnel, cooperative learning/peer‐assisted instruction, computer‐assisted instruction, feedback and progress monitoring, and incentive programmes (e.g., Dietrichson et al., 2017). Reading interventions directed to younger students often target content domains as phonemic awareness, phonics, fluency, vocabulary, and comprehension (e.g., Slavin et al., 2009). Slavin and Lake (2008) describe differences in elementary school math curricula in terms of how they emphasise domains such as problem solving, manipulatives, concept development, algorithms, computation and word problems. Gersten et al. (2009) used the following domains to divide mathematics interventions into categories: operations (e.g., addition, subtraction, and multiplication), word problems, fractions, algebra, and general math proficiency (or multiple components). Many school interventions have additional goals concerning other aspects of the student's life, such as reducing problematic behaviour of the students (Cheung & Slavin, 2012; Slavin & Lake, 2008; Wasik, 1997; Wasik & Slavin, 1993).

As indicated, many interventions combine theoretical perspectives. For example, interventions such as tutoring and peer‐assisted instruction interventions often have in common that they comprise an eclectic theoretical model that combines components from all three perspectives on learning presented in the previous section. They are comprehensive interventions that rely on mechanisms such as increased feedback and tailor‐made instruction (pedagogical theory), regulation of behaviour by for example rewards or interaction with role models (social learning theory), and development of cognitive functions such as learning how to learn (cognitive developmental theory).

Another way of viewing these and other types of interventions is that they address the differential family and neighbourhood resources of students with high and low risk of academic difficulties. Low‐risk students are more likely to have access to “tutors” all year round, as parents, siblings, and other family members help out with homework and schoolwork. Interventions to change mindsets, increase expectations, and mitigate stereotype threat may also substitute for low‐risk families and teachers already having such expectations or teaching low‐risk students such a mindset. Different types of extrinsic rewards may be a way to bolster motivation, which may be especially important for students whose families place less weight on educational achievement.

Furthermore, if the differences between students with high and low risk of academic difficulties can be understood as a consequence of differential access to a combination of resources, then remedial efforts may need to address several problems at once to be effective. Programmes that combine certain components may therefore be more effective than others. Another reason why it is interesting to examine combinations of components relates to an often suggested explanation for missing impacts: lack of motivation among participants (e.g., Edmonds et al., 2009; Fuchs et al., 1999). It is therefore possible that programmes will be more effective if they, for example, include some form of rewards for participating students, along with other components providing for instance specific pedagogical support.

Prior reviews have in particular covered reading interventions. Slavin et al. (2009) reviewed reading programmes for elementary Grades. They focused on all kinds of programmes and not only programmes for at‐risk or low‐performing students specifically. Wanzek et al. (2006) reviewed reading programmes directed to students in Grades K‐12 with learning disabilities, and Flynn et al. (2012), Inns et al. (2019), Scammaca et al. (2015), Slavin et al. (2011), and Wanzek et al. (2018) reviewed programmes for struggling readers in Grades 5‐9, K‐5, 4‐12, K‐5, and K‐3, respectively.3 These reviews thus covered low‐achieving students, but neither at‐risk students nor areas other than reading. Suggate (2016) reviewed the long‐run effects of phonemic awareness, phonics, fluency, and reading comprehension interventions from preschool up to Grade 7, but did not discern between interventions targeting students with/at risk of and without/not at risk of academic difficulties.

Mathematics interventions were reviewed in Slavin and Lake (2008) and Pellegrini et al. (2018) for general student populations in elementary school. Gersten et al. (2009) examined four types of components of mathematics instruction for students with learning disabilities, but did not include studies for students at risk of math difficulties (or other reasons for difficulties than learning disabilities). Dietrichson et al. (2017) included interventions targeting both reading and mathematics and based inclusion on the share of students with low SES, but did not consider whether students had academic difficulties or not. Fryer (2017) included both math and reading interventions for all types of student groups.4

All reviews that reported an overall effect size found that it was positive. Most also found substantial variation between interventions. Regarding intervention types, we provide a more detailed comparison to our results in Section 6.5 (including reviews focused on a specific intervention type), but to preview that discussion we describe some overarching results here. Among instructional methods, many reviews indicated that one‐to‐one or small‐group tutoring have relatively large effect sizes across both mathematics and reading interventions compared with other intervention types (Dietrichson et al., 2017; Fryer, 2017; Inns et al., 2019; Pellegrini et al., 2018; Slavin & Lake, 2008; Slavin et al., 2009, 2011). Peer‐assisted instruction or cooperative learning interventions also showed relatively large effect sizes in some reviews (Dietrichson et al., 2017; Inns et al., 2019; Pellegrini et al., 2018; Slavin & Lake, 2008; Slavin et al., 2009, 2011), but not in all (Gersten et al., 2009). Computer‐assisted or technology‐supported instruction have typically positive but smaller effect sizes than small‐group and peer‐assisted instruction (Dietrichson et al., 2017; Inns et al., 2019; Pellegrini et al., 2018; Slavin & Lake, 2008; Slavin et al., 2009, 2011).

Gersten et al. (2009) examined some components of mathematics instruction that do not map neatly into the categories used in the current review and some of the others. They found for example most support for explicit instruction, use of heuristics, and curriculum design. Regarding specific math domains, interventions targeting word problems had higher effect sizes than other math domains but not significantly so.

Reviews focusing on short‐term effects across reading domains reported positive effects in general but few reliable differences over reading domains (Flynn et al., 2012; Scammaca et al., 2015; Wanzek et al., 2006). An exception is that reading comprehension interventions were associated with significantly higher effect sizes than fluency interventions in Scammaca et al. (2015), but this difference disappeared when they only considered standardised tests. Suggate (2016) found that comprehension and phonemic awareness interventions showed relatively lasting effects that transferred to non‐targeted skills, whereas phonics and fluency interventions did not (mean follow‐up was around 11 months).

Academic difficulties and lack of educational attainment are significant societal problems. Moreover, as shown by the Salamanca declaration from 1994 (UNESCO, 1994), there has for decades been a great interest among policy makers to improve the inclusion of students with academic difficulties in mainstream schooling, and a desire to increase the number of empirically supported interventions for these student groups.

The main objective of this review is to provide policy makers and educational decision‐makers at all levels—from governments to teachers—with evidence of the effectiveness of interventions aimed to improve the academic achievement of students with or at risk of academic difficulties. To this end, we chose a broad scope in terms of the target group and the types of interventions we included. We included studies that measured the effects of interventions by standardised tests in reading and mathematics. The reason is that many interventions are not directed specifically to either subject and outcomes are therefore measured in both (Dietrichson et al., 2017). Including both students with and at risk of academic difficulties in the target group should also decrease the risk of biasing the results due to omission of studies where information about either academic difficulties or at‐risk status is available, but not both. Furthermore, making comparisons over intervention components within one review, rather than across reviews, should increase the possibilities of a fair comparison. For instance, controlling that effect sizes are calculated in the same way, that the definitions of intervention components are consistent, and that moderators are coded in the same way, is easier within the scope of one review than across reviews.

Earlier reviews with a comparable focus on students with or at risk of academic difficulties have included a more narrowly defined target group. Furthermore, their analyses either did not include intervention components together with other moderators in a meta‐regression, or only included very broad categories of instructional methods and content domains. Such analyses risk confounding the effects of intervention components with for example participant characteristics, and precludes testing whether components have significantly different effect sizes. Furthermore, some reviews have coded interventions regarding the instructional methods used, or regarding the type of content taught, and used such indicators in meta‐regressions (e.g., Dietrichson et al., 2017; Gersten et al., 2009; Scammaca et al., 2015). With the exception of Gersten et al. (2009), who included an indicator for word problems alongside instructional methods‐indicators, the analyses did not include both methods, and content domain indicators. They therefore risk confounding instructional methods with content domains.

Lastly, we are not aware of another review that have provided meta‐analytic estimates of medium‐ and long‐term effects specifically for students with or at risk of academic difficulties.

According to our protocol, included studies should use an intervention‐control group design or a comparison group design (Dietrichson et al., 2016). Included study designs were randomised controlled trials (RCT), including cluster‐RCTs; quasi‐randomised controlled trials (QRCTs), that is, where participants are allocated by means such as alternate allocation, person's birth date, the date of the week or month, case number, or alphabetical order; and quasi‐experimental studies (QES). To be included, QES had to credibly demonstrate that outcome differences between intervention and control groups is the effect of the intervention and not the result of systematic baseline differences between groups. That is, selection bias should not be driving the results. This assessment is included as a part of the risk of bias tool, which we elaborate on in the “Risk of bias” section, and no QES was excluded on this criterion in the screening process. A fair amount of studies within educational research use single group pre–post comparisons (e.g., Edmonds et al., 2009; Wanzek et al., 2006); such studies were however excluded in the screening process due to the higher risk of bias.

Control groups received treatment‐as‐usual (TAU) or a placebo treatment. We found no studies in which the control group explicitly received nothing (i.e., a no‐treatment control), as all students experienced regular schooling. That is, control groups got whatever instruction the intervention group would have gotten, had there not been an intervention. The TAU condition can for this reason differ substantially between studies (although many studies did not describe the control condition in much detail). Eligible types of control groups included also waiting list control groups, which only differed in the time frame in which researchers estimate the effects. That is, students in both waiting list and regular control groups were offered regular schooling but after the students in the waiting list control group had received the intervention, they could no longer be used as controls.

Comparison designs compared alternative interventions against each other. That is, they made it clear that all students get something other than TAU because of the intervention. Effect sizes from such studies are not fully comparable to effect sizes from intervention‐control designs. We therefore planned to analyse comparison designs separately from intervention‐control designs, and use them where they may shed light on an issue, which could not be fully analysed using the sample of intervention‐control studies. However, the number of studies that were, in this sense, relevant was small and we used them only in one analysis of the effects of group sizes in small‐group instruction interventions.

Due to language restrictions in the review team, we included studies written in English, German, Danish, Norwegian, and Swedish. To ensure a certain degree of comparability between school settings and to align TAU conditions in included studies, we only included studies published in or after 1980.

The population samples eligible for the review included students attending regular schools in Grades K‐6, who were having academic difficulties, or were at risk of such difficulties. Students attending regular private, public, and boarding schools were included, and students receiving special education services within these school settings were also included.

We included only studies carried out in OECD countries. This selection made it more likely that school settings and TAU conditions were comparable across included studies. Grades K‐6 corresponds roughly to primary school, defined as the first step in a three‐tier educational system consisting of primary education, secondary education and tertiary or higher education. We included studies with a student population in higher Grades than K‐6 as long as the majority of the students were in Grades K‐6. The age range included differed between countries, and sometimes between states within countries (ages range from 4–7 to 11–13, depending on country/state). Much fewer studies reported the participants’ ages than Grades, which was also our main reason to formulate the inclusion criteria in terms of Grade rather than age.

The eligible student population included both students identified in the studies by their observed academic achievement (e.g., low academic test results, low grade point average or students with specific academic difficulties such as learning disabilities), and students that were identified primarily on the basis of their educational, psychological, or social background (e.g., students from families with low socioeconomic status, students placed in care, students from minority ethnic/cultural backgrounds, and second language learners). We excluded interventions that only targeted students with physical learning disabilities (e.g., blind students), students with dyslexia/dyscalculia, and interventions that were specifically directed towards students with a certain neuropsychiatric disorder (e.g., autism, ADHD), as some interventions targeting such students are different from interventions targeting the general struggling or at‐risk student population (e.g., they include medical treatments like in Ackerman et al., 1991).

Because there was substantial overlap between students that were already struggling and groups considered at‐risk of difficulties in studies found in a previous review (Dietrichson et al., 2017), we chose to include both students with difficulties and students that were deemed at‐risk, or were considered educationally disadvantaged. If the two criteria were inconsistent, we gave priority to students having academic difficulties. For example, we excluded interventions that targeted high‐achieving students from low‐income backgrounds.

Some interventions included other students, who neither had academic difficulties nor were at risk of such difficulties. For example, in some peer‐assisted learning interventions high‐performing students were paired with struggling students. Studies of such interventions were included if the total sample (intervention and control group) included at least 50% students that were either having academic difficulties or were at risk of developing such difficulties, or if there were separate effect sizes reported for these groups.

We included interventions that sought to improve academic achievement or specific academic skills. This does not mean that the intervention had to consist of academic activities, but there had to be an expectation in the study that the intervention, regardless of the nature of the intervention content, would result in improved academic achievement or a higher skill level in a specific academic task. We however choose to exclude interventions that only sought to improve performance on a single test instead of improving a skill that would improve test scores. For similar reasons, we excluded studies of interventions where students are provided with accommodations when taking tests; for instance, when some students are allowed to use calculators and others not.

An explicit academic aim of the intervention did not per se exclude interventions that also included non‐academic objectives and outcomes. However, we excluded interventions having academic learning as a possible secondary objective. If the objectives were not explicitly stated, we used the presence of a standardised test in mathematics or reading as an indication that the authors expected the intervention to improve academic achievement. We excluded cases where such tests were included but the authors explicitly stated that they did not expect the intervention to improve reading or math skills.

Furthermore, we only included school‐based interventions. That is, interventions conducted in schools during the regular school year with schools as one of the stakeholders. This latter restriction excluded summer reading programmes, after‐school programmes, parent tutoring programmes, and other programmes delivered in the home of students.

Universal interventions that aimed to improve the quality of the common learning environment at the school level in order to raise academic achievement of all students (including average and above average students), were excluded. Interventions such as the one described in Fryer (2014) where a bundle of best practices were implemented at the school level in low‐achieving schools, where most students are struggling or at risk, was also excluded. This criterion also excluded whole‐school reform strategy concepts such as Success for All, curriculum‐based programmes like Elements of Mathematics (EMP), as well as reduced class size interventions.

This criterion also meant that we excluded interventions where teachers or principals receive professional development training in order to improve general teaching or management skills. Interventions targeting students with or at risk of academic difficulties may on the other hand include a professional development component, for example, when a reading programme includes providing teachers with reading coaches. Such interventions were therefore included.

Our protocol contained no criterion for the duration of interventions and we included interventions of all durations. We coded the duration of the interventions and this variable was included as a moderator in some of the analyses.

We included outcomes that cover two areas of fundamental academic skills:

Studies were only included if they considered one or more of the primary outcomes. Standardised tests included norm‐referenced tests (e.g., Gates‐MacGinitie Reading Tests and Star Math), state‐wide tests (e.g., Iowa Test of Basic Skills), and national tests (e.g., National Assessment of Educational Progress, NAEP). If it was not clear from the description of the outcome measures in the studies, we used online sources to determine whether a test was standardised or not. For example, if a commercial test has been normed, this was typically mentioned on the publisher's homepage. However, for older tests it was not always possible to find information about the test from electronic sources. In these cases, we included the test if there was a reference to a publication describing the test, which made it clear that the test had not been developed for the intervention or the study.

We restricted our attention to standardised tests in part to increase the comparability between effect sizes. Earlier related reviews of academic interventions have pointed out that effect sizes tend to be significantly lower for standardised tests compared with researcher‐developed tests (e.g., Flynn et al., 2012; Gersten et al., 2009; Scammaca et al., 2015). Scammaca et al. (2015) furthermore reported that whereas mean effect sizes differed significantly between the periods 1980–2004 and 2005–2011 for other types of tests, mean effect sizes were not significantly different for standardised tests. As researcher‐developed tests are usually less comprehensive and more likely to measure aspects of content inherent to intervention but not control group instruction (Slavin & Madden, 2011), standardised tests should provide a more reliable measure of lasting differences between intervention and control groups.

We excluded tests that provided composite results for several academic subjects other than mathematics and reading, but included tests of specific domains (e.g., vocabulary, fractions) as well as more general tests, which tested several domains of reading or mathematics. Tests of subdomains had significantly larger effect sizes compared with more general tests in Dietrichson et al. (2017). This result may indicate that it may be easier to improve scores on tests of subdomains than on tests of more general skills, or that tests of subdomains may be more likely to be inherent to intervention group instruction. At the same time, it seems reasonable that interventions that target subdomains of reading and mathematics are tested on whether they affect these subdomains. Therefore, we did not want to exclude either type of test, but coded the type of test and used it as a moderator in the analysis. However, to mitigate problems with test content being inherent to intervention and not control group instruction, we did not consider tests where researchers themselves picked a subset of questions from a norm‐referenced test as being standardised. The subset should either have been predefined (as in e.g., the passage comprehension subset of Woodcock‐Johnson Tests of Achievement) or the picked by someone other than the researchers (e.g., released items from the NAEP).

We included all postintervention tests and coded the timing of each test (see “Multiple time points” section).

All searches were restricted to publications after 1980. This year was chosen to balance the competing demands of comparability between intervention settings and comprehensiveness of the review. We used no further limiters in the searches.

Relevant published studies were identified through electronic searches of bibliographic databases, government and policy databanks. We searched the following electronic resources/databases:

All databases were originally searched from 1st of January 1980 to March 2016. As mentioned, we only included studies published in or after 1980 to ensure a certain degree of comparability between school settings and to align TAU conditions in included studies. We updated the searches in June/July 2018 using identical search strings. Some database searches were not updated in 2018 due to access limitations.

In Supporting Information Appendix, we report the search strings as well as details for each electronic database and resource searched. A

Note that the searches contained terms relating to secondary school, since the search contributed to a review about this older age group (Grades 7–12, see Dietrichson et al., 2020). There is overlap in the literature among the age groups, and in order to rationalise and accelerate the screening process, we decided upon performing one extensive search.

We also searched the following national/international repositories and review/trial archives/registries:

Our protocol stated that we should search two trial registries: The Institute for Education Sciences’ (IES) Registry of Randomized Controlled Trials (http://ies.ed.gov/ncee/wwc/references/registries/index.aspx↗), and American Economic Association's RCT Registry (https://www.socialscienceregistry.org↗). We were however unable to search the IES registry as it was not available (last tried 23 July 2018). We have asked IES about availability, but have to date not received a reply. We updated the search of American Economic Association's RCT Registry on 23 July 2018.

The following selected journals had the highest frequency of potentially relevant studies based on the initial pilot‐searches during the development of the search string and the protocol:

The search was performed on editions from 2015 to July 2018 (i.e., including an updated search) of the journals mentioned, in order to capture relevant studies recently published and therefore not found in the systematic search.

We performed a wide range of searches on the below institutional and governmental resources, academic clearinghouses and repositories for relevant academic theses, reports and conference/working papers. Most of the resources searched for grey literature include multiple types of references. The resources are listed under the category of literature most prevalent in the resource, even though multiple types of unpublished/published literature might be identified in the resource.

We contacted international experts to identify unpublished and ongoing studies. We primarily contacted corresponding authors of the related reviews mentioned in Section 2.4.1,5 and authors with many and/or recent included studies. The following authors replied: Douglas Fuchs, Lynn Fuchs, Russell Gersten, Nancy Scammaca, Robert Slavin, and Sharon Vaughn. Furthermore, during work with another review about the use of randomised controlled trials in Scandinavian compulsory school, authors were contacted about studies with sometimes overlapping inclusion criteria with the current review (see Pontoppidan et al., 2018).

In order to identify both published studies and grey literature we used citation‐tracking/snowballing strategies. Our primary strategy was to citation‐track related systematic reviews and meta‐analyses. 1446 references from 23 existing reviews were screened in order to find further relevant grey and published studies (see Section 2.4.1 and the list in Supporting Information Appendix A, subsection Grey Literature Searches). The review team also checked reference lists of included primary studies for new leads.

Under the supervision of the review authors, at least two review team assistants independently screened titles and abstracts to exclude studies that were clearly irrelevant. Any disagreement of eligibility was resolved by the review authors. We retrieved studies considered eligible in full text. Two review team assistants then independently screened the full texts under the supervision of the review authors. Any disagreement of eligibility was resolved by the review authors. The review authors piloted the study inclusion criteria with all review team assistants.

Two members of the review team independently coded and extracted data from included studies. A coding sheet was piloted on several studies and revised. Any disagreements were resolved by discussion, and it was possible to reach consensus in all cases. We extracted data on the characteristics of participants, characteristics of the intervention and control/comparison conditions, research design, sample size, outcomes, and results. We contacted study authors if a study did not include sufficient information to calculate an effect size. Extracted data was stored electronically, and we used EPPI Reviewer 4, Microsoft Excel, and R as the primary software tools.

We assessed the risk of bias of effect estimates using a model developed by Prof. Barnaby Reeves in association with the Cochrane Non‐Randomised Studies Methods Group. This model is an extension of the Cochrane Collaboration's risk of bias tool and covers risk of bias in non‐randomised studies that have a well‐defined control group. The extended model is organised and follows the same steps as the risk of bias model according to the 2008‐version of the Cochrane Handbook, chapter 8 (Higgins & Green, 2008). The extension to the model is explained in the three following points:

The refined assessment is pertinent when thinking of data synthesis as it operationalises the identification of studies (especially in relation to non‐randomised studies) with a very high risk of bias. The refinement increases transparency in assessment judgements and provides justification for not including a study with a very high risk of bias in the meta‐analysis.

The analysis of effect sizes involved comparing an intervention to control or comparison conditions. We conducted separate analyses for short‐ and follow‐up outcomes. The below sections apply to both types of outcomes, unless otherwise mentioned.

Errors in statistical analysis can occur when the unit of allocation differs from the unit of analysis. In cluster‐randomised trials, participants are randomised to intervention and control groups in clusters, as when participants are randomised by school. QES may also include clustered assignment of treatment. Effect sizes and standard errors from such studies may be biased if the unit‐of‐analysis is the individual and an appropriate cluster adjustment is not used (Higgins & Green, 2011).

Our protocol stated that we should adjust studies individually using the methods suggested by Hedges (2007). However, of the 61 studies with clustered assignment of treatment, less than a third contained any information about realised cluster sizes, and estimates of the ICC or the within‐cluster and between‐cluster variances (the ICC is the ratio between the between‐cluster and the total variance). Only a handful contained all the necessary information (both realised cluster sizes, and/or the ICC and the variances). We therefore adjusted all studies in a similar way and used an ICC of 0.09, which is very close to the mean of both reading and mathematics taken over Grades K‐6 in the pretest covariate models of tables 6 and 7 in Hedges and Hedberg (2007, pp. 72–73). In the sensitivity analysis, we report both results using unadjusted effect sizes and using a substantially higher ICC (0.3) than in the primary analysis.

We assessed missing data and attrition rates in the individual studies using the risk of bias tool. Studies had to permit a calculation of a numeric effect size for the outcomes to be eligible for inclusion in the meta‐analysis. Where studies had missing summary data, such as missing standard deviations, we derived effect sizes where possible from, for example, F ratios, t values, χ² values and correlation coefficients using the methods suggested by Lipsey and Wilson (2001). If these statistics were also missing, we asked the study investigators if they could provide us with the information. We were unable to retrieve information from 24 studies. These studies were included in the review but excluded from the meta‐analysis (see Supporting Information Appendix C: Studies with overlapping samples or lacking information for a summary description).

Many studies did not provide data about all moderators. See Section 4.3.10 and Table 1 for information about missing moderator data. As the number of included studies limited the number of moderators we could include in the analysis, we focused on moderators that had no missing data, were relevant for all types of studies, and were not highly correlated with other moderators.

In a sensitivity analysis, we used multiple imputation to assign values to moderators with relatively low levels of missing information. We defined “relatively low” as missing in less than 20% of interventions. We confined the sensitivity analyses further to moderators that were relevant to all intervention types (e.g., the number of sessions and hours per week is not relevant for incentive and progress monitoring interventions). We constructed imputed data sets using the 3.6.0 version of the mice package in R (first developed by Van Buuren & Groothuis‐Oudshoorn, 2011), which uses chained equations to predict missing values. We averaged our results over five imputed data sets using the methods described in Rubin (1996) and used the predictive mean matching method to assign missing values.

Heterogeneity was assessed with χ² (Q) test, and the I², and τ² statistics (Higgins et al., 2003). In Supporting Information Appendix L, we also provide the prediction intervals for the main effects and subgroup analyses (this analysis was not included in our protocol).

Reporting bias refers to both publication bias and selective reporting of outcome data and results. Bias from selective reporting of outcome data and results is one of the main items in the risk of bias tool.

To examine possible publication bias, we used funnel plots, Egger's test (Egger et al., 1997), and tested whether studies published in scientific journals had different effect sizes compared with other studies. We used the R package metafor to conduct these tests, and the restricted maximum likelihood (REML) estimation procedure with the Knapp and Hartung adjustment of standard errors (Viechtbauer, 2010; this procedure was recommended by e.g., Langan et al., 2019).

The simulation results in Pustejovsky and Rodgers (2019), published after our protocol, indicated that the original Egger's test often reject the null hypothesis of no asymmetry at higher rates than the chosen level of statistical significance (i.e., the Type I errors are inflated). We therefore also conducted a version of the “Egger sandwich”‐test suggested by Rodgers and Pustejovsky (2020), which had good Type I properties in their analyses. For the Egger sandwich‐test, we used the same RVE procedure as in the primary analysis (see next section for more details) and simply added a measure of the precision of each effect size, equal to (N/n₁n₂)^0.5, to the respective estimating equation.

We conducted the overall data synthesis in this review when effect sizes were available. Effect sizes coded with a very high risk of bias (score of 5 on any item in our 5‐point scale) were not included in the data synthesis. The primary analysis had the following steps. We described summary and descriptive statistics of the intervention‐level characteristics, and the risk of bias assessment, and then performed analyses divided by measurement timing (end‐of‐intervention or follow‐up), which corresponded to our first objective for the review. We then explored heterogeneity across instructional methods and content domains (corresponding to the second objective), and other study characteristics (corresponding to the third objective). We describe these subgroup and moderator analyses in the next section.

We used the RVE procedure in the R command robumeta (Fisher et al., 2017) in all our analyses. The RVE procedure allowed us to simultaneously include all effect sizes from each study and avoid problems with dependence between effect sizes for estimation and to calculate robust standard errors (Hedges et al., 2010). We used the random‐effects model weighting scheme option, as it seemed most likely that the effects of the included interventions were not the same across studies, but follow a distribution. A fixed effects model would therefore be less appropriate in our case (e.g., Borenstein et al., 2009). As there were many more effect sizes from studies conducting several tests on the same samples than effect sizes from studies reporting results from independent samples, we chose the correlated effects model instead of the hierarchical effects model in robumeta.

The RVE procedure requires an initial estimate, ρˆ, of the correlation between tests within the same study. We used ρˆ = 0.8 (as e.g., Dietrichson et al., 2017; Hedges et al., 2010; Wilson et al., 2011). We report 95% CIs throughout the analysis (i.e., “statistically significant” denote p < .05) and used the small sample adjusted standard errors and degrees of freedom suggested by Tipton (2015) to calculate CIs. We reported when the adjusted degrees of freedom were close to or below 4, as the results in Tanner‐Smith and Tipton (2014) and Tipton (2015) indicate that the standard errors are not reliable below this level.

Despite that the RVE procedure may have some disadvantages in terms of estimating heterogeneity parameters (see Tanner‐Smith et al., 2016), we chose to use the same framework for all analyses in order to make sure that disparate results were not caused by using different statistical models. We provide a sensitivity analysis for our main results in the “Publication bias” section and in Supporting Information Appendix J: Forest plots by intervention component, where we used study‐level effect sizes (a simple average of the effect sizes in each study) and the REML procedure in the R package metafor in the analysis (Viechtbauer, 2010).

One of the main objectives of the review was to assess the comparative effectiveness of intervention components. We therefore performed subgroup and moderator analysis to attempt to identify the characteristics of interventions and study methods that were associated with effect sizes. We again used the RVE procedure in robumeta and reported 95% CIs for regression coefficients. We performed two types of analyses: single‐factor subgroup analysis, in which we estimated RVE models including only an intercept on samples of effect sizes defined by the subgroup of interest, and multiple meta‐regression analyses, in which we estimated RVE models including additional moderators besides the intercept. Although both types of analyses are regression‐based, we sometimes refer to the latter as just “meta‐regressions” to ease the reading. Below we describe the variables we used to define subgroups and as moderators, and thereafter a roadmap for the analysis, which includes a discussion of the advantages and disadvantages with the different types of analyses.

Most included moderators were coded as indicator variables (most variables are natural indicators, e.g., whether the study design was an RCT or not). Continuous variables were mean‐centred to facilitate interpretation. Our protocol specified the following types of moderators:

It is important to note that the number of included studies and the number of effect sizes were not large enough to include all coded moderators in one meta‐regression (see Supporting Information Appendix D: Coding scheme for a description of all coded variables). In line with the objectives of the review and our protocol, we therefore focused the analysis of subgroups and heterogeneity on instructional methods and content domains. These components are substantive features of interventions that for example teachers and principals can affect, in contrast to other moderators (e.g., participant characteristics may be more difficult to affect for a school). They were also more often (quasi‐)experimentally manipulated in studies than other moderators in our sample. They may therefore be less likely to be confounded with other, omitted moderators. However, we want to emphasise that the moderator and subgroup analysis estimate the associations between moderators and effect sizes, and may not capture the causal effects (Thompson & Higgins, 2002).

To further reduce the number of moderators, we first excluded moderators with very low variation (i.e., for which nearly all observations have the same value) or where information was missing from studies. We also excluded moderators that were not relevant for all intervention types (e.g., there is no number of sessions in an intervention that provide students with incentives to read a certain number of books).

We characterised the included interventions using two general categories of treatment modalities or intervention components: instructional method and content domain. As described in our protocol, the components were not fully prespecified, but developed and adapted during the coding process. We used previous reviews and author‐reported classifications in included studies as a starting point, and an iterative process to construct component categories. Below, we describe the coded components by treatment modality, and how we used these components to develop the moderators we included in the analyses. Note that interventions often contained more than one component and they were coded in all component categories they contained. The categories below are therefore not mutually exclusive.

We only coded that an intervention included a component when it was clear from the study that the component was used. For example, Torgesen et al. (2007) write that their intervention contained extensive “professional development and support” but do not mention whether teachers were coached or not. Therefore, we did not code the interventions in this study in the coaching of personnel‐category.

Some studies examined the effects of the same programme (e.g., READ180, PALS). In some cases, the same programme was described differently across the studies. As implementations may differ, we used the descriptions provided in the studies as much as possible and the same programme may thus be coded in different categories. In other cases, information about for example group sizes was lacking from a study and we then inferred a likely group size from information about the programme in other studies of the same programme.

We performed the following sensitivity analyses.

Figure 1 displays the results of the search process. The total number of potentially relevant records was 24,414 after excluding 187 duplicates (database search: 17,444; grey literature: 3014; citation tracking: 1509; author contacts: 576; hand search: 1024; trial registries and others: 847).

We screened all records based on title and abstract. Of the ones that we did not exclude, 201 records were not retrievable in full text. Older reports and dissertations were overrepresented among the records we could not retrieve. We screened the remaining 4247 retrievable records in full text. A large number of studies were not relevant for this review due to the Grade of the participating students. Studies that were relevant except for the grade of participating students were included in a review covering Grades 7–12 (Dietrichson et al., 2020). In total, 607 studies met the inclusion criteria for this review and we extracted data from these studies.

We did not include all 607 studies in the meta‐analyses. We were unable to retrieve sufficient information from 24 studies/study authors to calculate an effect size. We excluded 17 studies, which used samples that overlapped with other included studies and had either higher risk of bias or contained less information. 104 studies were not included in the meta‐analyses due to their study design. These studies used comparison designs that contrasted two alternative interventions and the contrast was not recurring in enough studies for meta‐analysis to be meaningful (one study included both an intervention‐control contrast and a comparison design and we counted it among the studies included in the meta‐analysis). We did not include 257 studies in the meta‐analysis due to the risk of bias assessment. All eligible outcomes in these studies had, in our view, too high risk of bias (for more details, see further below).

Furthermore, some of the remaining 205 (202 intervention‐control and 3 comparison designs) studies contained overlapping samples, but included for example information about short‐term and follow‐up outcomes. These were all included in some meta‐analyses, but never in the same. We treated studies that reported short‐ and follow‐up outcomes from the same intervention in separate papers as one study in the analysis (i.e., when clustering the standard errors). This definition left us with 195 included clusters of studies in the analysis of intervention‐control studies (none of the comparison design studies had this problem).

We discuss the results of the risk of bias assessment further in Section 5.2. See also the Risk of bias tables in Supporting Information Appendix F for details of the assessment for effect sizes that we included in the meta‐analysis, as well as those that we deemed had too high risk of bias (Supporting Information Appendix G). We describe the comparison designs in the Supporting Information Appendix E: Description of comparison designs. For more information about studies with overlapping samples or that lacked sufficient information for the calculation of an effect size, see the Supporting Information Appendix C: Studies with overlapping samples or lacking information.

The data we coded for the 195 (clusters of) intervention‐control studies and the comparison design studies included in the meta‐analyses are included in Supporting Information Appendicesand. Below we describe the characteristics of the 195 intervention‐control studies, which we based most of the analyses on. H I

Figure 2 displays the included studies by publication year. Most studies were published in the last 10–15 years, 28 included studies were published before the year 2000 and only 7 before 1990. It is therefore unlikely that we missed many relevant studies by restricting our searches to studies published after 1980. Figure 3 shows the 195 studies by the mean Grade of participating students during the implementation of the intervention. There were more studies with participants in kindergarten and, in particular, first Grade than in the higher Grades. The mean Grade was 2.4.

Table 1 contains descriptive statistics of the intervention‐control studies included in the meta‐analysis. Many studies contained more than one intervention, the effects of which may have been tested with more than one standardised test from which we calculated the effect sizes. We denoted the number of studies that provided information about a certain characteristic with k, the number of interventions with i, and the number of effect sizes with n (note that these are not necessarily unique study populations, as some studies with more than one intervention group used only one control group). As most characteristics vary on the intervention level, we averaged – with three exceptions marked with *—over interventions to calculate the mean, standard deviation, and range. For example, in a study with two interventions where each intervention and control group take two tests, we averaged by intervention over the two tests. These averages are the basis for means, standard deviations, and ranges in Table 1 (which is why there is an i subscript in the table). There were in total 195 studies, 327 interventions, and 1334 effect sizes.

Included interventions were to a large extent conducted in the United States (86%). The remaining interventions were from Australia (0.6%), Canada (0.9%), Denmark (0.3%), Germany (1.2%), Ireland (0.3%), Israel (0.3%), Netherlands (2.1%), New Zealand (2.1%), Sweden (3.7%) and the United Kingdom (2.1%). The mean number of participants, schools, and districts were 280, 12, and 3, respectively, but sample sizes varied quite widely and many studies were small. Most included study designs were RCTs, only 7% of interventions were QES. A small share, 11%, reported having some form of implementation problems, and a large majority of the tests (91%) tested a single reading or mathematics domain, that is, they were not general in our terminology. In 22% of the interventions, follow‐up tests were conducted (i.e., students were tested more than 3 months after the end of intervention).

Participants were more likely to be boys (45% were girls), and a majority were minority (65%) and low‐income students (70%). Information about minority and, in particular, low‐income students was relatively often missing. Note that we often based the share of low‐income students on the share receiving free‐ and reduced price lunches in studies from the United States, which is not necessarily directly comparable to low‐income variables used in other countries (e.g., free school meals in the United Kingdom).

The effects of interventions were more often examined using reading tests, 18% used a mathematics test. Note that the separation of effect sizes was made based on the test, not the subject targeted, as there were several interventions that used tests in both reading and mathematics (i = 19), or used a composite reading and mathematics tests (i = 2). This is also a main reason why we do not separate results into reading and mathematics interventions to start with (we will return to this issue in the analysis of heterogeneity).

The mean duration was about 23 weeks, and the mean frequency and intensity equalled 69 sessions and 2 h per week. The range was wide for all three variables measuring intervention dosage and note that we lack information for quite a few interventions regarding frequency and intensity (for 28 and 27 interventions, respectively). Among half of the interventions were implemented by school staff.

Among the instructional methods, small‐group instruction is clearly the most studied method. Around 65% of all interventions included small‐group instruction. The shares otherwise ranged from 2% in the other method‐category to 15% in the CAI and peer‐assisted instruction category.

The proportions of interventions targeting reading domains were more similar, from fluency being targeted in 30% of all interventions to decoding that was targeted by 51% of all interventions. Most interventions (55%) targeted multiple reading areas. Fewer interventions targeted mathematics and the shares ranged from 3% for fractions to 15% for number sense and operations. Targeting more than one domain was common also for math interventions, 24% of interventions targeted more than one math domain. Regarding the non‐math and non‐reading domains, meta‐cognitive strategies was taught by 13%, social‐emotional skills by 5%, and general academic skills by 4%.

The fact that combinations of both instructional methods and content domains was relatively common imply that there are few single component interventions. As shown in the lower part of the table, it is only small‐group instruction that have been studied alone in a large number of studies. Small‐group instruction was the only instructional method in 50% of all interventions. Among the content domains, single domain interventions were rare.

Due to the large number of studies screened in full text, we were unable to describe all excluded studies. To exemplify how we applied the inclusion criteria, this section describes studies that met all but one of our criteria.

The included study designs contrasted intervention and control groups, or alternative interventions, to estimate effects. Compton et al. (2012) examined 129 first Grade children who were unresponsive to classroom reading instruction and were randomly assigned to 14 weeks of small‐group intervention. They defined nonresponders and responders using a standardised test of word identification, and then used logistic regression models to examine which information predicted the group students ended up in. As the study did not compare an intervention group to a control group, we excluded it.

We included interventions that sought to improve academic achievement or specific academic skills. We however excluded interventions that used accommodations to improve results on tests or only sought to improve test‐taking skills, like Scruggs and Mastropieri (1986). We also excluded interventions that primarily aimed to improve student attendance, like the intervention examined by Guryan et al. (2017). Although improving attendance may also improve student achievement, it need not do so on standardised tests. Furthermore, in groups of low attenders, improving attendance in the intervention group compared with the control group changes the composition of the students who take standardised tests, making it difficult to evaluate whether academic skills have improved (e.g., because the programme may affect student achievement through other channels than just attendance, an instrumental variables analysis need not work, even if the programme is randomly assigned).

Interventions had to be targeting students with or at risk of academic difficulties to be included. Al Otaiba et al. (2011) examined an intervention where teachers received help to differentiate reading instruction in kindergarten based upon students’ ongoing assessments of language and literacy skills. Although the intervention included students’ from at‐risk groups (e.g., students who qualified for special education) and some schools were Title I and Reading First schools (indicating that a relatively large share of the students were from disadvantaged groups), we excluded the study. The programme did not specifically target students with or at risk of academic difficulties, but all students in the participating schools, including high‐performing students.

Interventions should be school‐based to be included, meaning that they were conducted in school during regular school‐hours and semesters. Kim (2006) studied a voluntary summer reading intervention. Although some reading lessons took place in during the spring semester, most of the intervention period was outside the regular school year. We therefore excluded the study.

Studies had to test the effects of interventions using standardised tests in reading and mathematics. Fortner (1986) tested intervention effects with the Test of Written Language (TOWL). TOWL is standardised but primarily a test of writing. However, it includes a subtest of vocabulary but the students’ vocabularies are assessed based on their writing products. Therefore, we thought this subtest was more of a writing than a reading assessment, and excluded the study.

Figure 4 shows the distribution of the assessments for intervention‐control effect sizes included in the meta‐analysis by the items in the risk of bias tool. See Supporting Information Appendix F: Risk of bias in studies included in the meta‐analysis for a description of the ratings by study and item.

Few RCTs reported how they generated the random sequence used to assign students to intervention and control groups, only 7% of effect sizes were given a low‐risk assessment (QES have high risk by default on this item and therefore also on allocation concealment). More generally, the procedure of randomisation was often not described in detail. In almost all cases where the random sequence generation was described and was adequate, the allocation was likely concealed, as the randomisation was not done sequentially.

All studies had problems with the blinding of treatment status. No effect size received a rating of 1 on this item and very few studies provided an explicit discussion about blinding. There was some variation between effect sizes though: For around 66% of effect sizes, there was no indication that any participant group was blind to treatment status. About 32% of effect sizes had one group that was blind to treatment status (usually the persons performing the tests), and in 2%, several groups were likely blinded. The ratings for this item vary to some extent also within studies, as some studies, for example, used both tests performed by persons outside the study (e.g., state‐wide tests) and by involved, non‐blinded, study personnel.

The distribution of assessments for incomplete outcome reporting was more mixed. Only 5% had a high risk of bias rating on this item and almost all studies provided information. We rated a large majority (74%) of effect sizes to be free of selective reporting, but this does not mean that the studies followed a prespecified protocol and analysis plan. The figure omits the items examining if the study followed an a priori protocol and analysis plan, as just two studies mention a protocol and three an analysis plan explicitly written before conducting the analysis. Torgerson et al. (2011) was the only study for which we could retrieve both a protocol and an analysis plan. Lastly, about 17% of effect sizes were rated as having a high risk of bias for the other bias item.

The confounding item was only assessed for the 19 QES; 75% of effect sizes from these studies received a rating of 4. That is, a high risk of bias. Only 7% of effect sizes from QES received a rating of 1 or 2.

In sum, the included effect sizes and studies have relatively often a high risk of bias, but there is also variation. We return to the sensitivity of our results to different part of the risk of bias assessment in Section. 5.4

This section presents the results from the robust‐variance estimation of the overall short‐term effects—that is, from the end of intervention to 3 months after—and effects with a longer follow‐up period.

We included 1030 effect sizes, 189 clusters, and 206,186 student observations8 in the analysis of short‐term effects. Eleven individual studies did not provide results from a short‐term test. The weighted average short‐term effect size was positive and statistically significant (ES = 0.30, CI = [0.25, 0.34]). This effect size corresponds to a 58.4% chance that a randomly selected score of a student who received the intervention is greater than the score of a randomly selected student who did not (a null effect would imply a 50% chance, see e.g., Ruscio, 2008, for a conversion formula). The Q‐statistic was 797.3 (p < .01), the τ² 0.067 and the I² was 76.4. All three heterogeneity measures therefore indicated substantial heterogeneity.9

Figure 5 displays the distribution of short‐term effect sizes. The figure underscores that the effect sizes are heterogeneous. Although most effect sizes are centred around the mean (the red line), there are examples of very large positive effect sizes as well as large negative effect sizes, indicating substantial heterogeneity.

Most studies did not report follow‐up effect sizes: there were 195 effect sizes from 27 studies measured more than 3 months after the end of intervention, which included 19,902 student observations. The weighted average follow‐up effect size was positive and significant (ES = 0.27, CI = [0.17, 0.36]). This effect size corresponds to a 57.5% chance that a randomly selected score of a student who received the intervention is greater than the score of a randomly selected student who did not. The Q‐statistic was 47.8 (p < .01), the τ² 0.03, and the I² was 45.6, which, although lower than for the short‐term effects, indicated that there was some systematic variation in the effect sizes and significant heterogeneity.10

The average follow‐up effect size was almost as large as the average short‐term effect size. The reason is not only that studies measured outcomes very close to 3 months after the end of intervention. Exploratory analyses revealed that the average effect size measured between 4 and 12 months after the end of intervention was ES = 0.26 (CI = [0.15, 0.37]) and between 12 and 24 months was ES = 0.17 (CI = [−0.03, 0.37]). These analyses included 22 and 9 studies, respectively. That is, the effects were still substantial also for the longer follow‐up periods. There were only 5 studies measuring effects after more than 24 months (ES = 0.11, CI = [−0.14, 0.36]) and the adjusted degrees of freedom fell below 4. This result was therefore unreliable.

It is possible that the follow‐up measurements are mainly confined to interventions that were successful in the short‐term (we found no example of a study with a protocol that detailed a follow‐up measurement in advance). We found an average short‐term effect size of ES = 0.40 (CI = [0.24, 0.55]) among the 22 studies that provided both a short‐term measure and at least one follow‐up measure. That this effect size was larger than the effect size for all studies is an indication that successful interventions are more likely to be examined with follow‐up tests. In turn, this may explain part of the similarity between the short‐term and follow‐up average effect sizes in our sample.

Studies with follow‐up measurements were also different in two other ways. All but five studies used small‐group instruction (i.e., student groups of five or below) and all but eight studies tested effects only on reading measures (two of the eight studies tested both math and reading). Among the exceptions, two studies used peer‐assisted instruction, one study used groups of max eight students, one study used CAI and incentives, and one study changed only the content domain. Thus, evidence of medium‐ to long‐term effects pertains almost exclusively to small‐group instruction interventions that examined the effects on reading measures. Confining the analysis of follow‐up effects (>3 months after end‐of‐intervention) to studies using small‐group instruction yields an ES = 0.28 (CI = [0.16, 0.39]) and confining the analysis further to include only interventions using small‐group instruction and testing effects on reading measures yields an ES = 0.27 (CI = [0.12, 0.42]).

Summarising the analysis thus far, we found evidence of reasonably large and statistically significant short‐term and follow‐up effects. All measures indicated substantial heterogeneity of the short‐term effects, which included a mix of intervention types. The evidence for the follow‐up effects pertains almost exclusively to studies examining small‐group instruction and to effects on reading measures, and we found few studies that have examined effects more than two years after the end of intervention. Because of the small number of studies, and the few instructional methods and content domains used in the follow‐up studies, we focused the subgroup analysis and investigation of heterogeneity in the following section on the short‐term effects.

The previous analyses indicated substantial heterogeneity of the short‐term effects. This section examines if we can explain some of this heterogeneity using subgroup and moderator analysis. The analysis follows the five‐step roadmap laid out in Section: 4.3.10

To test sensitivity to how effect sizes were measured and calculated, we included four moderators indicating whether we used the raw means to calculate the effect size, standardised the SMDs with the control group standard deviation (i.e., used a Glass's δ), if the standardisation was unclear, or the effect size was standardised with a standard deviation from a super‐population. There were no Glass's δ, unclear effect size types, and effect sizes standardised with a super‐population among the follow‐up outcomes and among the single method peer‐assisted instruction effect sizes. In these regressions, we just included the raw means‐indicator. There were no Glass's δ and no effect sizes standardised with a super‐population among the single method small‐instruction effect sizes. Thus, this regression included the raw means‐indicator and the indicator for unclear effect sizes.

Including effect size measurement‐moderators in the analyses did not change the estimated average effect sizes much (all increased somewhat). Although the average effect sizes retained their statistical significance in all four analyses, the CIs became broader in all cases. This was expected, as many effect sizes in all analyses was measured or calculated differently and the effect size shown in the figures is the average among those that were not. In particular, we used the raw means to calculate a relatively large proportion of the effect sizes.

The raw means‐indicator was not statistically significant in any analysis, and changed sign between specifications. Effect sizes of unclear type were associated with smaller effect sizes in the three specifications in which we could include it and so were those standardised with a super‐population in the analysis of overall short‐term effects. The only statistically significant moderator was the unclear effect size type in the analysis of small‐group instruction interventions (β = −.27, CI = [−0.40, −0.14]).

In the multiple meta‐regression in Table 8, Glass's δ and unclear effect size type are relatively large but not significant, whereas raw means and standardising with a super‐population are small and not significant. Peer‐assisted instruction and small‐group instruction retained both their magnitude and statistical significance in the meta‐regression compared with the primary analysis.

We examined the distributions of effect sizes for the presence of outliers and the sensitivity of our main results by methods suggested by Lipsey and Wilson (2001): trimming the distribution by dropping the outliers and by winsorizing the outliers to the nearest non‐outlier value. We show the latter results in the figures below. Supporting Information Appendix K: Extra sensitivity analyses contains figures showing the effect size distributions of the four types of effect sizes and the results of additional analyses.

Although is difficult to come up with a definition of outliers that is not in some sense arbitrary, there are quite a few effect sizes, particularly in the small‐group instruction category and among the short‐term effects (many of which are the same), which seem like clear outliers. Outliers seem rarer among the follow‐up and peer‐assisted effect sizes, although there a few potential examples. The short‐term and small‐group distributions start to thin out around 1.5 and −0.5, respectively. We used these values as cut‐offs when winsorizing.

The studies with larger and smaller effect sizes than these cut‐offs were almost all small sample studies. All except one effect size was based on a total sample size of 60 or under. Otherwise, they were not exceptional in terms of other potential explanations, for example, study design, risk of bias, or whether we used raw means to calculate effect sizes or not.

Winsorizing outliers had a small impact on our results. All average effect sizes in the subgroup analyses are reasonably close to those in the primary analysis and were still statistically significant (the follow‐up effect size increased to 0.34, which is the largest difference). Peer‐assisted instruction and small‐group instruction were still sizeable and statistically significant in the multiple meta‐regression.

In Supporting Information Appendix K, we report results from analyses where we sequentially remove outliers down to effect sizes in between −0.25 and 1. The results are also again relatively close to those in the primary analysis. Thus, outliers do not seem to be driving our results. It is also worth noting that when we removed outliers, the heterogeneity decreased by quite a lot. For example, with the harshest cut‐off, the I² was 34.4% and the τ² was 0.02 in the meta‐regression including both instructional methods, content domains, and study characteristics.

We tested sensitivity to clustered assignment of treatment by the methods described in Section 4.3.5. In the effect size estimates shown in Figures 10, 11, 12, 13 (named “4. Clustered”), we did not adjust for clustering. We report results in text from specifications in which we instead adjusted effect sizes using a substantially higher ICC (0.3) than in the primary analysis.

As can be seen in Figures 10, 11, 12, 13 and for peer‐assisted instruction and small‐group instruction in Table 8, not adjusting for clustered assignment of treatment left our estimates virtually unchanged. Although adjusting for clustering decreases the individual effect sizes, the average effect sizes decreased slightly when we used unadjusted effect sizes. The reason is likely connected to the fact that the variances of studies with clustered assignment treatment is larger when we adjust for clustering. Therefore, adjusting for clustering gives the adjusted effect sizes smaller weights in the meta‐analyses. As effect sizes from studies using clustered assignment of treatment tend to be smaller in our sample, the average effect size decreased when these studies receive more weight.

A more surprising result was that the CI actually increased a little with unadjusted effect sizes in the analysis of follow‐up effects (the other intervals became slightly broader). The reason may be that the between‐study heterogeneity decreased as well (τ² increased from 0.031 to 0.042). If effect sizes using clustered assignment were further away from the average than effect sizes using individual assignment when not adjusted, they will receive less weight when we adjust. If the ensuing reduction in τ² is large enough, it will dominate the effect of increasing the individual variances by adjusting for clustering.

Increasing the ICC to 0.3 strengthened the above tendencies, although the differences to the primary analysis were again very small. All our main results retained their significance also with this substantially higher ICC (see Supporting Information Appendixfor these results). K

We used the items with numerical ratings from the risk‐of‐bias assessment to examine if the ratings were associated with effect sizes. As described in Section, we coded indicator variables that contrasted effect sizes given a higher risk of bias rating to effect sizes with a lower risk. We defined higher risk as 4 or unclear for the items blinding, incomplete outcome reporting, and other bias, and as ratings higher than 1 for the selective outcome reporting item. We coded all indicators so that they equalled 1 for ratings indicating a higher risk of bias and included them in the multiple meta‐regressions. 4.3.11

As discussed in Section, we omitted the QES from this sensitivity analysis. The coefficient on the constant in the meta‐regressions for overall short‐term effects, overall follow‐up effects, peer‐assisted instruction, and small‐group instruction should therefore be interpreted as the average effect size in RCTs with a relatively low risk of bias. In the analysis of peer‐assisted instruction, we could not include the blinding indicator because there were only two studies rated low risk. Including it together with the other indicators meant that the reference category would have been empty. The constant in the meta‐regressions including both intervention components and study characteristics without missing values is the average effect size in RCTs that did not use any of the included components/characteristics and was at the mean of the mean‐centred variables. 4.3.11

All average effect sizes in Figures 10, 11, 12, 13 increased slightly when we adjusted for the risk of bias ratings compared with the primary analysis. That is, effect sizes with less risk of bias tended to be slightly larger, although the differences were minor. The CIs became broader in all analyses, reflecting that we rated relatively few studies as having a (relatively) low risk of bias on all included items. However, all average effect sizes in RCTs with a low risk of bias were statistically significantly different from zero. Both peer‐assisted instruction and small‐group instruction retained both the magnitude and statistical significance in the multiple meta‐regression. The risk of bias‐item indicators all had small and statistically insignificant associations with effect sizes in this regression.

In the multiple meta‐regressions reported in Table 5, we omitted potentially important moderators because they had missing values. In this section, we report results from specifications using multiple imputation to account for missing values of moderators with relatively low rate of missing values, as described in Section 4.3.6. We imputed values for the following moderators: an indicator for implementation problems, share of girls, share of minority students, duration, and an indicator for whether school staff implemented the intervention (as opposed to e.g., researchers or volunteers). Information about these moderators were missing from <20% of interventions and they are potentially relevant for all types of interventions. Moderators such as the number of sessions and hours per week were also missing from <20% of interventions but are not relevant for all types of interventions (e.g., there are no sessions in incentive and progress monitoring interventions).

The results in column 1 of Table 9 indicate that adjusting for these moderators does not change our main results. The coefficients on peer‐assisted instruction and small‐group instruction are of similar size and both are still statistically significant. Furthermore, all other coefficients are close to the values displayed in column 2, Table 5. The coefficients on the imputed moderators are all small and not statistically significant.

This section examines whether there is heterogeneity across the control group conditions by examining the control group's progression from pre‐ to posttest. As in our analysis of heterogeneity of the overall effect sizes, we confined this analysis to the short‐term effects. We calculated a control group “effect size”, as the difference between pre‐ and posttest divided by the control group posttest standard deviation. We used the posttest standard deviation to be able to better compare the progress with the effect sizes found in the primary analysis.

We then (a) tested whether this control group “effect size” was heterogeneous across studies and (b) used multiple meta‐regression to examine whether study characteristics explained some of the heterogeneity. We included 142 studies that supplied the necessary information in the analysis. That is, this sample is different from our primary analysis sample and the results reported should be viewed with some caution in relation to the heterogeneity of control group progression in the primary analysis sample.

We found substantial heterogeneity in the control group progression across studies. We first estimated a specification including just a constant using the same RVE procedure as in the primary analysis (not shown in Table 9). The constant therefore shows the weighted average control group progression, which is 0.62 (CI = [0.50, 0.74]). There is substantial variation around this average: the τ² is 0.70, the I² is 96%, and the Q‐statistic is 3949, which is highly significant. There continues to be heterogeneity when we confine the set of interventions to those using peer‐assisted instruction (mean = 0.47, τ² = 0.07, I² = 55.4, and Q = 31.4) or small‐group instruction (mean = 0.61, τ² = 0.28, I²= 88.2, and Q = 534.3) as their only instructional method.

The results in column 2 of Table 9, which mimics the specification shown in Table 5, column 2, shows that very few of the instructional methods and content domains were significantly associated with the control group progression. The only two significant associations are with number sense and mean Grade. Number sense is positively associated and mean Grade negatively associated with effect sizes. These results are not surprising, given that students on average make less progress, the higher the Grade (Lipsey et al., 2012), and that number sense is typically a focus in the early years of primary school.

Thus, the results indicated that control group progression was strongly heterogeneous, which may imply that the quality of control group instruction is an important explanation of the heterogeneity we see in most of our analyses. However, it was reassuring that we did not find strong evidence of an association between the control group progression and intervention components, and in particular that we did not find a significant association with peer‐assisted instruction and small‐group instruction.

This section examines publication bias by testing whether unpublished studies have different effect sizes compared with published studies, and by using funnel plots and Egger's test (Egger et al., 1997). However, we want to acknowledge that these tests are for several reasons difficult to interpret as direct evidence of publication bias, or the lack thereof. The effect size estimates and corresponding standard errors we have analysed are not necessarily the ones used by authors, editors, and reviewers to decide whether a paper should be published. We used only effect sizes from standardised tests that had low enough risk of bias, which were included in studies found through our search and screening process. There may thus be publication bias in the literature that would not show up in our sample and other processes than publication bias can cause asymmetries in funnel plots. We discuss the interpretation of funnel plots further below.

We found no evidence that effect sizes from studies published in scientific journals were larger than effect sizes from studies not published in journals (e.g., in government reports, working papers, and dissertations). We added an indicator equal to 1 if the study was published in a journal to the specification in column 2 of Table 5 for the full sample of short‐term effects. The estimate, reported in column 3 of Table 9, indicated that studies published in journals, conditional on all other moderators without missing values, have slightly lower effect sizes (β = −.06), but not significantly so.

For the funnel plots and test of asymmetry, we averaged effect sizes and variances over studies and estimated a random‐effects model by using the REML option with the Knapp and Hartung adjustment of standard errors in the R package metafor (Viechtbauer, 2010). As mentioned, this procedure also provided a sensitivity test of our primary analysis.

Table 10 shows the average effect sizes, CIs, and heterogeneity statistics for the REML procedure compared with the RVE procedure used in the primary analysis. The effect sizes and CIs from the two procedures are very close in all four cases. The heterogeneity statistics indicate more heterogeneity in the RVE procedure, which seems reasonable as the RVE procedure takes into account variation in effect sizes within studies. However, our conclusions would be similar using study‐level averages. There is substantial heterogeneity across short‐term and small‐group instruction effect sizes. The heterogeneity across follow‐up effect sizes is smaller, but still statistically significant, whereas the heterogeneity among the effect sizes from peer‐assisted instruction intervention is relatively small and not significant.

Figure 13 displays funnel plots of the study‐level short‐term effect sizes (upper left corner), follow‐up effect sizes (upper right corner), peer‐assisted instruction effect sizes (lower left corner), and small‐group instruction effect sizes (lower right corner). The effect sizes are shown on the x‐axis and standard errors on the y‐axis. The center‐line displays the weighted average in each analysis (i.e., the effect size from column 6 in Table 10).

There are indications of asymmetries primarily for short‐term effects and small‐group instruction. There seem to be more outliers with positive effects and more large than small studies with effect sizes around null. Although they seem less asymmetric, the drastically smaller number of studies in the follow‐up and peer‐assisted categories makes these plots harder to interpret. Conducting Egger's test (Egger et al., 1997; we used the regtest option in metafor), we rejected the null hypothesis of no asymmetry for the short‐term effect sizes (p < .001), and small‐group instruction effect sizes (p = .002) but not for follow‐up effect sizes (p = .165) and peer‐assisted instruction (p = .518). We got qualitatively similar results when we used the “Egger sandwich” test suggested by Rodgers and Pustejovsky (2020): we again rejected the null hypothesis for the short‐term effect sizes (p < .001), and small‐group instruction effect sizes (p = .007) but not for follow‐up effect sizes (p = .107) and peer‐assisted instruction (p = .606).

Asymmetric funnel plots may have other causes than publication bias. In general, asymmetry is a sign of small‐study effects, of which there can be many causes beside publication bias (Sterne et al., 2005). Small‐study effects would show up as heterogeneity (Egger et al., 1997). In line with this idea, the analyses displaying more heterogeneity across effect sizes in Table 10 and that have more studies outside the funnel lines in Figure 14 also have lower p values in Egger's test. A general reason to expect small‐study effects is that researchers often perform statistical power analyses before embarking on an intervention. As interventions with large expected effects require smaller sample sizes for a given level of power, small studies will have larger effects if researchers are reasonably good at guessing the effect sizes (Hedges & Vevea, 2005).

In our context, there are several further reasons why studies with larger sample sizes and consequently smaller standard errors could have smaller effect sizes. One reason is that larger samples tend to be more heterogeneous and may therefore have larger standard deviations (e.g., because they include school district variation and not just school variation as argued by Lipsey et al., 2012). Larger standard deviations would mean smaller standardised mean differences in large studies, even if the effects were exactly the same.

A second reason is that it may be easier to get large positive effects in studies with small samples. Students and teachers can be given more attention and be better monitored by researchers in small studies and there is less risk of coordination and implementation problems. More attention and better monitoring seem likely to increase effect sizes (see Thomas et al., 2018, for an interesting discussion and results in line with this hypothesis) and coordination and implementation problems may cause both smaller effect estimates of otherwise effective programmes and decrease the chances of publication, regardless of publication bias.

A third reason has to do with the recruitment of schools and teachers to studies. In small‐scale studies, researchers typically recruit schools or teachers directly, which implies that only schools or teachers that want to participate are included in the sample. In large‐scale studies, larger administrative units like school districts or municipalities are more likely to be the unit of recruitment. In turn, these larger units may determine which schools and teachers that participate, meaning that some schools and teachers that do not want to participate are included in the sample. If being motivated to participate is important for how well schools and teachers implement the intervention, which seems reasonable (e.g., Kennedy, 2016), then, regardless of publication bias, we should expect smaller effects in large‐scale studies. More generally, it may be easier for small‐scale interventions to select sites that are particularly likely to benefit whereas large‐scale studies is more informative about the population‐wide effects. Depending on the type of policy‐maker (or researcher), both types of effects are interesting, but they are not likely to be the same.

Such connections between sample sizes and effect sizes would violate the assumptions needed to interpret asymmetric funnel plots and Egger's test as publication bias (it would also violate the assumptions underlying the selection models suggested by, e.g., Hedges, 1992 and Andrews & Kasy, 2019). Furthermore, if sample sizes are systematically different across intervention components, the analyses we presented in this review may risk confounding the associations of intervention components with sample sizes.

We examined this issue further by including two study‐level moderators related to some of the reasons stated above in our analyses. The first moderator is equal to one if the effect size/study included a larger than the median number of schools (median = 8), districts (1), or regions (1). If the study contained no information, it was coded as zero along with studies below the median. About 57% of the effect sizes and 52% of studies had a larger number of units regarding at least one of these three units. The second moderator is an indicator for clustered assignment of treatment. As mentioned, 61 studies (around 31%) used a clustered assignment of treatment, which amounted to 17% of the effect sizes (i.e., clustered studies included on average fewer standardised tests/effect sizes). Both of these indicators typically means that the sample size is larger and may imply more coordination problems, as well as a different recruitment process (unfortunately, we did not code how participants were recruited or whether a power analysis was conducted).

We first added the two moderators to the specification reported in column 2, Table 5. As can be seen in Table 9, both indicators were negatively associated with effect sizes but only the indicator for clustered assignment was statistically significant. Reassuringly, none of our other results changed much compared with our primary analysis. We then proceeded to add the same two moderators in the specification underlying Egger's test in the two analyses where this test indicated funnel plot asymmetry. Although the moderators continued to be negatively associated with effect sizes, we still found a significant asymmetry in the funnel plots of overall short‐term effect sizes (p < .001) and small‐group instruction effect sizes (p = .004). As the heterogeneity may depend on outliers, we also ran the test using winsorized effect sizes (with cut‐offs at −0.5 and 1.5). This did not change the outcome of the test, which was still significant for both short‐term effect sizes (p < .001) and small‐group instruction (p = .010).

Of course, these results do not prove that there is publication bias, the moderators are imperfect indicators of the phenomena we want to capture and there may, as discussed, be other reasons for the funnel plot asymmetry.

The sensitivity analysis showed that the positive and statistically significant overall average short‐term and follow‐up effect sizes, and the positive and statistically significant associations between peer‐assisted instruction and small‐group instruction were generally robust. The exceptions were that there were too few peer‐assisted instruction effect sizes with a relatively low risk of bias on the blinding item for us to include this item in the analysis, and that there were indications of asymmetric funnel plots in the analyses of short‐term and small‐group instruction effect sizes. While we therefore should be more cautious in the interpretation of our results, these results should not be interpreted as a confirmation that there is publication bias. There are other reasons for asymmetric funnel plots, which seem likely to apply in our case.

We found significant average effect sizes of CAI across interventions that included this component (ES = 0.15) and in single method interventions (ES = 0.13), but smaller and not significant associations in meta‐regressions where we adjusted for other components and study characteristics. Dietrichson et al. (2017) found a combined average effect size in reading and mathematics of 0.11 for CAI interventions targeting students with low SES. Slavin et al. (2011) and Inns et al. (2019) reviewed interventions for struggling readers and similarly defined instructional technology interventions or programmes. They found average effect sizes of 0.09 and 0.05, respectively. Slavin and Lake (2008) found an overall effect size of 0.19 for CAI interventions targeting mathematics and general student populations, but stated that effects were similar for disadvantaged students and students with a non‐majority ethnic background. Our results for CAI are thus reasonably close to those from comparable analyses in earlier reviews.

The average effect size of coaching of personnel was around 0.20 and statistically significant in interventions that included this instructional method. We found no evidence that coaching was associated with substantial or significant effects in meta‐regressions or in single method interventions. Combining math and reading outcomes, Dietrichson et al. (2017) found an average effect size of 0.16 for coaching interventions targeting students with low SES. They did not examine single method interventions or included coaching in meta‐regressions. Our results for coaching of personnel are thus close to the comparable analysis in their review (Kraft et al., 2018, also reviewed coaching interventions but did not report results for at‐risk groups).

We found a relatively large average effect size in interventions including an incentive component (ES = 0.33). The average effect size was smaller and not significant in single method interventions (ES = 0.05) and in the meta‐regressions. Dietrichson et al. (2017) found an average effect size of 0.01 for incentive interventions targeting students with low SES. Although their coding and analysis methods were most comparable to the one we used to obtain the effect size of 0.33, our result seemed to be driven by the inclusion of interventions that combined incentives with other methods, in particular small‐group instruction. Most of these studies were not included in Dietrichson et al. (2017). Our single method and meta‐regression results are close to their results.

Instruction in medium‐sized groups had a relatively large and significant average effect size in interventions including such a component (ES = 0.32), but smaller and not significant in single method interventions (ES = 0.10) and in the meta‐regressions. Dietrichson et al. (2017) found an average effect size of 0.24 for a similarly defined category (called small‐group instruction) in their review of interventions targeting students with low SES. As the methods used to derive their result was closest to our ES = 0.32, the results are reasonably close.

Peer‐assisted instruction had a relatively large effect size averaged over interventions that included this component (ES = 0.44), in single method interventions (ES = 0.39), and retained significance and a similar size in the meta‐regressions. Dietrichson et al. (2017) found an average effect size of 0.22 for cooperative learning interventions targeting students with low SES. Slavin et al. (2011) found a large average effect size of cooperative learning interventions for struggling readers (ES = 0.58). Inns et al. (2019) found an average effect size of 0.29 for “classroom approaches”, where four out of five studies were of cooperative learning/same‐age peer‐tutoring programmes. Gersten et al. (2009) found very large effects for cross‐age tutoring (ES = 1.02) in their review of math interventions for school‐age learning disabled students, but they included only two such studies. The effects for same‐age (within‐class) peer‐assisted instruction was lower, 0.14, and decreased further when they adjusted for other methods and study characteristics in a meta‐regression.

The target group in Gersten et al. likely had more severe academic difficulties than the average student in our review (as well as the target group in Slavin et al., 2011, and Inns et al., 2019). One explanation of the differences across reviews is therefore that peer‐assisted instruction may work less well for students with the greatest difficulties. Another possible explanation may be that Gersten et al. also included effect sizes based on non‐standardised tests, but they adjusted for this in the meta‐regression and non‐standardised tests typically yield larger effects. Gersten et al. also included interventions targeting older students (although most of their included studies included participants in our Grade range). Note that Gersten et al. only included math interventions, but our results were similar for peer‐assisted instruction across reading and math tests.

Interventions including progress monitoring had a significant average effect size of 0.17, whereas there were only two progress monitoring interventions where this method was the only one (ES = 0.28). The associations with effect sizes was small and insignificant in our meta‐regressions. Dietrichson et al. (2017) found an average effect size of 0.32 for progress monitoring interventions targeting students with low SES, but they included only four studies of such interventions and this method was in all cases combined with other methods. Gersten et al. (2009) examined math interventions for students with learning disabilities. Their category teacher feedback combined (ES = 0.23) was reasonably close to how we defined progress monitoring. As in our case, when Gersten et al. included other moderators in a meta‐regression the teacher feedback components (Gersten et al. split the component in two in 7their meta‐regression) lost their statistical significance.

Small‐group instruction showed consistently significant and relatively large effect sizes in analyses of interventions including this component (ES = 0.38), in single method interventions (ES = 0.38), and retained both size and significance in meta‐regressions. Dietrichson et al. (2017) found an average effect size of 0.36 for tutoring interventions targeting students with low SES in an analysis similar to our Table 3. Inns et al. (2019) found an average effect size for “one‐to‐small group” tutoring of 0.20 and 0.25 for one‐to‐one tutoring. Slavin et al. (2011) found an average effect size for one‐to‐one tutoring by teachers of 0.39, by paraprofessionals of 0.38, and volunteers of 0.16, and an average effect size for small‐group tutoring of 0.31 (both Inns et al., 2019, and Slavin et al., 2011, reviewed programmes/interventions for struggling readers in Grades K‐5/6). Wanzek et al. (2016) found larger average effect sizes of one‐to‐one (ES = 0.50), one‐to‐two or three (ES = 0.61), and one‐to‐four or five (ES = 0.44) student small‐group instruction, but they included only interventions in Grades K‐3 in their review of students with or at risk of reading difficulties. Wanzek et al. (2018) found average effect sizes of 0.59 for one‐to‐one tutoring and 0.33 for small‐group tutoring in the review of intensive (100 sessions or more) reading interventions for at‐risk students in Grade K‐3. The results of our review was thus in between the effect sizes reported in earlier reviews, which may be explained by our broader inclusion criteria and larger number of studies.

Other reviews with a target group consisting of more general student populations have also analysed small‐group instruction and tutoring. As tutoring nearly always is targeting students with or at risk of difficulties, we discuss them as well. Fryer (2017) reported an average effect size of 0.31 for math achievement and 0.23 for reading for “high‐dosage tutoring”, which was quite close to our definition. Pellegrini et al. (2018) found an average effect size of 0.25 for combined one‐to‐one and small‐group tutoring in math. Nickow, Oreopoulos, and Quan (2020) examined RCTs in pre‐K to Grade 12 of tutoring, defined as one‐on‐one or small‐group instructional programming by teachers, paraprofessionals, volunteers, or parents, and found an overall pooled effect size of 0.37. Ritter et al. (2006) reviewed the effectiveness of volunteer tutoring programmes for improving the academic skills of student enroled in Grades K‐8 (in the United States), and found effect sizes of 0.30 for reading outcomes and 0.27 in mathematics. As our small‐group instruction category contained almost only tutoring interventions, these results agreed reasonably well with ours.

We found few differences between math and reading effect sizes, and between effect sizes from interventions targeting more narrowly defined content domains. Interventions targeting the reading domains comprehension, decoding, fluency, spelling and writing, and vocabulary had average effect sizes ranging from 0.20 to 0.32. We examined the following mathematics domains: algebra/pre‐algebra, fractions, geometry, number sense, operations, and problem solving, which had average effect sizes ranging from 0.15 to 0.50. Only comprehension (ES = 0.21), decoding (ES = 0.31), number sense (ES = 0.51), and operations (ES = 0.17) had been examined in more than one single domain intervention. The meta‐regressions revealed few significant differences across content domains. The exception was fractions, which had a significantly stronger association with effect sizes compared with the other math domains. However, given the small number of interventions targeting fractions and that just one of them targeted only fractions, we caution against strong conclusions based on this result.

Few reviews have examined effect sizes by the domains targeted by interventions for our target group. Gersten et al. (2009) examined the following math domains: operations, word problems, fractions, algebra, and general math proficiency. The categories with similar names as ours also had similar definitions. Their word problems category was similar to our problem‐solving category and their general math proficiency similar to our multiple math category. They found larger effect sizes for word problem interventions in a meta‐regression, but the differences to other domains were not significantly different.

Regarding reading domains, Wanzek et al. (2016) examined effects of interventions on standardised measures of language and comprehension, in Grade K‐3. They coded studies by the outcomes, not the by the targeted domain. However, it seems likely that interventions target the areas that are tested, so their coding may be reasonably close to ours. They found an average effect size of 0.38 for language and comprehension outcomes among interventions in Grade K‐3. While this was slightly larger than our effect size for comprehension, students in their review were in lower Grades and we found larger effects for the lower Grades.

Scammaca et al. (2015) included no study using standardised tests in 4–5th Grade, which was their analytical category closest to our Grade range. The overall average effects in Scammaca et al. (2015) measured by standardised tests were 0.25 in Grade 6–8, which was close to our overall effect size on reading tests. Their effect sizes reported by reading domain were larger than ours for reading comprehension (ES =0 .47) and word study (ES = 0.68), and reasonably similar for fluency (ES = 0.17) and multiple components (ES = 0.14). These analyses were not grouped by Grade however, and thus included secondary students as well. Flynn et al. (2012) found larger effects on standardised tests for comprehension (ES = 0.73), decoding (ES = 0.43), and word identification (ES = 0.41) than we did, but found negative effects on fluency (ES = −0.29; they included students in Grade 5–6 as well). Both Scammaca et al. (2015) and, especially, Flynn et al. (2012) included substantially fewer studies in the comparable Grades and it is also unclear what instructional methods were included in their analyses.

Interventions targeting more domain‐general skills than reading and mathematics had positive effect sizes in our review. The average effect sizes for general academic skills (ES = 0.21) and meta‐cognitive skills (ES = 0.24) were statistically significant, while social‐emotional skills (ES = 0.24) was not significant. These domains were rarely the only targeted domain and we found no evidence that interventions targeting domain‐general skills had larger effect sizes than interventions that only targeted reading and mathematics domains. However, we included only standardised tests in reading and mathematics in the analysis, and targeting domain‐general skills may have important effects on other types of tests.

Few of the earlier reviews with similar target groups as ours included similar categories of domain‐general skills. Dietrichson et al. (2017) included a category called psychological/behavioural interventions (ES = 0.05), which was somewhat similar to a combination of our meta‐cognitive and social‐emotional categories. Pellegrini et al. (2018) included a social‐emotional learning category but their category included whole‐school reforms, which were not included in our review, and their review included interventions targeting general student populations. Thus, their average effect size of 0.03 is difficult to compare with ours.