The importance of independent replication of treatments in plant science Free

Among other foci, the Journal of Experimental Botany aims to advance understanding of plant–environment interactions, including abiotic stress, mineral nutrition, and the response of plants to global change. Advancing understanding in these areas often requires manipulation of the growth environment. Experiments range in scale and include: growth chambers (Yiotis et al., 2020), glasshouses (Rho et al., 2019), whole-plant chambers (Sharwood et al., 2017), and open-air field experiments (Ruiz-Vera et al., 2020). The application of environmental manipulations can be constrained by available resources, operational costs, and time. In designing experiments, researchers attempt to balance logistical challenges with the desire to address scientific questions and maximize statistical power.

Unfortunately, by design or default, pseudoreplication (Hurlbert, 1984) remains a problem in plant science. The Journal of Experimental Botany expects independent replication of experimental treatments, randomized experimental designs, and use of appropriate analytical approaches. Here we highlight the issue for our readers, reviewers, and authors; and remind authors that a full description of the experimental design and statistical methods should be included in research papers. This is now aided by the journal’s new policy of excluding the Materials and methods section from the word count of research papers.

What is pseudoreplication and why is it a problem?

Pseudoreplication is the collection of what the researcher considers to be independent samples from different treatments that they wish to compare, when in fact the samples are not independent because they come from a single experimental unit (Davies and Gray, 2015). Pseudoreplication is a problem because: (i) random effects or events affecting just one treatment can lead to incorrect conclusions; and (ii) the treatment of subunits as independent experimental units artificially inflates the statistical significance of numerical differences (Hurlbert, 1984; Lindroth and Raffa, 2017; Silk et al., 2020).

Identifying the experimental unit

The experimental unit is the unit to which the treatment has been applied (Box 1). Correctly identifying the experimental unit is critical, but it is not always intuitive (Lindroth and Raffa, 2017). The nature of the experimental unit can change with scale or experimental design. To illustrate this point, we provide three examples, where the experimental unit is either a field plot (Fig. 1A), a growth chamber (Fig. 1B), or a plant (Fig. 1C). Terms such as replicate, repeats, technical replicates, biological replicates, semi-biological replicates, treatment replicates, measurements, or samples are often used to describe experimental design, but have ambiguous meanings (Zar, 1999; Cumming et al., 2007; Vaux et al., 2012). For example, in Fig. 1B, the biological replicate is a subunit, but in Fig. 1C it is the experimental unit. The statistical evaluation of a treatment effect, and the presentation of error bars associated with a treatment, should be at the level of the experimental unit (Hurlbert, 1984; Cumming et al., 2007; Vaux et al., 2012).

Random effects and events

At the simplest level, there are two basic statistical requirements for a good experiment, replication and randomization, or, more accurately, interspersion (Hurlbert, 1984; Mead et al., 2003). The hypothetical validity of using unreplicated treatments is dependent upon the experimental units being identical at the time of manipulation and remaining identical throughout the duration of the experiment, except for differences resulting from the manipulation (Hurlbert, 1984). Meeting these criteria is necessary to ensure that measured effects in an unreplicated experiment can be attributed to the manipulation, and not to random variation between experimental units. For this reason, such validation is practically impossible if n=1 and a treatment effect is indistinguishable from random effects (Potvin and Tardif, 1988; Cottenie and De Meester, 2003; Vaux et al., 2012; Ramage et al., 2013). Replication of the experimental unit, and interspersion—the distribution of experimental units in space—provide the best insurance against chance events producing spurious effects that could be attributed to the treatment or manipulation (Hurlbert, 1984).

Consider a field experiment (Fig. 1A) with replicated (n=5) but undispersed experimental units (plots). In this example, all the plots receiving treatment level 1 are located at the north end of the field and all those receiving treatment levels 2 or 3 are located in the middle and the south end of the field, respectively. The problem with this experimental design is that unknown differences in land use history or drainage between the north and the south end of the field could lead to a spurious treatment effect. Even if the field had no pre-existing variation, events occurring during the experiment could impact the treatments unevenly. For example, overspray of irrigation or a pesticide from an adjacent field to the north will affect plots used for treatment 1 to a much greater extent than those used for treatments 2 and 3. The problems with this hypothetical experimental design are well recognized and easily mitigated with alternative designs, such as a randomized block (Mead et al., 2003). Fortunately, the experimental design depicted in Fig 1A is rarely encountered in field trials. However, an extreme form of this kind of simple segregation often arises when plants are grown in controlled environments where logistical constraints begin to exert an influence on experimental design.

Consider a growth chamber experiment (Fig. 1B) where the effect of elevated carbon dioxide concentration ([CO₂]) is investigated using plant growth chambers set at three different [CO₂] levels. The experimental design described in Fig. 1B is an example of what Hurlbert referred to as isolative segregation (Hurlbert, 1984), and is unfortunately common in plant science. In this example, the researchers have taken care to set the environmental conditions of the chambers to be the same, except for the CO₂ treatment, but the potential for spurious effects resulting from uncontrolled or unmeasured effects remains. As with the research field example (Fig. 1A), spurious effects can manifest as pre-existing differences between chambers; for example, prior to this experiment, chamber 1 was used by another researcher for a plant pathogen experiment and not properly cleaned after use, the air intake for chamber 2 is adjacent to a piece of equipment that generates ozone, or a researcher installed different bulbs in chamber 3 that do not have the same spectral qualities as those installed in chamber 1 or 2. Differences can also emerge during an experiment, such as a reduction in irradiance in chamber 1 due to a bulb failing, or contamination of a CO₂ tank with ethylene in chamber 2 (Morison and Gifford, 1984). Since all the plants for a given treatment are located in one chamber, pre-existing or emergent effects that are unique to one chamber are guaranteed to affect all of the plants experiencing the treatment assigned to that chamber; isolating and segregating the treatments exacerbates the dangers of simple segregation (Hurlbert, 1984). Furthermore, isolation in a single chamber increases the risk of Type I errors (Box 1) because isolation promotes uniformity among the plants within the chamber, reducing variance and increasing statistical power.

Note that the principal of interspersion is also important to consider when evaluating genetic modifications. Segregation and isolation of transgenic plants from their wild-type controls would confound the ability to confidently attribute an observed phenotype to the genetic manipulation, and it is therefore critical to grow transgenic plants in the same chamber as their wild-type control plants.

Statistical considerations

In statistics, a Type I error occurs when a test falsely identifies a significant difference between treatments when there is no real difference. The specified significance level (α) determines the probability of committing a Type I error—when α=0.05, we are accepting the probability of incorrectly identifying a difference between treatments 5% of the time. A Type II error (Box 1) occurs when a test fails to identify a difference between treatments when it actually exists; the probability of committing a Type II error is β. Higher probabilities of committing a Type I error are associated with lower probabilities of committing a Type II error, and vice versa (Zar, 1999; Mead et al., 2003).

The statistical power (1–β) is the probability of correctly identifying a difference between treatments for a given α. Power is determined by the following parameters: the difference between the treatment means; the pooled variance of the treatments; the number of levels of the treatment (k); and the number of experimental units (n) (Zar, 1999; Cohen, 1988; Cottingham et al., 2005). The resulting degrees of freedom associated with the treatment, commonly referred to as the group error degrees of freedom (v₁, k–1), and the experimental units, commonly referred to as the error degrees of freedom [v₂, k(n–1)] also influence statistical power. Statistical power increases notably when the magnitude of the treatment effect increases, or when variance decreases, and importantly as n and the associated v₂ increase. This was presented graphically by Pearson and Hartley (1951) and reproduced by Zarr (1999). Therefore, incorrectly treating subunits as experimental units falsely inflates statistical power and dramatically increases the likelihood of Type I errors. For these reasons, authors are expected to clearly identify the experimental unit, and report n, and the degrees of freedom associated with their experimental design and statistical analysis.

In our growth chamber example (Fig. 1B), the plants within each chamber are not independently subjected to the CO₂ treatment and, for the purposes of statistical analysis, cannot be considered as true experimental replicates (Box 1). In this example, the unit of replication is the chamber, and n=1 for each level of the treatment (Cumming et al., 2007; Wernberg et al., 2012; Johnson et al., 2016; Lindroth and Raffa, 2017).

Common approaches to address issues of replication and interspersion

It is common for treatments to be rotated between growth chambers during an experiment as a means to reduce the potential for spurious chamber effects (Drag et al., 2020; Yiotis et al., 2020). For example, researchers running the experiment described in Fig. 1B may attempt to equalize unintended chamber effects across the three CO₂ treatments by rotating the plants, and their CO₂ treatment, among the three chambers during the experiment. This action may indeed reduce the possibility for spurious chamber effects, but it does not eliminate it as the effect of variation among chambers may have a temporal component (Potvin and Tardif, 1988). This could be addressed by staggering the treatments so that, in our example, all treatments are in each chamber for the same stage of growth (Johnson et al., 2016). However, there is still no control for what Hurlbert terms ephemeral events or demonic intrusion (Hurlbert, 1984). However, whilst chamber swapping may serve to minimize the chance of chamber effects impacting the results, the practice in no way guarantees it and does not address the central problem that the treatments are not independently replicated (Hurlbert, 1984; Johnson et al., 2016).

Faced with limited resources for plant growth, repeating experiments in time and randomly assigning the treatment to growth chambers is one approach that can be used to generate independent replication (Johnson et al., 2016). When replication of treatments in time or space is not possible, alternative designs can be considered. A regression approach can be used to analyse the response to continuous factors (Cottingham et al., 2005). For example, Drag et al. (2020) used a regression approach to analyse the effect of an unreplicated gradient of [CO₂] spanning eight treatment levels. Mixed-effect models can account for pseuodoreplication by the use of random effects that take into account the hierarchical nature of the experimental design (Harrison et al., 2018); however, mixed-effect models are not a cure for pseudoreplication resulting from poor experimental design (Silk et al., 2020) and can only help when a treatment has been repeated in more than one experimental unit (Johnson et al., 2016). In short, you cannot analyse your way out of a poorly designed experiment.

Is pseudoreplication a pseudoissue?

There has been considerable debate over the dogmatic identification of pseudoreplication and editorial treatment of work where it is identified (Oksanen, 2001, 2004; Cottenie and De Meester, 2003; Hurlbert, 2004; Davies and Gray, 2015). This debate has been focused on opportunistic experiments resulting from natural disturbance, or landscape-scale phenomena. In their contribution to the debate, Davies and Gray (2015) emphasize the importance and value of publishing such papers, but advocate for a cautious approach when reporting results from unreplicated natural experiments and for open acknowledgement by the authors when pseudoreplication is an issue (e.g. Eastman et al., 2021). However, they also highlight the genuine problem of simple pseudoreplication—such as that presented in Fig. 1B—in designed experiments (Davies and Gray, 2015).

Key conclusion

The credibility of a study is fundamentally determined by the experimental design (Christie et al., 2020). For that reason, JXB expects independent replication of experimental treatments and also requires a full description of the experimental design and the statistical approach so that reviewers and readers can properly understand how an experiment was replicated and analysed.

Acknowledgements

AR was supported by the United States Department of Energy contract no. DE-SC0012704 to Brookhaven National Laboratory. MLG was supported by a Biotechnology and Biological Sciences Research Council (BB/T015357/1) grant. JEL was supported by the Max Planck Society.

References