The TCR trial

The Test-Control-Reference (TCR) trial is the experimental base of EFSA’s prescribed two-step difference and equivalence testing methodology. Despite differences in the number of sites, reference varieties, etc., all TCR trials share some common properties. Let (Δ_TC, Δ_CR, Δ_TR) denote genotype group differences of the test versus control, control versus references, and test versus references, respectively. Let (D_TC, D_CR, D_TR) be the sample estimates and , and be the respective variances and co-variances. Under common statistical assumptions, i.e. independent and identically-distributed responses, two basic properties of all TCR trials are:

1. Genetic relationship: Both Δ_TC and Δ_TR contain the trait effect, but Δ_TR also contains Δ_CR due to Δ_TR = Δ_TC + Δ_CR, and Δ_CR represents the control background effect;
2. Experimental design: The paired arrangement of the test and control materials in the field, along with the above genetic relationship, leads to the following relationships:

Notice in property [a] that the differences of the three genotype groups are two dimensional: one dimension is the control background effect; the other dimension is the GM trait effect. While both Δ_TC and Δ_TR are measures of the GM trait effect, in principle, only one dimension should be left for the comparative evaluation of the trait. Conceptually, a new parameter, possibly a combination of Δ_TC and Δ_TR (since each one cannot represent the other), must exist for the best measure of the GM trait difference, without confounding by the background effect.

Property [b] is a direct result of the field arrangement. For example, and are the sampling variations of D_CR and D_TR, respectively, and should be equal due to the paired arrangement in the field and the common background effect. Similarly, the covariance of D_TC with D_TR should be the same as with D_CR. Other relationships can also be derived among variances and covariances.

The control material is the result of conventional plant breeding and there is no obvious preference for one control over other commercially-available varieties. Thus, the control background effect should be considered as random, and its potential value is no different from any individual reference in the reference genotype group. The random control background has been considered by [2], [5], and [8] in establishing the equivalence criterion, but not properly modeled (as will be shown in the following trait model) and incorporated in the statistical testing. In this investigation, this has been an underlying assumption in every step of the analysis, including the development of hypotheses, estimation of the trait effect, and the derivation of test statistics.

The remainder of this section addresses three fundamental questions: (1) how do we define the GM trait effect for a given background; (2) how do we model the random background effect in addition to other factors; (3) what is the sample performance of the estimation procedure?

GM trait difference for a given control background

Given Δ_TR, a mixture of the GM trait- and background effects can be decomposed using conditioning or regression. Under the assumption of a bivariate normal distribution of (D_TC,D_CR), consider the following hypothetical model:

Let , and denote the conditional mean and variance of D_TC and D_TR given D_CR. Using the notation of conditioning or regression, the model can be specified as: , and β depends on the correlation. The two equations in this model are linearly related, thus only one model equation is necessary since D_TR = D_TC + D_CR. This implies that, given D_CR, evaluation based on both D_TC and D_TR would be the same, and can be replaced by the evaluation of the intercept . The application of the above model depends on the correlation structure among the three genotype groups, which was derived in the last section. The objective of the conditioning is to eliminate the dependence of Δ_TR (when its estimate is applied in the equivalence testing) on Δ_CR (the control background effect). This is achieved by making use of the estimated background effect.

Let denote the correlation coefficient between D_TC and D_CR. The following equations are direct applications of bivariate normal theory [9]: (1) (2) where . Substituting the corresponding equations in property [b], (1) and (2) become: (3) (4)

By conditioning on the estimated background effect, we do not obtain total independence of Δ_CR in . However, in (4) becomes only fractionally-dependent on Δ_CR, and is even less than , which is known to be free of background variation.

Also, notice that the expectations of and , with respect to D_CR, are equal to the marginal means Δ_TC and Δ_TR, assuming D_CR follows a normal distribution with mean Δ_CR and variance . While the difference (D_CR − Δ_CR) would be reduced by increasing the size of the TCR trial, such as the number of sites, the background effect D_CR in expectation (i.e. Δ_CR) can never be reduced because the same control appears in all sites. Our objective is to find a measure of the trait effect less-dependent, or ideally independent, of the background effect.

Although is a regression-type of estimate, D_CR is a random variable which can take different values, making it different from classical regression. Let d_TC, d_TR, and d_CR be values of D_TC, D_TR, and D_CR estimated from a TCR trial. Two cases of special interest in (3) and (4) are when D_CR = d_CR and D_CR = 0, where D_CR = d_CR assumes the control background as in the current trial, and D_CR = 0 implies that the random control background is adjusted to the mean of the reference distribution.

(5)

Intuitively, (5) is a regression of D_TR on D_CR. is the regression coefficient based on the correlation structure, represents the expectation of the sample estimate of the intercept (or the expected mean of D_TR at D_CR = 0), and represents the expectation of D_TR given D_CR at the observed mean d_CR in the current study. The implication of will be discussed later in (8) and (9). The intercept is more interesting here; it has two components: the first part is Δ_TC, known to be a pure estimate of the GM trait effect; and the second part is a portion of Δ_CR, due to the covariance structure defined in property [b]. This is the familiar form of a simple regression: the intercept, say α, is equal to (μ_y − βμ_x) [10]; i.e. the dependent variable mean (μ_y) minus the regression coefficient (β) times the independent variable mean (μ_x). Here, the correlation and the regression coefficient are negative. The second part of (5) separates from Δ_TC as a potentially better measure of the trait effect. It’s shown later when and how much would make a maximum difference as compared with Δ_TC and Δ_TR.

The implication of (5) is technically-interesting and practically-important. By conditioning, three parameters of (Δ_TC, Δ_TR, Δ_CR) are resolved into a two-dimensional space of (, Δ_CR). In this two-dimensional space, the value of is determined mostly by Δ_TC and depends on Δ_CR only to the extent of the correlation between D_TC and D_CR. Besides, given the background, the expectations of D_TC and D_TR are the same. Thus, from a statistical point of view, or can be considered as the unique parameter of interest to replace Δ_TC and Δ_TR in the comparative assessment.

A trait model for the TCR trial

Characteristics of the TCR trial have posed some difficulties [3, 4] in statistical modelling, partly due to properties [a] and [b], and because of the differential field arrangement of the test and the control materials, versus references, among sites. At least three linear mixed models have been proposed for the TCR trial. The discussion of these models is delayed until later, but two common limitations are noteworthy (which were also concerns of the agricultural biotechnology industry [4]). The first limitation is the inability to decompose the trait effect and random background effect; these effects are therefore confounded in the equivalence testing. In fact, EFSA requires a two-step procedure—using EFSA Model 1 (assuming both the trait difference and the control background effect as fixed) for difference testing and Model 2 (assuming the independent random background effect separately for the test and the control) for setting equivalence limits [2]. The second limitation is the inability to separate the GM trait-related interaction from the control- and reference-related ones, which are all inherent to the design of the TCR trail. A GM trait-related interaction of two nearly-isogenic lines (i.e. the test and the control materials) with the site, if present, would certainly be different from a G×E interaction in a conventional plant breeding context. Even though the interaction may not always be an important source of variation, the statistical model should provide such an option, if needed. Instead, EFSA requests a separate, post hoc interaction analysis, restricting data to test and control only, and assuming the site effect is fixed [2]. These limitations are structural shortcomings of the existing methods. For this reason, a trait model is proposed, which defines effects on linear contrasts of genotype groups corresponding to either the trait- or control background effect. This contrasts with the current genotype models which define effects on each genotype group.

Two factors can be defined on the three genotype groups based on property [a]: the GM trait T with two levels (GM: the test material, and Non-GM: the control material and the references), and the background G also with two levels (Non-Ref: The test and the control varieties, and Ref: reference varieties). Then, G and T(G) represent two orthogonal, single-degree-of-freedom decompositions of the genotype group effects. While G represents the contrast of the test and the control as a pair versus the references, T(G) is the contrast of the test versus the control within the Non-Ref group.

The trait model can be expressed as: Model A where Y is the observed response; μ is overall mean; [S and B(S)] are environmental effects for site and block within each site, g is the background effect, and e is the residual. G and T(G) are main effects, and GS and TS(G) are corresponding interactions. Based on property [a] of the TCR trial, the background effect is defined as:

Only μ, G and T(G) are assumed to be fixed; all other factors are random and assumed to be independently distributed with mean 0 and corresponding variance. One advantage of the expression of Model A over the model equations (one equation for each genotype) in Kang and Vahl [5] and Vahl and Kang [8] is the avoidance of a specified covariance structure derived from the random common background effect between the test and the control materials.

The primary factor differentiating Model A from the existing methods can be summarized as follows. First, Model A assumes, in the effect g, that the random control background is identical for the control and the test, and has the same variance as the references, which is part of property [a] of the TCR trial. Second, there are three possible types of interactions in the TCR trial: the GM trait-by-site TS(G), the control background-by-site GS, and the reference-by-site. Inclusion or exclusion of each or all of them is optional depending on the data. Model A includes the first two types of interactions because of the balanced arrangement of the test and control materials across sites, which support the inclusion of the trait-by-site and the control background-by-site interactions in the analysis. A minor modification could also include the reference-by-site interaction, when a proper field arrangement of references is available across sites. Since our focus here is on modelling and estimating effects G and (G), and interactions GS and TS(G), Model A is sufficient. Additional discussion is provided in subsequent sections.

The components of variances and co-variances are as follows in a design with the same number, but different varieties of references, at each site. Overlapping references among sites will be discussed later. Let n_s denote the number of sites, n_b the number of blocks and n_sg the number of references at each site, and n_g the total number of references. (6) where , and are corresponding variance components of GS and TS(G), g and e, respectively, in Model A. A simplified case may help the understanding of (6) when no G×E is assumed, i.e. and . Then, (6) becomes (7)

Results in (7) show that under Model A, is the same as that of EFSA Model 1, but under Model A is the same as that of EFSA Model 2. This is because Model A includes not only the fixed trait effect but also the random background of the test and the control, i.e. the test and the control materials are assumed to be both fixed and random per the effect, rather than the identity of the genotype group.

When fitting Model A, (d_TC, d_CR, d_TR) are the sample means. Let and and be the sample variances and covariance. When these estimates are applied to Eq in (5), the sample estimates of the conditional means can now be obtained: (8)

Eq (8) indicates that d_TR is the estimate of Δ_TR given D_CR at its observed value d_CR. It simply says that, given the control background as d_CR, the observed trait difference d_TR would be a sum of two correlated differences d_TC and d_CR, which is what EFSA uses in their equivalence testing [2]. Given D_CR = 0, however, is only slightly affected by d_CR. It can be shown that the correlation between (the estimate of the so-defined GM trait effect ) and d_CR (the estimate of the background effect Δ_CR) is zero. That is, they are independent from each other when normality is assumed. Now, we have two independent estimates for two parameters, one for the background effect and another for the GM trait effect.

To estimate the variance of , the variance and co-variance of d_TC and d_CR can be obtained directly from (6) or (7). Even though the ratio is a random variable itself, its variance consists of higher order terms and the variances of d_TC and d_CR are the leading terms in the overall variance of . Therefore, an approximation can be obtained based on the variation of d_TC and d_CR: (9)

Note that has the same form as in (4) with each variance replaced by the corresponding sample estimates due to the nature of the regression type of estimation. Thus, denotes the sample estimate of the variance of the conditional difference. Though d_CR is the control background effect of a TCR trial and could never exactly equal zero, by property [a], d_CR is random and follows a distribution with mean zero and variance estimated from the references.

Since both D_TR and D_CR are the respective differences from the same reference mean, the regression of D_TR on D_CR is then equivalent to the regression of the test mean on the control mean. The conditional mean is the test mean given the control mean at the estimated reference mean. Therefore, would have minimum variance among estimates for given D_CR at any value other than zero. Furthermore, the sample variance would have the same form as the conditional variance of (3) and (4), i.e. . Both (8) and (9) are likelihood estimates of the corresponding parameter functions since each component is the likelihood estimate from the mixed model analysis.

Now, the following hypothesis can be tested using a t-test: (10) (11) with degrees of freedom (. When the Kenward-Rodger [10, 11] method is applied, the degrees of freedom can be obtained from the analysis for , and an additional adjustment of minus one is due to the regression.

Asymptotic and sample performance of the conditional method

First, the conditional and unconditional variances are compared, and the correlations are examined as functions of the variation settings. From Eqs (4) and (8), the asymptotic correlation coefficients can be calculated as: (12)

Fig 2 illustrates the relationship of standard deviations and relative to , and of correlation coefficients and , as functions of σ_g: σ_e. A few results are worth noting. In general, has the lowest variance as compared with D_TC and D_TR. Even though the variation of D_TR is mostly larger than that of D_TC due to the background effect, it is actually smaller when σ_g is small. When genotypic variation is negligible, the residual becomes the main source of variability even among references. As the total number of replicates of references is larger than that of the control (a common case in practice), becomes true, which is also revealed by Eqs (6) and (7).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Standard deviations of simple and conditional trait differences and their correlation.

Standard deviations (Left) of (dash) and (solid) relative to (100% dot line), and correlation coefficients (Right) of (dash) and (solid), as functions of σ_g: σ_e.

https://doi.org/10.1371/journal.pone.0210747.g002

Sample performance of the conditional mean is examined in Fig 3. Fig 3 shows 100 pairs of simulated values of D_TR and versus D_CR, all scaled by (or equivalently) to provide an intuitive impression of the effect of conditioning. Four variation settings were assumed by fixing and adjusting the value of accordingly. The Least-Square Mean difference, D_TR, shows positive parallel change with D_CR due to the common background, as long as a is large enough, say , or . The parallel change becomes dramatic as increases to 1. The conditional mean , however, shows no correlation with the background. Results in Fig 3 strongly support the message in Fig 1 that EFSA, by using D_TR as a test statistic, is mostly testing the background effect, as long as nonnegligible reference variation exists. In contrast, the conditional method using is independent of the background, and is thus a true test of the trait effect even if the reference variation is around zero.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Scatter plots of simple and conditional trait differences as functions of the background effect.

100 simulated mean differences D_TR (in Black) and conditional mean difference (in Red) versus the background effect D_CR in the unit of and 95% prediction ellipses for each setting of .

https://doi.org/10.1371/journal.pone.0210747.g003

Conditional difference testing

In this section, the practical performance of the difference testing methods is compared using the following hypotheses: (13)

Although hypotheses (13) are for difference testing, the results are directly related to equivalence testing, as is shown next.

The empirical Type I error of the difference testing was evaluated by simulation at α = 0.1 (the same as in [2]). Unlike those results in Fig 2, simulations are subject to sampling variation. The total variation mirrored the example in [2] and assumed a coefficient of variation (CV) of 40%, including both the genetic and residual components with an overall mean 10 and variation setting the same as in Fig 3. The site variation was assumed to be with no block variation. For the power analysis, the trait effect was varied from 0 to 3, which is equal to 0 to 0.75 standard deviations of the total effects of the reference and residual. The trait model and the conditional method were applied to each simulated dataset and the Kenward-Roger adjustment was applied in the difference testing. Simulations were repeated 10,000 times for each set of parameter values, and results were summarized over replicate simulations.

All three methods performed as expected with the Type I error being around the nominal level (approximately 7 to 13% except one case H₀₂: Δ_TR = 0 and ) (Table 1). Slight deviations from the nominal level are likely due to the bias in the likelihood estimates of the variance components; especially when the actual value is close to zero, the non-zero restriction is applied in the estimation, the assumed experimental design will dictate the degrees of freedom of the variance estimates, as well as potential biases. Results of the power analysis in Fig 4 confirm those from Figs 1–3: among the three difference tests, the conditional difference test is always the most efficient, and the simple difference test H₀₂: Δ_TR = 0 is often the least efficient. When is low, say < 0.1, the difference in statistical power is substantial between testing for H₀₁: Δ_TC = 0 and . As gets larger, such as , this difference diminishes, and the two tests become very similar.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Empirical power of three difference tests.

Statistical power of testing H₀₁: Δ_TC = 0 (dash), H₀₂: Δ_TR = 0 (dot), (solid) under four variation settings. Markers represent the simulation results, and the curves were generated as smooth lines by SAS procedure SGPLOT [12].

https://doi.org/10.1371/journal.pone.0210747.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The empirical Type I errors of difference testing under the trait model assuming Δ_TC = 0 at α = 0.10.

https://doi.org/10.1371/journal.pone.0210747.t001

Results in Fig 4 are directly relevant to those of EFSA’s equivalence testing. Large background variation in D_TR often makes the testing H₀₂: Δ_TR = 0 very inefficient with extremely low power, regardless of the trait difference. This implies that, when the trait difference is small, large variation in D_TR will result in a failure to reject the non-equivalence hypothesis at a rate much greater than the nominal α = 0.10 level. Alternatively, a large trait effect might confound with the background variation, resulting in a rejection of the non-equivalence hypothesis.

Revisiting EFSA

The two-step difference and equivalence testing approach was illustrated with a maize grain composition example [2, 3]. The purpose of revisiting that example here is to explore the performance of the trait model and conditional method. Before the re-analysis, results reported in EFSA (2010) were confirmed, including the post hoc testing of the trait-by-site interaction.

In the re-analysis with the trait model, SAS PROC GLIMMIX [12] was used since the procedure provides a likelihood ratio test on variance components using option COVTEST. One statistically-significant interaction (Neutral Detergent Fiber) was found at the 0.05 level (the same as used by EFSA) for the trait-by-site interaction, and none of the control background-by-site interactions were significant. In contrast, eight statistically-significant interactions were found using the EFSA method [2]—treating site as a fixed effect while restricting the data to test and control only (i.e., references are excluded). The discrepancy highlighted here may be the result of the data or the statistical procedure. Nevertheless, the interaction is an effect of the design, and should be reflected in the statistical model (as in the trait model) instead of in a fragmented, post hoc procedure, as recommended by EFSA [2].

EFSA could not reach a conclusion on equivalence for nine of the 53 analytes (including two analytes with reference variance estimated as zero) [2]. Among those analytes, only two were found to be significantly-different from the control (Table 2). Using s_g estimated from the corresponding model, the standardized differences d_TR/s_g of the seven analytes under the EFSA model are all greater than or close to 2.0 (from 1.92 to 2.75) in absolute value. Under the trait model, five of the seven show a background effect d_CR/s_g greater than 2 in absolute value. Obviously, the inability to reach conclusions on equivalence is the result of large control background effects, and has nothing to do with the GM trait itself. In the conditional difference testing, nine analytes showed a statistically-significant difference at the α = 0.10 level. The standardized differences are mostly less than 1 in absolute value and only two (Niacin and Vitamin B2) were slightly greater than 1 (ranging from 0.27 to 1.38 except Phytic Acid with reference variation estimated as zero).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Reanalysis of EFSA example by the conditional method and comparing results with those of EFSA method.

https://doi.org/10.1371/journal.pone.0210747.t002

G×E effects in the TCR trial

The lack of a genotype-by-environment interaction term is one of the main concerns highlighted by Ward et al. [4]. In fact, three types of genotype-by-environment interactions (G×E) can be defined based on the design of the generic TCR trial: the GM trait-by-site, the control background-by-site, and the references-by-site. The trait model proposed in this investigation can accommodate all or any subset of the interactions based on evidence in the data. One consequence among various models is that the trait-by-site interaction is the leading term in the standard error of the estimated trait difference under the trait model. The presence or absence of this interaction term often affects the error degrees of freedom used in testing the trait difference, even if the true trait effect is negligible or absent. This is likely why only eight significant trait differences were detected under the trait model, while 22 analytes were significantly-different under EFSA Model (1) [2]. As illustrated in [4], by neglecting to include any interaction terms, the difference tests conducted under EFSA Model (1) will always be based on an error term that mixes all potential sources of interaction and the residual with a correspondingly large degrees of freedom. If the GM trait-by-site and reference-by-site interactions are both non-zero and of similar magnitude, the error term is under-estimated and the associated degrees of freedom tends to be higher (relative to the true distribution of the test statistic), leading to an inflated false positive error rate that exceeds the nominal level. Other circumstances may result in the difference test being overly conservative. In fact, the only time when the difference test performs as intended under [2] is when all interactions involving site are zero [4].

The control background-by-site interaction may not be of interest to the GM safety assessment per se. It is, however, a component in the covariance structure, and will affect the standard error of the estimated background effect. The impact of this interaction must be determined on a case-by-case basis.

The reference-by-site interaction is different from the above two types of interactions. First, the TCR design does not require overlapping references among sites. Without overlapping references, this interaction cannot be properly estimated, and a good estimate of the interaction depends on a large degree of overlap. Second, the reference-related interaction will confound with the reference variation when absent in the statistical model. Such confounding should not have any consequence on the comparative assessment.

Equivalence criterion

The equivalence of the test and references should be determined in the context of natural variation, i.e. reference variation, when reference variation is a significant part of the total variation. When the reference variation is estimated as zero, EFSA (declares the equivalence criterion and the equivalence status as undetermined [2]. Presumably, when it is near zero relative to the residual variation, the equivalence criterion and the conclusion would be poorly determined. In practice, many endpoints have negligible reference variance. For example, the ratio of was estimated between 0 to 9.6 across 53 analytes in the EFSA example under Model A. It makes sense to use the reference variation for the equivalence limit only when is much greater than zero, say . When is small, the equivalence limit should consider other sources of variation, such as those discussed by [14].

Clearly, the conditional mean will need a different equivalence criterion because of its independence from the background effect. A plausible equivalence criterion for the conditional mean likely should consider the following: residual variation–the size of the standard error of the conditional mean difference, and the reference variation–providing biological context [15]. Ideally, such a criterion should be predetermined, like the 80–125% rule [16] used in the regulation of the generic drugs, except that the criterion for a GM crop would be on the variation instead of mean level when sufficient reference variation exists. The criterion should be general enough for all types of characteristics in terms of the variation setting . In addition, the criterion should be able to take account of a GM trait-by-site interaction, when needed. These questions are topics of ongoing research.