2.1. Matrix of measured data

The intra-class correlation has been used to characterize a wide range of different experimental data. It has been applied for example to directly measured physical data, such as muscular strength [20] and force [21]; to rated time of physical activity [22] and to reported subjective data such as sum scores of pain and anxiety symptoms [23]. We will refer to all such data as the result of measurements. One important restriction is that the result of a measurement is a quantity which can be represented in a meaningful way by a real number, i.e. by numerical data rather than categorical data [24]. A general description which covers most experimental situations of interest, provided that the ICC is applicable, should then be the following.

From a population P, a number n of subjects (i = 1,2,…,n) have been randomly selected. On these subjects some kind of measurement of a specific quantity x (for example, blood pressure) is performed. On each subject, the measurement is made k times (j = 1,2,…,k). The result obtained for subject (i) in measurement (j) is a real number x_ij (a blood pressure value). The complete experimental result for all the subjects and all the measurements may thus be written [4] as a matrix with n rows and k columns, i.e., a rectangular arrangement of numbers x_ij such as shown in Table 2. The "subjects" are almost always assumed to be individuals (in the literature sometimes referred to as, e.g., "targets" or "people"), and the "measurements" are often referred to as for example "conditions" or "raters", depending on the experimental or clinical situation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Experimental data matrix with n rows and k columns.

https://doi.org/10.1371/journal.pone.0219854.t002

We now wish to calculate an ICC value from this sample of n × k measured data. This sample ICC value is a quantity that characterizes this particular set of data, but, more interestingly, it is also a statistical estimate of the intra-class correlation coefficient ρ which characterizes the method of measurement more generally, i.e. when it is applied to any randomly selected sample of subjects from the population P. Thus, the ICC value may tell something about the reliability of the method.

2.2. Analysis of variance

Regardless of which ICC version is to be used, we first need to perform ANOVA (Analysis of Variance) on the measured data matrix, calculating the various possible sums of squares (SS) and, from them, the mean squares (MS). The experimental situation described in section 2.1 motivates the use of the so-called repeated measures ANOVA [1]. We do not yet have to consider which of these MS we actually will need when calculating the desired ICC; the SS and MS are all easily calculated anyhow, and for clearness we list their definitions in Appendix 1 (S1 Appendix), as well as the most important relations among them. In so doing we also define the presently used notation. We prefer a four-letter notation (MSWS = Mean Square Within Subjects; MSBS = Mean Square Between Subjects; MSBM = Mean Square Between Measurements; MSWM = Mean Square Within Measurements) except for MSE (= Mean Square, Error) and MST (= Mean Square, Total). This notation seems to be more unambiguous and also easier to interpret and memorize than the somewhat varying three-letter notations that are more commonly used.

From now on we assume that all the required mean squares have been calculated from a measured data matrix such as indicated in Table 2.

2.3. Model 1, leading to formula ICC(1)

2.4. Model 2, leading to formulas ICC(A,1) and ICC(C,1)

2.4.1. Statistical model and population ICC.

Conventionally, Model 2 (the two-way random model) is specified by the following statistical model [4–6]. Each value x_ij in the experimental data matrix is assumed to be the sum of five terms: (7)

The symbols μ and r_i have the same meaning as in Model 1. Thus, the r_i term is sampled from a normal distribution with mean = 0 and standard deviation σ_r. However, one new term c_j is added. Moreover, the term v_ij that describes the random fluctuations in each measurement has been replaced by the sum of two different contributions rc_ij and e_ij.

The term c_j represents a systematic error (bias), common to all measurements labeled (j); i.e. it affects column (j) in the data matrix. For example, these measurements may have been performed by a rater, who, for some reason, has a tendency to obtain somewhat higher x_ij values. Following Bartko [26], who named c_j an "additive bias", we will refer to the c_j terms simply as "bias terms". In Model 2 it is assumed that these bias terms are random, in the sense that each c_j value is sampled from a normal distribution with a mean value = 0 and standard deviation σ_c; thus, if the entire experiment is repeated with k measurements on each of n new subjects, then the biases c_j will in general not be the same. For example, raters may have been replaced by new raters with other biases.

The term rc_ij is a so-called interaction term; it takes into account that the effect of bias may not be the same for all subjects. In Model 2 it is assumed that for each x_ij, the value of rc_ij is to be sampled from a normal distribution with mean = 0 and standard deviation σ_rc. The term e_ij, finally, is a residual (error) term. It is, for each x_ij separately, sampled from a normal distribution with mean = 0 and standard deviation σ_e.

We wish to stress the fact that these verbal descriptions of the various terms should be regarded as a statistical interpretation of the model given by Eq (7). What defines the model from a purely mathematical point of view is the algebraic form of Eq (7) and the precise specifications of how the various terms are to be sampled from various normal distributions.

Therefore, we may use the well-known mathematical fact that if a value rc_ij is sampled from a normal distribution with mean = 0 and variance σ_rc², and another value e_ij is sampled from another normal distribution with mean = 0 and variance σ_e², and they are added as in Eq (7), then the result is equivalent to sampling one single value v_ij from a normal distribution with mean = 0 and variance σ_v² = σ_rc² + σ_e². Consequently, we may, with the above sampling prescriptions, quite generally write the statistical model Eq (7) more simply as (8) where v_ij is sampled from a normal distribution with mean = 0 and standard deviation σ_v. The rc_ij and e_ij terms have been merged into one single term v_ij. The meaning of the v_ij term is mathematically precisely the same in Model 1 and Model 2, namely a term whose value is sampled from a normal distribution with a standard deviation σ_v. Comparison of Eq (8) and Eq (1) then shows that the only thing that differs Model 2 from Model 1 is the presence of the bias terms c_j.

The population intraclass correlation coefficient corresponding to the model given by Eq (8) is again given by the variance of interest divided by the total variance, i.e. [4] (9)

This population intraclass coefficent is said to describe the "absolute agreement" between different measurements [6]; hence the "A", which here appears as an index in ρ_2A. The meaning of this will be more apparent when Eq (9) is compared with the "consistency" intraclass correlation defined below.

In order to obtain a statistical estimate of ρ_2A, the EMS relations for Model 2 are needed. They are (see S2 Appendix) (10) where we have, for completeness and later convenience, included the expected mean square MSWM (Mean Square Within Measurements).

Also for convenience we will henceforward refer to the v_ij terms as "noise" terms; regardless of their origin, they are (roughly) analogous to what is known in signal theory as “noise”, i.e. an in general unwanted random variation that tends to drown the signal (the true scores). For example, a high ICC(1) value corresponds to a high signal-to-noise ratio.

The complete set of EMS relations for Model 1 (including, as a detail, Eq (3)) are obtained from Eq (10) simply by putting σ_c ² = 0. This is also seen in Fig 1, where all models, EMS relations and ICC formulas discussed here are summarized.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Survey of models, mean square relations and ICC formulas.

https://doi.org/10.1371/journal.pone.0219854.g001

2.5. Model 3, leading to formula ICC(A,1) and formula ICC(C,1)

2.5.1. Fixed biases.

Model 3 is, as mentioned, described as a two-way mixed model [6], meaning that a random sampling of subjects is again assumed, while the biases are assumed to be fixed. We will introduce a theoretical modification that makes our Model 3 slightly different from the ICC(3,1) case of Shrout and Fleiss [5] as well as the Case 3 model described by McGraw and Wong [6]. However, we still obtain precisely the same ICC formulas. We should add that we will finally argue (in Section 6) that in a reliability study there is actually no difference between Model 2 and Model 3.

Formally, the conventional Model 3 looks at first sight just like Eq (7), i.e.

(19)

The μ, r_i and e_ij are defined as before, and r_i and e_ij are sampled as before from normal distributions with mean values = 0 and standard deviations σ_r and σ_e. However, the bias terms c_j and the interaction terms rc_ij are different.

In Model 3, the terms c_j in Eq (19) are assumed to be fixed rather than sampled from a normal distribution. The meaning of this is perhaps confusing. As mentioned, it will be argued that in the analysis of a reliability study there is no difference between Model 2 and Model 3, i.e. there is no difference between fixed and random biases. This may be intuitively understood from the fact when a matrix such as indicated in Table 2 is generated using Eq (19), it does not matter for the subsequent calculation of an ICC value whether the constants c_j (j = 1, 2, … k) are randomly sampled from a normal distribution or chosen in some other way.

It may however be meaningful to speak about fixed bias terms c_j if they remain the same when the measurements are continued with new subjects. An example, given by Shrout and Fleiss [5] and commonly referred to in the literature, is the situation where raters are judging subjects and where "the selected raters are the only raters of interest" [5]. In practice, this should mean that the same raters (j = 1,2, …, k) are used in subsequent measurements.

Turning to the value of the interaction term rc_ij, it is assumed to be sampled from a normal distribution with mean = 0 and standard deviation σ_rc. This would seem to be equivalent to Model 2. However, in the conventional approach [5,6] to Model 3 there is a restriction on c_j as well as rc_ij, which is discussed in the next section.

2.5.3. Model 3, present approach: No restriction.

In the presently discussed version of the two-way mixed model (Model 3), we will therefore not apply the restrictions Eq (20) and Eq (21). This means a considerable simplification of the theory. It will still lead to the same ICC formulas as those described for Model 3 by McGraw and Wong [6], i.e., to ICC(A,1) and ICC(C,1).

In the absence of the restriction Eq (21), and thus in the absence of anti-correlation, we may again replace rc_ij and e_ij in Eq (19) by a single “noise” term v_ij sampled from a normal distribution with standard deviation σ_v. Thus, Model 3 becomes simply (22) where, however, the c_j now are fixed. The absolute agreement population ICC with Model 3 is therefore similar to Eq (9), i.e. to that obtained with Model 2, except that (according to the conventional approach) the Model 2 bias term variance σ_c² is replaced (see S2 Appendix) by (23) i.e. the variance of the fixed bias terms c_j about their average . The absolute agreement population ICC with Model 3 is thus (24) Eq (24) is identical to the population ICC named "Case 3A, absolute agreement" by McGraw and Wong [6]. From S2 Appendix we find the EMS relations for Model 3 to be (25) Solving Eq (25) for σ_r², σ_c² and σ_v² we get the variance estimates (26) Using Eq (26) in Eq (24) we obtain precisely the same ICC(A,1) formula as the one we obtained with Model 2, i.e., Eq (12). This result was also reached by McGraw and Wong [6]. It is not surprising, in view of the fact that in going from Model 2 to Model 3, we have replaced σ_c² by θ_c² in the EMS relations Eq (25) as well as in the definition of the absolute agreement ICC, Eq (24).

We now turn to the consistency version of the population intraclass correlation coefficient, which with Model 3 is defined in the same way as for Model 2, i.e. as (27) Remembering that σ_v ² = σ_e ² + σ_rc ², Eq (27) is identical to the population ICC referred to as "Case 3A, consistency" by McGraw and Wong [6]. In fact, Eq (27) is identical to Eq (13), and using Eq (25) it gives again the ICC(C,1) formula of Eq (14), also in agreement with McGraw and Wong [6]. This result is also reached by Bartko [4], with the added assumption that the interaction term is negligibly small. This assumption has not been used here.

Thus, if a choice is to be made between the two-way random and the two-way mixed models one should be aware that both models lead to precisely the same sample ICC formulas. The same absolute agreement ICC and consistency ICC values will be obtained for a given data matrix, regardless of which model is regarded as appropriate. One may therefore reasonably wonder whether there really is any difference between Model 2 and Model 3. As mentioned, we will return to this question in Section 6. We will also demonstrate fixed bias results by Monte Carlo simulation in Section 4. A flow diagram summarizing the connection between models and the resulting formulas is shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Relation between ICC models and ICC formulas.

Three statistical models used in the intraclass correlation theory are indicated: Model 1 (one-way model); Model 2 (two-way random model); and Model 3 (two-way mixed model). The figure shows the relation between these models and the three well-known sample ICC formulas, i.e. ICC(1), ICC(A,1) and ICC(C,1).

https://doi.org/10.1371/journal.pone.0219854.g002

One choice which definitely does make a difference in the value of the ICC (unless bias are very small or absent) is the choice between the absolute agreement formula ICC(A,1) and the consistency formula ICC(C,1) (Fig 2). That choice may seem to depend only on whether absolute agreement is desired or if consistency is sufficient. However, there is one further aspect that should be noted.

If the biases, i.e. the numbers c_j, are fixed in continued measurements, then a reasonable experimentalist approach should be either to try to correct the measured values x_ij for these biases, or to eliminate (remove) the sources of bias. Let the corrected x_ij value be called y_ij. Then, according to Eq (22), the corrected values are given by (28) But from this equation it is clearly seen that the corrected values y_ij may in fact be generated by Model 1, i.e. without bias. The appropriate population intraclass correlation coefficient for the corrected values should therefore be given by Eq (27), i.e. that coefficient which is estimated by ICC(C,1). Consequently, one may regard ICC(C,1) as an estimate of the population intraclass correlation coefficient that would be obtained if the bias terms could be eliminated or corrected for. This, which applies to Model 2 as well as Model 3, seems to be a somewhat stronger statement than saying merely that ICC(C,1) is a measure of consistency. It will be clearly demonstrated also by the simulations discussed below. In fact, it was proposed by Alexander already in 1947 [17].

2.6. Models specified from experimental design

Much discussion has been devoted to the question of which model (Model 1, 2 or 3) should be specified, i.e. correctly associated with a given experimental design [2,4,5,6,19,25,26,27]. We wish to show that a simple argument based on the algebraic formulas that define the models may be helpful. Neverthless, we also wish to point out that it will later be shown that such a specification of a model should be considered to be only preliminary and may have to be reconsidered in the light of experimental data.

In this discussion it will be convenient to differ between biases associated with raters and biases associated with the measurement procedure (for example, effects of learning or fatigue in a series of measurements on the same subject). We will indicate rater biases with a single prime, for example c'_j, while procedure biases are indicated by a double prime, for example c''_j. With a linear model such as assumed in Eq (1) and Eq (8), the total bias will be the sum of the two kinds of biases, i.e. c_j = c'_j + c''_j.

Example 1: one rater each column. Each of n subjects is judged (measured) once by each of k raters. (This design may be used e.g. in an inter-rater reliability study.) There might be non-zero rater biases c'_j as well as procedure biases c''_j. A measured value should be the sum of all terms, i.e. (29) We can see that Eq (29) has the form of Eq (8). This means that Model 2 or 3 are expected to be appropriate, depending on whether c_j is regarded as random or fixed. We note that with this design, rater biases cannot be separated from procedure biases. If both biases are absent (c'_j = c''_j = 0) we get, of course, Eq (1); then Model 1 is appropriate.

Example 2: one single rater. Each of n subjects is judged (measured) k times. There is only one rater, with a single bias we may call c'; this rater performs all the measurements. This design might be used e.g. in an intra-rater reliability study; it is also common in medical reliability studies where the identity of the rater is not regarded as important. We first assume that there are no procedure biases, i.e. c_j'' = 0. Then (30) Only the apparent population mean value is affected, being shifted from μ to μ' = μ + c'. Eq (30) has the form of Eq (1), i.e. Model 1 is expected to be appropriate. On the other hand, if there are procedure biases, i.e. if not all the c_j'' are zero, then a term c_j'' should be added to Eq (30). We then recognize Eq (8), i.e. Model 2 or 3 should be appropriate.

Example 3: one rater for each subject. Each subject is measured k times by the same rater, but different subjects have different raters (for example from different clinics). Thus, there are n raters, and the rater biases are indexed as c'_i, i = 1, 2, … n according to which subject they are measuring. Assume there are no procedure biases. This gives (31) where r'_i = r_i + c_i. We recognize Eq (1), so Model 1 is expected to be appropriate, but we should observe from Eq (31) that what is estimated is a population ICC given by (32) where σ'_r² = σ_r² + σ_c², i.e. the sum of the variance of the subject's true scores and the variance of the raters; these two variances cannot be separated with such an experimental design.

Now suppose that procedure biases c''_j are present. Adding this term to Eq (31) shows that Model 2 or 3 are then expected to be appropriate.

Example 4: A different rater for each subject and each measurement. This should be a very unusual design, but it may be shown to illustrate the principle of the argument. Each of the n × k measurements is made by a different rater with a bias c'_ij sampled from a normal distribution with variance σ_c²; thus, there are n × k raters. Assume that there are no procedure biases. This gives (33) where v'_ij = c'_ij + v_ij is sampled from a normal distribution with variance σ'_v² = σ_c² + σ_v². Eq (33) has the form of Eq (1), which shows that Model 1 is expected to be appropriate, but it should be observed that the noise term includes the random variation of rater bias. The random variation of rater biases cannot be separated from noise.

2.7. What ICC value is high enough?

Qualitative interpretations of ICC values are often recommended, for example (under certain restrictions [2]) “poor” (ICC < 0.5), "moderate" (0.5–0.75), “good” (0.75–0.9) and "excellent" (ICC > 0.9). However, there are also quantitative ways of considering an ICC value obtained from a reliability study. We take Model 1 as an example. The population ICC given by Eq (2) may be written. (34) Suppose that it is required that the typical error or uncertainty in the method of measurement should be less than 10% of the spread among the true scores, i.e., σ_v < 0.1σ_r. (One example could be the measurement of body temperature, where a typical range of interest might be a few degrees, while the measurement of the temperature may have an uncertainty of about 0.1°C.) Inserting σ_v < 0.1σ_r into Eq (34) we find that the population ICC should then be about 1/(1 + 0.01) ≈ 0.99 or larger. The required ICC value is thus very close to unity. However, it may be questioned if an ICC value that close to unity is informative; rather, one should consider the ratio σ_v /σ_r.

Suppose instead that we assume that ρ₁ > 0.8 is high enough. Using this in the above equation gives σ_v/σ_r < 0.5. This means that the error σ_v might be as much as 50% of the spread σ_r among the true scores. Such a large error might be acceptable in some contexts but not in others; consequently, we may have to reconsider our assumption. A higher ICC value might be required.

Considerations such as these might be of assistance when judging if an ICC value is high enough in a given clinical context.

3.1. Simulation of ICC probability distributions from Model 1

Monte Carlo simulation is well-known in statistical research [28–31]. Still, a brief comment on the difference between for example a clinical reliability test and the present simulation is motivated.

In a typical clinical or research reliability test, the researcher starts by planning an experiment according to a specific design. Subjects willing to take part in the study have been found, inclusion and exclusion criteria have been applied and ethical permissions have been secured. Finally, measurements have been made, resulting in an experimental data matrix of size n × k (Table 2) from which the researcher, using ANOVA and a specified statistical model corresponding to the design, may derive a standard error of measurement (σ_v), an ICC value and a confidence interval for the population ICC.

In the simulation we do essentially the reverse. We start from a specified statistical model (Model 1, 2 or 3). Using this model, we let a computer simulate (create) "experimental" data which then are analyzed. For the simplest model, Model 1, we begin by choosing explicit numerical values for the population mean value μ, the population standard deviations σ_r and σ_v, the number of subjects n and the number of measurements k. In order to study essentially all possible results of this model, a large number N of "experimental" data matrices are simulated, each of size n × k. In the present simulations N = 10000 for each case of interest; this provides sufficient statistical accuracy for our purposes.

The simulation of each "experimental" data matrix is for Model 1 performed in accordance with Eq (1). Thus, each matrix element x_ij is explicitly calculated to be the sum of a population mean value μ, a signal term r_i and a noise term v_ij. The values r_i are randomly sampled from a normal distribution with mean value zero and the chosen standard deviation σ_r, while the values v_ij are randomly sampled from a normal distribution with mean value zero and the chosen standard deviation σ_v. (The element of randomness in the simulation is the reason for the traditional name “Monte Carlo simulation”.) For transparency and completeness, the method presently used for this random sampling is briefly indicated in Appendix 3 (S3 Appendix).

Each simulated "experimental" matrix will be a little different because of the random nature of the r_i and the v_ij numbers. When finished, it is (within the simulation program) subjected to ANOVA. An ICC(1) value and an F value are calculated and stored; for comparison, ICC(A,1) and ICC(C,1) values are calculated and stored as well. Thus, in one single simulation with N = 10000 we get, for example, a set of 10000 simulated ICC(1) values. This set may be regarded as a probability distribution of the ICC(1) values generated by the specified model. From the ICC(1) distribution the average ICC(1) value is calculated and its 95% central range limits (see below) are estimated numerically. Apart from a minute for manually feeding the input data (μ, σ_r, σ_v, n, k, and N in the case of Model 1), the simulation and analysis of N = 10000 “experiments” takes typically one or a few seconds on an ordinary personal computer.

The advantage of such simulations is that they may help us to understand how the ICC “works”, we may get a survey of what to expect in an experiment, and we may better understand how to make a proper choice of models, experimental designs and analysis.

3.2. Data obtained from a Model 1 ICC probability distribution

From the N = 10000 simulated mean squares, ICC values and F values various averages are calculated. For example, let ICC_J (1) be the sample ICC(1) value obtained from the J:th simulated matrix. The average ICC(1) value is then (35) The same notation (< … >) is used for other averages over the N simulated matrices. Thus, for example <MSBS> is, as mentioned, the average mean square between subjects. These averages are presented together with a standard deviation calculated in the usual manner. The standard deviations are only meant to give a rough idea of the spread of simulated data around the average value.

The confidence limits of the population ICC are of greater interest. In order to estimate these confidence limits, the 95% central range limits of the simulated ICC(1) distributions may be used. With a large number N of simulated ICC(1) values, 2.5% are expected to be above the upper central range limit, while 2.5% are expected to be below the lower central range limit. In the present simulations, a simple numerical method has been used to estimate these limits: with N = 10000, the upper 95% central range limit is estimated as being equal to the lowest simulated ICC(1) value among the 250 highest ones, while the lower limit is estimated as being equal to the highest ICC(1) value among the 250 lowest ones. The connection between these sample ICC central range limits and the population ICC confidence interval will be demonstrated in Section 4.

It is informative to study for example the average value <ICC(1)> and the central range limits as functions of the input data. For a systematic survey of ICC(1) we note again that the population intraclass coefficient for Model 1 may be written as in Eq (34), from which it is obvious that ρ₁ depends only on the ratio σ _v /σ _r. All cases of interest may therefore be simulated and studied by varying this ratio within a reasonable interval, for example 0 < (σ _v /σ _r)² < 2. (A value of (σ _v /σ _r)² larger than 2 is deemed not to be of interest since the expected average ICC(1) value would then be less than 0.33.)

It is also obvious from Eq (34) that, as regards the ICC value, the separate values of μ, σ_r and σ_v do not matter. We may therefore, without any loss of generality, choose fixed values for μ and σ_r, while varying σ_v, i.e. the ratio σ _v /σ _r. For convenience we have therefore throughout all present simulations assumed μ = 100 and σ_r = 10. For example, a value of σ_v = 5 will then give (σ _v /σ _r)² = 0.25 and ρ₁ = 0.8.

3.3. Methods for simulation of Model 2 and Model 3

The methods are similar to that used for Model 1. In Model 2 each matrix element is calculated by the formula Eq (8), i.e. as a sum of a population mean value μ, a signal term r_i, a bias term c_j and a noise term v_ij. The sampling of the r_i and v_ij terms are performed as for Model 1. The terms c_j are sampled from a normal distribution with standard deviation σ_c and mean value zero. Input data for the simulation are thus N, n, k, μ and the standard deviations σ_r, σ_c and σ_v. For each simulated matrix, mean squares (MS) and the ICC(A,1) and ICC(C,1) values are calculated. For comparison, we also calculate ICC(1), even though it is not valid when biases are present.

For a systematic survey of ICC(A,1) we may note that the population ICC for absolute agreement may be written (36)

This depends on two ratios, namely σ _c /σ _r and σ _v /σ _r; for the survey, these ratios may be varied independently over suitable intervals.

In Model 3, each matrix element x_ij is calculated as a sum of a population mean value μ, a signal term r_i, a fixed bias term c_j (i.e., equal for all the simulated matrices) and a noise term v_ij. The signal term r_i and the noise term v_ij are sampled from their respective normal distributions. Input data for the simulation are thus N, n, k, μ, the standard deviations σ_r and σ_v, and the k fixed (but arbitrary) c_j values. Again, we use N = 10 000, μ = 100 and σ_r = 10 throughout. There is an infinite number of ways of choosing the fixed c_j values. For our purposes it will be sufficient to study only two cases.

3.4. Fortran code

An interactive Fortran 77 code, SIMANOVA, was written to perform these simulations. It was repeatedly rewritten in order to perform the various tasks given to it. The source code of a rather general version is given in Appendix 4 (S4 Appendix). Output data for plotting are given in a form suitable e.g. for Excel. Note that two random seeds have to be given to the random generator, for example two four-digit integers.

4.1. Simulation study of Model 1

Fig 3 shows results (part of the print-out, somewhat edited) of one specific Monte Carlo simulation run using Model 1. Fig 3 (as well as Fig 4; see text below) may be reproduced by running SIMANOVA with the input data shown, including the random seeds. As seen from Fig 3, input data in this run were n = 20, k = 3, N = 10 000, μ = 100, σ_r = 10, and σ_v = 5, with the two random seeds 4566 and 2345. A different choice of random seeds will give statistically the same result, but not exactly the same numbers. As mentioned, the values μ = 100, σ_r = 10 and N = 10000 are used throughout all simulations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Part of the computer printout from a SIMANOVA run using Model 1.

Corresponding ICC distributions are shown in Fig 4. See text for discussion.

https://doi.org/10.1371/journal.pone.0219854.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Probability distributions of ICC(1), ICC(A,1) and ICC(C,1) obtained with a simulation based on Model 1, i.e. in the absence of bias.

Input data and results from this simulation are shown in Fig 3. With Model 1, the simulated distributions are seen to be identical apart from small differences due to finite statistics (finite N).

https://doi.org/10.1371/journal.pone.0219854.g004

From Eq (34) the population intraclass correlation coefficient is ρ₁ = 1/(1 + 5²/10²) = 0.8. In a similar way, the expected average mean squares MSBS, MSWS, MSBM, MSWM and MSE are calculated using the EMS relations (see Fig 1) from the variances σ_r = 10, and σ_v = 5; their values are shown in Fig 3. The simulated average mean square values (< MSBS >, etc.) are in excellent agreement with the expected values, apart from very small statistical fluctuations due to the finite (albeit large) value of N.

For each of the N matrices, all three sample ICC values ICC(1), ICC(A,1) and ICC(C,1) are calculated. Not only do their average values <ICC(1)>, <ICC(A,1)> and <ICC(C,1)> agree very closely, as should be expected with Model 1, but the values tend in fact to agree closely also within each single simulated matrix. This is exemplified in Fig 3 by the result for the 10 first simulated matrices.

It may be noted that all three average ICC values are slightly different from the population ICC value = 0.8. This is due to a simple numerical effect. The average ICC(1), for example, is given by (37)

In general, this is not precisely equal to the alternative expression (38) (i.e., it does matter whether the average is taken before the division or after.) Since the average simulated MS values in Eq (38) agree very well with the expected MS values, AICC(1) agrees better with the population ICC, ρ₁, than does <ICC(1)>. However, <ICC(1)> is by definition the average ICC(1) value calculated from the ICC(1) distribution. Moreover, the difference between the values calculated by Eq (37) and Eq (38) is in practice small compared to the widths (95% central ranges) of the simulated ICC distributions. A similar argument applies to ICC(A,1) and ICC(C,1). The simulated 95% central ranges are shown in Fig 3 for all three calculated ICC versions.

For the same simulation run as shown in Fig 3, Fig 4 shows the simulated distributions of the values of ICC(1), ICC(A,1) and ICC(C,1). The distributions are presented as histograms, for graphical clearness with points (symbols) rather than staples. The relative number N_i/N, where N_i is the number of simulated ICC values found in the histogram channel (interval) i, is plotted against the value of the ICC corresponding to the center of the channel. Thus, the simulated distributions may be regarded as estimates of the normalized probability distributions of the ICC values for the given values of the model parameters. It can be seen that the three distributions obtained with Model 1 are nearly identical, apart from very small statistical variations. The distribution maxima are positioned near the population ICC = 0.8.

The effect of increasing the noise (interaction + error), i.e. increasing the standard deviation σ_v while keeping everything else fixed, is shown in Fig 5. As follows from Eq (2), the population intraclass coefficient ρ₁ then decreases. In accordance with this, the ICC(1) distribution shifts to a lower average value < ICC(1) > and is widened as well, increasing its 95% central range.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Probability distributions of ICC(1) values obtained with simulations using Model 1, showing the effect of increasing noise (error).

In all three cases n = 20, k = 3 and σ_r = 10. The standard deviation of the noise term is increased from σ_v = 5 (giving population ICC ρ₁ = 0.8) to σ_v = 7.5 (ρ₁ = 0.64) and σ_v = 10 (ρ₁ = 0.5).

https://doi.org/10.1371/journal.pone.0219854.g005

It is reasonable to expect that an increased number of subjects should lead to a better statistical accuracy. Fig 6 shows the effect of increasing n for the case ICC(1) = 0.5. The average value < ICC(1) > is not affected, but the ICC(1) distribution narrows, decreasing the 95% central range. Thus, the probability increases that the sample ICC will be close to the population ICC. A similar effect is obtained by increasing the number of measurements k, as shown by Fig 7. However, the number of measurements is for practical reasons usually limited, so the effect is less dramatic.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Probability distributions of ICC(1) obtained with Model 1, showing the effect of increasing the number of subjects.

The number of subjects increases from n = 20 to n = 100 and n = 400. In all three cases k = 3 and the standard deviations of error and subject's score are 𝛔_v = 𝛔_r = 10, giving the population ICC = 0.5. As can be seen, an increasing n leads to a decrease in the width of the probability distribution.

https://doi.org/10.1371/journal.pone.0219854.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Probability distributions of ICC(1) obtained with Model 1, showing the effect of increasing the number of measurements k.

In all four distributions, n = 20 and 𝛔_v = 𝛔_r = 10 (giving population ICC = 0.5). Increasing k leads to a decreasing width.

https://doi.org/10.1371/journal.pone.0219854.g007

The 95% central range of ICC(1) is not the same thing as the confidence interval of ρ₁, but the latter can be deduced from the former. In Fig 8 the curves show the upper and lower limits of the 95% central range of simulated ICC(1) distributions, for the cases k = 3 and n = 10, 20, 50 and 100, as functions of the population intraclass correlation coefficient ρ₁ assumed in the simulation. To find the confidence intervals, the diagram may be used as follows.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Graphic presentation of confidence limits.

The curves show the upper and lower 95% central range limits of the ICC(1) probability distributions as functions of the population ICC, using Model 1. The number of measurements is everywhere k = 3 while the number of subjects n range from 10 to 100. Read horizontally, the diagram provides the 95% confidence limits of the population ICC for a given sample ICC(1) value. For example, if n = 10 and the sample ICC(1) is found to be 0.70, then the confidence limits of the population ICC are graphically read to be approximately 0.38 and 0.91. The diagram is also valid for ICC(C,1), i.e. the consistency ICC obtained with Model 2 and Model 3.

https://doi.org/10.1371/journal.pone.0219854.g008

Suppose, for example, that a single experimental matrix with n = 10, gives the sample value ICC(1) = 0.80. In Fig 8, the 95% lower and upper confidence limits of ρ₁ are found where a horizontal line at ICC(1) = 0.80 crosses the n = 10 upper and lower 95% central range limits, i.e. (graphically) at ρ₁ ≈ 0.55 and ρ₁ ≈ 0.94. This is understood from the fact that, as can be seen from Fig 8, the probability of getting a sample ICC(1) = 0.8 if ρ₁ < 0.55 is less than 2.5%; similarly, it is less than 2.5% if ρ₁ > 0.94. With ICC(1) = 0.80 and n = 100, the confidence limits are in the same way graphically found to be ρ₁ ≈ 0.74 and ρ₁ ≈ 0.85.

This graphical method of estimating the ICC(1) confidence interval agrees well with the results obtained with the formulas given by McGraw and Wong [6] and the algorithms used in SPSS for Model 1. In fact, it should be pointed out that Monte Carlo simulation is in principle an exact method, apart from statistical "fuzziness". We will show that even in the presence of bias the graphical presentation in Fig 8 can be used for the consistency intraclass correlation coefficient ICC(C,1). However, it cannot be used for ICC(A,1). The graphical method (exemplified here only for k = 3) is not meant to replace confidence interval formulas [6] or, for example, SPSS software, but it demonstrates the connection with the ICC probability distributions as well as the general features of the confidence interval. Thus, Fig 8 shows that the confidence interval is centred approximately at the observed ICC value. Moreover, the confidence interval narrows with an increasing n and also with a higher sample ICC value. For low n and a low sample ICC value, it is however so wide that the population ICC may have almost any value below an upper limit.

An interesting point is worth mentioning. Suppose, for example, that the population ICC is ρ₁ = 0.7 and that n = 10. Reading vertically, the 95% central range of the sample ICC probability distribution is found graphically to be approximately between 0.28 and 0.88 (Fig 8). For each sample ICC between these limits we obtain a certain confidence interval, as described above. Inspection of Fig 8 shows that one single population ICC value is common to all of these confidence intervals–namely ρ₁ = 0.7. A moment of reflection shows that this result should be expected. The true (population) ICC is likely to be found in the overlap of any two 95% confidence intervals.

4.2. Simulation study of Model 2 (two-way random)

As discussed in Section 2.4, the only difference between Model 1 and Model 2 is the presence, in Model 2, of bias (systematic error) terms c_j sampled from a normal distribution with standard deviation σ_c and mean value = zero. To see the effect of such biases, we may therefore start with Model 1, i.e. σ_c = 0, and then turn on a non-zero bias, sampling the c_j from normal distributions with successively larger bias variance σ_c² (keeping everything else fixed). This is demonstrated in Fig 9 and Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Probability distributions of ICC(A,1) and ICC(C,1) obtained with Model 2, showing the effect of increasing bias.

In all cases n = 20, k = 3, σ_r = 10 and σ_v = 5. The bias standard deviation σ_c is increased from σ_c = 0.1 to σ_c = 5 and then to σ_c = 10. As may be seen, the ICC(C,1) distributions are insensitive to bias. The ICC(A,1) distributions are, with increasing bias, shifted towards lower values and broadened.

https://doi.org/10.1371/journal.pone.0219854.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. A survey of features of Model 2.

https://doi.org/10.1371/journal.pone.0219854.t003

Fig 9 shows the distributions of ICC(A,1) and ICC(C,1) obtained with three different versions of Model 2, namely with a "weak" bias (σ_c/σ_r = 0.01), a "moderate" bias (σ_c/σ_r = 0.5), and a "strong" bias (σ_c/σ_r = 1.0). In the "weak" bias case, the results of Model 2 are practically identical to the results of Model 1 (σ_c/σ_r = 0)–at least with the modest number of subjects n = 20. As can be seen from Fig 9, the effect of an increasing bias σ_c is a lower average value <ICC(A,1)> and a wider ICC(A,1) probability distribution. The average ICC value in each distribution agrees approximately with the population ICC. Using Eq (9) we find ρ_2A = 10²/(10² + 0.1² + 5²) ≈ 0.80, ρ_2A = 10²/(10² + 5² + 5²) ≈ 0.67 and ρ_2A = 10²/(10² + 10² + 5²) ≈ 0.44 for the three ICC(A,1) distributions, respectively. The increased width of the ICC(A,1) distributions with increasing σ_c may be attributed to the increasing contribution from the random variation of the c_j terms.

The ICC(C,1) distributions are insensitive to the presence of bias; they remain the same, regardless of the strength of the bias. Using Eq (13) we find ρ_2C = 10²/(10² + 5²) = 0.8 for all three ICC(C,1) distributions. They coincide with the ICC(1) (zero bias) distribution as well as with the ICC(A,1) distribution when biases are very small. This indicates that the confidence limits of ρ_2C (the consistency population ICC of Model 2) will be the same as the confidence limits of ρ₁ (population ICC of Model 1, i.e. in the absence of bias). Thus, Fig 8 may be used to graphically derive not only the confidence limits corresponding to an ICC(1) value calculated from an experimental matrix compatible with Model 1, but also the confidence limits corresponding to an ICC(C,1) value calculated from a matrix compatible with Model 2. This observation is in agreement with the confidence limit formulas given by McGraw and Wong [6] provided that the number of subjects n is not too small.

Since we may turn the bias down all the way to zero without changing the ICC(C,1) distribution, ICC(C,1) may, as already discussed in Section 2.5.3, be regarded as an estimate of the intraclass correlation that would be obtained if the effects of bias could be removed or corrected for [17]. However, it should also be observed that the ICC(C,1) value might give an exaggerated and misleading impression of the reliability of a method unless it is clearly stated that the value given is the consistency ICC value.

Table 3 gives an survey of the main features of Model 2 for the case n = 20, k = 3, showing the average values < ICC(A,1) > and < ICC(C,1) > as well as the 95% central range limits obtained from the simulated distributions. The average values and central range limits are shown as functions of the two parameters (σ_v/σ_r) and (σ_c/σ_r) (see Eq (36)). The average values of ICC(1) are for comparison shown also for σ_c > 0, even though the formula ICC(1) is then invalid. It may be noted that excellent ICC(A,1) values are with any certainty only found in the upper left corner; relative noise as well as relative bias have to be at most about 0.1. Occasionally, a high ICC(A,1) value could be obtained even with a strong relative bias; the 95% central range is very wide. The results for ICC(C,1) are, as already discussed, independent of σ_c and equal to the results obtained for ICC(1) with Model 1. Average invalid ICC(1) values (in brackets) are everywhere lower than the ICC(A,1) and ICC(C,1) values. For σ_c = 0 (Model 1) we find < ICC(1) > = < ICC(A,1) > = < ICC(C,1 >. We also find equal 95% central range limits, i.e. equal distributions as also exemplified in Fig 4.

Table 4 gives an example (for n = 20 and k = 3) of what to expect when the ratio ICC(C,1)/ICC(A,1), where ICC(C,1) and ICC(A,1) have been calculated from the same matrix, is used as a qualitative indication of the presence of bias. The simulated average ratio <ICC(C,1)/ICC(A,1)> obtained with Model 2 is shown, as well as the probability of finding an ICC(C,1) value larger than ICC(A,1). The average ratio increases with increasing bias and decreases with increasing noise. In the absence of bias the average ratio is very close to unity; the (very narrow) distribution is such that it is actually more probable to find an ICC(C,1) value that is lower than ICC(A,1). With a low relative bias (≈ 0.1) the difference between ICC(C,1) and ICC(A,1) is seen to be of order 1%. With a high relative bias (≈ 1) it might be as much as a factor ≈ 2. As mentioned, it is strongly recommended to use an F-test as well (see Section 2.4.5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Survey of model 2 for n = 20 and k = 3: the average ratio <ICC(C,1)/ICC(A,1)> and the probability for ICC(C,1) to be larger than ICC(A,1) in a single data matrix.

https://doi.org/10.1371/journal.pone.0219854.t004

4.3. Simulation study of Model 3 (two-way mixed model)

In order to demonstrate features of Model 3, we will discuss the following example. Let us assume that there are two research groups, A and B, each consisting of k = 3 raters. These two groups intend to carry out the same kind of measurements on the same population (with σ_r = 10), and for this purpose each of them makes a reliability study on n = 20 subjects (not the same subjects).

For the present argument we will make a comparison with Model 2. We assume that the raters in each group have been randomly chosen from a population of raters normally distributed with a bias standard deviation σ_c = 5. The probability distributions of all possible ICC(A,1) and ICC(C,1) values in a reliability study with such raters are given by Model 2 and shown in Fig 9, where they are marked "bias = 5".

Assume that each group has made a reliability study. We now consider the fact that each group, A and B, has a certain set of biases c₁, c₂ and c₃. We assume that group A has a set (a) of biases, namely c₁ = 1, c₂ = 6 and c₃ = -1, while group B has a set (b) of biases c₁ = 10, c₂ = 6 and c₃ = - 10. The set (a) is not unlikely, while the set (b) is very improbable but still possible.

Using Model 3 (simulation with fixed biases), these two sets of biases have been used to simulate the two ICC(A,1) and two ICC(C,1) probability distributions shown in Fig 10. We may note that the ICC(C,1) distributions again coincide with each other and also with the ICC(C,1) distributions in Fig 9.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Effect of fixed bias: ICC(A,1) and ICC(C,1) distributions simulated using Model 3.

Two simulated Model 3 cases are shown, (a) and (b). In both cases, n = 20, k = 3, σ_r = 10 and σ_v = 5. In case (a), the fixed bias values are c₁ = 1, c₂ = 6 and c₃ = -1. In case (b), they are c₁ = 10, c₂ = 6 and c₃ = -10. A larger spread among the fixed bias values gives a larger shift of the ICC(A,1) distribution towards lower values. The ICC(C,1) distributions are however insensitive to bias.

https://doi.org/10.1371/journal.pone.0219854.g010

The ICC(A,1) probability distributions in Fig 10 are however distinctly different from each other. They are the probability distributions from which group A and group B will actually obtain their ICC(A,1) values in a reliability study, if their biases are regarded as given. Thus, they are also the probability distributions for the ICC(A,1) values that would be obtained in a second reliability study. What Fig 10 shows is simply that if group A and group B would repeat their respective reliability studies, they would probably obtain an ICC(A,1) value not far from the one found in their first study. With a third set C of three randomly selected raters we may of course expect another result, as indicated by the Model 2 distribution in Fig 7.

We may understand the details of the Model 3 distributions in Fig 10 as follows. In case (a), the bias values c₁ = 1, c₂ = 6 and c₃ = -1 give, using Eq (23), θ ² = 13. Using Eq (24), the absolute agreement population ICC is then ρ_3A = 10²/(10² + 13 + 5²) ≈ 0.725. Using Eq (27), the consistency population ICC is ρ_3C = 10²/(10² + 5²) = 0.8. We can see from Fig 10 that the maxima of the ICC(A,1) and ICC(C,1) distributions in case (a) are positioned near these ρ_3A and ρ_3C values. The width of the ICC(A,1) distribution obtained with Model 3 is somewhat smaller than the width of the ICC(A,1) ("bias = 5") distribution obtained with Model 2 in Fig 9, reflecting the fact that since the c_j are fixed, the width is due only to the variation in the subject scores r_j and the noise v_ij.

In case (b), the fixed bias values c₁ = 10, c₂ = 6 and c₃ = -10 give in the same way θ ² = 112, from which the absolute agreement population ICC is found to be ρ_3A ≈ 0.422. The consistency population ICC is again found to be ρ_3C = 0.8 and again the ICC(A,1) and ICC(C,1) distributions are approximately centred on these values.

Thus, when the choice of raters has been made, it may turn out that the ICC(A,1) values obtained by the two research groups A and B are very different from each other, due to their different biases. This is more likely to happen if the number of raters in each group is small and the raters have been sampled from a population with a large standard deviation σ_c.

In general we should expect that the ICC(A,1) 95% confidence intervals obtained by group A and group B will overlap. Within this overlap we may, as indicated in Section 4.1, expect to find the Model 2 population intraclass coefficient ρ_2A.

5.1. Conventional approach

In the conventional approach to the analysis of a reliability study it is common to specify one statistical model (the one-way random, the two way random or the two-way mixed model) in accordance with the design of the study [2,6]. For example, in the SPSS software it is required that the one-way random model is chosen if the ICC(1) value is to be calculated. Likewise, if the ICC(A,1) value is desired, one has to choose the absolute agreement two-way random or two-way mixed model. To obtain the ICC(C,1) value, one has to choose the consistency two-way random or two-way mixed model. Similar recommendations are given in the flow charts presented by McGraw and Wong [6] and Koo et al [2].

5.2. Present approach

We find that the initial specification of a statistical model is inconvenient and not necessary. Of course, the researcher may, from the design of a reliability study, expect that a certain model should be appropriate (see Section 2.6). Reasonably, the design has been such that it might be hoped that not only error (noise) but also systematic differences between measurements (bias) are negligible, in which case Model 1 (one-way random) can be used. However, it may turn out that the model has to be reconsidered. Unexpected biases may turn up; also, the effect of biases may be larger than foreseen.

We therefore propose that while the design of the method should, of course, be planned with utmost care, the analysis of the result of the reliability study should be impartially open to all different possibilities. Our recommendation is similar to that previously given by Weir [19].

The analysis procedure may for example be as follows. We start by calculating the mean squares MS from the matrix of experimental data and, from them, the sample ICC values (ICC(1), ICC(A,1) and ICC(C,1), using the standard formulas shown e.g. in Fig 1. If we find that ICC(1) ≈ ICC(A,1) ≈ ICC(C,1), i.e. approximately equal (within a few percent; see Table 4), then biases are likely to be small or even negligible. If an F-test with F = MSBM/MSE shows that Model 1 cannot be rejected, then ICC(1) may be reported as a result, with the comment that biases are negligible. We may note, though, that all three ICC values may then be regarded equally valid estimates of the population intraclass coefficient. The estimated magnitude of the spread in true scores and the noise are obtained by calculating the standard deviations σ_r and σ_v, using Eq (5).

Distinctly different ICC(1), ICC(A,1) and ICC(C,1) values from the same matrix indicate the presence of non-negligible biases (see Table 4), i.e. systematic differences between different measurements j = 1,2, …,k. Usually, one then finds ICC(1) < ICC(A,1) < ICC(C,1); however, ICC(1) is no longer a valid formula. The F test may be used to definitely reject Model 1, confirming the probable presence of bias. We need not specify whether Model 2 (two-way random) or Model 3 (two-way mixed) is used instead of Model 1; it makes no difference.

Thus, we avoid the error of reporting the ICC(1) value in the presence of bias. We report instead two ICC values, ICC(A,1) and ICC(C,1), together with their confidence intervals and a comment on the presence of bias. The estimated magnitudes of spread in true scores and of bias and noise are obtained by calculating the standard deviations σ_r, σ_v and σ_c, using Eq (11).

5.3. A clinical example

To exemplify the analysis procedure we use clinical data from Elfving et al [20]. The measured data, shown in Table 5, have been extracted from a reliability study of a method using electromyography. The data are a subset of electromyographic recordings made on back muscles on three separate days, namely measurements made in the morning on 10 subjects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. A clinical example from physiotherapy.

https://doi.org/10.1371/journal.pone.0219854.t005

For the present analysis we have used SPSS version 24 as a calculation tool. Application of ANOVA (as part the ICC calculation in SPSS) gave the mean squares MSBS = 212.61, MSBM = 39.15, MSE = 24.45 and MSWS = 25.92. Results for single-score ICC with 95% confidence intervals were, choosing the one-way random model, ICC(1) = 0.706 (0.387–0.906) and, choosing the two-way random model, ICC(A,1) = 0.708 (0.392–0.907) and ICC(C,1) = 0.720 (0.396–0.912). We may note that the confidence interval obtained for ICC(1) and ICC(C,1) agrees well with a graphical reading from Fig 8. It is clear from Fig 8 that somewhat less wide confidence intervals could have been obtained with a larger number of subjects. We need not use the two-way mixed model; it gives exactly the same results as the two-way random model.

We observe that ICC(1), ICC(A,1) and ICC(C,1) are closely equal (as are also their confidence intervals), which suggests that there are no systematic differences between the recordings from day to day, i.e. no bias; thus ICC(1) might be correctly reported. ICC(C,1) is slightly higher which indicates that there might possibly be a weak bias; the ratio ICC(C,1)/ICC(A,1) is 1.017 (compare Table 4). To make more certain, we look at the between days F-value which is F = MSBM/MSE = 1.601 with p = 0.229. This low F-value and relatively high p value shows that the systematic differences between days are not significant; i.e. the one-way random model (no bias) cannot be rejected.

Additional insight is provided by calculating the estimated standard deviations. Assuming first that the one-way model is applicable we find manually, using the mean squares quoted above and Eq (5), the standard deviations σ_r = 7.89 Hz and σ_v = 5.09 Hz. Thus, even if biases are assumed to be absent, the error σ_v is substantial, i.e. 5.09/7.89 = 65% of the spread σ_r among the true scores of the subjects. To discern the magnitude of a possible bias, we use instead the two-way random model and apply Eq (11), which, with the mean squares quoted above, gives σ_r = 7.92 Hz, σ_v = 4.94 Hz and σ_c = 1.21 Hz. Thus, if biases exist, they are small, and the F-test has already told us that the one-way model cannot be rejected. It may be noted that both σ_v values are estimates of the standard error of measurement, one (5.09 Hz) obtained from MSWS and the other (4.94 Hz) from MSE. The standard error of measurement in connection with ICC is discussed by for example [19] and [32].

However, even if there were no significant biases, the result was not encouraging. The ICC value obtained, ICC ≈ 0.71, was not considered to be good enough for the intended use of the method as a diagnostic tool [20].