Ethical and legal considerations.

Nile tilapia is not a protected species, but rather fished commercially. Sampling was conducted in collaboration with respective local authorities and therefore no special permission was required. All Nile tilapia samples which went into this investigation where harvested by local fishermen and used for commercial purposes. In all cases, the fish were already dead when obtained from the fishermen. For obtaining the Nile tilapia images a special treatment of the animals was therefore not required.

Data collection.

To provide conclusions which are relevant for typical field work, the data which we consider in this study consists of 209 Nile tilapia fish images that were collected at six different Ethiopian lakes under natural conditions. An overview of the locations and the number of samples is found in Table 1. To maintain identical orientation all Nile tilapia samples were photographed from the left side with a Nikon D5200 digital camera. Images were taken with a 18mm lens at constant focal length of 30cm and labeled according to the lake of origin. To compare purely ML based analysis strategies with classical morphometrics, all fish images were annotated with the 14 two dimensional landmarks that were previously used by [50]. The landmarks are depicted in Fig 2 and further described in Table 2. The data which we use in this study consists thus of 209 images and a 209 × 28 dimensional matrix with landmark coordinates. Images and landmarks were processed and subsequently used as input to infer classifiers which learn to predict the respective lake of origin. We are however not interested in the predictions per se. The objective of a biologically meaningful analysis lies in 1) identifying whether and how much information Nile tilapia body characteristics contain about the lake of origin and 2) an identification of Nile tilapia body characteristics which allow to distinguish the origin of samples. To deduce this information we apply the analysis pipelines which are shown in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Landmark positions on an image of a Nile tilapia fish.

A classical morphometrics analysis of Ethiopian Nile tilapia is based on the 14 landmark positions which we illustrate in this image of a Nile tilapia fish.

https://doi.org/10.1371/journal.pone.0249593.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characterization of the 14 landmark positions in Fig 2.

https://doi.org/10.1371/journal.pone.0249593.t002

Landmarks and GPA.

To remove effects from landmarks like differences in sample size or orientation, GPA was performed using function procGPA() from the R package shapes, [41]. To discuss GPA briefly we follow [39–41] and represent landmark m ∈ [1, M] of Nile tilapia sample n ∈ [1, N] as two dimensional column vector . All landmarks of sample n can thus be represented as [2 × M] dimensional matrix Y_n = (ψ_n,1, …, ψ_n,M). The goal of the GPA is to convert all N initial configurations Y_n to shapes. GPA adjusts to this end all configurations to obtain identical location and scale. A subsequent correction for rotational variation completes GPA. GPA is thus a three step procedure.

1. The initial configurations Y_n are translated such that all sample centroids c(Y_n) = 1/M∑_m ψ_n,m coincide with the origin of the coordinate system.
2. The translated configurations are rescaled such that configuration Y_n gets a unit centroid size. Using I_M as M-dimensional identity matrix the centroid size is defined as where denotes the centering matrix.
3. The final GPA shapes, Z_n, are obtained by rotation such that the agreement between Z_n and the mean shape μ_Z = 1/N∑_n Z_n is optimized. Optimization minimizes to this end for every shape trace(μ_ZZ_nΓ_n) w.r.t. Γ_n ∈ Sr(R²) where Sr(R²) denotes the set of all rotation matrices of the 2 dimensional Euclidean space.

The resulting set of shapes Z = [Z₁, .., Z_N] characterize Nile tilapia landmarks invariant to differences in location, size and angular rotation of the fish in the image. To fit to the analysis which is shown in Fig 1 we convert Z to a [N × 2M] dimensional matrix X where column X[:, 2m − 1] denotes all x and column X[:, 2m] all y coordinates of the m-th GPA-shape feature.

Gaussian process latent variable models.

To compare the landmark based analysis with analyses which rely entirely on ML methods, we extract features from Nile tilapia images with Gaussian process latent variable models (GP-LVM) [55, 56]. GP-LVM is an unsupervised ML method which represents an [N × D] dimensional data matrix Y by a [N × L] dimensional matrix X and a probabilistic model p(Y|X, θ). We use N to denote the number of samples, D as dimension in observation space, L ≪ D as latent variable dimension and θ to denote the GP-LVM parameters.

GP-LVM models the D columns of Y, y_d, independently as Gaussian process. We will to this end adopt the notation (1)

All data columns, y_d, are hence considered observations of a zero mean Gaussian process with an additive Gaussian noise term with variance λ⁻¹. The vector f_d in Eq (1) represents the noiseless signal in column d. The Gaussian process (GP) p(f_d|X) is governed by an [N × N] dimensional covariance matrix, K_f,f(X, θ). The covariance matrix is a deterministic function of X and θ and in GP-LVM assumed to be identical for all data columns. GP-LVM learns thus to represent all columns of Y with the same latent variables X. GP-LVM is parameterized via a covariance function K_f,f[n, ν](X, θ) = k(X[n,:], X[ν,:], θ) where X[n,:] and X[ν,:] denote rows n and ν of X. The vector θ represents the parameters of the covariance function. Due to the large number of parameters and many local optima, determining optimal values for X and θ by maximizing the likelihood p(Y|X, θ) is cumbersome and has a tendency to overfit [57, 58].

Current applications of GP-LVM rely thus on a later extension by [38] which combines variational ideas and sparse GPs [58, 59] to reduce the number of parameters considerably. Denoting the latent representation of sample n as X[n,:], [38] acknowledge that X is a [N × L] matrix of random variables and specify a factorizing prior . Approximating the true posterior p(Y|X) as allows the authors of [38] to formulate a variational lower bound F(Q) < log(p(Y)) (2)

While the Kullback Leibler (KL) divergence KL(Q(X)||p(X) is analytically tractable, nonlinear interactions between X and Y require approximating further. The suggestion in [38] to use a sparse GP for p(f_d|X) is achieved by augmenting the model for every dimension d by M < N inducing variables u_d and a common [M × L] dimensional matrix of inducing inputs Z. Using a squared exponential [43] as covariance function (3) allows [38] to derive an analytic lower bound for . Eq (3) uses as parameters of the covariance function. While α_l have an interpretation as inverse squared length scale in the latent dimension l, represents a signal amplitude. The significance of introducing a sparse GP and a lower bound for is that [38] obtain an analytic expression which bounds log(p(Y)) from below and allows for maximization w.r.t. the variational parameters Z, [μ_n, S_n∀n] and the parameters of the covariance function θ.

The GP-LVM implementation by [38] requires in addition to chose between the deterministic training conditional (DTC) or the fully independent training conditional (FITC) parametrization of sparse GPs [60]. The maximum value of depends furthermore on the number of inducing values M and on the dimension of the latent space L. For applying GP-LVM we use the BayesianGPLVM model of the Python GPy package [61], induce sparsity by DTC and initialize the modes of Q(X) by the first L coordinates in principal component space. Except for the number of iterations which we set to 10000, optimization uses BayesianGPLVM.optimize with default parameters. This choice will optimize the lower bound of with respect to the variational parameters Z and [μ_n, S_n∀n] by using the L-BFGS-B optimizer of Python SciPy with default settings and iterations set to 10000. To obtain a reasonable number of inducing variables we set L to 36 and increase M until reaches a plateau. The resulting M is then fixed and we determine the optimal latent dimension as . The optimized Bayesian GP-LVM represents the 209 Nile tilapia images as a product of Gaussian distributions, . GP-LVM is meant to extract features from Nile tilapia images which allow predicting the lake of origin of the fish images. GP-LVM will however target image features irrespective of their biological relevance [36]. We refrain therefore from the proposition in [38, 59] to use GP-LVM as class conditional densities in a fully generative classifier. Instead we propose to use selected dimensions of the expectation E[X]_Q(X) as inputs for diagnostic classifiers after we convinced ourselves that the features represent genuine Nile tilapia morphology. As is pointed out by [43], we may use the size of the inverse length scales α_l to order the L dimensions of X by feature relevance. To elucidate the correspondence between latent dimensions and features in the Nile tilapia images we make use of the generative nature of GP-LVM. When fixing all but the l-th dimension of X to the respective sample mean, projecting the variability of dimension l with p(Y|X) to image coordinates illustrates how pixel variation in different image regions manifests itself in GP-LVM dimension l. Visualizations of the resulting saliency maps allow us to differentiate between latent GP-LVM dimensions which represent genuine Nile tilapia morphology and GP-LVM dimensions which are influenced by technical artifacts.

Predictive classification.

To obtain robust analysis results we consider two classical machine learning methods to infer classifiers which are trained to predict the lake of origin when presented with GP-LVM or GPA transformed landmarks. To include a competitive approach which is based on classical neural networks, we use the hybrid Monte Carlo implementation for multi layer perceptrons (HMC-MLP) by R. Neal [44, 45]. As a second classical machine learning method we apply a Gaussian process classifier (GPC) [42, 43]. Using C = 6 as the number of lakes, predicting the lake of origin of Nile tilapia samples is a 1-of-C classification problem. A discrete column vector y denotes in this setting the lake of origin of every Nile tilapia sample. To obtain a measure of uncertainty when classifying test cases, the proposed approaches learn to predict the posterior probabilities of class. In preparation of model fitting the labels y have to be coded in a zero-one fashion such that Y is a [N × C] dimensional matrix of zeros, except for column y[n] in row n which we set to one. Denoting the classification inputs of one sample as L-dimensional vector x, the probabilities, P(y|x), of 1-of-C problems are for HMC-MLPs obtained with the softmax transformation [42]. An implementation of HMC-MLP inference is provided by R. Neal at https://www.cs.toronto.edu/~radford/fbm.software.html.

Polychotomous 1-of-C classification with GPCs is most often approached with C binary classifiers. The model predicts in this case the zero-one coded Y matrix column wise by using a logistic sigmoid [42] as target transformation. Denoting the L-inputs of sample n as x_n = X[n,:]^T, a GPC defines a Gaussian process in some latent map f(x_n). If we condition on all training samples sample, f(ξ) has for a novel input ξ a univariate Gaussian distribution. GPC predictions of C binary class probabilities are obtained by integrating out f(ξ) from P(y_c, f(ξ)|ξ). A subsequent normalization provides the 1-of-C probabilities as . We apply in our analysis the GPC which is provided in GPy [61], use expectation propagation [62] with default parameters for inference and L-BGFS-B as optimizer.

GPC as well as HMC-MLP allow attenuating input features which do not aid classification. R. Neal coined in [44] the term automatic relevance determination (ARD) to denote this behavior. Introducing ARD in GPCs is a matter of using the squared exponential in Eq (3) as covariance function. The inverse squared length scales [α₁, .., α_L] which are inferred during inference can be used as ARD metric with larger values indicating more important inputs. Allowing for ARD during HMC-MLP inference requires specifying a hierarchical prior which governs the scale of a zero mean Gaussian prior over all input specific parameters. Our application relies on a prior which in the limit of an infinite number of hidden units converges to a Gaussian process. We consider separate ARD scales for input to hidden and input to output parameters and otherwise follow the default suggestions in the documentation [44]. While the number of inputs depends on whether we use GPA or GP-LVM features, all MLPs have 20 hidden units and six output units. The HMC simulations draw 2500 samples, discard the first 25% as burn in and extract independent predictions and ARD relevance scales from the remaining MCMC samples. Inference represents linear relevance, σ_l,1, and nonlinear relevance, σ_l,2, for every dimension l separately. Since HMC-MLP ARD values are prior scales, we suggest combining both relevance measure by summing squares, . The same aggregation allows to interpret the relevance metrics of both landmark coordinates together. The α_l length scales in the squared exponential covariance function have a nonlinear effect on the covariance matrix of the Gaussian process. For combining GPC ARD metrics we suggest therefore to minimize the difference between the inferred covariance matrix and an approximation which results from using for all respective input dimensions the combined inverse length scales. To combine the inverse squared length scales [α_l, .., α_l+j] for input dimensions l to l + j to one value, we obtain the approximate optimum as (4)

Eq (4) expresses the original length scales as diagonal matrix L = diag(α_l, ..α_l+j) and denotes all samples of the corresponding input subsets as η_n. The ARD metrics which are obtained from ten fold cross testing are averaged and used to rank input features.

Convolutional neural networks.

Convolutional neuronal networks (CNN) were originally introduced in [25, 26] as optimal architecture for extracting information from images. Renewed interest in CNNs was triggered by [63] and CNNs since then regularly scoring top results in data analysis competitions. The CNN architecture which we apply for predicting the lake of origin from Nile tilapia images is the Keras [53, 54, 64] implementation of the ConvNet-D (VGG-16) architecture in [24]. Motivated by the success of VGG-16 with this resolution, we decided to use gray scale images with 224 × 224 pixels as CNN input. To allow using the pre-trained ImageNet [52] model of VGG-16 which expects three RGB channel images with a 224 × 224 resolution as input, we replicate the same gray scales in all three channels. Our CNN architecture, which we illustrate in Fig 3, follows largely the proposition in [24]. To fit to our Nile tilapia data, we replaced however their fully connected layers by a flatten layer, a dense layer with 1024 neurons, a dropout layer with a 50% dropout rate and a final dense layer with six neurons and softmax transformation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Sketch of the CNN architecture we use for predicting the lake of origin from 224 gray scale Nile tilapia images.

Our architecture uses 16 layers. Except for the fully connected layers which use 1024 nodes and a 6 class softmax output layer, we apply the Keras version of VGG-16 by [24] which is inferred on ImageNet data [52].

https://doi.org/10.1371/journal.pone.0249593.g003

Albeit relying on a pre-trained CNN, additional adaptation to our data set is crucial to reach good model performance. Satisfying results can be obtained with a procedure that was proposed by [24]. Training optimizes cross entropy, uses 70 epochs, a batch size of 8, the RMSprop optimizer with a learning rate of 10⁻⁴ and otherwise default settings. To achieve invariance to location, scale and orientation differences we follow [65] and use data augmentation with the following parameters: rescale = 1/255, rotation_range = 20, width_shift_range = 0.2, height_shift_range = 0.2, horizontal_flip = True and fill_mode = ‘nearest’.

The strength of CNNs to extract very subtle features lead to excellent performance characteristics which come however at the expense that the resulting models are complex and difficult to interpret. To avoid reporting results which are technically superior but result from the inclusion of rogue features, recent publications [36, 37, 48, 66–69] accentuate the importance of investigating how CNNs reach their decisions. We apply to this end the Grad-CAM [69] implementation in [70] and the implementation of layer wise propagation of relevance (LPR) [36, 37, 48] in [71]. Both approaches provide visual impressions about image regions which are important for trained CNNs to arrive for a test case at the respective prediction.

Statistical assessments of saliency maps.

LPR [48] and Grad-CAM [69] provide quantitative assessments at pixel level about how different regions of a sample image contribute to the calculated predictions. By mapping the variation of individual latent dimensions into image space, GP-LVM provides comparable information for the entire data set. The LPR, GRAD-CAM and GP-LVM derived spatial importance metrics are positive, usually illustrated as saliency maps and interpreted manually. The decision whether a saliency map gives cause for concern may thus depend on the person assessing the map. To improve the reproducibility of determining image regions which are important for predicting the lake of origin of Nile tilapia samples, we suggest to assess the saliency maps with a statistical test.

To derive a test statistic a feature which is represented by the convolution layers of VGG-16 or by the latent space of GP-LVM is considered as a set of N neighboring pixels, . The tuple (ξ_n, η_n) refers to the x and y coordinate of every pixel which characterizes the feature . Spatial characteristics of extracted features like local persistence and sparsity is apparent in image processing literature (e.g. [54], chapter 5.4). Local persistence of feature coordinates implies that every feature pixel, , has a small neighborhood ϵ((ξ, η)) which contains other pixels which are also part of feature . We may express this observation as . Image regions which are important for model predictions will thus be characterized as comparably dense clusters of large LPR, GRAD-CAM or GP-LVM derived importance metric values. Such saliency maps may be described as marked spatial point processes (e.g. [72] chapter 6.4) on where denotes the coordinate space of the spatial process which gives rise to a saliency map and is the domain of the importance metric. Saliency maps are hence samples from a marked point process , where determines the location of important image coordinates and determines the location specific distribution of the importance metric. This notation allows expressing lack of importance of image regions as null hypothesis that the spatial process on which gives rise to an observed feature in the saliency is a homogeneous Poisson process. The expectation that important image regions are characterized by the existence of sparse clusters of large metric values suggests using a one sided alternative.

To obtain reproducible verdicts about image regions which contribute substantially to predictions, the proposed test is applied on a pixel per pixel basis in a non-parametric manner.

1. We start by regarding a saliency map to be a sample which was generated under the alternative hypothesis .
2. A sample of the null hypothesis in the marked space is obtained by randomly reshuffling all x and y coordinates and rearranging the importance metric values of the original saliency map.
3. To assess the null hypothesis against the alternative at position (x, y), we compare the (x, y) surrounding local averages in the saliency map which represents the null hypothesis against the respective average value of the original saliency map. For calculating averages we convolve the saliency maps with a k × k pixel wide Gaussian kernel to obtain g_H1(x, y) for the alternative hypothesis and g_H0(x, y) for the null hypotheses. Using a one sided alternative implies that we obtain evidence in favor of the null hypothesis H0 if g_H0(x, y)≥g_H1(x, y).
4. A p-value map is obtained by iterating sample generation under the null hypothesis N times, where N is sufficiently large and counting for every location the number of instances n(x, y) for which we obtain evidence in favor of H0. Location specific p-values are obtained as ratio (5)
5. To obtain a reproducible visualization from the saliency map we use a p-value threshold of 0.001 to obtain a binary mask which indicates image regions which are by LPR, GRAD-CAM or GP-LVM considered important for the calculated predictions. The sample specific nature of LPR and GRAD-CAM suggests to display the discretized p-value map on top of the image to highlight important image features. The saliency map which we obtain for GP-LVM assesses the data set in its entirety. In this case we use the binary mask to set all saliency values in irrelevant image regions to zero.

The implementation of this test relies on the Python OpenCV interface for calculating convolutions and sets k = 5 and N = 10⁴.

Performance metrics.

To assess whether geographic variation exists in Ethiopian Nile tilapia body features, we perform in this paper an indirect analysis. The existence of population specific morphotypes would imply that image derived features contain information about the lake of origin of fish. If such information exists, we expect to obtain reasonably good performance characteristics. Drawing respective conclusions depends therefore on quantitative performance metrics which allow us to compare classifiers. While we did already point out that we use ten fold cross testing [46] to obtain unbiased estimates, we will now summarize three complementary metrics of classification performance.

1. Classification accuracy (Acc) is the most commonly applied performance metric for classifiers [46]. Evaluating classification accuracy requires us to determine the rate of agreement between predicted and true labels. If we use N to denote the number of samples and use as predicted and y_n as true label and define if labels agree and if labels disagree, generalization accuracy is expressed as (6)
   Rating a classifier as reasonably good requires that its Acc is larger than the Acc obtained when predicting the majority label.
2. While Acc captures predictive accuracy, taking posterior probabilities for class, P(y|x_n), into consideration provides further insight. Mutual information (MI) [73], I(x, y), between input x and class labels y provides us with such a metric. We obtain (7) where use of log₂ leads to quantifying MI in bit. To rate a classifier by MI as reasonably good we require that I(x, y) is larger than zero.
3. Assessing classifiers is in statistical terminology equivalent to conducting an experiment. Repeating the assessment will thus inevitably change Acc and I(x, y). To ensure that differences are systematic requires confirming that randomness could not have caused the differences. For assessing classifiers a and b McNemar [74, 75] proposed an optimal paired significance test. McNemars statistic consists of counts n_a and n_b. While n_a counts the number of samples which were correctly predicted by classifier a and wrongly classified by classifier b, n_b counts the number of samples that were wrongly classified by classifier a and correctly predicted by classifier b. McNemars procedure is based on the idea that differences between both classifier arise by chance if (n_a, n_b) has a Binomial distribution with count n_a + n_b and probability 0.5. Denoting the probability mass of the binomial distribution as and assuming a two sided alternative hypothesis, we get the McNemar p-value (Sig) as (8)
   A reasonably good classifier will in comparison with predicting the majority label for all samples obtain a McNemar p-value, p < 0.05. For visualization purpose, we truncate p-values to be smaller than 0.99 and apply a logit transformation logit(p) = log(p) − log(1 − p).

Generalization accuracy, mutual information and McNemars significance test allow investigating whether Nile tilapia images contain information about the origin of fish samples. Mutual information provides as the name suggest a direct quantification of information content. Generalization accuracy and McNemars test require however to compare the image and GPA-shape based classifiers against predicting the majority label. Predicting the majority class is an input independent lower bound for generalization performance. To substantiate that Nile tilapia morphology depends on the lake of origin, respective classifiers must hence significantly outperform predictions which only depend on counting labels [46].