Theory for tuning afferent connectivity based on spatial covariance structure

To theoretically examine covariance-based learning, we start with the abstraction of the MAR dynamics P⁰ ↦ Q⁰ in Eq (10). As depicted in Fig 3A, each training step consists in presenting an input pattern P⁰ to the network and the resulting output pattern Q⁰ is compared to the objective in Fig 3B. For illustration, we use two categories (red and blue) of 5 input patterns each, as represented in Fig 3C and 3D. To properly test the learning procedure, noise is artificially added to the presented covariance pattern, namely an additional uniformly-distributed random variable with a magnitude of 30% compared to the range of the noiseless patterns P⁰, independently for each matrix element while preserving the symmetry of zero-lag covariances; compare the noisy pattern in Fig 3A (left matrix) to its noiseless version in Fig 3C (top left matrix). The purpose is to mimic the variability of covariances estimated from a (simulated) time series of finite duration (see Fig 2), without taking into account the details of the sampling noise. The update ΔB_ik for each afferent weight B_ik is obtained by minimizing the distance (see Eq (25) in Methods) between the actual and the desired output covariance (11) where U^ik is an m × m matrix with 0s everywhere except for element (i, k) that is equal to 1; this update rule is obtained from the chain rule in Eq (26), combining Eqs (27) and (30) with P⁻¹ = 0 and A = 0 (see Theory for learning rules in Methods). Here η_B denotes the learning rate and the symbol ⊙ indicates the element-wise multiplication of matrices followed by the summation of the resulting elements —or alternatively the scalar product of the vectorized matrices. Note that, although this operation is linear, the update for each matrix entry involves U^ik that selects a single non-zero row for U^ikP⁰B^T and a single non-zero column for BP⁰U^ikT. Therefore, the whole-matrix expression corresponding to Eq (11) is different from , as could be naively thought.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Learning variances in a feed-forward network.

A: Schematic representation of the input-output mapping for covariances defined by the afferent weight matrix B, linking m = 10 input nodes to n = 2 output nodes. B: Objective output covariance matrices for two categories of inputs. C: Matrix for the 5 input covariance patterns P⁰ (left column) for the first category, with their images under the original connectivity (middle column) and the final images after learning (right column). The training leads to a larger variance (darker pixel) for output 1 (bottom left pixel of matrices in the right column) than for output 2 (top right pixel). D: Same as C for the second category. The training leads to a larger variance for output 2 than for output 1 except for the last pattern. E: Evolution of individual weights of matrix B during ongoing learning. F: The top panel displays the evolution of the error between Q⁰ and at each step. The total error taken as the matrix distance E⁰ in Eq (25) is displayed as a thick black curve, while individual matrix entries are represented by gray traces. In the bottom panel the Pearson correlation coefficient between the vectorized Q⁰ and describes how they are “aligned”, 1 corresponding to a perfect linear match.

https://doi.org/10.1371/journal.pcbi.1008127.g003

Before training, the output covariances are rather homogeneous as in the examples of Fig 3C and 3D (initial Q⁰) because the weights are initialized with similar random values. During training, the afferent weights B_ik in Fig 3E become specialized and tend to stabilize at the end of the optimization. Accordingly, Fig 3F shows the decrease of the error E⁰ between Q⁰ and defined in Eq (25). After training, the output covariances (final Q⁰ in Fig 3C and 3D) follow the desired objective patterns with differentiated variances, as well as small cross-covariances.

As a consequence, the network responds to the red input patterns with higher variance in the first output node, and to the blue inputs with higher variance in the second output (top plot in Fig 4B). We use the difference between the output variances in order to make a binary classification. The classification accuracy corresponds to the percentage of output variances with the desired ordering. The evolution of the accuracy during the optimization is shown in Fig 4C. Initially around chance level at 50%, the accuracy increases on average due to the gradual shaping of the output by the gradient descent. The jagged evolution is due to the noise artificially added to the input covariance patterns (see the left matrix in Fig 3A), but it eventually stabilizes around 90%. The network can also be trained by changing the objective matrices to obtain positive cross-covariances for red inputs, but not for blue inputs (Fig 4D); in that case variances are identical for the two categories. The output cross-covariances have separated distributions for the two input categories after training (bottom plot in Fig 4E), yielding the good classification accuracy in Fig 4F.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison between learning output patterns for variance and cross-covariance.

A: The top matrices represent the two objective covariance patterns of Fig 3B, which differ by the variances for the two nodes. B: The plots display two measures based on the output covariance: the difference between the variances of the two nodes (top) and the cross-covariance (bottom). Each violin plot shows the distributions for the output covariance in response to 100 noisy versions of the 5 input patterns in the corresponding category. Artificial noise applied to the input covariances (see the main text about Fig 3 for details) contributes to the spread. The separability between the red and blue distributions of the variances indicates a good classification. The dashed line is the tentative implicit boundary enforced by learning using Eq (30) with the objective patterns in panel A: Its value is the average of the differences between the variances of the two categories. C: Evolution of the classification accuracy based on the difference of variances between the output nodes during the optimization. Here the binary classifier uses the difference in output variances, predicting red if the variance of the output node 1 is larger than 2, and blue otherwise. The accuracy eventually stabilizes above the dashed line that indicates 80% accuracy. D-F: Same as panels A-C for two objective covariance patterns that differ by the cross-covariance level, strong for red and zero for blue. The classification in panel F results from the implicit boundary enforced by learning for the cross-covariances (dashed line in panel E), here equal to 0.4 that is the midpoint between the target cross-covariance values (0.8 for read and 0 for blue).

https://doi.org/10.1371/journal.pcbi.1008127.g004

As a sanity check, the variance does not show a significant difference when training for cross-covariances (top plot in Fig 4E). Conversely, the output cross-covariances are similar and very low for the variance training (bottom plot in Fig 4B). These results demonstrate that the afferent connections can be efficiently trained to learn categories based on input (co)variances, just as with input vectors of mean activity in the classical perceptron.

Discriminating time series observed using a finite time window

Now we turn back to the configuration in Fig 2A and verify that the learning procedure based on the theoretical consistency equations also works for simulated time series. This means that the sampled activity of the network dynamics itself is presented, rather than their statistics embodied in the matrices P⁰ and Q⁰, as done in Figs 3 and 4 and as a classical machine-learning scheme would do with a preprocessing step that converts time series using kernels. Again, the weight update is applied for each presentation of a pattern such that the output variance discriminates the two categories of input patterns. The setting is shown in Fig 5A, where only three input patterns per category are displayed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Learning input covariances by tuning afferent connectivity.

A: The same network as in Fig 3A is trained to learn the input spatial covariance structure P⁰ of time series governed by the dynamics in Eq (12). Only 3 matrices P⁰ = WW^T out of the 5 for each category are displayed. Each entry in each matrix W has a probability f = 10% of being non-zero, so the actual f is heterogeneous across the different matrices W. The objective matrices (right) correspond to a specific variance pattern for the output nodes. B: Example of simulation of the time series for the inputs (light brown traces) and outputs (dark brown). An observation window (gray area) is used to calculate the covariances from simulated time series. C: Sampling error as measured by the matrix distance between the covariance estimated from the time series (see panel B) and the corresponding theoretical value when varying the duration d of the observation window. The error bars indicate the standard error of the mean over 100 repetitions of randomly drawn W and afferent connectivity B. D: Evolution of the error for 3 example optimizations with various observation durations d as indicated in the legend. E: Classification accuracy at the end of training (cf. Fig 4C) as a function of d, pooled for 20 network and input configurations. For d ≥ 20, the accuracy is close to 90% on average, mostly above the dashed line indicating 80%. F: Similar plot to panel E when varying the input density of W from f = 10 to 20%, with d = 20.

https://doi.org/10.1371/journal.pcbi.1008127.g005

To generate the input time series, we use a superposition of independent Gaussian random variables with unit variance (akin to white noise), which are mixed by a coupling matrix W: (12)

We randomly draw 10 distinct matrices W with a density of f = 10% of non-zero entries, so the input time series differ by their spatial covariance structure P⁰ = WW^T. At each presentation, one of the 10 matrices W is chosen to generate the input time series using Eq (12). Their covariances are then computed using an observation window of duration d. The window duration d affects how the empirical covariances differ from their respective theoretical counterpart P⁰, as shown in Fig 5C. This raises the issue of the precision of the empirical estimates required in practice for effective learning.

As expected, a longer observation duration d helps to stabilize the learning, which can be seen in the evolution of the error in Fig 5D: the darker curves for d = 20 and 30 have fewer upside jumps than the lighter curve for d = 10. To assess the quality of the training, we repeat the simulations for 20 network and input configurations (W and z, resp.), then calculate the difference in variance between the two output nodes as in Fig 4B and 4C. Training for windows with d ≥ 20 achieve very good classification accuracy in Fig 5E. This indicates that the covariance estimate can be evaluated with sufficient precision from only a few tens of time points. Moreover, the performance only slightly decreases for denser input patterns (Fig 5F). Similar results can be obtained while training the cross-covariance instead of the variances.

Discrimination capacity of covariance perceptron for time series

The efficiency of the binary classification in Fig 4 relies on tuning the weights to obtain a linear separation between the input covariance patterns. Now we consider the capacity of the covariance perceptron, evaluated by the number p of input patterns (or pattern load) that can be discriminated in a binary classification. For the classical perceptron and for randomly-chosen binary patterns that must be separated in two categories, the capacity is 2m, twice the number m of inputs [33, 34]. An analytical study of the capacity of the covariance perceptron using Gardner’s theory of connections from statistical mechanics is performed in a sister article [35]. That study shows that a single readout cross-covariance can in theory discriminate twice as many patterns per synapse as the classical perceptron, but that this capacity does not linearly scale with the number of outputs. Finding such optimal solutions is an NP-hard problem in general and optimization methods like the gradient descent employed here may only achieve suboptimal capacities in practice. In addition, an important difference compared to that study, which focused on “static” patterns, concerns the time series used for training and testing here, which involves empirical noise (see Fig 5C). Thus, we here employ numerical simulation to get a first insight on the capacity of the covariance perceptron with “noisy inputs”, varying in the same manner the number of input neurons and patterns to learn.

Here we consider the same optimization of output variances on which the discrimination is based as in Fig 5, instead of the cross-covariances studied using Gardner’s theory [35]. The evolution of the classification accuracy averaged over 10 configurations is displayed in Fig 6A, where darker gray levels correspond to larger network sizes as indicated in the legend. For each configuration, the mean accuracy of the last three epochs is plotted in Fig 6B where the observation duration is d = 20. At the load p = m, the performance decreases with the network size: for instance, it remains in the case of m = 100 inputs (darker curve) around 75% for a load of m patterns and way above 50% for 2m patterns. Interestingly, the performance significantly increases when using larger d, for example improving by roughly 10% for m = 50 and 100 in each of the two plots of Fig 6C. This means that the empirical noise related to the covariance estimation over the observation window (see Fig 5B) becomes larger when the number m of inputs increases, but it can nonetheless be compensated by using a larger window.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Numerical evaluation of capacity.

A: Evolution of the classification accuracy over the optimization epochs. During each epoch, all p = m patterns are presented in a random order. We use the same network as in Fig 5, but with distinct input numbers m as indicated in the legend. The observation duration is d = 20. The error bars correspond to the standard error of the mean accuracy over 10 configurations. B: Comparison of the classification accuracies as a function of the relative pattern load p/m (x-axis). Note that 2 corresponds to the theoretical capacity p = 2m of the classical perceptron with a single output, but the architecture considered here has 2 outputs; for an in-depth study of the capacity and its scaling with the number of output nodes, please refer to our sister paper [35] whose results are discussed in the main text. The plotted values are the mean accuracies for each configuration, averaged over the last three epochs in panel A. The error bars indicate the standard error of the mean accuracy over 10 repetitions for each configuration. C: Similar plot to panel B when varying the observation duration d for two cases p = m and p = 2m.

https://doi.org/10.1371/journal.pcbi.1008127.g006

Stability of ongoing learning

Before examining classification, we consider the problem of stability of ongoing learning (or plasticity). Unsupervised Hebbian learning applied to recurrent connections is notoriously unstable and adequate means to prevent ongoing plasticity from leading to activity explosion are still under debate [36, 37] —note that, if those studies concern especially spiking networks, their conclusions also apply to non-spiking networks as considered here. Supervised learning, however, can lead to stable weight dynamics [38, 39]. Stability can be directly enforced in the objective function, but can also be a consequence of the interplay between the learning and network dynamics. Because our objective functions are based on the output covariances, we test whether they also yield stability for the weights and network activity.

The learning procedure is tested with simulated time series as in Fig 5. The weight updates are given by equivalent equations to Eq (11) that determine the weight updates for the afferent and recurrent connectivities, B and A respectively; see Eqs (30), (32), (33) and (34) in Theory for learning rules in Methods. We recall that they rely on the consistency equations (23) and (24), which are obtained in Network dynamics (Methods) under the assumption of stationary statistics. Fig 7 illustrates the stability of the learning procedure, while the error decreases to the best possible minimum. As a first example, we want to map 10 input patterns —corresponding to 10 distinct matrices W in Eq (13), each giving a specific pair of input covariance matrices (P⁰, P¹)— to the same objective covariance matrix , thereby dealing with a single category as illustrated in Fig 7A. Note that with the choice of W with small weights here, all input covariance matrices P⁰ are close to the identity and the optimized connectivity must generate cross-correlations between the outputs. Adapting Eqs (26) and (27) in Methods to the current configuration, the weight updates are given by (14) where the derivatives are given by the matrix versions of Eqs (30) and (32) in Theory for learning rules (Methods): (15)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Stability of ongoing learning for both afferent and recurrent connectivities.

A: Network configuration with m = 20 inputs and n = 5 outputs. Each input pattern corresponds to a randomly chosen m × m matrix W with 10% density of non-zero connections that determines the input covariance matrices (P⁰, P¹) of the time series generated from Eq 13. The objective is a randomly-drawn m × m symmetric matrix (also ensuring its definite positivity). Here we consider a single category (in red) to test whether the weight learning rule can achieve a desired input-output mapping, leaving aside classification for a moment. B: Example evolution of the afferent and recurrent weights (green and purple traces, respectively). Simulation of the time series as in Fig 5, observed for a duration d = 50. The network has to map 10 input pattern pairs (P⁰, P¹) similar to the example in panel A to a single output . C: Evolution of the tuning of the network output. The learning procedure aims to reduce the normalized error between the output covariance Q⁰ and its objective (left plot, similar to Fig 3F), here calculated as where ∥⋯∥ is the matrix norm. We also compute the Pearson correlation coefficient between the vectorized matrices Q⁰ and (right plot). The thick black traces correspond to the mean over 10 repetitions of similar optimizations to panel A, each gray curve corresponding to a repetition. D-E: Similar plots to panels A-B for the tuning of both for a single repetition. The input time series are generated in the same manner as before. Contrary to panel A, the output objective pair is chosen as two homogeneous diagonal matrices with larger values for than , corresponding to outputs with autocorrelation and no cross-correlation —this example is inspired by previous work on whitening input signals [40]. The observation duration is d = 100.

https://doi.org/10.1371/journal.pcbi.1008127.g007

Both formulas have the form of a discrete Lyapunov equation that can be solved at each optimization step to evaluate the weight updates for A and B. Also recall that the derivation of the consistency equations in Network dynamics (Methods) assumes P² = 0 and is thus an approximation because we have P² = W² P⁰ here. As the input matrix W must have eigenvalues smaller than 1 in modulus to ensure stable dynamics, our approximation corresponds to ∥P²∥ = ∥WP¹∥ ≪ ∥P¹∥. The purpose of this example is thus to test the robustness of the proposed learning in that respect. The weight traces appear to evolve smoothly in Fig 7B. In the left plot of Fig 7C, the corresponding error between the output Q⁰ and the objective firstly decreases and then stabilizes. The evolution of the Pearson correlation (right plot of Fig 7C) further indicates that the matrix structure of the output Q⁰ remains very close to that of , once it reached saturation, even though the network may not perfectly converge towards the objectives as indicated by the residual error.

A second and more difficult example is explored in Fig 7D and 7E, where the objective is a pair of matrices and . The weight optimization then involves the equivalent of Eq (15) for Q¹. The issue of whether there exists a solution for the weights A and B to implement the desired mapping is more problematic because the defined objectives imply many constraints, namely Eqs (23) and (24) must be satisfied for all ten pairs (P⁰, P¹) with with the same weight matrices. This results in less smooth traces for the weights (Fig 7D) and a weak decrease for the normalized error (left plot in Fig 7E), suggesting that there is not even an approximate solution for the weights A and B. Nonetheless, the weight structure does not explode and the Pearson correlation between the output covariances and their respective objectives indicate that the objective structure is captured to some extent (right plot in Fig 7E).

Computational and graph-local approximations of the covariance-based learning

First, we consider an approximation in the calculation of the weight updates that does not require solving the Lyapunov equation. An important question is which role the elements that quantify the non-linearity due to the recurrent connectivity A play in determining the weight updates. As explained around Eq (36) in Methods, we consider the approximation of Eq (15) that ignores second-order terms in the recurrent connectivity matrix A in the Lyapunov equation (16)

Now the calculation of the weight updates is much simpler, involving only a few matrix multiplications and additions. To test the validity of this approximation, we repeat the same optimization as in Fig 7A with 10 input patterns to map to a single output pattern with objective . As illustrated for an example in Fig 8A, the comparison of Eq (16) (red trace) with Eq (15) (black trace) hardly show any difference in the performance. This is confirmed in Fig 8B for 10 repetitions of the same training with randomly chosen input and output patterns. Although the approximation for the solution of the Lyapunov equation may seem coarse, it yields a very similar trajectory for the gradient descent. This gives the intuition that this computational approximation gives the correct general direction in the high-dimensional space and, since the weight updates are computed at each optimization step and these steps are small, the gradient descent does not deviate from the “correct” direction. Even though this example involves non-full connectivity where only about 30% of the weights are trained (others being kept equal to zero), the same simulation with full connectivity (not shown) gives similar results with no distinguishable difference between the full computation and the computational approximation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Approximations of the gradient descent.

A: Comparison of the evolution of the error in optimizing Q⁰ for three flavors of the gradient descent: the “full” solution (in black) using Eq (15), the computational approximation (in red) using Eq (16) and the local approximation (in purple) using Eq (17). Network and pattern configuration similar to Fig 7A, involving 10 input patterns to map to a single output pattern by training both afferent and recurrent connections in a network of m = 20 inputs and n = 5 outputs. Here the network has sparse connectivity, corresponding to a probability of existence for each connection equal to 30%; weights for absent connections are not trained and kept equal to 0 at all times. B: Asymptotic error estimated from the last 10 optimization steps in panel A for 10 repetitions of similar configurations to panel A. C: Schematic representation of the local approximation in Eq (17) to compute the weight updates of afferent connections and recurrent connections targeting neuron i (here with two examples B_ik and A_ij): only covariances from network neurons with a connection to neuron i (like the “parent” neuron i′ via the dashed blue arrow) are taken into account. D: Example of network connectivity (binary matrices in black on the right) that determine which elements of the covariance error matrix (in color on the left) are used to calculate the weight update in Eq (17). Crosses in the left matrix indicate discarded elements, that correspond to absent recurrent connections. Note that variances are never discarded.

https://doi.org/10.1371/journal.pcbi.1008127.g008

Second, we consider a local approximation where the information necessary to compute the weight updates is only accessible from presynaptic neighbor neurons in the network as illustrated in Fig 8C: (17) where S_i is the subset of “parent” neurons with a connection to neuron i. The rationale here is that information related to the activities of a neuron pair can be evaluated at the point of contact that are recurrent synapses. Of course, this only makes a difference in the case of non-full connectivity (around 30% density in Fig 8) and this local approximation requires the computational approximation since solving the Lyapunov equation in the full calculation of Eq (15) requires the knowledge of all connections within the network. Using the expressions in Eq (16) for recurrent connections while ignoring afferent connections, we have (18)

This means that the necessary information for computing the weight update is the summed activity received by each parents neuron i′ ∈ S_i, which has to be centralized by the downstream i. The same observation is valid for the summed activity received by each neuron i′ from the input neurons. This approximation is local in the graph in the sense that it only uses information from neighbor (parent) neurons. An example of the matrix elements in the covariance error that contribute to the weight update is displayed in Fig 8D. Although the performance decreases compared to the computational approximation as illustrated in Fig 8A and 8B, this local optimization still performs reasonably well.

Classification of time series with hidden dynamics

From the dynamics described in Eq (5), a natural use for A is the transformation of input spatial covariances (P⁰ ≠ 0 and P¹ = 0) to output spatio-temporal covariances (Q⁰ ≠ 0 and Q¹ ≠ 0), or vice-versa (P⁰ ≠ 0, P¹ ≠ 0, Q⁰ ≠ 0 and Q¹ = 0). S1 Appendix provides examples for these two cases that demonstrate the ability to tune the recurrent connectivity together with the afferent connectivity (from now on with 100% density), which we further examine now.

We consider input time series that are temporally correlated and spatially decorrelated, meaning that P¹ conveys information about the input category (i.e. reflecting the hidden dynamics), but not P⁰. The theory predicts that recurrent connectivity is necessary to extract the relevant information to separate the input patterns. To our knowledge this is the first study that tunes recurrent connectivity in a supervised manner to specifically extract temporal information from lagged covariances when spatial covariances are not informative about the input categories. Concretely, we here use 6 matrices W (3 for each category) to generate the input time series that the network has to classify based on the output variances, illustrated in Fig 9A. Importantly, we choose W = exp(μ1_m + V) with exp being the matrix exponential, V an antisymmetric matrix and μ < 0 for stability. As a result, the zero-lag covariance of the input signals is exactly the same for all patterns of either category, proportional to the identity matrix as illustrated in Fig 9B. This can be seen using the discrete Lyapunov equation P⁰ = WP⁰W^T + 1_m, which is satisfied because WW^T = exp(2μ1_m + V + V^T) = e^2μ1_m. In contrast, the time-lagged covariances P¹ = WP⁰ differ across patterns, which is the basis for distinguishing the two categories.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Learning input spatio-temporal covariances with both afferent and recurrent connectivities.

A: Network architecture with m = 10 input nodes and n = 3 output nodes, the latter being connected together by the recurrent weights A (purple arrow). The network learns the input spatio-temporal covariance structure, which is determined here by a coupling matrix W between the inputs as in Eq (13). Here we have 3 input patterns per category. The objective matrices (right) correspond to a specific variance for the output nodes. B: The matrices W are constructed such that they all obey the constraint P⁰ ∝ 1_m. C: Evolution of the error for 3 example optimizations with various observation durations d as indicated in the legend. D: Classification accuracy after training averaged over 20 network and input configurations. For the largest d = 100, the accuracy is above 80% on average (dashed line). The color contrast corresponds to the three values for d as in panel C. E: Accuracy similar to panel D with no recurrent connectivity (A = 0). F: Same as panel D with a random fixed matrix A and switching off its learning. G: Same as panel D with the computational approximation in Eq (16) that does not require solving the Lyapunov equation.

https://doi.org/10.1371/journal.pcbi.1008127.g009

The output is trained only using Q⁰ according to Eq (15), meaning that the input spatio-temporal structure is mapped to an output spatial structure —also following the above considerations about the existence of adequate weights to implement the desired input-output covariance mapping. The covariances from the time series are computed using an observation window of duration d in the same manner as before in Fig 5B. Note that it is important to discard an initial transient period to remove the influence of initial conditions on both x^t and y^t. In practice, we use a larger window duration d compared to Fig 5, as it turns out that the output covariances are much noisier here. The influence of d can also be seen in Fig 9C, where the evolution of the error for the darkest curves with d ≥ 60 remain lower on average than the lighter curve with d = 20. To assess the quality of the training, we repeat the simulations for 20 network and input configurations and then calculate the difference in variance between the two output nodes for the red and blue input patterns. The accuracy gradually improves from d = 20 to 100 in Fig 9D. When enforcing A = 0 in Fig 9E, classification stays at chance level. This is expected and confirms our theory, because the learning for B only captures differences in P⁰, which is the same for all patterns here. When A is sufficiently large (but fixed), it contributes to some extent to Q⁰ such that the weight update for B can extract relevant information as can be seen in Eq (15), raising the performance above chance level in Fig 9F. Nonetheless, the performance remains much worse than when the recurrent connections are optimized. These results demonstrate the importance of training recurrent connections in transforming input spatio-temporal covariances into output spatial covariances. Last, Fig 9G shows that the computational approximation in Eq (16) performs as well as the full gradient in Eq (15) that involves solving the Lyapunov equation for this task.

Covariance-based decoding and representations

The perceptron is a central concept for classification based on artificial neuronal networks, from logistic regression [19] to deep learning [24, 25]. The mechanism underlying classification is the linear separability of the input covariance patterns performed by a threshold on the output activity, in the same manner as in the classical perceptron for vectors. All “dimensions” of the output covariance can be used as objectives for the training, cross-covariances and variances in Q⁰, as well as time-shifted covariances in matrix Q¹. As with the classical perceptron, classification relies on shaping the input-output mapping, for example by potentiating afferent weights from an input with high variance to two outputs to generate correlated activity between the outputs. Note that, in general, the existence of an achievable mapping between the input patterns and the objective patterns is not guaranteed, even when tuning only afferent connections with . Nonetheless, the weight optimization aims to find the best solution as measured by the matrix distance with respect to the objectives (Fig 7). Nonetheless, our results lay the foundation for covariance perceptrons with multiple layers, including linear feedback by recurrent connectivity in each layer. The important feature in its design is the consistency of covariance-based information from inputs to outputs, which enables the use of our covariance-based equivalent of the delta rule for error back-propagation [23]. The generalization to higher statistical orders seems a natural extension for the proposed mathematical formalism, but requires dedicated input structures and is thus left for future work.

Although our study is not the first one to train the recurrent connectivity in a supervised manner, our approach differs from previous extensions of the delta rule [21] or the back-propagation algorithm [23], such as recurrent back-propagation [26] and back-propagation through time [27–29]. Those algorithms focus on the mean activity based on first-order statistics and, even though they do take temporal information into account (related to the successive time points in the trajectories over time), they consider the inputs as statistically independent variables. Moreover, unfolding time corresponds to the adaptation of techniques for feedforward networks to recurrent networks, but it does not take the effect of the recurrent connectivity as in the steady-state dynamics considered here. We have shown that this stationary assumption is not an issue for practical applications, even though signals may strongly deviate from Gaussian distributions like the MNIST dataset. Further study about finding the best regularities in input signals for classification, like comparing covariances and profiles of the average trajectories for MNIST digits, is left for future work. In the context of unsupervised learning, several rules were proposed to extract information from the spatial correlations of inputs [42] or their temporal variance [43]. Because the classification of time-warped patterns can be based on the second-order statistics of the input signals [43], we foresee a potential application of our supervised learning scheme, as the time-warping transformation preserves the relative structure of covariances between the input signals (albeit not their absolute values).

The reduction of dimensionality of covariance patterns —from many input nodes to a few output nodes— implements an “information compression”. For the same number of input-output nodes in the network, the use of covariances instead of means makes a higher-dimensional space accessible to represent input and output, which may help in finding a suitable projection for a classification problem. It is worth noting that applying a classical machine-learning algorithm, like the multinomial linear regressor [19], to the vectorized covariance matrices corresponds to nm(m − 1)/2 weights to tune, to be compared with only nm weights in our study (with m inputs and n outputs). We have made here a preliminary exploration of the “capacity” of our covariance perceptron by numerically evaluating its performance in a binary classification when varying the number of input patterns to learn (Fig 6). The capacity for the classical perceptron has been the subject of many theoretical studies [34, 44, 45]. For the binary classification of noiseless patterns based on a single readout, the capacity of the classical perceptron is equal to 2m, twice as much as its number of inputs. In contrast, we have used a network with two outputs that classifies based on the covariance or the variance difference in Fig 4. A formal comparison between the capacities of the covariance and classical perceptrons has been made in a separate paper [35]. Note that a theory for the capacity in the “error regime” was also developed for the classical perceptron [46], which may be relevant here to deal with non-perfect classification and noisy time series (Figs 5 and 6).

Learning and functional role for recurrent connectivity

Our theory shows that recurrent connections are crucial to transform information contained in time-lagged covariances into covariances without time lag (Fig 9). Simulations confirm that recurrent connections can indeed be learned successfully to perform robust binary classification in this setting. The corresponding learning equations clearly expose the necessity of training the recurrent connections. For objectives involving both covariance matrices, and , there must exist an accessible mapping (P⁰, P¹)↦(Q⁰, Q¹) determined by A and B. The use for A may also bring an extra flexibility that broadens the domain of solutions or improve the stability of learning, even though this was not clearly observed so far in our simulations. A similar training of afferent and recurrent connectivity was used to decorrelate signals and perform blind-source separation [40]. This suggests another possible role for A in the global organization of output nodes, like forming communities that are independent of each other (irrespective of the patterns).

The learning equations for A in Theory for learning rules (Methods) can be seen as an extension of the optimization for recurrent connectivity recently proposed [47] for the multivariate Ornstein-Uhlenbeck (MOU) process, which is the continuous-time version of the MAR studied here. Such update rules fall in the group of natural gradient descents [48] as they take into account the non-linearity between the weights and the output covariances. We have shown that a much simpler approximation of the solution of the Lyapunov equation for the weight updates gives a quasi identical performance (Fig (8)). This approximation greatly reduces the computational cost of the covariance-based learning rule. It is expected that it may be insufficient when the recurrent connectivity grows and results in strong network feedback, in which case Eq (35) may be expanded to incorporate higher orders in A.

Another positive feature of our supervised learning scheme is the stability of the recurrent weights A for ongoing learning, even when there is no mapping that satisfies all input-output pairings (Fig 7). This is in line with previous findings [39, 49] and in contrast with unsupervised learning like STDP that requires stabilization or regularization terms, in biology known as “homeostasis”, to prevent the problematic growth of recurrent weights that often leads to an explosion of the network activity [36, 37, 50]. It also remains to be explored in more depth whether such regularizations can be functionally useful in our framework, for example to improve classification performance.

Extensions to non-linear neuronal dynamics and continuous time

In Section the capacity has been evaluated only for the case of linear dynamics. Including a non-linearity, as used for classification with the classical perceptron [21], remains to be explored. Note that for the classical perceptron a non-linearity applied to the dynamics is in fact the same as applied to the output; this is, however, not so for the covariance perceptron. The MAR network dynamics in discrete time used here leads to a simple description for the propagation of temporally-correlated activity. Several types of non-linearities can be envisaged in recurrently connected networks of the form (19)

Here the local dynamics is determined by ψ and interactions are transformed by the function ϕ. Such non-linearities are expected to vastly affect the covariance mapping in general, but special cases, like the rectified linear function, preserve the validity of the derivation for the linear system in Network dynamics (Methods) in a range of parameters. Another point is that non-linearities cause a cross-talk between statistical orders, meaning that the mean of the input may strongly affect output covariances and, conversely, input covariances may affect the mean of the output. This opens the way to mixed decoding paradigms where the relevant information is distributed in both, means and covariances. Extension of the learning equations to continuous time MOU processes requires the derivation of consistency equations for the time-lagged covariances. The inputs to the process, for consistency, themselves need to have the statistics of a MOU process [51]. This is doable, but yields more complicated expressions than for the MAR process.

Learning and (de)coding in biological spiking neuronal networks

An interesting application for the present theory is its adaptation to spiking neuronal networks. In fact, the biologically-inspired model of spike-timing-dependent plasticity (STDP) can be expressed in terms of covariances between spike trains [8, 9], which was an inspiration of the present study. The weight structure that emerges because of STDP is determined by and reflects the the spatio-temporal structure of the input spike trains [11, 52]. STDP-like learning rules were used for object recognition [53] and related to the expectation-maximization algorithm [54]. Although genuine STDP relates to unsupervised learning, extensions were developed to implement supervised learning with a focus on spike patterns [4, 49, 55–58]. A common trait of those approaches is that they mostly apply to feedforward connectivity only, even though recently also recurrently-connected networks have been considered.

Instead of focusing on the detailed timing in spike trains in output, our supervised approach could be transposed to shape the input-output mapping between spike-time covariances, which are an intermediate description between the full probability distribution of spike patterns (too complex) and firing rate (too simple). As such, our approach allows for some flexibility concerning the spike timing (e.g. jittering) and characterization of input-output patterns, as was explored before for STDP [11]. An important implications of basing information on covariance-based patterns is that they do not require a reference start time, because the coding is embedded in relative time lags. The robustness of such representations in spiking activity is also compatible with the large variability of spiking activity observed in experiments. This contrasts with supervised learning schemes of spike trains with detailed timing that have attracted a lot of recent interest [4, 49, 55]. Our theory thus opens a novel and promising perspective to learn temporal structure of spike trains and provides a theoretical ground to genuinely investigate learning in recurrently connected neuronal networks.

In addition to the computational approximation of the covariance-based learning rule, another key question for biological plausibility is whether our scheme can be implemented in a local rule, meaning that the weight updates should be calculated from quantities available by the pre- and post-synaptic neurons. Moreover, the empirical covariances should ideally be computed online. In the learning equations such as Eq (11), the matrix U^ikP⁰B^T involved in the update of weight B_ik can be reinterpreted as a product of input and output, since its matrix element indexed by (i′, j′) is simply after using y^t = Bx^t according to Eq (5) with A = 0. Such average quantities can be obtained in an online manner by smoothing the product of activities over several time steps. In the presence of recurrent connectivity, the learning rule for a given connection can be approximated to use only information from “parent” neurons that connect to the target neuron of the tuned connection, as detailed in Eq (18). Although this leads to a decrease in training performance (Fig (8)), the output covariance pattern can still be shaped toward an objective. One can also note that there is no reason for separating afferent and recurrent connections in the biology, hence the calculations in Eq (18) that ignore afferent connections. It remains to further explore how to approximate more efficiently the covariance-based learning rule in a local manner in the network.

Here we have used arbitrary covariances for the definition of input patterns, but they could be made closer to examples observed in spiking data, as was proposed earlier for probabilistic representations of the environment [14]. It is important noting that the observed activity structure in data (i.e. covariances) can not only be related to neuronal representations, but also to computations that can be learned (here classification). Studies of noise correlation, which is akin to the variability of spike counts (i.e. mean firing activity), showed that variability is not always a hindrance for decoding [30, 31]. Our study instead makes active use of activity variability and is in line with recent results about stimulus-dependent correlations observed in data [59]. It thus moves variability into a central position in the quest to understand biological neuronal information processing.

Network dynamics

Here we recapitulate well-known calculations [32] that describe the statistics of the activity in discrete time in a MAR process in Eq (5), which we recall here: (20)

Our focus are the self-consistency equations when the multivariate outputs are driven by the multivariate inputs , whose activity is characterized by the 0-lag covariances P⁰ and 1-lag covariances P¹ = (P⁻¹)^T, where T denotes the matrix transpose. We assume stationary statistics (over the observation period) and require that the recurrent connectivity matrix A has eigenvalues in the unit circle (modulus strictly smaller than 1) to ensure stability. To keep the calculations as simple as possible, we make the additional hypothesis that P^±n = 0 for n ≥ 2, meaning that the memory of only concerns one time lag. Therefore, the following calculations are only approximations of the general case for , which is discussed in the main text about Fig 9. Note that this approximation is reasonable when the lagged covariances Pⁿ decrease exponentially with the time lag n, as is the case when inputs are a MAR process.

Under those conditions, we define and express these matrices in terms of the inputs as a preliminary step. They obey (21)

Because we assume P^±n = 0 for n ≥ 2, we have the following expressions (22)

Using the expression for R, we see that the general expression for the zero-lagged covariance of depends on both zero-lagged and lagged covariances of : (23)

The usual (or simplest) Lyapunov equation [32] in discrete time corresponds to P⁻¹ = P^1T = 0 and the afferent connectivity matrix B being the identity with n = m independent inputs that are each sent to a single output. Likewise, we obtain the lagged covariance for : (24)

Note that the latter equation is not symmetric because of our assumption of ignoring P^±n = 0 for n ≥ 2.

Theory for learning rules

We now look into the gradient descent to reduce the error E^τ, defined for τ ∈ {0, 1}, between the network covariance Q^τ and the desired covariance , which we take here as the matrix distance: (25)

The following calculations assume the tuning of B or A, or both.

Starting with afferent weights, the derivation of their updates ΔB_ik to reduce the error E^τ at each optimization step is based on the usual chain rule, here adapted to the case of covariances: (26) where η_B is the learning rate for the afferent connectivity and the symbol ⊙ defined in Eq (11) corresponds to the sum after the element-wise product of the two matrices. Note that we use distinct indices for B and Q^τ. Once again, this expression implies the sum over all indices (i′, j′) of the covariance matrix Q^τ. The first terms can be seen as an n × n matrix with indices (i₁, i₂): (27)

The second terms in Eq (26) correspond to a tensor with 4 indices, but we now show that it can be obtained from the above consistency equations in a compact manner. Fixing j and k and using Eq (23), the “derivative” of Q⁰ with respect to B can be expressed as (28)

Note that the first term on the right-hand side of Eq (23) does not involve B, so it vanishes. Each of the other terms in Eq (23) involves B twice, so they each give two terms in the above expression —as when deriving a product. The trick lies in seeing that (29) where δ denotes the Kronecker delta. In this way we can rewrite the above expression using the basis n × m matrices U^ik that have 0 everywhere except for element (i, k) that is equal to 1. It follows that the n² tensor element for each (i, k) can be obtained by solving the following equation: (30) which has the form of a discrete Lyapunov equation: (31) with the solution and Σ being the sum of 6 terms involving matrix multiplications. The last step to obtain the desired update for ΔB_ik in Eq (26) is to multiply the two n × n matrices in Eqs (30) and (27) element-by-element and sum over all pairs (i₁, i₂) —or alternatively vectorize the two matrices and calculate the scalar product of the two resulting vectors.

Now turning to the case of the recurrent weights, we use the same general procedure as above. We simply substitute each occurrence of A in the consistency equations by a basis matrix (as we did with U^ik for each occurrence of B), once at a time in the case of matrix products as with the usual derivation. The derivative of Q⁰ in Eq (23) with respect to A gives (32) where V^ij is the basis n × n matrix with 0 everywhere except for (i, j) that is equal to 1. This has the same form as Eq (31) and, once the solution for the discrete Lyapunov equation is calculated for each pair (i, j), the same element-wise matrix multiplication can be made with Eq (27) to obtain the weight update ΔA_ij.

Likewise, we compute from Eq (24) the following expressions to reduce the error related to Q¹: (33) and (34)

These expressions are also discrete Lyapunov equations and can be solved as explained before.

In numerical simulation, the learning rates are fixed to 0.01.