Information theoretic measures.

Information theory has been used recently in a number of approaches in robotics in order (i) to understand how input information is structured by the behavior [7], [19] and (ii) to quantify the nature of information flows inside the brain [8]–[10] and in behaving robots [11], [12]. An interesting information measure is the empowerment, quantifying the amount of Shannon information that an agent can “inject into” its sensor through the environment, affecting future actions and future perceptions. Recently, empowerment has been demonstrated to be a viable objective for the self-determined development of behavior in the pole balancer problem and other agents in continuous domains [20].

Driving exploration by maximizing PI can also be considered as an alternative to the principle of homeokinesis as introduced in [21], [22] that has been applied successfully to a large number of complex robotic systems, see [23]–[26] and the recent book [27]. Moreover, this principle has also been extended to form a basis for a guided self-organization of behavior [27], [28].

Intrinsic motivation.

As mentioned above, the self-determined and self-directed exploration for embodied autonomous agents is closely related to many recent efforts to equip the robot with a motivation system producing internal reward signals for reinforcement learning in pre-specified tasks. Pioneering work has been done by Schmidhuber using the prediction progress as a reward signal in order to make the robot curious for new experiences [29]–[31]. Related ideas have been put forward in the so called play ground experiment [32], [33]. There have been also a few proposals to autonomously form a hierarchy of competencies using the prediction error of skill models [34] or more abstractly to balance skills and challenges [35]. Predictive information can also be used as an intrinsic motivation in reinforcement learning [36] or additional fitness in evolutionary robotics [37].

Embodiment.

The past two decades in robotics have seen the emergence of a new trend of control in robotics which is rooted more deeply in the dynamical systems approach to robotics using continuous sensor and action variables. This approach yields more natural movements of the robots and allows to exploit embodiment effects in an effective way, see [38], [39] for an excellent survey. The approach described in the present paper is tightly coupled to the ideas of exploiting the embodiment, since the development of behavioral modes is entire dependent on the dynamical coupling of the body, brain, and its environment.

Spontaneity.

We would like to briefly discuss the implications of using a self-determined and deterministic mechanism of exploration to the understanding of variability in animal behavior. Self-determined is understood here has “only based its own internal laws”. In the animal kingdom, there is increasing evidence showing that animals from invertebrates to fish, birds, and mammals are equipped with a surprising degree of variety in response to external stimulation [40]–[43]. So far, it is not clear how this behavioral variability is created. Ideas cover the whole range from the quantum effects [44] (pure and inexorable randomness) to thermal fluctuations at the molecular level to the assumption of pure spontaneity [45], rooting the variability in the existence of intrinsic, purely deterministic processes.

This paper shows that a pure spontaneity is enough to produce behavioral variations, and as in animals, their exact source appears “indecipherable” from an observer point of view. If the variation of behavior in animals is produced in a similar way, this would bring new insights into the free will conundrum [13].

Nonstationarity and Time-local Predictive Information (TiPI).

Most applications done so far were striving for the evaluation of the PI in a stationary state of the system. With our robotic applications, this is neither necessary nor adequate. The robot is to develop a variety of behavioral modes ideally in a open-ended fashion, which will certainly not lead to a stationary distribution of sensor values. The PI would change on the timescale of the behavior. How can one obtain in this case the probability distributions of ? The solution we suggest is to introduce a conditioning on an initial state in a moving time window and thus obtain the distributions from our local model as introduced below. More formally, let us consider the following setting. Let be the current instant of time and be the length of a time window steps into the past. We study the process in that window with a fixed starting state so that all distributions in eq. (2) are conditioned on state . For instance, instead of in eq. (2), we have to use(3)and the related expression for , where is the joint probability distribution for the process in the time window, conditioned on . In the Markovian case eq. (3) boils down to . As to notation, the conditional probabilities depend explicitly on time so that is different from in general if , with equality only in the stationary state. As a result we obtain the new quantity, written in a short-hand notation as(4)which we call time-local predictive information (TiPI). Note the difference to the conditional mutual information where an averaging over would take place. Analogously we define the time local entropy as (5)

Error propagation dynamics.

Let us introduce a new variable describing the deviation of the actual dynamics, eq. (6), from the deterministic prediction in a certain time window. We define for a time window starting at time (8)for any time with and . As to notation, denotes a single variable not to be confused with the Dirac function. Intuitively captures how the prediction errors occurred since the start of the time window are propagated up to time . Figure 1 illustrates the transformed state and the relevant distributions of the belonging process . Interestingly the TiPI on the process is equivalent to the one on the original process , see eq. (A5) in Text S1. The dynamics of the can be approximated by linearization as

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. The time window and the error propagation dynamics used for calculating the TiPI, eq. (11).

In principle, the process is considered many times with always the same starting value but different realizations of the noise . Note that, when using the one-shot gradients, only one realization is needed.

https://doi.org/10.1371/journal.pone.0063400.g001

(9)using the Jacobian

(10)Assuming the prediction errors (noise) to be both small and Gaussian we obtain an explicit expression for the TiPI on (11)where is the covariance matrix of and is the covariance matrix of the noise. The derivation and further details are in section A in Text S1. The results for linear systems in [14] can be obtained from the general case considered here by .

When looking at eq. (11) one sees that the entropies are expressed in terms of covariance matrices. This is exact in the case of Gaussian distributions. In the general case this may be considered as an approximation to the true TiPI. Alternatively, we can also consider eq. (11) as the definition of a new objective function for any process if we agree to measure variability not in terms of entropies but more directly in terms of the covariance matrices.

Gradient ascending the TiPI.

Based on the TiPI, eq. (11), a rule for the parameter dynamics is given by the gradient step to be executed at each time (13)where is the update rate and . The term from eq. (11) has been omitted assuming that is essentially noise which is not depending on the parameters of the controller. This is justifiable in the case of parsimonious control as realized by the low-complexity controller networks. These generate typically well predictable (low noise) behaviors as shown in the applications studied below.

In order to get more explicit expressions, let us consider the case of very short time windows. With there is no learning signal since meaning that . So, is the most simple nontrivial case. The parameter dynamics is given by(14)where and the auxiliary vector are given as

(15)(16)(17)(18)stipulating the noise is different from zero (though possibly infinitesimal) and employing the self-averaging property of a stochastic gradient, see below. The general parameter dynamics for arbitrary is derived in section B in Text S1. However, in the applications described below, already the simple parameter dynamics with will be seen to create most complex behaviors of the considered physical robots.

In a nutshell, eq. (13) reveals already the main effect of TiPI maximization: increasing means increasing the norm of (in the -metric see eq. (A21) in Text S1). This is achieved by increasing the amplification of small fluctuations in the sensorimotor dynamics which is equivalent to increasing the instability of the system dynamics, see also the more elaborate discussion in [47].

Learning vs. exploration dynamics.

Usually, updating the parameters of a system according to a given objective is called learning. In that sense, the gradient ascent on the TiPI defines a learning dynamics. However, we would like to avoid this notion here, since actually nothing is learnt. Instead by the interplay between the system and the parameter dynamics, the combined system never reaches a final behavior corresponding to the goal of a learning process. Therefore we prefer the notion exploration dynamics for the dynamics in the parameter space that is driven by the TiPI maximization.

One-shot gradients.

The formulas for the gradient (eqn. (14) and (A21) in Text S1) were obtained by tacitly invoking the self-averaging properties of the gradient, i. e. by simply replacing with in eq. (A16) in Text S1. This still needs a little discussion. Actually, the self-averaging is exactly valid only in the limit of sufficiently small , with eventually being driven to zero in a convenient way. However, our scenario is different. What we are aiming at is the derivation of an intrinsic mechanism for the self-determined and self-directed exploration using the TiPI and related objectives. The essential point is that self-exploration is driven by a deterministic function of the states (sensor values) of the system itself.

Equation (14) obtained from the gradient of the TiPI fulfills these aims very well–any change of the system parameters and hence of the behavior is given in terms of the predecessor states in the short time window. With finite (and often quite large) eqs. (14)–(18) are just a rough approximation of the original TiPI but, in view of our goal, the one-shot nature of the gradient is favorable as it supports the explorative nature of the exploration dynamics generating interesting synergy effects.

Synergy of system and exploration dynamics.

A further central aspect of our approach is the interplay between the system and the parameter dynamics driven by the TiPI maximization process. In specific cases, the latter may show convergence as in conventional approaches based on stationary states. An example is given by the one-parameter system studied in [46] realizing convergence to the so called effective bifurcation point. However, with a richer parametrization and/or more complex systems, instead of convergence, the combined system (state+parameter dynamics) never comes to a steady state due to the intensive interplay between the two dynamical components if is kept finite. An example will be given in the Results section.

Typically, the TiPI landscape permanently changes its shape due to the fact that increasing the TiPI means in general a destabilization of the system dynamics. If the latter is in an attractor, increasing the TiPI destabilizes the attractor until it may disappear altogether with a complete restructuring of the TiPI landscape. This is but one of the possible scenarios where the exploration dynamics engages into an intensive and persistent interplay with the system dynamics. This interplay leads to many synergistic effects between system and exploration dynamics and makes the actual flavor of the method.

Self-directed search.

The common approach to solve the exploration–exploitation dilemma in learning problems is to use some randomization of actions in order to get the necessary exploration and then decrease the randomness to exploit the skills acquired so far. This is prone to the curse of dimensionality if the systems are gaining some complexity. Randomness can also be introduced by using a deterministic policy with a random component in the parameters, as quite successfully applied to evolution strategies and reinforcement learning [48], [49].

Our approach is also to use deterministic policies (given by the function ) but aims at making exploration part of the policy. So, instead of relegating exploration to the obscure activities of a random number generator, variation of actions should be generated by the responses of the system itself. This replaces randomness with spontaneity and is hoped (and will be demonstrated) to restrict the search space automatically to the physically relevant dimensions defined by the embodiment of the system.

Formally, we call a search self-directed if there exists a function so that the change in the parameters(19)is given as a deterministic function of the states in a certain time window (of length ) and the parameter set itself. In this paper, is given by the gradient of the predictive information in the one-shot formulation.

In more general terms, we believe that randomization of actions makes the agent heteronomous, its fate being determined by an obscure (to him) procedure (the pseudo-random number generator) alien to the nature of its dynamics. The agent is autonomous in the ‘genuine’ sense only if it varies its actions exclusively by its own internal laws [50]. In our approach, according to eq. (19), exploration is driven entirely by the dynamics of the system itself so that exploration is coupled in an intimate way to the pattern of behavior the robot is currently in. The danger might be that in this way the exploration is restricted too much. As our experiments show, this is not so for active motion patterns in high dimensional systems. This fact can be attributed to the destabilization effect incurred by the TiPI maximization, see above and [47]. For stabilizing behaviors, however, the exploration may be too restrictive.

Exploration dynamics for neural control systems.

In the present setting, we assume that both the controller and the forward model of our robot are realized by neural networks, the controller being given by a single-layer neural network as(23)the set of parameters now given by and . In the concrete applications to be given below, we specifically use (to be understood as a vector function so that )

Moreover, the forward model is given by a layer of linear neurons, so that(24)

The matrices , and the vector represent the parametrization of the forward model that is adapted on-line by a supervised gradient procedure to minimize the prediction error as(25)

In the applications, the learning rate is large such that the low complexity of the model is compensated by a very fast adaptation process.

In contrast to the forward model parameters, the controller parameters are to be adapted to maximize the TiPI. For that the map (eq. (22)) is required which becomes with Jacobian matrix(26)where is the postsynaptic potential and(27)is the diagonal matrix of the derivatives of the activation functions for each control neuron.

In the applications given below, we are using the short-time window, with the general exploration dynamics given by eq. (14). The explicit exploration dynamics for this neural setting with are given as(28)(29)where all variables are time dependent and are at time , except which is at time . The vector is defined as(30)(see eq. (17)), and the channel specific learning rates are

(31)The derivation and generalization to aribrary activation functions are provided in section C in Text S1. The update rules for are given by a sum of such terms, with appropriate redefinitions of the vector , see eq. (A20) in Text S1.

The Hebbian nature of the update rules.

In order to interpret these rules in more neural terms, we at first note that the last term in eq. (28) is of an anti-Hebbian structure. In fact, it is given by the product of the output value of neuron times the input into the -th synapse of that neuron, the (which are positive, as a rule) being interpreted as a neuron specific learning rate. Moreover, we may also consider the term as a kind of Hebbian since it is again given by a product of values that are present at the ports of the synapse of neuron . The factor can be considered as a signal directly feeding into the input side of the synapse . Moreover, given as is obtained by using as the vector of output errors in the network and propagating this error back to the layer of the motor neurons by means of the standard backpropagation algorithm.

These results make the generalization to more complicated, multi-layer networks straightforward. However already the simple setting produces an overwhelming behavioral variety, see the case studies below.

More intuitively the Hebbian term acts as a self-amplification and increases the Lyapunov exponents. In the linear case [14] this leads eventually to the divergence of the dynamics such that the PI does not exist any longer. With the nonlinearities, the latter effect is avoided, but the system is driven into the saturation region of the motor neurons. However, the second term in eq. (28), by its anti-Hebbian nature, is seen to counteract this tendency. The net effect of both terms is to drive the motor neurons towards a working regime where the reaction of the motors to the changes in sensor values is maximal. This is understandable, given that maximum entropy in the sensor values requires a high sensorial variety that can be achieved by that strategy.

Deterministic self-induced hysteresis oscillation.

Now we show that in the one-dimensional case the parameter dynamics is independent of white noise. This implies we can in the state dynamics make the limit of vanishing noise strength and obtain a fully deterministic system. Again we only consider the two-step window (). Using (eq. (9)) we find that the TiPI, according to eq. (A15) in Text S1is independent of the noise. Analogously to eqs. (28)–(31) we obtain the update rules for and as the gradient ascent on and thus the full state-parameter dynamics (with ) is given by

(34)(35)(36)with .

Apart from the definition of (that just modulates the speed of the parameter dynamics), the extended dynamical system agrees in the one-dimensional case with that derived from the principle of homeokinesis, discussed in detail in [27]. Let us therefore only briefly sketch the most salient features of the dynamics. Keeping fixed at some supercritical value, as above the most important point is that, instead of converging towards a state of maximum TiPI, the dynamics drives the neuron through its hysteresis cycle as shown in Fig. 2, which we call a self-induced hysteresis oscillation, see Fig. 3 (A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. State and parameter dynamics in the one-dimensional system.

(A) Only dynamics (fixed ); the bias oscillates around zero and causes the state to jump between the positive and negative fixed points. The TiPI is seen to increase steadily until it eventually drops back when the state is jumping. (B) With full dynamics (). increases until it oscillates around its average at where the hysteresis cycle starts. Parameters: , , in (B), .

https://doi.org/10.1371/journal.pone.0063400.g003

For the full dynamics (with eq. (35)) the results are given in Fig. 3 (B) showing that the feedback strength in the loop converges indeed toward the regime with the hysteresis oscillation. This demonstrates that the latter is not an artifact present only under the specific parametrization. In fact, we encounter this phenomenon in many applications with complex high-dimensional robotic systems, see the experiments with the Armband below and many examples treated in [27].

Interestingly this behavior is not restricted to simple hysteresis systems but is of more general relevance. For instance, in two-dimensional systems a second order hysteresis was observed, corresponding to a sweep through the frequency space of the self-induced oscillations [27]. It would be interesting to relate this fast synaptic dynamics to the spike-timing-dependent plasticity [51] or other plasticity rules [52] found in the brain.

About time windows.

Before giving the applications to embodied systems, let us have a few remarks on the special nature of the time windowing technique as compared to the common settings. Let us consider again the bistable system with the bias as the only parameter and with finite noise. Figure 4 depicts a typical situation with so that the wells are of different depth. The figure depicts the qualitative difference between the classical attitude of considering information measures in very large time windows, large enough for the process to reach total equilibrium, as compared to our nonstationarity approach where the TiPI is estimated on the basis of a comparatively short window. Note that the time to stay in a well is exponentially increasing with the depth of the well and decreasing exponentially with the strength of the noise [53]. Mean first passage times can readily exceed physical times (on the time scale of the behavior) by orders of magnitudes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. The probability density distributions with different time windows of the stochastic process in an asymmetric double well potential.

The mean first passage time of switching between wells is one characteristic time constant of the process [53], increasing exponentially with the barrier height. If observing the process in a window of length , the distribution of (A) will be observed. In that situation, the TiPI is maximal if the wells are of equal depth (). However, with windows of length , the system state will be predominantly in one of the wells generating the distributions shown in (B), (C). Gradient ascending the TiPI will decrease the well depth as long as the probability mass is still concentrated in that well. This is what drives the hysteresis cycle depicted in Fig. 2.

https://doi.org/10.1371/journal.pone.0063400.g004

While in the former case convergence of the hysteresis parameter towards the equilibrium condition is reached, there is no convergence in the nonstationary case. Instead, one obtains a self-induced hysteresis oscillation. This is generic for a large class of phenomena based on the synergy effects between system and exploration dynamics which open new horizons for the explorative capabilities of the agent. In the context of homeokinesis, this phenomenon has already been investigated in many applications, see [27]. This paper provides a new, information theoretic basis and opens new horizons for applications as the matrix inversions inherent to the homeokinesis approach are avoided.

The Armband.

The Armband considered here is a complicated physical object with degrees of freedom, see Fig. 5. The physics of the robot is simulated realistically in the LpzRobots simulator [54]. The program source code for this and the next simulation is available from [55]. Each joint is controlled by an individual controller, a single neuron driven by TiPI maximization, as with the robot chain treated in [47]. The controller receives the measured joint angle or slider position and the output of the controller defines the target joint angle or target slider position to be realized by the motor. The motors are implemented as simulated servomotors in order to be as close to reality as possible. Moreover, the forces are limited so that, due to the interaction with obstacles or the entanglement of the system’s different degrees of freedom, the true joint angle may differ substantially from the target angle. These deviations drive the interplay between system and exploration dynamics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. The Armband.

The robot has 12 hinge and 6 slider joints, each actuated by a servo motor and equipped with a proprioceptive sensor measuring the joint angle or slider length. The robot is strongly underactuated so that it can not take on a wheel like form where locomotion were trivial.

https://doi.org/10.1371/journal.pone.0063400.g005

In the experiments, we use the controller given by eq. (23) and the update rules for the parameter dynamics as given by eqs. (35) and (36). The adaptive forward model is given by eq. (24) with and the appropriate learning rules eq. (25). In order to demonstrate the constitutive role of the synergy effect, we started by studying the system with fixed and . In contrast to the chain of mobile robots, with fixed parameters there is no parameter regime where the Armband shows substantial locomotion. This result suggests that, as compared to the chain of mobile robots, the specific embodiment of the Armband is more demanding for the emergence of the collective effect.

In order to assess the effects appropriately, note that potential locomotion depends on the forces the motors are able to realize. For instance, if the robot is strongly actuated, the command for each of the motors drives each joint to its center position so that the shape of the robot is nearly circular, locomotion readily taking place under the influence of very weak external influences. In order to avoid such trivial effects, we use an underactuated setting so that gravitational or environmental forces are deforming the robot substantially, see Fig. 5.

The situation changes drastically if the dynamics is included. As demonstrated by Fig. 6, substantial locomotion sets in only if is large enough so that the exploration dynamics is sufficiently fast for the synergy effect to unfold. Also, as the experiments show, the effect is stable for a very wide range of and under varying external conditions. It is also notable, that the Armband robot shows a definite reaction to external influences. For instance, obstacles in its path are either surmounted or cause the robot to invert its velocity, see Fig. 7. The latter effect is observed in particular in the underactuated regime defined above, so that the reflection is not the result of the elastic collision but it is actively controlled by the involvement of the exploration dynamics. The role of the latter is also demonstrated by the fact that locomotion stops as soon as the update rate is put to zero, see Fig. 8 and the corresponding video S1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. Role of the fast synaptic dynamics: depending on the speed of the synaptic dynamics defined by , the locomotion properties are changing drastically.

Depicted is the distance traveled by the robot in 10 min simulated time on an empty plane. The inset gives a close up view for low , demonstrating that the locomotion starts only if exceeds a certain threshold value. Shown is the mean and standard deviation of 10 runs each. Update frequency 25 Hz.

https://doi.org/10.1371/journal.pone.0063400.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 7. Regular locomotion pattern and interaction with the environment.

Plotted are the center positions of the 6 rigid segments in space for an interval of 40 sec. One line is highlighted for visibility. The trajectory starts while the robot is moving to the left (A) and is hitting the wall (B) (black box) and locomotes to the right (C) showing a very regular pattern. Then it overcomes an obstacle (D) and hits the wall (E) and moves back (F). The behavior is cyclic. Parameter: .

https://doi.org/10.1371/journal.pone.0063400.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 8. Armband robot surmounting an obstacle and inverting speed at a wall.

Screen shots from the simulation for Fig. 7. The order is row-wise from left to right. The last two pictures show the situation after switching off the parameter dynamics for a few seconds (the robots stops) and enabling it again (starts moving).

https://doi.org/10.1371/journal.pone.0063400.g008

The Armband has also been investigated recently using artificial evolution for the controller [56], demonstrating convincingly the usefulness of the evolution strategy for obtaining recurrent neural networks that make the Armband roll into a given direction. There are several differences to our approach, both conceptually and in the results. While in the evolution strategy the fitness function was designed for the specific task and many generations were necessary to get the performance, in our approach the rolling modes are emerging right away by themselves. Moreover, the modes are sensitive to the environment, for instance by inverting velocity upon collisions with a wall, they are flexible (changing to a jumping behavior on several occasions) and resilient under widely differing physical conditions. Interestingly, these behaviors are achieved with an extremely simple neural controller, the functionality of a recurrent network being substituted by the fast synaptic dynamics.