2.1 Genuinely multi-objective problems

The desired behaviour of the robotic system may entail multiple requirements—different sub-goals—that may also be conflicting (e.g., they may represent the costs and benefits of applying a certain behavioural strategy). Therefore, a good trade-off must be found. Although the problem to be solved is a genuinely multi-objective optimisation problem, the classical approach in evolutionary robotics consists in the definition of a tailored fitness function that scalarizes, a priori, the different objectives (behavioural terms) to obtain a single-valued function [11], which can be optimised by means of well-known SOO algorithms. However, this scalarization—often a weighted sum—is somewhat arbitrary. The advantage of MOO is that it requires no such choice, and leaves the evolutionary process free to explore different trade-offs between the objectives, allowing the designer to choose a specific trade-off a posteriori on the basis of the analysis of the obtained solutions (see also Section 6 for a discussion about the selection of the best trade-off in relation to the evolutionary robotics domain). Given that the goal is to ultimately find the best trade-off between objectives, these MOO approaches can be considered goal-refiners, because they change the requirements of the task by introducing the Pareto optimality criterion and discarding, or not even defining, any a priori trade-off among objectives.

Most importantly, in evolutionary robotics different trade-offs may correspond to radically different behavioural strategies, which may not be attainable through single-objective evolution. Therefore, MOO leads to a large exploration of the possible solutions to a given robotics problem, and allows generating a wide set of behaviours, all optimising the trade-off between the given objectives. A simple MOO approach would be to optimise different scalarizations for a number of a priori defined weights, using SOO algorithms, and return the best solution found for each, in order to approximate the Pareto front. However, scalarizations based on weighted sums have well-known theoretical limitations [24]: (i) solutions that do not lay in the convex hull of the Pareto front are not optimal for any scalarization, and (ii) an even distribution of weights does not ensure an even distribution of solutions in the Pareto front. These limitations might not be crucial in practice for evolutionary robotics, since the Pareto front is often convex, the obtained solutions are just approximations to the optimal, and designers do not generally seek a perfectly even spread of solutions in the Pareto front but rather a sufficiently varied set of high-quality solutions. Nonetheless, we show later in the paper that, even for the typical scenarios of evolutionary robotics, such a simple MOO approach based on SOO algorithms is still inferior in practice to an MOO approach not relying on scalarizations.

Different research studies in the literature consider genuinely multi-objective problems. A first line of research exploits MOO to optimise at the same time the desired behaviour and some relevant task-specific feature of the controller, e.g., minimise the number of hidden units in a neural network [25, 26], or promote the efficient usage of state variables in a controller evolved trough genetic programming [27]. These studies demonstrate how MOO can be beneficial in identifying a good trade-off between the evolvability of the desired behaviour and the complexity of the controller. The latter should be high enough—to support the evolution of the desired behaviour—but not higher—to reduce the dimensions of the search space and ensure convergence. Teo and Abbass [25] present an extensive comparison between SOO and MOO approaches in which MOO approaches optimise both the behaviour and the controller structure, whereas SOO approaches fix the controller structure a priori and only evolve the behaviour. This comparison is, however, limited to one specific aspect of the advantages of MOO and does not address the problem of defining the correct fitness function to obtain a desired behaviour. Indeed, in the above studies the selective pressure to evolve the desired behaviour is provided by a single objective: if this objective is ill-defined or presents a low evolvability per se, no controller structure can help.

Another line of research exploits MOO for genuinely multi-objective tasks such as the evolution of virtual creatures displaying complex abilities, e.g., object grasping and legged locomotion [28, 29]. A genuinely multi-objective approach was also used to evolve complex morphologies of virtual creatures by introducing a cost for morphology complexity along with the reward for effective locomotion, with the goal of testing how different morphologies are evolved to match different environments [30]. Similarly, the parameters defining the morphology and kinematics of a flapping-wing robot have been optimised with a genuinely MOO approach [31]. The same flapping-wing robot has been evolved to show both target-following and altitude-control behaviours [32]. MOO was exploited to optimise the parameters of the controller of a four-wheeled mobile robot to perform aggressive maneuvers over slippery surfaces. A good trade-off was required between the two objectives representing the speed and accuracy in following the prescribed trajectory [33, 34].

A different line of research exploits MOO to evolve behaviours and robots with desired task-agnostic features. For instance, transferability of an evolved controller from simulation to reality is a much desired property, but it is difficult to obtain due to discrepancies between the simulation and the physical platform (often referred to as the reality-gap problem [3]). Thus, a possible approach consists in adding an objective that explicitly favours transferability [35], thus effectively expanding the design problem. A similar approach has been used for desired properties like morphological diversity [36], robustness through controller reactivity [37], resiliency to hardware failures [38], and generalisation to problem instances never encountered before [39]. The approaches used to obtain these properties are in general task-agnostic (e.g., search for novelty, behavioural diversity, transferability [8, 9, 35]), because the above properties do not depend on the given task but are generally desired across applications. The same approach has been used to evolve behavioural consistency and memory [40], although this particular approach may be considered task-specific because defining the additional objective requires knowledge about the scenarios for which the behaviour must be consistent.

These studies demonstrate that MOO can be successfully applied in an evolutionary robotics context, but they do not provide insights or experimental evidence about the advantages of MOO over SOO approaches. The only available comparison was performed by evolving a robot controller for two conflicting tasks: protecting another robot by remaining close to it and collecting objects scattered in the environment [41]. A genuinely MOO approach was compared against multiple scalarized problems obtained as the weighted average of the performance in the two sub-tasks, and solved using a SOO algorithm. The results did not reveal significant differences in the performance achieved by the two approaches, but noted the higher computational cost of running the SOO algorithm multiple times to produce different trade-offs between the two tasks [41]. In this paper, we provide an extensive comparison that better highlights the advantages of MOO over SOO across multiple tasks. In particular, we present a case study (see Section 3) where the multi-objectivization of an original SO fitness formulation does not provide additional evolvability, yet defining the design problem as a genuinely multi-objective problem disentangles the conflicting aspects of the fitness function. The multi-objective formulation leads to a larger diversity of solutions, which could not be obtained with the original SO formulation, and to a higher probability of discovering the desired behaviour.

In this paper, we limit our experimental studies to task-specific approaches, and we demonstrate how MOO improves the evolutionary search and, at the same time, obtains a larger behavioural diversity and a better exploration of the search space. We argue that these properties are delivered by MOO per se, and we provide an experimental demonstration by conducting a systematic comparison between single- and multi-objective evolution. We refer the interested reader to the studies mentioned above for what concerns approaches making use of task-agnostic techniques.

2.2 Multi-objective approximation by proxies

In evolutionary robotics, a suitable fitness function to optimise is often not available a priori, i.e., the task does not come with a detailed performance metric. Indeed, the real objective is the behaviour of the robotic system, and the fitness function represents just a means to obtain this behaviour. Evolutionary robotics represents a prototypical case in which “it is the solution that has primacy, and the objectives are only a means of orienting the search in order to discover this solution” [18]. In such conditions, the design problem can be faced by introducing multiple objectives that are proxies for the true “unavailable” objective. These proxies can be complementary, each covering some particular aspect of the behaviour to be obtained. “Thus, it should be expected that the desired solution(s) will score relatively highly under all of the ‘proxy’ objectives, and an MOO approach may therefore be suitable” [18].

Differently from genuinely multi-objective approaches, which assist in finding the (unknown) optimal aggregation among multiple objectives, multi-objective approximation by proxies should be counted among the task-specific process helpers [3]. The proxy objectives are actually intermediate goals or sub-goals that only indirectly optimise the overall goal, which is unavailable or insufficiently well defined. An MOO approach is particularly helpful in such conditions because the designer may not know a priori neither the relevance of each intermediate goal, nor how to aggregate them into a single fitness function. All the downsides of an a priori aggregation mentioned above become even more critical here, because no aggregation of the multiple objectives measures the desired behaviour. In practice, determining if an evolutionary robotics study reports a genuinely multi-objective problem or an approximation by proxies may be somehow arbitrary, because it is sometimes unclear whether an objective is a pre-existent requirement or just a proxy.

One of the first applications of MOO in evolutionary robotics concerned the incremental evolution of controllers for an unmanned aerial vehicle (UAV) through multi-objective genetic programming [42]. In this case, the overall goal was too complex to be described by a particular objective (or even to be evolved using a single evolutionary stage), since the UAV was required to locate, reach and track a radar source, hence, multiple ancillary objectives were defined for evolving the behaviour in several stages. A clear example of task-specific proxies introduced to evolve a desired behaviour is given by Moshaiov and Ashram [43] when tackling a single-robot navigation problem (a problem inspired by the same pioneer study we refer later in this paper). In their case, an additional objective with respect to the original fitness formulation was introduced to reward the passage through pre-defined waypoints, in order to favour a looping behaviour. A similar approach was used when comparing multi-objective evolutionary strategies and genetic algorithms, in which a robot was required to display both collision-free navigation and object retrieval abilities [44]. Another relevant study concerns the evolution of a walking controller for a humanoid robot [45, 46]. In that study, several proxies are introduced that reward specific characteristics of the desired gait, still the real objective is efficient locomotion. Task-specific proxies have also been used to evolve efficient navigation employing an artificial potential field [47], in which the proxies rewarded different aspects of the navigation ability, such as goal approaching or obstacle avoidance.

In this paper, we discuss a flocking case study in Section 4 that demonstrates how multi-objective approximation by proxies is a viable method to devise the selective pressures required to obtain a desired behaviour. We also show that it is superior to the a priori scalarization of the same objectives.

2.3 Multi-objectivisation

Even when a single-objective function is available for measuring the performance in the given robotics problem, it may not be suitable for evolutionary optimisation because it may not provide evolvability to the system. Generally speaking, the evolvability issue is related to the search landscape of the optimisation problem, which can be either very rugged, presenting many local optima, or on the contrary largely flat, offering no gradient for the evolutionary search. The latter case gives rise to the bootstrap problem, which is the absence of selective pressures among randomly initialised individuals at the beginning of the evolutionary optimisation [11, 13, 14].

Multi-objectivisation is a technique that allows transforming a difficult SOO problem into a more tractable MOO problem [18] by either decomposing the SOO problem into multiple objectives in order to disentangle the concurrent aspects giving rise to local optima, or by introducing ancillary objectives to kick-start or guide the evolutionary search. Both the decomposition and the ancillary objectives are process helpers, as they are just relevant for guidance, and the final solution(s) may largely ignore them, focusing on the original objective alone. The key distinction between multi-objectivisation and multi-objective approximation by proxies is the existence of an original objective that measures the final design. In the latter case, such objective does not exist or it is not sufficiently well-defined. In practice, this distinction depends on what the goal of the designer is.

Multi-objectivisation has been exploited by adding both task-specific and task-agnostic objectives used for guidance. In [48], an ancillary task-specific objective was added to provide hints on how to solve a complex task. This objective rewarded the usage of a knowledge base of behaviour segments automatically generated on a simpler version of the same task. In recent interactive evolutionary experiments [49, 50], the age of the genotype was exploited to promote the preservation of novel solutions and give them a chance to be evaluated by the interactive users. In these experiments, the feedback from the user was not considered as an additional helper objective and the experiments used a simple scalarization. Other recent studies [9, 51, 52] exploited MOO in conjunction with techniques to maximise behavioural diversity among the evolved solutions, to promote a better exploration of the search space and tackle the bootstrap problem. In these studies, one objective was dedicated to the fitness of the goal-task, while a second objective corresponded to a diversity measure. An extensive comparison of SOO and MOO approaches for encouraging behavioural diversity, varying also the controller structures and the diversity measures, showed that MOO performs best in such cases [9].

Multi-objectivisation has been exploited to tackle the bootstrap problem in evolving controllers for complex incremental tasks, that is, tasks that require the completion of multiple sub-tasks to achieve the main goal [13]. In similar conditions, it is possible to define a MOO problem in which the different objectives correspond to the achievement of the individual sub-tasks. The only limiting factor of this otherwise well grounded work resides in the experimental scenario, since the existence of clearly defined sub-tasks that can be characterised as conflicting objectives explicitly favours MOO over SOO approaches. In this paper, we provide further evidence on the relevance of MOO to tackle the bootstrap problem in a task that intrinsically leads to low evolvability, but that does not prevent SOO to find a solution (see Section 5). In this way, we extend the demonstration of the benefit of MOO over SOO as a general approach for evolutionary design methods.

3.1 Fitness function and multi-objective formulation

The original fitness function is designed to reward fast motion and obstacle avoidance on the basis of the information available to the robot. Therefore, at each control step t, the current angular speed of the wheels and the readings from the IR proximity sensors are collected, and a single-objective performance measure is computed as follows: (1) where T is the total number of control steps. Ω(t) is the reward for moving straight and fast at time t, and is computed on the basis of the current angular speed of the robot wheels: (2) where ω_l and ω_r are the left- and right-wheel angular speeds, and ω_m is the maximum possible speed of the differential-drive motion controller. Here, the first factor rewards fast motion because it is maximised by high absolute values of the angular speed. The second factor rewards straight motion as is maximised by small differences between the left and right angular speeds. Φ(t) is the reward for being away from any obstacle at time t: (3) where ϕ_i(t) is the activation of the i^th IR proximity sensor, and ϕ_m is the maximum possible value. The closer the obstacle, the higher the sensor activation, the smaller the reward. 𝓝_𝓣 is a scalarization that averages over time the product of these two components, therefore requiring that both components are non-null for a large amount of time. The interesting aspect of this function is that both its components are low at the same time while the robot is negotiating an obstacle. Therefore, 𝓝_𝓣 is maximised by controllers that ensure a fast reaction to the perception of an obstacle that leads the robot away from it.

Starting from the above formulation, we define two objectives by decomposing the function 𝓝_𝓣 as follows: (4) (5) The objective 𝓝_Ω encodes information about the wheels speed over time, and corresponds to the evaluation of the robot’s ability to move straight and fast. On the other hand, 𝓝_Φ encodes information about the proximity to obstacles and walls, and is maximised by behaviours that keep the robot away from them. The two objectives introduce an interesting trade-off, that was not considered in the original formulation. Indeed, there may be behaviours that move less but stay maximally away from obstacles, or fast-moving robots that however keep the walls closer. As we will see below, the MOO approach will explore these possible trade-offs producing several qualitatively different behaviours.

Finally, we introduce a single-objective formulation that scalarizes the two objectives introduced above by means of a weighted sum, as follows: (6) with varying γ ∈ [0, 1]. The rationale is that different trade-offs between the two objectives can be explored by varying γ, but we can still use the SOO approach to optimize Eq (6) for each value of γ. Although such an approach has well-known theoretical limitations [24], it could be in practice more efficient in finding alternative solutions to the navigation problem than the purely MOO approach.

3.2 SOO vs. MOO

To compare the SOO and MOO approaches, we run several repetitions of the same evolutionary algorithm, changing only the way in which potential solutions are evaluated and selected for reproduction. In the SOO case, solutions are selected according to the fitness function. In the MOO case, solutions are selected according to the “hypervolume measure”, which represents the “size of the dominated space” [54]. This type of selection is used by state-of-the-art multi-objective evolutionary algorithms [55, 56], but here we just replace the selection step of a standard SOO algorithm. A detailed description of the evolutionary algorithm is available in S1 Text. For each experimental condition, we run twenty evolutionary runs for a fixed number of generations, and we consider the solutions of the last generation for further analysis. All solutions are re-evaluated to obtain an unbiased assessment of their performance using the available performance metrics, i.e., the (scalarized) fitness and the individual objective measures. The comparison between the SOO and MOO approaches is carried out on these measures by looking at the probability that a specific approach dominates the other one. All the details of the comparative approach we employ are available in S1 Text.

3.3 Results

3.3.1 Performance analysis.

Following the methodology described above, we compare the SOO and MOO approaches by looking at the solutions in the objective space given by 𝓝_Ω and 𝓝_Φ. The comparison between the MOO approach and the SOO approach using the traditional fitness formulation 𝓝_𝓣 is given in Fig 2a. Here, we plot the differential ability of the two approaches in attaining portions of the objective space. It is possible to observe that the solutions found by the SOO approach prevail only in the region in which both speed and distance from obstacles are maximised (see the dark areas on the right part of Fig 2a). This region is also attained by the MOO approach, but less frequently. Indeed, the MOO approach explores widely the objective space, and proves best in all the other regions in which either speed or distance from obstacles are maximised. These regions are never attained by the single-objective evolution. By looking at these results, we can conclude that MOO does not provide a competitive advantage over SOO if we are only interested in the original single-objective performance. A very similar discussion can be done for the comparison with the weighted sum 𝓝_γ, with γ = 0.5. Also in this case, the single-objective approach finds and optimises only solutions that maximise at the same time the two objectives, being deficient in the other parts of the objective space (Fig 2b).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Navigation experiment.

Differences between the MOO and the SOO approach with (a) the traditional fitness function 𝓝_𝓣 and (b) the fitness 𝓝_γ, with γ = 0.5. The comparison is carried out by means of empirical attainment functions (EAFs) [57–59], which give the (estimated) probability that a single run of an optimiser attains (dominates or equals) a specific point in the objective space (for details, see S1 Text). The difference between EAFs is reported in the figure, where grey-levels represent the magnitude of the difference: darker colours indicate larger differences. For example, a black point on the left indicates that the difference between the EAF of the left optimiser minus the EAF of the right optimiser is at least 0.8, that is, the left optimiser has attained that point in at least 80% more runs than the right one. The solid lines shown in the plot delimit the points attained, respectively, in at least one run (grand-best attainment surface) and in every run (grand-worst attainment surface) by any of the two optimisers. Any difference between the optimisers is located between these two lines. The dashed lines, which are different on each side, delimit the points attained in 50% of the runs (median attainment surface) of the optimiser shown in that side.

https://doi.org/10.1371/journal.pone.0136406.g002

The situation is different if we vary the weighted sum of the two objectives. We tested different weights by systematically varying γ, as shown in Fig 3. In this case, the MOO approach has a large advantage on most parts of the objective space, generally dominating the weighted sum solutions. For γ = 0.2, mainly obstacle avoidance is rewarded, and single-objective evolution remains trapped in a local optimum in which only 𝓝_Φ is maximised (Fig 3a). Conversely, for γ = 0.6 and γ = 0.8, SOO mainly finds the other local optimum in which only 𝓝_Ω is maximised (Fig 3c and 3d). In all these cases, the SOO approach does not produce good solutions for the navigation problem. The weight γ = 0.4 provides better results, with the SOO approach showing a slight advantage in attaining areas of the objective space close to the central part of the Pareto front (Fig 3b), but this looks less efficient than the case for γ = 0.5 shown in Fig 2b. These results demonstrate that the choice of the right compromise between different objectives is very difficult, and even relatively small variations may lead to radically different results. This aspect is discussed further in the case study presented in Section 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Navigation experiment.

Differences between the EAFs of the multi-objective approach and the single objective approach with weighted fitness 𝓝_γ and varying weight γ between components (see caption of Fig 2 for an explanation of the EAFs difference).

https://doi.org/10.1371/journal.pone.0136406.g003

3.3.2 Behaviour diversity.

As expected, individual single-objective runs do not explore different trade-offs. With 𝓝_𝓣, the selective pressure is focused on the region of the objective space where 𝓝_Ω ≈ 𝓝_Φ. In these conditions, the evolved solutions produce behaviours that present at the same time high values of Ω(t) and Φ(t). This happens when a robot moves as fast as possible and as far as possible from obstacles. When a corner must be negotiated, the evolved strategy makes the robot quickly rotate on the spot. In particular, we observed that the avoidance action is performed by turning always in the same direction, notwithstanding the maze requiring a left or a right turn. This leads to a fast and efficient collision-avoidance action, because the robot does not need to determine whether the turn is on the left or on the right. However, despite optimal with respect to the single-objective function, this strategy trades the avoidance speed with the exploration abilities in the maze. In fact, the robot remains trapped in some parts of the circuit because it is unable to correctly negotiate both left and right turns. Evolving the neural network controllers with 𝓝_𝓣 produces only behaviours of this kind, which we will refer to as quick-avoidance behaviours. An example of this behavioural strategy is provided by S1 Video.

Multiple scalarizations with 𝓝_γ provide an approximated Pareto set, but are not able to cover the whole front [24]. On the contrary, the MOO approach explores the objective space widely and provides a better approximation of the Pareto set, therefore producing behaviours of different kind that optimise different trade-offs of 𝓝_Ω and 𝓝_Φ (see also Fig 4a). By observing the behaviour corresponding to the different solutions found, we can identify mainly four categories:

1. the solutions that maximise 𝓝_Ω only, which correspond to controllers that make the robot move fast, but do not present any obstacle avoidance ability;
2. the solutions that maximise 𝓝_Φ only, which correspond to controllers that keep the robot away from obstacles, but do not present any navigation ability;
3. the solutions that mainly maximise 𝓝_Ω, possibly at the cost of a lower 𝓝_Φ, which correspond to controllers producing the quick-avoidance behaviour described above;
4. the solutions that mainly maximise 𝓝_Φ at the cost of a lower 𝓝_Ω, which correspond to robots that move following the closest wall, either on the left or on the right. This wall-following behaviour leads to a very good exploration of the maze, which can be performed at the cost of a reduced speed in order to keep a constant distance from the wall (see also S1 Video).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Navigation experiment.

(a) The exploration ability of the evolved controllers with the MOO and the SOO approaches. Each dot represents a solution in the objective space, and the darkness of the circle corresponds to the exploration performance, measured as the fraction of the maze that was effectively covered by the robot. (Left) Solutions obtained with MOO. (Centre) Solutions obtained with SOO using 𝓝_𝓣. (Right) Solutions obtained with SOO using 𝓝_γ for every γ ∈ {0.2,0.4,0.5,0.6,0.8}. (b) Empirical tail distribution of exploration performance, that is, probability that the exploration performance is larger than a given performance threshold τ.

https://doi.org/10.1371/journal.pone.0136406.g004

For the engineering of good navigation in a maze, the MOO approach has the advantage of providing a varied set of possible solutions in each evolutionary run. Some of the evolved behaviours are not useful for navigation purposes, such as those belonging to the categories (i) and (ii) described above. However, the wall-following behaviour clearly outperforms the quick-avoidance behaviour in terms of exploration of the maze, and represents the natural choice that would be made a posteriori by a designer interested in good maze-navigation strategies. This gives MOO an advantage over SOO approaches that can be quantified. To this purpose, we compute an exploration performance value that measures the exploration abilities of the best evolved solutions by looking at the percentage of the maze that is visited by the robot in a sufficient amount of time to cover the full length of the circuit. Fig 4a shows the exploration performance for all the solutions obtained with MOO and SOO. Note that we consider all the solution obtained with 𝓝_γ as a unique experimental condition, in the attempt to approximate the Pareto set by systematically varying γ. The best controllers have been chosen in the MOO case from the approximated Pareto set of the different evolutionary runs, while in the SOO case they where selected as the solutions with the highest average performance. This corresponds to the choice one would make a priori when selecting or discarding solutions from the last population, as described in S1 Text. Fig 4a shows that all the best solutions of the SOO approach with 𝓝_𝓣 present poor exploration, as they all belong to the quick-avoidance category. Similarly for the case of SOO with 𝓝_γ, even though a few solutions with good exploration can be observed. Instead, in the MOO case, a large fraction of solutions provide a good exploration, suggesting that the wall-avoidance behaviour was often discovered.

Given that the MOO approach produces much more solutions than the SOO approach, we can compare their efficiency by looking at the probability of obtaining a solution presenting a good exploration behaviour. We estimate this probability by computing the fraction of evolved solutions that present an exploration performance higher than a threshold τ, at varying τ. We plot this probability as an empirical tail distribution (i.e., the complementary of the cumulative distribution function) in Fig 4b. We note that 𝓝_𝓣 produces only solutions that visit less than 20% of the maze, which corresponds to poor exploration abilities. For larger τ, the probability of obtaining good navigation behaviours is higher for the MOO approach than for 𝓝_γ, which only occasionally produces the wall-following behaviour. Thus, we conclude that MOO delivers better results for what concerns the diversity of behaviours evolved, even when compared against a scalarization approach exploring various values of γ. In this particular case, better exploration also corresponds to the discovery of desired solutions that could not be easily obtained with the SOO approach.

4.1 Motion and cohesion objectives and fitness function

In order to reward motion and cohesion in a group, the simplest way is to rely on the absolute positions of the robots, which are available in our simulation environment. At the beginning of a trial, robots are placed within a circle of 1 m radius, choosing their position and orientation at random. To reward group motion, it is sufficient to look at the displacement of the centre of mass of the group during a fixed-duration trial: (7) where is the position of the centre of mass at time t, D_m(t) is the maximum distance a single robot can travel in t seconds, and T is the length of a trial. Cohesion instead is maximised when the average distance of the robots from the centre of mass of the group is minimised: (8) where X_i(t) is the position of robot i at time t, N is the total number of robots, and d_m is the maximum tolerated distance. Cohesion is computed at the end of the trial, assuming that the group must remain aggregated for the whole duration of the trial (and possibly beyond it).

Also in this case, we can obtain a single-objective formulation from the two above objectives through a scalarization: (9) with γ ∈ [0, 1]. To properly select the weight γ, we would need to know a priori the relative importance of the two components of the fitness. Since this relative importance is generally unknown, the standard approach is therefore to systematically vary γ. Consistently with the previous case study, we provide here the results for γ ∈ {0.2,0.4,0.5,0.6,0.8}.

4.2 Results

We again followed the methodology described in Section 3.2 and detailed in S1 Text. The results of the comparison between the different approaches is presented in Fig 6. Generally speaking, the MOO approach outperforms the SOO approach with 𝓕_γ, presenting a higher probability of attaining all regions of the objective space (Fig 6a). If we look at the comparison with specific values of γ as shown in Fig 6b to 6f, only for γ ≥ 0.6 the differences fall below 60% in favour of the MOO approach for most of the objective space (see Fig 6e). By contrast, the choice of γ = 0.5 (the natural choice one would make without any a priori knowledge) results in poor performance, with most of the objective space attained by the MOO approach with at least 60% more probability than the SOO approach. The reason is that low values of γ bias the search towards solutions presenting high cohesion and no motion. These solutions behave as local optima from which it is difficult to escape, even when more importance is given to the motion objective 𝓕_m with higher values of γ. Indeed, solutions providing only cohesion and no motion are obtained also with γ = 0.8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Flocking experiment.

Differences between the EAFs of the MOO approach and the SOO approach with weighted fitness 𝓕_γ. (a) Comparison with the combined results of all scalarizations. (b-f) Comparison for specific values of the weight γ between components (see caption of Fig 2 for an explanation of the EAFs difference).

https://doi.org/10.1371/journal.pone.0136406.g006

The performance argument alone gives already a point in favour of the MOO approach, as it proves capable of attaining more frequently all areas of the objective space. However, not all possible trade-offs correspond to good coordinated motion behaviours. From an engineering perspective, those solutions providing low motion or low cohesion should not be retained: the former correspond to packed groups that do not move, the latter correspond to groups that split by moving in multiple directions. It is therefore useful to look at the probability of producing acceptable solutions by using the SOO and MOO approaches. To this purpose, we examine the attainment surfaces of both approaches as they give an absolute indication about the algorithm ability to produce solutions with a given trade-off between the objectives (Fig 7). Here, we compare the MOO approach with the SOO approach aggregating all values of γ. While the best attainment surfaces are comparable, the probability of producing solutions that maximise both objectives at the same time is lower for the single-objective approach. With the MOO approach, even the worst attainment surface contains solutions that feature reasonably good cohesion and motion. This indicates that the MOO approach can produce good solutions in every run, thanks to a wide exploration of the objective space, which is prevented by a single-objective approach. Additionally, also in this case it is possible to observe different kinds of behaviours evolved with the MOO approach, similarly to what was shown in Section 3 (refer also to [64]). Some of the evolved behaviours are available as supplementary material in S2 Video.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Flocking experiment.

Attainment surfaces of (a) the MOO approach and (b) the SOO approach for all values of γ. The k%-attainment surfaces shown in the figure allow visualising the EAF over the objective space [59]. As explained earlier, the EAF indicates the (estimated) probability of dominating a point in the objective space in a single run of an algorithm. A k%-attainment surface denotes the Pareto front of the points that have been attained by (dominated by or equal to) at least k% of the runs, that is, the points that have a value of at least k/100 of the EAF. Hence, the worst attainment surface indicates the Pareto front of the points that have been attained in 100% of the runs. Conversely, the best attainment surface indicates the Pareto front of the points that have been attained by at least one run. More details on the EAFs are available as supplementary material in S1 Text.

https://doi.org/10.1371/journal.pone.0136406.g007

5.1 Main and ancillary objectives in strictly collaborative tasks

As mentioned above, the task we have defined requires that N = 6 robots collaborate to switch off M = 18 beacons, randomly scattered in a circular arena (radius: 6 m). The beacons are automatically switched off when at least n robots are within a radius r_v = 25 cm from the beacon. The intrinsic performance metric for this task corresponds to the fraction of switched-off beacons at the end of a trial: (10) (11) where B₀(t) represents the set of beacons b whose status s_b(t) is off at time t, and T is the length of a trial. The state of a beacon is always on (s_b = 1) unless at least n robots are close enough at the same time: (12) where A′ is a subset of cardinality n of the set of robots A, and d_i,b(t) = ∥X_i(t) − X_b(t)∥ is the euclidean distance between robot i and beacon b. Similarly to [23], 𝓢_c,n represents the collaboration rate within a trial of length T. Behaviours in [23] were designed by hand. Here, we use 𝓢_c,n as the fitness function to evolve behaviours in the SOO approach, and we test it with two collaboration levels (i.e., n = 2 and n = 3) to explore different task complexities.

Single-objective evolution with 𝓢_c,n is affected by the bootstrap problem, as will be shown in Section 5.2. To overcome this problem, we resort to multi-objectivisation by adding an ancillary objective that is somehow related to the task. To this purpose, we note that collaborations can be obtained if robots are capable of visiting multiple beacons during a trial. Indeed, if a robot stops at the first encountered beacon and never leaves, it is very likely that the group will end in a deadlock condition in which all robots are waiting at different beacons and no collaboration takes place. Therefore, a robot must be capable of visiting multiple beacons during a trial. We compute this exploration performance as follows: (13) (14) where V_i(t) is the set of beacons visited by robot i at time t. A beacon b is considered visited by robot i if the minimum distance achieved from the beacon is less than r_v. By visiting multiple beacons, the robots maximise their exploration abilities. However, a robot needs to remain close to a beacon for some time in order to establish a collaboration with its teammates. Therefore, a tradeoff is present between the maximisation of the exploration ability and the waiting time for collaboration. The MOO approach is most suited to explore this tradeoff and eventually provide solutions capable of maximising 𝓢_c,n, which remains our main goal.

5.2 Results

As mentioned above, we study two different task complexities by setting the collaboration level to n = 2 and n = 3. We run single-objective evolution using 𝓢_c,n as fitness function. For each evolutionary run, we first observe the trend of the fitness of the best individual in the population during the evolutionary optimisation, as shown in Fig 9. We notice that the bootstrap problem only mildly affects evolution with n = 2. Several evolutionary runs do show a flat fitness surface for many generations, which indicates that the whole population scored a null fitness. However, by random drift, some solutions with non-null fitness appeared and optimisation rapidly started. This indicates that the bootstrap problem exists, but is not too severe. On the contrary, when n = 3 the situation is worse, as can be seen in the bottom part of Fig 9. Most of the evolutionary runs suffered of the bootstrap problem, and present a very flat fitness surface. Only a minority of runs were able to discover a suitable strategy that was optimised through the generations. This also indicates that there exist possible solutions to the proposed problem, but they are not easily obtained by a SOO approach.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Strictly Collaborative Task.

Fitness of the best individual across generation for all evolutionary runs, for n = 2 (top) and n = 3 (bottom). Note that the maximum fitness achievable with n = 3 is lower than with n = 2, due to the increased complexity of the task.

https://doi.org/10.1371/journal.pone.0136406.g009

Once confirmed the existence of the bootstrap problem, in both a mild and severe version, it is worth comparing the performance of the MOO approach with respect to the SOO one. We are interested in studying whether multi-objective evolution is capable of producing solutions as efficient as those evolved in the single-objective case (especially for n = 2) and whether the bootstrap problem can be bypassed by systematically evolving good solutions in every evolutionary run (especially for n = 3). The results of the comparison for n = 2 are shown in Fig 10a and 10b. By looking at the difference between the EAFs of the MOO and SOO approaches (Fig 10a), we can observe that the second objective does not prevent to evolve good quality solutions. Indeed, both approaches similarly attain the regions of the objective space where 𝓢_c,2 is maximised. Additionally, the MOO approach also finds solutions that provide good exploration abilities to the robots by maximising 𝓢_v, which cannot be achieved by the SOO approach. This also confirms the existence of a tradeoff between 𝓢_c,n and 𝓢_v and therefore supports the usage of a MOO approach instead of a scalarization of the two objectives, which may result in poor solutions, as we have discussed in the previous case studies. The similarity in performance between the MOO and SOO approaches is also confirmed by looking at Fig 10b, which shows the performance of the best evolved controllers for each run computed over 500 different trials. We can therefore conclude that the mild bootstrap problem observed when n = 2 can be efficiently overcome by both approaches, and no substantial difference in performance can be observed between the two with respect to 𝓢_c,2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Strictly Collaborative Task.

Left: Differences between the EAFs of the MOO and the SOO approaches (see caption of Fig 2 for an explanation of the EAFs difference). Right: performance of the best individuals from the different evolutionary runs with respect to 𝓢_c,n, ordered by decreasing median and mean performance.

https://doi.org/10.1371/journal.pone.0136406.g010

The situation is completely different when the bootstrap problem becomes severe, as is the case for n = 3. The comparison between MOO and SOO shown in Fig 10c and 10d demonstrates that the ancillary objective 𝓢_v is definitely helpful. Indeed, the EAF differences shown in Fig 10c demonstrate a strong advantage of the MOO approach in every area of the objective space. Similarly, the performance comparison among the best evolved individuals shown in Fig 10d indicates that suitable solutions are obtained in every evolutionary run with MOO, which means that the bootstrap problem is bypassed thanks to the introduction of the second objective. Indeed, 𝓢_v allows guiding the evolutionary optimisation when there is no fitness gradient, because exploration is a beneficial ability to promote collaborations. In this case, random drift alone is not sufficient to produce good solutions, at least not systematically in every run. We can conclude that MOO is a more reliable approach to produce satisfactory behaviours in every evolutionary run, thanks to the ability to bypass the bootstrap problem.