1.2.1 Theoretical results.

1. We formally show that different implementations of Chapman’s strategy [5], one of the main heuristic strategies for the simplified two-dimensional ball catching problem, all form a unified mathematical framework. They only differ with respect to the type of derivative of the angular representation used as a control signal (Section 4.2).
2. We prove that Chapman’s strategy generates an agent reference trajectory that, if attained by a controller, guarantees interception for both ideal flight trajectories and systematically perturbed trajectories. This includes perturbations that resemble air resistance (Section 4.2).
3. We further show that Chapman’s strategy does not require computation of the angle’s acceleration (the second derivative) but only of its velocity. This insight allows us to develop two robust variants of angle-based controllers: COV-IO and COV-OAC (Section 4.3).
4. We show that the angular representation employed by Chapman’s strategy results in a stationary control problem and can be implemented with bang-bang control (Section 4.3).
5. Our analysis shows that this simplified control scheme comes at the cost of intractable dynamics: none of the angular representations used for control is Markov (Section 4.3.8), and thus no closed-form description of the dynamics based on the angular representation exists.
6. In contrast, we show that a Cartesian representation of agent and ball results in simpler dynamics (linear for ideal parabolic flight, locally linear for air resistance). This comes at the expense of non-stationary PD-control required to catch accurately (Section 4.1).

1.2.2 Empirical results.

1. An evaluation of a wide variety of systematic and random perturbations shows that Chapman’s strategy generalizes over systematic perturbations of the ball trajectory, such as air resistance, whereas strategies based on the Cartesian representation generalize better over perturbations induced by Gaussian noise, both in two and three dimensions (Section 4.4).
2. When formulating ball catching in two dimensions as a model-free reinforcement learning problem, searching for a policy that directly maps from camera sensor input to control output, a generic black-box optimization with regularization learns a policy equivalent to Chapman’s strategy (Sec. 5).

2.2.1 Cartesian representation.

The agent and ball positions and velocities correspond to the Cartesian representation of the ball catching scenario. It will form the basis of the Cartesian control strategies, the strategies brought forward by the generalist approach. These try to estimate from the agent’s sensors and assume a dynamic model of ball and agent that closely matches the real dynamics given in Eqs (1), (2) and (3) to be available to the agent. In practice, it is not possible for the agent to get a perfect estimate of all quantities. In particular, the agent’s distance to the ball is difficult to estimate using monocular vision [13]. We will see later how these approaches perform with imperfect estimates or when their dynamic model does not match the real dynamics.

2.5.1 Angular representation.

We refer to the pair θ, β as the angular representation, possibly replacing θ, β by higher derivatives, for example . In the two-dimensional case, the angular representation only consists of θ (or its derivatives). It will form the basis of the angular control strategies presented later.

2.7.1 Chapman’s strategy.

[5] presented an important finding that triggered a whole research field dedicated to angular control for projectile interception. In his paper, he presented what we call Chapman’s strategy: he was able to show that an agent arrives at the ball’s impact point exactly at impact time if the agent runs such that the tangent of the angle θ between ground plane and the ball rises at a constant rate. Moreover, he showed that in this case the agent’s velocity is constant. Fig 2 provides a visual explanation of this insight, and we prove it formally in Section 4.2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. An intuitive explanation of Chapman’s strategy.

(Adapted from [7]). It states that for an ideal parabolic ball trajectory and an agent moving towards the impact point with constant velocity the tangent of the vertical viewing angle θ(t) = tan α(t) rises with a constant rate. This is illustrated by the blue line segments whose length is linear both in θ since s_t = (R + D₀)θ(t) and in time since s_nt = ns_t; hence, θ is linear, too. We will show this formally in Section 4.2.

https://doi.org/10.1371/journal.pone.0197803.g002

To implement Chapman’s strategy, we need to address two issues. First, we need to find a control law that ensures that the agent runs such that θ rises linearly. Several controller implementations have been suggested, one of the most popular ones being optical acceleration cancellation (OAC, [17]). It implements Chapman’s strategy by tracking the acceleration of the tangent of the vertical viewing angle and adjusting the agent’s motion such that remains zero. We review OAC and other implementations of Chapman’s strategy in Section 4.2.

The second issue is that we need is how to control the lateral orientation of the agent. Chapman suggests to adjust the lateral movement such that the bearing angle remains constant, a strategy known as constant bearing angle (CBA).

The obvious advantage of Chapman’s strategy is that it does not require an estimate of the impact point. However, the optimal-control based Cartesian controllers come with strong guarantees that reliably predict their behavior, whereas little theoretical knowledge about Chapman’s strategy is available. Hence, a main focus of our work is to provide a theoretical understanding of how and why Chapman’s strategy works. We additionally verify our findings in a large array of simulation experiments.

2.7.2 Linear Optical Trajectory.

Chapman’s strategy is not the only angular control strategy that has been suggested. The most important competitor is the linear optical trajectory (LOT) strategy [7]. Similarly to Chapman’s strategy, LOT uses the viewing and bearing angles to adjust the agent’s running trajectory. The main difference to Chapman’s strategy is that LOT does not command the vertical and the bearing angle independently, but adjusts the agent’s running speed such that the ball follows a linear trajectory from its viewpoint, which is equivalent to keeping constant. This can result in a different running path, and it has been argued that it is more biologically plausible as it does not treat θ and β independently.

Although various studies for ball but also frisbee catching [18–20] support the LOT model, it is still debated whether LOT or Chapman’s strategy (or optimal control) account better for explaining human behavior [16, 21]. We do not attempt to answer whether one strategy resembles human behavior better than another. Instead, we aim to understand the relationship of generalist and specialist strategies for ball catching. We therefore focus on Chapman’s strategy (OAC with CBA) and use it as a representative of the specialist strategies for ball catching in the remainder of this paper.

3.2.1 Cartesian control strategies.

As explained in Section 2.6, Cartesian control strategies are based on a prediction of the ball’s landing point, or more generally on the prediction of the ball trajectory. A large body of work in psychology advocates that prediction plays an important role for cognition [28] and particularly for motor control [28–30]. Experimental studies showing that prediction plays an important role in ball catching have been presented, too [31, 32]. In ball catching, a predictive strategy has the great advantage that it can effectively cope with cases where the ball goes out of the field of view [16]. In general, the main advantage of viewing cognition as predictive modeling is that it provides a generally applicable computational framework for explaining human behavior. Moreover, it can often be directly transferred to artificial systems. Hence, the trajectory prediction strategy is the method of choice in most robotic applications for ball catching [33, 34]. These applications, however, only study catching in closed-room environments and thus do not address difficulties faced in scenarios such as baseball: due to the long flight time the various aerodynamic forces such as drag or Magnus forces affect the ball trajectory significantly [12]. Therefore, it is not clear if the predictive approach to ball catching can be successfully applied in such scenarios.

3.2.2 Angular control strategies.

The view of equating cognition with predictive modeling is not undisputed. As mentioned above, [10] advocates that humans rely on simple heuristics and considers Chapman’s strategy as an instance of this heuristic approach. The simplicity of Chapman’s strategy has triggered a large body of work in psychology and engineering to study whether humans use this strategy and how it could be implemented [6, 35–37]. Most of them study the setting presented in Section 2, but extensions that incorporate human constraints have been suggested, too [8].

Several studies investigate implementations of Chapman’s strategy and their performance under different conditions, mostly in physical simulation experiments. One of the earliest studies [38] notices the effect of air resistance on the ball trajectory, which Chapman neglected. However, the author wrongly concludes that Chapman’s strategy is not applicable in the case of air resistance [39] and does not study the influence of additional perturbations. [40] suggests different control architectures for implementing Chapman’s strategy and tests them in two- and three-dimensional simulations. He reports that Chapman’s strategy is more robust when implemented with bang-bang, rather than proportional control, especially when modeling human constraints such as sensorimotor delays, velocity and acceleration constraints. Except for air resistance, he does not consider perturbations on the ball trajectory, though. [41] suggests a different variant of Chapman’s strategy which controls the velocity rather than the acceleration of the tangent, the control of optical velocity (COV) strategy. He evaluates it in a set of simulated experiments with random noise and wind perturbations, with the main goal of showing its superiority to LOT. We will derive a mathematical relationship between COV and OAC in Section 4.3 and propose a method that combines the strengths of both approaches. [42] implement OAC on a simulated mobile robot with differential drive, focusing on deriving the full (image) Jacobians required to map image errors to wheel motor commands. They only evaluate their system in two exemplary experiments. [43] study the performance of a bang-bang OAC model for a variety of different initial configurations of the agent and various perturbations, but they neglect air resistance.

Finally, simplified variants of Chapman’s strategy have been implemented in real-world robotic experiments. [44] test different variants of Chapman’s strategies and suggest to preferably use bang-bang, following a similar line of reasoning as [40]. [45] propose a system for ground ball interception but use an angle prediction strategy (AP) rather than OAC: the error is computed as the difference of the current and a linear prediction of the tangent of the angle in the next time step. We will discuss this strategy in Section 4.3 and show that it is equivalent to the COV strategy. Finally, [39] implement the OAC strategy on a mobile robot and study the behavior when catching a balloon. They also report that active servoing of the camera resulted in more stable behavior than using a passive camera.

4.1.1 State, observation and dynamics.

The formalization of the ball catching problem (Sections 2.1 and 2.2) directly provides the state and dynamics required to instantiate the LQG framework. LQG requires the state representation x to be Markov: x_t must fully define the system’s state at time t such that adding more information about the past of x_t cannot improve the prediction of any future state. A suitable state representation fulfilling this criterion is the Cartesian representation presented in Section 2.2. Section 2.1 also presented the dynamics model governing ball and agent motion based on this state. The dynamics are linear in the ideal case and locally linear when assuming drag. Moreover, LQG allows us to relax the assumption that we can perfectly estimate all quantities of the Cartesian representation. Instead we assume the agent’s observation of these quantities to be perturbed by Gaussian noise.

We thus define the state representation x and the observation z as (11) (12) To formalize the state dynamics for LQG, we use the ball dynamics stated in Section 2.1 and bring them into the linear form required by LQG. Similarly, we need to define a linear observation model mapping states to observations. All details on how to adapt the state dynamics and observation model for the LQG framework are provided in the supplementary materials (S2 Text).

Remark: in our formalization, the Cartesian representation defines agent and ball position with respect to a global reference frame, whereas the angular representation of the ball is defined relative to the agent. However, this is only a matter of convenience because separating agent a and ball b makes the equations more comprehensible. We could easily define the ball in a relative reference frame , omit the agent from the state and obtain equivalent dynamics.

4.1.2 Cost function.

Section 2.3 formalized the ball catching task as the minimization of the distance between agent and ball at impact time while respecting the agent’s motion constraints (Eqs 4–6): (13) (14) (15) where denotes the terminal distance cost. The main objective is a finite-horizon term and thus suited for LQG, but LQG cannot directly incorporate the hard agent motion constraints. Instead, LQG requires a single cost function in quadratic form. To address these requirements, we define the cost for LQG as a sum of and an additional regularization term that penalizes control effort, (16) (17) where the hyperparameters w_dist and w_ctrl balance the influence of the two cost terms.

Note that omitting would result in a controller that tries to move the agent directly to the impact point within a single time-step—a motion that is impossible in the general case given the agent’s motion constraints.

4.1.3 Computing time-to-impact.

Finally, LQG requires a way to estimate the current time horizon. In the ball catching scenario, this corresponds to the time-to-impact T. For the ideal case, we can compute it directly solving the ball dynamics equation (Section 2.1) for b_y(t) = 0. We thus obtain, (18) and solve for T, assuming the current ball position b_y(t) and velocity to be known.

In the presence of drag, no closed-form solution of the dynamics is known. In this case, we compute T by running a forward simulation of the ball trajectory using the current estimate of x and the (drag) dynamics model.

4.1.4 Controller analysis.

We are interested in the performance of LQG both in the ideal case and in the presence of perturbations. We consider two different types of perturbations.

Random perturbations: It is by design that LQG is robust to Gaussian noise both in the dynamics as well as the observation [47]. This suggests that LQG is robust to any form of Gaussian perturbations.

Systematic perturbations: To get an intuition for how LQG is affected by systematic perturbations, we look at how it is influenced by drag. Since we know that the drag induces different ball dynamics than the ideal case we inspect the optimal control gains obtained when assuming a model either with or without drag.

We begin by stating the insights that hold irrespective of whether we assume drag or not. First, we observe that LQG computes separate, identical gains for u_x and for u_z; this makes sense for the ball catching problem as the two coordinates are decoupled [30]. Second, we see that, for both models, the gains can be decomposed into a constant, proportional and derivative term which vary over time k_p(t), k_d(t) and k_c(t). Therefore, we see that LQG learns a (biased) PD-controller of the following form (19) where we have omitted the time indices to improve readability (u_z can be defined in an analogous way).

We can now analyze how these gains differ depending on whether the model assumes drag or not. Fig 3 shows the gains in relation to the time to impact. We see that k_p and k_d are the dominating terms for LQR, and that k_p has a (reciprocal) quadratic and k_d a linear dependence on t. This reflects the quadratic form of the idealized ball trajectory equation. When the model incorporates drag (Fig 3B), the influence of the constant term k_c increases with the time to impact. This correlates with the fact that with increased flight time the influence of drag increases, too. Due to the non-linear nature of the drag force no linear control term can account for it, and different drag parameters (mass, radius, drag coefficient) thus affect this term and the behavior of the controller.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Analysis of the control gains computed for the ideal case and for drag.

(A) LQR without drag. (B) iLQR with drag.

https://doi.org/10.1371/journal.pone.0197803.g003

We thus hypothesize that wrong model assumptions induces a systematic error in LQG and degrade the performance of LQG for unmodeled systematic perturbations.

4.3.4 Constant optical velocity with Initial Observation (COV-IO).

The first implementation exploits Lemma 4.2.5 which state that the tangent reference can be estimated robustly when observing at the beginning of the ball trajectory. This overcomes the issue of COV and AP of assuming a fixed a value for . We term this implementation COV-IO, for Constant Optical Velocity with Initial Observation of .

We further increase the robustness of COV-IO by filtering both the estimate as well as the current reference velocity (detailed in the supplementary material S2 Text).

4.3.5 Combining COV and OAC—COV-OAC.

The second implementation exploits Lemma 4.3.3, which establishes a relation between COV and OAC. It allows us leverage the fact that OAC does not require explicit computation of and combine it the COV’s advantage of not having to compute a second derivative of the tangent velocity . The reason we want to avoid the second derivative is because it amplifies random noise, in particular at high control rates.

We thus suggest COV-OAC, which estimates using the delayed tangent velocity : (47) If , the controller reduces to a standard (unnormalized) OAC controller, otherwise if we obtain COV-IO. In practice, we choose a value that approximates OAC close enough while still compensating high-frequent random fluctuations. To increase robustness to noise even further, we use moving average filters for estimating and , similar to COV-IO.

We hypothesize that COV-IO and COV-OAC result in the most robust implementations of Chapman’s strategy, in particular when random noise is present. To validate this hypothesis, we will compare our two novel implementations of Chapman’s strategy to COV(/AP), OAC and the Cartesian controllers in simulated experiments in Section 4.4.

4.3.6 Remark on the biological plausibility of Chapman’s strategy.

We would like to remark that the existence of COV-IO and COV-OAC counters an argument brought forward by [50] regarding the biological plausibility of Chapman’s strategy. The authors conclude that humans do not use Chapman’s strategy because they lack the ability to visually perceive accelerations accurately enough for control. The fact that COV-IO and COV-OAC only rely on the computation of (averaged) velocities clearly disproves this argument.

4.3.7 Angular control in three dimensions: Constant bearing angle.

So far, we have been concerned with the two-dimensional scenario. As explained in Section 2.7.1, the bearing angle can be used to extend any implementation of Chapman’s strategy to three dimensions. The idea is to combine the control output of Chapman’s strategy, which we denote by u_Chapman with a second control output u_CBA which maintains a constant bearing angle β between the agent and the ball. The two control outputs are then applied to perpendicular axes, formally (48) where v = [v_x, v_z]^T = [a_x − b_x, a_z − b_z]^T is the vector pointing from the agent to the ball’s shadow on the ground plane, v_⊥ = [v_z, −v_x] is the orthogonal vector on the ground plane (see Fig 1A) and (49) for some constant gain k_CBA.

Implementation: We apply two techniques to make CBA robust to random noise. First, we compute by finite differencing at some low frequency : (50) Second, we apply a moving-average filter similar to OAC-COV (Section 4.3.5). In total, this introduces three hyperparameters, the time delay Δ_β as well as the averaging window sizes h_β and .

4.3.8 Chapman and the Markov property.

We conclude the theoretical analysis of the angular control strategies by discussing their relationship to the Markov property. As discussed before, the Markov property is an important prerequisite for using optimal control and reinforcement learning. For example, the theoretical guarantees for LQG do not hold if we cannot provide a Markov state representation. Therefore, it is interesting to study whether the angular representation used by Chapman’s strategy (depending on the implementation, θ, or ) fulfills the Markov property. If this was the case, we might consider applying LQG or a similar variant to the angular representation.

However, it turns out that neither θ, nor are Markov. The full proofs are lengthy and can be found in the supplementary material (S1 Text). As an example, we provide an intuitive explanation for why θ is not Markov. The idea is to show that, even if the agent does not move and the environment is deterministic, we cannot predict the future θ(t′) from a single observation θ(t) for any t′ > t. Such a case can easily be found: we just look at different initial conditions ν₀ ≠ ν₁, φ₀ ≠ φ₁ that result in trajectories θ₁(t), θ₂(t) which cross at some point θ₁(t₁) = θ₂(t₂) but have different slopes . Similar type of reasoning can be carried out to show that and are not Markov, either.

We will discuss the implications of this finding in Section 7.

4.4.1 Experimental set-up.

We now present the experimental set-up, including a description of the physical simulation environment, the set of tested perturbations and the controller implementations.

All code has been made publicly available at https://github.com/shoefer/ball_catching.

Simulation: In our simulation we choose the ball parameters to standard baseball rules with mass m_b = 0.15 kg and radius r = 0.0366 m. When simulating drag, we assume experimentally established parameters for baseballs [51]: air density , frontal area A = πr² and drag coefficient c_w = 0.5. To model the agent, we use the approximate values of olympic sprinters who can run with a maximal acceleration of roughly and reach a maximum velocity of [52, 53]. The simulation runs at a frame rate of 60 Hz (time constant ).

Initial Conditions: For each strategy and noise scenario we test different ball-agent configurations. In all configurations, the agent is in principle able to catch the ball, given its velocity and acceleration constraints, but we include extreme cases where the agent must operate at its limits to be successful.

In two dimensions, we use the configurations suggested by [43]. The initial ball launching angle is always set to , and we vary (i) the initial ball velocity ν = {20, 24, 28, 32, 36, 40} and (ii) the distance of the agent to the ball’s impact point D₀ ∈ {−15, −10, −5, 0, 5, 10, 15} (D₀ = a₀ − R, see Fig 1B).

In three dimensions, we additionally vary the initial position of agent in the direction orthogonal to the ball trajectory (z-axis in Fig 1B), parametrized by the angle ψ. We report the averaged results over .

In the cases where random noise is applied to the ball trajectory, we must ensure that the initial conditions regarding the agent’s position hold. We therefore run the simulation twice in order to determine the actual ball impact point: we first record the ball trajectory including the actual noise, and then position the agent at the right distance to the actual impact point and replay the ball trajectory recorded in the first run.

Measuring Performance Criteria: In order to evaluate the performance of a controller we use the terminal distance cost defined in Section 2.3.

Perturbations We distinguish two types of perturbations.

Perturbations of the ball trajectory: These include (i) drag, (ii) wind gust (wind over short period of time) and (iii) turbulence (spin, Magnus force, and so on).

Perturbations of the agent’s sensory and motor capabilities: These include (iv) sensor noise, (v) motor noise, (vi) sensorimotor delay and (vii) reduced control rate. Table 1 summarizes all perturbations and their implementations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Overview of implemented perturbations.

https://doi.org/10.1371/journal.pone.0197803.t001

To keep the number of experimental conditions at reasonable size, we vary the perturbations to (iii) turbulence, (iv) sensor noise and (v) motor noise jointly, that is always use the same standard deviation σ for them. We therefore refer to these three perturbations as Gaussian perturbations.

The main set of experiments studies the influence of individual perturbations (listed in Table 1). Additionally, we test a worst case scenario with Gaussian noise σ = 0.1, wind and 400 ms delay (we run this scenario both with and without drag and reduced control rate).

In order to obtain statistically significant results, we run every controller five times for every experiment.

4.4.2 Controller implementations.

We test the angular controllers COV, OAC, COV-IO and COV-OAC and the Cartesian controllers (i)LQG (Section 4.1) and MPC [16]. For the controllers that have external parameters, we conducted preliminary experiments and determined the set of parameters that performed good both at ideal and perturbed cases. Additionally, we make the model parameters used to generate the Cartesian controllers explicit by writing (i)LQG_{no drag} for a model that assumes an ideal ball trajectory, (i)LQG_{drag:baseball} for a model that assumes drag parameters of a baseball, and so on.

Furthermore, we need to take special care for evaluating the MPC controller. First, it uses a slightly more complex agent representation. To guarantee a fair comparison to the other methods, we tested a variety of different parameters and selected the ones that yielded the best average performance (detailed in the supplementary material S2 Text). Second, the implementation provided by the authors is computationally expensive. To be able to collect sufficient amounts experimental data, we only simulate the strategy at a frame rate of 10 Hz (as in the original paper). We therefore mark the strategy with a * in all comparisons where it is evaluated with a different frame rate than the other strategies.

Table 2 summarizes the controllers and the parameters used.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Controller implementations.

For angular controllers only parameters for the two-dimensional case are shown, see text for parameters in three dimensions.

https://doi.org/10.1371/journal.pone.0197803.t002

Controller Implementation in 3D: The Cartesian controllers (i)LQG and MPC can be readily applied to both the two- and the three-dimensional case. To extend the angular strategies, we combine them with constant bearing angle (CBA, Section 4.3), using parameters Δ_β = 0.15 s and .

4.4.3 Results.

We now present the main results of our experiments. We study the results as to whether they confirm our two main hypotheses: that both angular and Cartesian controllers are optimal in the ideal case, but that different perturbations affect them in different ways.

Since the results for the two- and three-dimensional cases are largely equivalent, most of the plots depicted in the following only show results from the two-dimensional case. For the reader’s convenience, the results in three dimensions are provided in the supplementary material (S1 Fig).

Both Angular and Cartesian Control Solve the Ideal Case: We first study the performance of the controller implementations in the ideal case, without perturbations. Fig 8 shows the terminal distance averaged over all initial conditions. We see that (i)LQG, MPC, OAC, COV-IO and COV-OAC perform optimally, achieving close to zero terminal distance. This result confirms our theoretical analysis: (i)LQG and MPC guarantee optimal performance since their assumptions are fulfilled. COV-OAC is optimal as it is a direct implementation of Chapman’s strategy (Section 4.3). The good performance of COV-IO, which sets according to the initial observation, has been predicted by Lemma 4.2.5. We also see that for COV-IO the agent attains a value close to the consistent agent velocity (Fig 9), as proved in Lemma 4.2.3. Only COV performs inferior because it uses a fixed value for —as predicted by Corollary 4.2.4. We therefore exclude COV from the following experiments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Comparison of ball catching strategies in 2D for ideal case.

https://doi.org/10.1371/journal.pone.0197803.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. COV-IO: Agent velocity and tangent of vertical viewing angle velocity, initial conditions .

https://doi.org/10.1371/journal.pone.0197803.g009

We conclude that all sensible implementations of the angular and Cartesian controllers succeed in the ideal scenario.

No Free Lunch for Angular and Cartesian Control: Next, we inspect the influence of perturbations on controller performance. Figs 10 and 11 depict the average performance in the two-dimensional case when applying perturbations individually.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Comparison of ball catching strategies in 2D: Sensitivity to different individual perturbations.

https://doi.org/10.1371/journal.pone.0197803.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Comparison of ball catching strategies in 2D: Sensitivity to Gaussian perturbations.

https://doi.org/10.1371/journal.pone.0197803.g011

First, we observe that most of the angular and Cartesian controllers are largely unaffected by a reduced control rate, wind, and sensorimotor delays (Fig 10). Only delays of 400 ms pose a problem for the controllers, but solely because the delayed observation in combination with the agent’s motion constraints render extreme initial configurations unsolvable. We will thus focus on Gaussian perturbations (turbulence, sensor and motor noise) and drag in the following.

Angular controllers are impaired by high Gaussian perturbations: Fig 11 shows that OAC performs significantly worse for low amounts of Gaussian noise, whereas COV-IO and COV-OAC perform optimally for the low and medium setting. The reason is that COV-IO and COV-OAC operate on the first derivative of θ and use moving average filtering, whereas OAC operates on the second derivative, which amplifies noise much stronger. However, the performance of COV-IO and COV-OAC degrades for high Gaussian noise, whereas (i)LQG only degrades a bit and MPC is barely affected.

Cartesian controllers are impaired by wrong model assumptions regarding drag: The plots in Fig 12 show that the performance of all Cartesian controllers degrades significantly when the underlying model assumes wrong drag parameters. Fig 13 illustrates the behavior of the controllers with wrong drag assumptions in three dimensions. If the Cartesian controller assumes drag but we do not simulate it, the controller systematically overestimates the range R of the trajectory (Fig 13, top row). In the opposite case, the Cartesian controllers underestimate the range (Fig 13, bottom row).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Comparison of ball catching strategies in 2D for drag, and performance of (i)LQG with wrong drag model in ideal case.

https://doi.org/10.1371/journal.pone.0197803.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Agent trajectories (bird’s eye view) for different strategies in the three-dimensional case with and without drag.

Initial conditions are set to . Top row: with drag. Bottom row: without drag.

https://doi.org/10.1371/journal.pone.0197803.g013

Finally, Fig 14 shows how the controllers are affected by combinations of perturbations. We see that in the worst case scenario where all perturbations are applied with the highest value, both types of controllers perform similarly bad. Figs 15 and 16 show that the same holds true for the control effort. This confirms our result that there is no free lunch for any type of controller: angular controllers are impaired by high Gaussian noise, Cartesian controllers by wrong drag parameters, yet both perform equally bad when averaged over the various scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Sensitivity to combinations of perturbations (2D).

https://doi.org/10.1371/journal.pone.0197803.g014

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Control effort in ideal case (2D).

https://doi.org/10.1371/journal.pone.0197803.g015

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. Control effort affected by combinations of perturbations (2D).

https://doi.org/10.1371/journal.pone.0197803.g016

5.1.1 Biases for learning.

In our experiment, we rely on a set of assumptions and learning biases which we summarize in Table 3. The reason why this list might seem long at first sight is because most of these biases are commonly left implicit in applications of optimization and machine learning. For example, linear functions are very common as they can be learned efficiently and are effective at solving many tasks while still being easy interpret. Their interpretability enables us to show that the agent indeed learns an angular controller.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Biases used in the reinforcement learning experiment for ball catching.

https://doi.org/10.1371/journal.pone.0197803.t003

Inevitably, these assumptions impose a bias on the policy and thus exclude certain types of solutions. In particular, the assumption of a linear policy implies that it is stationary, which excludes any type of Cartesian controller (Section 4.1.4). Although we might be able to learn a Cartesian controller by removing assumptions (for example using a recurrent neural network) it is not the scope of this experiment to show that reinforcement learning is more likely to find an angular or Cartesian controller. Instead, we aim to show that we can state an optimization problem with reasonable, task-general assumptions, and that the solution to this problem is a heuristic angular controller. This shows that angular controllers can be considered as optimal with respect to these assumptions.

We turn to an overview of the learning experiment. For every component of our scenario, we will refer back to the list of biases presented in Table 3 and detail how each bias is implemented. To guide the reader, we will mark each reference to a bias with an underscore.

5.1.2 Camera model.

In order to perform learning directly on sensor data we equip the agent with a simulated camera sensor, illustrated in Fig 17. The camera provides a full 180°view, with the ball occupying a certain set of pixels in the camera image depending on its position in the environment. We term this image the raw observation and denote it by with dimensionality N. The 180° view ensures that the full observability bias is fulfilled. This bias is helpful in two regards: it is based on the same position-controlled agent model used in the previous section and thus allows us to transfer our previous insights for analyzing the learned policies. Further, fully observable problems are much easier to solve using learning than partially observable problems.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 17. Exemplary raw observations generated by the simulated camera model.

The agent a observes the ball b_t at different times t and receives a 1D image with a field of view of 180°.

https://doi.org/10.1371/journal.pone.0197803.g017

All implementational details about the camera model are provided in the supplementary material (S3 Text).

Input Augmentation: The successful controllers presented in the previous section operate both on an input signal as well as on higher derivatives of this signal—for example, the angular controllers operate on θ, or , and the Cartesian controllers on and . To enable the learner to come up with a successful control policy, too, we augment the raw observation such that the policy is able to compute derivatives, if necessary. The controller is thus provided with these augmented observations as input. In order to distinguish them from the raw observations , we denote the augmented observations by o.

We augment the observation in two ways. First, we combine the current image with a temporal derivative of the image , computed using image differencing . The notation for referring to this type of augmentation is , where ⊕ denotes the concatenation of two vectors. Second, we add a second delayed camera image to the observation: either the image at initial time t_init or the image that is delayed by t_delay.

Enriching the input in this way implements the augmented input bias. Note that we also evaluate the performance operating directly on the non-augmented observations. The complete list of tested augmentations is provided in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Observation augmentations.

https://doi.org/10.1371/journal.pone.0197803.t004

5.1.3 Reinforcement learning on observations.

The task of the agent is to learn a controller π(o) = u, called policy in the reinforcement learning setting, which maps every observation o to a control signal u. (Here the control signal u is scalar since we only consider the two-dimensional scenario.) We now explain how we define the policy π and use reinforcement learning to find π.

Linear Policy: In our experiments, we exploit the linear P-control policy bias by decomposing π such that it mimics the structure of a P-controller. This allows the learner to compute the control input and the control gain separately: (51) where denotes the scalar gain factor and the pixel-specific weight vector; N corresponds to the size of the (augmented) observation o.

We train one policy per augmented observation type and evaluate which one perform best (according to their terminal distance at impact time).

Policy Search Using CMA-ES: We apply model-free, policy-based reinforcement learning, using the stochastic optimization method Covariance Matrix Adaption—Evolutionary Strategy (CMA-ES) [58, 59]. Evolutionary methods such as CMA-ES optimize the policy π by maintaining a set of candidate policies, and by iteratively sampling new policies based on their performance with respect to a given cost until converging to a local optimum of this cost.

To apply CMA-ES to the ball catching problem we could, in principle, set to the terminal distance cost , which we already used in Chapter 4 to assess the performance of angular and Cartesian controllers. However, applying this cost directly results in slow or bad performance, as explained in the following. Therefore, we use the regularized worst-case terminal distance cost (52) which consists of a modified version of the terminal distance cost and an additional regularization term , weighted by hyperparameter λ. These two terms will be explained in the following. They address practical learning issues of applying CMA-ES to the ball catching problem and incorporate the biases curriculum learning and vision-related regularization.

Optimizing for Multiple Initial Conditions: Our ultimate goal is to find a policy π that works well for a wide range of initial conditions. However, only evaluates the performance of π for a single initial condition. We therefore evaluate every candidate policy considered by CMA-ES on a set of initial conditions , and compute the worst-case terminal distance cost for this set: (53) By considering only the initial condition where the policy performs worst, we avoid that the optimization procedure settles on a local optimum where the policy performs very well on some, but very bad on other initial conditions.

Curriculum Learning with CMA-ES: Optimizing for the worst-case cost is costly and thus increases the runtime of CMA-ES if the set of initial conditions is very large. To resolve this problem, we use curriculum learning to adapt the set of initial conditions in every iteration of CMA-ES: we sort all initial conditions by difficulty in ascending order. We then define sets of increasing size as follows: only contains the simplest initial condition and we add more difficult conditions one-by-one to every subsequent set . To use these sets, we apply CMA-ES on , starting with i = 0, and proceed to once the cost drops below a pre-defined threshold (in our experiments ).

Vision-Related Regularized Training Cost: Although is better-suited for learning than it still results in poor performance because we do not exploit any knowledge about the fact that we learn a policy that is applied to a camera image o. An important property of camera sensors is the spatial arrangement of pixels: it results in any visible object being projected on a set of adjacent rather than disconnected pixels in o. Exploiting this knowledge as a vision-related regularization bias for learning is very common in computer vision [57], and we do so too by defining the spatial continuity regularization term (54) where {0, …, N − 1} denote the indices of the image o and w_i denotes the policy’s weight associated with the i-th image pixel. We can write this formula more concisely if we define Var as the (uncorrected) sample variance and w′ as the spatial derivative over vector w: (55)

5.1.4 Angular baseline policies.

To analyze whether the previously outlined reinforcement learner indeed finds an angular, “heuristic” controller, we will compare the weights of the learned policies to the weights of two baseline policies.

The baseline policies mimic the COV-IO and COV-OAC controllers (Section 4.3), and use the same camera model and the same two-step linear controller as the learned policies. This makes it easy to directly compare the policies by inspecting their weights v and w.

We now illustrate how to compute the baseline policies, using the example of COV-IO. We know from Section 4.3 that COV-IO is implemented by zeroing the control error using a bang-bang controller. Therefore, the COV-IO baseline policy must be able to compute the quantities and from the augmented observation and subtract them from each other. Therefore, we compute a linear mapping such that , for some suitable augmented observation o. The COV-IO control output can then be computed by applying bang-bang control on Ω(o).

We show in the supplementary material (S3 Text) how to choose o and how to compute Ω to implement COV-IO and COV-OAC directly on observations. Importantly, we will compare the mapping Ω (in particular the pixel-specific parameter vector w_Ω) with the learned policy parameters to decide whether they resemble the COV-IO and COV-OAC strategies.

To avoid confusion with the controllers implemented in the previous section, operating directly on θ, and higher derivatives, we refer to the two baseline policies operating on observations as COV-IO^o and COV-OAC^o.

5.2.1 Simulation and scenario.

Simulation: We run the dynamic simulation with a time constant of , using the same simulation as in the previous experiments described in Section 4.4.1.

Camera Sensor: For the camera model, we use resolution ρ = 0.27 which results in a size of N = 18 pixels for the raw observation . We test different input augmentations by varying the type of observation (observation only, derivative only or both) and by adding a second delayed camera image to the observation, which we call time-embedding (no delay, observation at t_init or at t − t_delay). We test different variants, such as adding the image at initial time t_init or the image that is delayed by t_delay. We use the following parameters: , and .

Note that depending on type of input augmentation, the dimensionality of o varies between N = 18, N = 36 and N = 64.

This results in the nine different augmentations summarized in Table 4, which we all test in our experiments.

Performance and Testing Procedure: We test the success of each strategy in the same way as in the experiments presented in Chapter 4: using the terminal-distance cost evaluated on the full set of initial conditions.

Initial Conditions and Perturbations: To train the agent we choose the initial conditions to be a reduced set of the ones used in the previous experiments (see Section 4.4.1). Reducing the set has two purposes: first, it accelerates learning, second, it tests whether the learner overfits or is able to generalize to a range of unseen conditions.

The reduced set of initial conditions uses the same lauching angle as the full set, but only varies the initial ball velocity by ν = {20, 30, 40} and the initial distance of the agent to the ball’s impact point D₀ ∈ {−15, −7.5, 0, 7.5, 15} m. Additionally, we vary whether drag forces apply to the ball or not. In total, this results in 30 unique combinations of initial conditions.

5.2.2 Reinforcement learning.

We now briefly detail the settings of the reinforcement learner.

CMA-ES Hyperparameters: For CMA-ES, we use a candidate parameter set of size 10, and set the maximum number of CMA-ES iterations to 10000. We set the initial variance for v to and for w to . Note that setting the value of very high allows the agent to learn either a proportional or an approximation of a bang-bang controller.

As convergence criteria, we allow a tolerance of 0.1 with respect to variance in the cost and of 0.1 in the input parameters. Moreover, we vary the spatial regularization parameter λ ∈ {0.01, 0.1}.

Note that we also tested adding L1- and L2-regularization to but this did not result in improved performance.

Curriculum: To construct the curriculum of initial conditions we sort the set of (reduced) initial conditions according to initial distance D₀, initial ball velocity ν and the presence of drag (no/yes), resulting in a sequence of 30 sets of initial conditions of increasing difficulty.

Every time CMA-ES converges for one set of initial conditions from the curriculum, we proceed to the next set and multiply the variance parameters σ_v and σ_w by 1.2. In preliminary experiments this gave the better performance as it avoided premature convergence, that is convergence before the end of the curriculum is reached.

We also compare to optimization without a curriculum, that is setting to all initial conditions; however, this consistently leads to significantly worse performance and we thus omit the analysis of these results in the following section.

For each combination of input augmentation, regularization parameter setting and initial condition selection (curriculum or not), we run CMA-ES five times with different random seeds and report the best result for every run.

5.3.1 Learned policies are equivalent to COV-IO and COV-OAC.

The quantitative analysis shows that it is possible to learn a successful ball catching policy directly on observations, using generic reinforcement learning. We now turn to a qualitative analysis, by studying what type of policy has been learned, and whether one of the policies implements one of the angular controllers presented in the previous section. To answer this question, we will study the control outputs and the parameters learned by the two best policies and compare them to the parameters of the baselines policies.

The two best policies resulted from applying CMA-ES on the augmented observations and , we thus focus on these ones. In the following, we state our main results and provide all details in the supplementary material (S3 Text).

First, we observe that the CMA-ES policies compute a bang-bang control policy: for both the value of v is very high (|v|>10¹¹). This facilitates our comparison because we implemented the baseline policies as bang-bang controllers, too. Next, we compare the policies with respect to the control output they compute. We use the data used to train the baseline policies, discretize the output of both controllers y ∈ {−1, 0, 1} and use a zero-one loss (56) to compare the outputs ( denotes the indicator function which evaluates to 1 if the expression inside the brackets is true and to 0 otherwise). We obtain a similarity of 0.89 for COV-IO^o vs. and 0.77 for COV-OAC^o vs. . This shows the outputs are similar, in particular for COV-IO^o, although not exactly the same.

Finally, we compare the pixel-specific weight vectors w. Figs 19 and 20 show a comparison of the learned policy weights (blue solid curves) to the weights of the baseline policies (green dashed curves). At first sight, they do not look similar at all. We hypothesize that this results from the fact that not all entries in w are relevant for control. This can be the case if the ball always occupies the same area in the camera image, for example because it never passes above the agent’s head. In this case, the ball always appears in the right part of and .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 19. Analysis of policy parameters, with CMA-ES applied to .

denotes the correlation coefficient, m and b the slope and intercept of the linear fit.

https://doi.org/10.1371/journal.pone.0197803.g019

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 20. Analysis of policy parameters, with CMA-ES applied to .

https://doi.org/10.1371/journal.pone.0197803.g020

To investigate this hypothesis, we apply the learned policies on the set of all initial conditions and compute a histogram over each pixel in o. This histogram is shown in the left bottom halves of Figs 19 and 20. It confirms our hypothesis that the ball never passes the agent’s head because the left parts are completely empty. This is a result of the fact that the ball is always thrown from the same side and that the agent moves in such a way that it never lets the ball pass its head. This is an indication for the stable catching behavior—but also for some form of “overfitting” to the learning scenario.

Since the ball always occupies a certain part of the image, all weights outside this part of the image are irrelevant for performance. We thus compare the learned weights to the baseline weights only with respect to the relevant part of the image. The result is shown in Figs 19 and 20, upper right half. We see that, after accounting for the offset induced by focusing on a limited set of weights, the weights align almost perfectly for COV-IO^o, and approximately for COV-OAC^o. This means that, CMA-ES applied to has learned a controller that is highly similar to COV-IO^o, and a controller similar to COV-OAC^o when applied to .

5.3.2 Conclusion.

Our analysis shows that CMA-ES learns policies that are highly similar to the COV-IO and COV-OAC angular controllers presented in Section 4.3. The learned policies merely differ in the fact that they do not generalize to the setting when the ball is thrown from the left. However, we see no principle reason why the learned policies should not generalize to this slightly more complex version if we train on more initial conditions. We thus conclude that model-free reinforcement learning is able to find angular control policies. This implies that the angular control strategy is an optimal solution to the two-dimensional ball catching problem without high-frequency Gaussian perturbations.