2.1 Participants

18 participants were recruited locally in Oxford, United Kingdom. Participants were healthy volunteers with full sight and full stereo hearing. The choice to test a fully-sighted participant group had a twofold motivation: firstly, the test group was able to provide a visual control condition in our experiments by performing the navigation task with access to visual information in each virtual environment. Secondly, the group was seen as a necessary first step given that this is a proof-of-principle study exploring novel sonification methods for 3D spatial representation and navigation within a new experimental paradigm.

The mean age of participants was 28.78 ± 8.00 with a male/female distribution of 11/7. All participants had normal or corrected-to-normal vision. Due to the physical limitations imposed by the testing equipment, it was necessary that participants with corrected vision were able to wear contact lenses for the testing rather than glasses. Participants were rated on the amount of experience they had had with the developed SSD prior to the testing and their experience with first-person-controller computer games and virtual-reality devices (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Participant demographics.

Naivety rated on 1-5 scale of experience: 1 (> 9 hours), 2 (> 6 hours), 3 (> 3 hours), 4 (> 0 hours), 5 (= 0 hours). Virtual-reality/gaming frequency rated 1-5: 1 (no experience), 2 (rarely), 3 (several times a year), 4 (monthly), 5 (regularly).

https://doi.org/10.1371/journal.pone.0199389.t001

On the day of testing, participants were verbally instructed on the types of environments that they would be required to navigate (Section 2.2), the two sonification methods to be tested (Section 2.3), the equipment to be used (Section 2.4) as well as the experimental protocol (Section 2.5). Testing was conducted over two 1.5 hour sessions per participant. A brief re-instruction was conducted at the start of the second session. Verbal feedback was collected during and after each session. In addition to this, a voluntary and anonymous follow-up survey was sent to participants after their completion of both sessions.

Ethics This work received ethical clearance through the University of Oxford Central University Research Ethics Committee. All individuals involved in the study provided written informed consent to publish these case details.

2.2 Virtual reality environments

Participants were tasked with spatially navigating to randomised end points in VR environments, using each of the developed visual-to-audio mappings (sonification conditions) or using visual information (visual-only baseline condition). Two types of VR environments were constructed: a maze and an obstacle corridor. Our motivation for using virtual environments to test our methods was two-fold:

1. Virtual environments offer the ability to build randomised environments and obstacles on the fly which is an advantage over real-world testing.
2. Using a virtual environment bypasses the problem of detecting the 3D structure of the environment, allowing us to focus on the methods of conveying such spatial information to our participants.

Maze. The maze environments were generated within a 5 × 7 grid of virtual cubes, each cube 3 × 3 × 3m in size making a real-world sized arena of 15 × 21 × 3m. Each maze was constructed such that there existed only one constant-length (7 cubes) path to a goal which was a golden star (Fig 1A and 1C). For each maze trial, the path was randomly selected from a pool of 20 pre-generated maze paths saved on the Tango tablet (see Section 2.4). Upon selection, the participant was placed at the starting point of the selected virtual maze.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Virtual reality visualisation and implementation of simulated echolocation and distance-dependent hum volume modulation (humming).

(A) Example of a randomly-generated maze with a single path to goal of 7 cube steps. Orange glow indicates humming boundary in the humming sonification. Golden star indicates goal. Red cross + black dot indicate starting position. (B) Example of a corridor with randomly positioned obstacles. Orange glows and red dotted lines indicate humming boundaries in the humming sonification. Golden star indicates goal. Red cross + black dot indicate starting position. (C) Virtual reality visualisation of the maze. (D) Implementation of simulated echolocation. A pool of 900 particles is projected along a 30 × 30 grid array of rays from the (x, y, z) position of the Tango. (E) Virtual reality visualisation of the obstacle corridor.

https://doi.org/10.1371/journal.pone.0199389.g001

Obstacle corridor. The maze environment simplifies the task of navigation to making a series of right-angled turns. To mimic a scenario closer to the real-world challenge of detecting and avoiding randomly-placed obstacles, based on the work of [78], we constructed obstacle corridor environments. These virtual environments were laid out in a 6m-wide corridor bounded by a left and right wall with the length of the corridor segmented as follows: an initial 3m of empty space, a 7m segment of obstacles, a second 3m of empty space, and finally the goal, again a golden star. For each trial, the obstacle segment was populated by 5 randomly-positioned columnar objects of 0.8m diameter and 1m height. Additionally, for each trial, the star was placed at 13m from the starting line at a random point along the corridor’s 6m width (Fig 1B and 1E).

The placement of obstacles was randomised such that there were always multiple possible paths through the obstacles to the goal, and these paths were sufficiently diverse and also non-trivial (for example, participants could not simply navigate around the edges of the obstacle segment where they would encounter no obstacles). As with the mazes, for each trial, the obstacle corridor arrangement was randomly selected from a pool of 20 pre-generated arrangements saved on the Tango. Upon selection, the corridor was dynamically constructed and the participant placed at the starting point in virtual space.

2.3 Stereosonic vision

Our aim was to convert the 3D structure of the virtual visual environment into a stereo soundscape providing spatial information to our participants for the task of navigation. We call this stereosonic vision. This section presents the two visual-to-audio mappings, or sonification methods, we implemented and explored in the VR environments: the first, a simulated echolocation and the second, a distance-dependent hum volume modulation of hums attached to objects. Resulting sounds were presented on stereo headphones worn by the participant.

2.3.1 Simulated echolocation.

There is a large body of work supporting echolocation as a useful mobility technique for blind people [55, 57, 79]. Using self-generated sounds such as mouth clicks, echolocators use sonic reflections to infer the spatial properties of their surroundings. Learning to navigate using echolocation in the real world, however, is difficult and takes much training [55, 56, 58]. In this work, we simulated echolocation in the two VR environments. This alleviated the real-world difficulties of echolocation since we were able to slow down the speed of echolocation feedback, and also select the types of echo sounds. Our motivation was to make simulated echolocation easier to interpret and acoustically more pleasant than real world echolocation. The following describes our VR implementation of simulated echolocation, which from here on we will refer to as simply ‘echolocation’.

Much like echolocation in the real world, we implemented simulated echolocation in the virtual world such that it was dependent on the user’s body and head direction. The Tango emitted a virtual “chirp” in the form of a sharp click (audible via the user’s headphones) at a rate of 0.57Hz, or every 1.75 seconds. With each click, a pool of 900 virtual particles was projected radially outwards from the (x, y, z) position of the Tango (where x and y were the lateral coordinates of the Tango in space, and the z coordinate was its height above the ground). Each of the 900 particles was projected along its own radial line, following a 30 × 30 grid arrangement (Fig 1D). The radial projection of the particles had a field of view (FOV) of 90° vertically and horizontally (Fig 2C and 2D) with the rays spaced 3° apart in the horizontal and vertical direction, respectively. As the Tango was moved through space, the pool of particles was recast from the Tango’s updated (x, y, z) position. Since the Tango was head-mounted on the participant (see Section 2.4 and Fig 2A), a head/body rotation caused a corresponding Tango rotation. In this way, the virtual particle projection was always outwards and forwards relative to the participant’s head direction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Hardware set-up with the Project Tango and the Durovis Dive 7 head mount.

(A) Participant wearing the Tango mounted in the Durovis Dive 7 headset with a pair of stereo headphones. (B) Rear-facing camera hardware of Project Tango tablet. (C & D) 90° field of view vertically and horizontally from the position of the Tango.

https://doi.org/10.1371/journal.pone.0199389.g002

Following a click, each particle within the pool was projected along its ray but only generated a sound, again audible via the user’s headphones, if it ‘bounced’ off a virtual object in the VR environment. This was intended to be analogous to the echoes of sounds that would be reflected off real-world objects in traditional echolocation. In terms of implementation in our sound engine, this simply meant that a particle was ‘silent’ when travelling along its ray trajectory and emitted its sound when a virtual object in the virtual world obstructed that trajectory. This was thresholded to virtual objects within a 3m radius of the Tango’s (x, y, z) position. The sound of a particle bouncing off an object was sonified as a “pop”, intended to be harmonious and easy on the ear. The pairing of clicks and pops aimed to convey three pieces of information about the 3D spatial features of the environment: firstly, the presence of an object in a particular radial direction, secondly, an estimate of the distance to that object, and thirdly, some information about the object’s 3D form. This was done by employing the stereo capability of the headphones (i.e. a pop sonified on the left or right corresponded to an obstructing object on the left or right of the participant), the time delay between the initiating click and the corresponding pop as with traditional echolocation (a long delay indicated that the object was further away from the participant than a shorter delay), and the volume of the pop sound itself. Importantly, participants were required to attend to not just a single particle’s pop, but the pops of all particles (of the 900) that have landed on virtual objects in front of the participant. The speed of the click and particle projection was chosen such that there was no temporal overlap between the pops corresponding to different clicks. In this way, a participant had to use the click and the collection of popping sounds to construct and continually update a distance-based map of his/her immediate (< 3m) surroundings in the VR environment. A video example of echolocation is provided here: https://www.youtube.com/watch?v=WFHEJ8pOego

2.3.2 Distance-dependent hum volume modulation.

Differing from the “snapshot” nature of the point-and-project simulated echolocation, the second sonification method we explored aimed to encode the spatial layout of objects using continuous audio beacons. Here, hums of different pitches were attached to different objects, and the volume of these hums was modulated based on a participant’s distance to the objects: shorter distances to objects correlated to their hums being louder. Furthermore, an object’s hum was only triggered when the participant entered a defined “humming zone” around the object, and as he/she moved closer to the object, the volume of the hum was linearly increased. A linear volume roll-off curve was selected over a log-based roll-off since it allowed changes in distance to an obstacle (given the bounds of its humming zone) to be more easily discerned. In the maze, the humming zone extended 1.2m from the wall (Fig 1A). In the obstacle corridor, the humming zone extended 3m radially from each obstacle, and 1.5m from the walls (Fig 1B). Using the stereo headphones, the panoramic position of the hum enabled participants to determine its direction and hence the spatial position of the obstacle.

We varied the types of humming sounds based on the objects emitting them. The walls in both the maze and the corridor were assigned a deep, resonant hum, while the obstacles in the obstacle corridor were each assigned a hum of unique pitch (to all the other obstacles and the walls). We opted for this approach because it was crucial for participants to be able to differentiate obstacles, especially when the humming zones of obstacles overlapped.

Our rationale for choosing humming sounds over other types of sounds was two-fold: firstly, we wanted to create a continuous or smooth soundscape capable of capturing the presence and spatial location of multiple objects in an environment. The humming sounds were ideal for this since they could be looped seamlessly (compared to single beeps or pulses). Their continuous nature also allowed us to fuse audio representations across multiple objects in the environments. Secondly, we wanted to create an acoustically pleasant or harmonising soundscape with the motivation that our sonification mapping should be ambient and non-intrusive to users—essential in a real-world implementation of a SSD. For this reason, we selected hums since they have little temporal structure and are almost absorbed into the audio background unless specifically attended to. The hums themselves were additive blends of sinusoids, each of characteristic frequency, rather than pure tones which we viewed as harsher on the ear. The blends of frequencies were chosen based on what the researchers deemed to be pleasant sounding. Fig 3 summarises the core frequency components, or notes, of the hums used.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Spectral frequency displays of the additive sinusoidal hums used in the distance-dependent hum volume modulation sonification method.

Each column (horizontal axis) corresponds to a hum, with its frequency components/notes shown on the vertical axis. The blue band indicates the hum’s core frequency, while the red and orange bands indicate other prominent frequencies.

https://doi.org/10.1371/journal.pone.0199389.g003

For brevity, we refer to this method of distance-dependent hum volume modulation as simply the humming sonification. A video example of humming is provided here: https://www.youtube.com/watch?v=aR5r10daK7Y

2.4 Hardware & software

Unity (Unity Technologies, San Francisco, CA, USA), a 3D game engine, was used to create the virtual environments, and its built-in sound engine provided spatial audio features which were used to encode the environments using the two sonification methods. The spatial audio features allowed for the localisation of sound sources on the horizontal plane by taking the source and regulating the gains of the left and right ear contributions (relayed via the headphones) based on the distance and angle between it and the participant in the VR environment. The sonified environments were then deployed on a Google Tango tablet (Google, Mountain View, CA, USA). The tablet employs visual-inertial odometry using a 180° FOV fish-eye camera and its Inertial Measurement Unit (Fig 2B) to map its 3D position and rotation in real-time based on an initialisation point.

The software packages used for development included the Tango Tablet Development Kit (kernel version: 3.10.24-g36a1dd3; April 14, 2015), Unity 5.0.1 Personal Edition with C#, Android SDK tools (release 23.0.2 for Mac), and the Tango Unity SDK (release Ramanujan, version 1.17, July 2015). The Tango itself was running Android KitKat (version 4.4.2, API level 19).

Rather than using a keyboard or joystick-based spatial navigation paradigm, we aimed to create a fully immersive VR experience in which participants’ movements in the real world corresponded to their movements in the virtual world. To do this, the Tango was mounted in a Durovis Dive 7 VR headset (Fig 2A). The headset allowed the device to be fixed to participants’ heads such that their 3D positional (translational) movement and 3D head rotation were tracked. The headset included a pair of lenses which projected the image on the Tango screen to a comfortable viewing distance for participants. The headphones (Sennheiser HD 206 Stereo Headphones) were wire-connected to the Tango. Importantly, the headphones were over-ear with a noise attenuation feature, thus ensuring that any external sounds from the environment were largely suppressed.

2.5 Experimental protocol

Testing was carried out in a large indoor hall or a large outdoor space, thus comfortably fitting the virtual environments and reducing the risk of participants colliding with physical objects. The physical size of the hall was 20 × 25 metres. The outdoor space was a flat manicured lawn, the weather was mild, and no other people, besides the experimenter and the participant, were present. The testing location was selected based on the availability of the hall, but also allowed the robustness or generalisation of our methods to be tested in two different environments. Of the 18 participants, 5 were tested outdoors, and no difference in navigational performance was noted.

Six experimental conditions were undertaken by each participant. Each condition included a minimum of six repeated trials. Participants were instructed to reach the star goal as quickly as possible while avoiding walls and/or obstacles. Collisions with either were conveyed to the participant by the Tango vibrating gently. This vibration was intended to notify participants of a collision and, without being too noxious, to motivate them to avoid collisions as much as possible. Trials were discarded from analysis if 1) the Tango tracking was lost due to a device or initialisation failure, or 2) the participant failed to reach the goal within 150 seconds. Out of the total 641 valid trials across the 18 participants and 6 conditions (visual, humming and echolocation in both environments), 39 of these trials were discarded. Of these 39 trials, 28 were discarded due to device/initialisation failure, while the remaining 11 were discarded due to a participant taking over 150 seconds to reach the goal. The distribution of the 11 no-goal-reached trials was as follows: [5, 1, 1, 2, 2, 0] for the first to sixth trial, respectively.

Stage 1: Visual-only condition A purely visual condition was conducted to familiarise participants with both of the virtual environments and to determine a visual baseline for task performance. The virtual environments were visible on the Tango screen mounted in the Durovis headset and participants were instructed to walk toward the star at a natural walking pace while avoiding walls and/or obstacles. The star emitted a ping that served as an audio beacon, and played the completion chime when reached. No other sounds were present. Six trials of the visual-only condition in each environment (maze and obstacle corridor) were conducted with the order of environments randomised across participants.

Stage 2: Spatial audio training Following the visual-only condition, an opaque piece of cardboard was taped over the screen of the Tango for the remainder of the experimental conditions. Henceforth, only audio information was available to participants via the headphones. In order to acquaint participants with the concept of spatial sound, a training stage was conducted: participants were instructed to walk toward a pinging goal positioned 13 metres directly ahead of them in an obstacle-free environment. Training was continued until participants were able to comfortably localise the goal using the stereo and distance-based volume changes of the pinging sound. This did not prove to be too difficult, with all 18 participants doing one training run, 10 doing a second, and 3 doing a third.

Stage 3: Sonification conditions Six trials of each of the two sonification conditions were conducted, with a familiarisation period for each done prior to the test trials. The testing protocol was laid out as follows:

• Echolocation: a familiarisation period followed by 6 trials done in the maze and another 6 trials done in the obstacle corridor.
• Humming: a familiarisation period followed by 6 trials done in the maze and another 6 trials done in the obstacle corridor.

The order of the echolocation and humming sets was randomised across participants, as was the order of the environments tested within each set.

The familiarisation period for each sonification condition was conducted in a simplified version of the obstacle corridor set-up: a single obstacle was located centrally at 4m away from the starting point. In the echolocation training period, this obstacle and the walls received and projected the echolocation click and popping sounds as described in Section 2.3.1. In the humming training period, objects hummed depending on the participants’ distance to them as described in Section 2.3.2. Familiarisation was considered complete when both the participant and the experimenter agreed that the participant understood the task aims and sonification rules. This proved to be a relatively short period: for echolocation training, all 18 participants did one training run, 13 did a second and 3 did a third, while for humming training, all 18 participants did one training run, 13 did a second, 2 did a third and 1 did a fourth. A two-sample Kolmogorov-Smirnov test does not reject the null hypothesis (p = 1) that the number of training runs for each sonification condition comes from the same distribution. This suggests no significant difference in training duration between the two sonification conditions.

2.6 Experimental metrics

The 3D location (x, y, z) and 3D head rotation (yaw, pitch, roll) of the participant was recorded from the Tango at a rate of 2Hz. From these data, participants’ 6D path trajectories through each of the environments in each of the conditions were reconstructed. Data were written to a text file on the Tango at the completion of each trial and later processed with MATLAB (MathWorks, Natick, MA, USA) using custom scripts. Analysis included looking at the “bird’s eye view” path of participants, the length of this path, and participants’ instantaneous, mean and peak velocities. Measures designed to probe visual awareness and navigational strategies were additionally investigated through the derivation of 1) instantaneous and mean head rotation angle, and 2) deviation distances from obstacles and walls, both of which are expanded on below.

Head rotation angle. Since sound in the virtual environments was fully stereo, an indicator of exploration was taken to be the amount of lateral head rotation that participants made whilst navigating the virtual environments. Given that the Tango was head-mounted, the Tango’s rotation corresponded to participants’ head rotation. The Euler angle for rotation about the vertical axis (yaw) was extracted at each time point and compared to the instantaneous path angle (that is, the direction of walking calculated using the positional data). The difference in angle was then transformed such that a left head rotation fell in the range from 0° to −90°, and a right head rotation fell in the range from 0° to +90°. The mean left and right head rotation was calculated per trial, and their absolutes summed to give a total head rotation value between 0° and +180°.

Deviation distances. Detection distance may be defined as the distance at which a participant explicitly identifies an obstacle ahead of them [80]. One proxy for detection distance is a subject’s “deviation distance”, the distance of a participant to an obstacle when he/she begins to adjust his/her trajectory to avoid it. This value was calculated by extrapolating participants’ path trajectories, determining if their trajectories intersected with the area occupied by obstacles at each time point, and if so calculating the distance at which participants deviated from this “collision course” [78]. In the corridor environment, this calculation was applied to the five columnar obstacles. In the maze, however, it was necessary to reformulate the calculation of deviation, and a midline deviation was used instead. The midline path, considered the optimal path, is the path equidistant to all walls at any given time point. At each time step, the perpendicular distance of a participant’s path trajectory to this midline was calculated, and the means of the deviations left and right of the midline found. In the visual maze condition, participants often “clipped” the corners, and for this reason the corners of the maze were not included in the midline deviation calculation.

2.7 Statistical analysis

ANOVA analyses were performed using SPSS (IBM Corp., Armonk, NY, USA) to investigate the relationship between experimental condition and the above-mentioned metrics across participants. To account for the across-trial variability of each participant within a given condition, the mean over their six trials for each measure was calculated, and these values averaged across participants. Additionally, because the maze and the obstacle corridor are fundamentally different, with performance in each not necessarily comparable, the analysis was split accordingly (see Section 3.1 and Section 3.2). Results are reported as mean ± standard error, with asterisks used to indicate statistical significance as follows: p < .05(*), p < .01(**) and p < .001(***). The p-values in pairwise comparisons for multiple comparisons were corrected using the Sidak correction. Greenhouse-Geisser estimates were used to correct for violations of the assumption of sphericity in the results.

3.1 Maze

Participants were able to navigate the maze environment in both visual and sonification conditions. At the start of each trial, participants were placed equidistant from the walls and forward-facing at the starting point. By nature of the maze and sonification set-up, this meant that participants received no audio information if they did not move from this point and gained no extra benefit/advantage by remaining there. We therefore formally began each trial when the participant began walking (we took this to be when their walking speed increased above 20% of their peak-to-peak velocity for that trial). Across participants, this start-delay was 2.42s with echolocation and 2.28s with humming, a statistically insignificant difference between the sonifications. Fig 4A–4C show the path trajectories of participants for the visual, echolocation and humming conditions, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Maze navigational behaviours and results.

(A, B, & C) Examples of participant trajectories through a maze, using visual-only cues (A), echolocation audio-only cues (B), and humming audio-only cues (C). (D, E, F, & H) Basic efficiency and strategy of navigation through the maze in the visual (purple), echolocation (blue) and humming (green) conditions: time to reach goal in seconds (D), path length in metres (E), mean velocity in metres per second (F), and summed left and right midline deviation in metres (H). Error bars indicate standard error. Statistical significance indicated as follows: not significant (n.s.), p < .05(*), p < .01(**) and p < .001(***). (G) Summed left and right head rotation in degrees. Head rotation from head straight (0°) direction, comparing visual (purple), echolocation (blue) and humming (green) conditions in the maze. Error bars indicate standard error. Statistical significance indicated as follows: not significant (n.s.), p < .05(*), p < .01(**) and p < .001(***). (I, J, & K) Learning curves over 6 trials comparing visual (purple), echolocation (blue) and humming (green) conditions in the maze: time to complete in seconds (I), summed left and right midline deviation in metres (J) and summed left and right head rotation in degrees (K). Linear regression line with R²-statistic shown for each condition in corresponding colours. Error bars indicate standard error. Significance symbols mark significance of effect between first and sixth trial. Statistical significance indicated as follows: not significant (not shown), p < .05(*), p < .01(**) and p < .001(***).

https://doi.org/10.1371/journal.pone.0199389.g004

3.2 Obstacle corridor

Compared to the maze, the obstacle corridor offered an environment much closer to a possible real-world scenario. The obstacles, by way of the randomness of their positioning, made for a far less predictable path to the goal compared to the mazes, with multiple possible paths available. As with the maze, each trial was formally started when participants’ walking speed increased above 20% of their peak-to-peak velocity for that trial. Across participants, this start-delay was 2.07s with echolocation and 2.18s with humming, again a statistically insignificant difference. For illustration purposes, Fig 5A–5C show the path trajectories of participants in selected corridor arrangements.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Obstacle corridor navigational behaviours and results.

(A, B, & C) Examples of participant trajectories through particular obstacle corridor arrangements using visual-only cues (A), echolocation audio-only cues (B), and humming audio-only cues (C). (D, J, & K) Active interaction with obstacles in the corridor in the visual (purple), echolocation (blue) and humming (green) conditions: number of obstacles encountered at least once per trial (D), obstacle deviation distance (J), and percentage of obstacle patch time per trial spent <0.4m from left or right wall (K). Red dashed line shows 3m echolocation and humming threshold. Error bars indicate standard error. Statistical significance indicated as follows: not significant (n.s.), p < .05(*), p < .01(**) and p < .001(***). (E, F, G, & H) Basic efficiency of navigation through the obstacle corridor in the visual (purple), echolocation (blue) and humming (green) conditions: total number of collisions over 6 trials per participant (E), time to reach goal in seconds (F), path length in metres (G), mean velocity in metres per second (H). Error bars indicate standard error. Statistical significance indicated as follows: not significant (n.s.), p < .05(*), p < .01(**) and p < .001(***). (I) Summed left and right head rotation from head straight (0°) direction in the corridor in the visual (purple), echolocation (blue) and humming (green) conditions. Error bars indicate standard error. Statistical significance indicated as follows: not significant (n.s.), p < .05(*), p < .01(**) and p < .001(***). (L, M, & N) Learning curves over 6 trials comparing visual (purple), echolocation (blue) and humming (green) conditions in the obstacle corridor: time to complete in seconds (L), summed left and right head rotation in degrees (M), obstacle deviation distance in metres (N). Red dashed line shows 3m echolocation and humming threshold. Linear regression line with R²-statistic shown for each condition in corresponding colours. Error bars indicate standard error. Significance symbols mark significance of effect between first and sixth trial. Statistical significance indicated as follows: not significant (not shown), p < .05(*), p < .01(**) and p < .001(***).

https://doi.org/10.1371/journal.pone.0199389.g005

3.3 Qualitative experience

A final component enhancing the quantitative results presented above is the qualitative feedback received from participants following their two testing sessions. Since the spread of SSD naivety and gaming experience across participants was small, it was not expected to be correlated with participant performance. Looking beyond this, it was informative to look at participants’ overall experience with each sonification method, and its ease or intuitiveness in each environment. Fig 6A and 6B show the ratings received from 14 out of 18 participants when asked about the intuitiveness of each method. The maze had a largely even spread across ratings from “easy” to “challenging even with training”, with 36% and 43% of participants finding it easy for echolocation and humming, respectively. A common comment was that the mazes were more basic since one simply needed to walk in straight lines. One particular participant whose performance was notably poor in the mazes across both sonification conditions commented that her spatial memory and internal orientation were poor, and this made it difficult for her to spatially connect the corridors she was exploring. As a result, after a period of hesitation/turning/head rotation, she often began to navigate down a corridor from which she had just come. The obstacle corridor, on the other hand, had a more distinct qualitative split, with echolocation perceived as manageable with training by 79% of participants, while humming was perceived as such by only 36% of participants, with 50% rating it as challenging even with training. The perceived ease/intuitiveness of the sonification methods, therefore, contradicts participants’ true navigational performance using them in the corridor environment. These results also suggest that of the two environments, the maze was easier to navigate, likely because of its limited layouts, while the obstacle corridor posed a bigger challenge. It should be noted that no participant rated either sonification method as “impossible”.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Qualitative participant feedback.

(A)“How easy/intuitive did you find the echolocation sonification in each environment?” (B)“How easy/intuitive did you find the humming sonification in each environment?” Green: easy. Yellow: manageable with training. Orange: challenging even with training. Red: impossible.

https://doi.org/10.1371/journal.pone.0199389.g006

4.1 Development of an experimental paradigm to test sonification mappings

In this work, we successfully built a visual-to-audio SSD using a head-mounted Tango tablet and a pair of stereo headphones. We established an experimental testbed for the device by using standardised metrics to analyse participants’ detailed navigational behaviour through randomly generated VR environments. Participants’ navigational data was obtained by extracting the 3D positional and 3D rotational tracks of a head-mounted Tango tablet. These tracks corresponded to participants’ body position and head rotation at fine-grained time steps over the length of each trial. This testbed allowed us to successfully measure the utility of two sonification methods for the task of spatial navigation.

Our VR experimental paradigm provides important improvements over keyboard- and joystick-based paradigms which to date have been ubiquitous in studies on spatial navigation [82, 83], capturing the important contribution of proprioceptive feedback to VR navigation and mobility [72, 73]. Importantly, this was achieved using a portable, inexpensive tablet that can be used in stand-alone fashion in any sufficiently large open space, and did not require the large amount of dedicated infrastructure of previously published locomotion-controlled auditory VR [68, 74–76]. This makes locomotion-controlled, multi-sensory immersive VR accessible to smaller, independent research groups, or those who do not use immersive VR as their principal methodology.

Like other VR paradigms, our VR paradigm lends itself well to the automatic and randomised generation of simulated environments, and avoids real-world mobility hazards to participants. Moreover, the continuous tracking of real walking behaviour in the virtual world allows for the automated extraction of measures that capture the dynamics of walking and navigating, such as path length, speed, head rotation, and deviation distances from objects. In the current study, we generated simplified models of real-world scenarios, a necessary first step, however the flexibility of the experimental paradigm will enable the testing of audio-only navigation in more complex virtual environments in the future, for example multi-room scenarios with objects of varying shapes and sizes.

While we have used this navigation testing paradigm to investigate the utility of our sonification mappings, the paradigm would be equally suited for testing navigation using existing SSDs. Directional spatial-to-audio SSDs like the EyeCane have already been successfully tested in VR environments [51, 84, 85], so it would only require the use of a tracking device such as the Tango used here to convert this into a locomotion-based VR paradigm. To test realistic echolocation-type SSDs [59, 61–63] in VR, the most important additional requirement would be the accurate modelling of acoustics in the virtual environment to accurately render the temporal and spectral information present in real-world echoes [86], something that has already been achieved in this specific context [60, 87]. Finally, to test navigation in SSDs that take 2D images or videos as input, whether auditory such as the vOICe or EyeMusic [42, 43], or tactile such as [37–41], the image of the VR environment as rendered on screen can simply be used as a substitute to the regular video input of such SSDs.

Additionally, the wealth of data on the dynamics of navigation automatically extracted from the tracking device and its lightweight, low-cost control via real locomotion opens up other exciting use cases. We believe this approach will not only prove useful for studies of mobility and sensory substitution, but also for studies of navigation and spatial cognition where a contribution of proprioceptive information is relevant; for instance, in immersive studies of spatial learning and memory [88, 89] and how they may be affected by clinical conditions such as Alzheimer’s Disease [90, 91] or depression [92].

4.2 Comparison of stereosonic navigational behaviour between sonification mappings

Visual/sighted control. Participant walking velocity in the visual control condition approached 1m/s, close to the average human walking speed of 1.4m/s [81]. These measurements verify the robustness of the visual control against which performance in each of the sonification conditions is compared.

Simulated echolocation. The emitted click and distance-dependent “pops” of simulated echolocation necessitated the following: firstly, participants needed to sample their surroundings by orienting their head and secondly, using the returning waves of pops, participants needed to stitch their audio-based representations together in order to construct a representation of the full scene. For these reasons, echolocation is unlike the more passive humming sonification method, and more like the task of looking and “seeing”. By comparing participant performance, however, the simulated echolocation sonification appeared to be less intuitive than the humming sonification. This was supported by overall slower echolocation navigation speeds in both the maze and the obstacle corridor, indicative of participants being more hesitant. This hesitancy could have been the result of a number of factors. Firstly, compared to the humming sonification, there is a higher cognitive load in processing the click-pops of echolocation. Participants reported that the click-pop delays were often difficult to interpret, however, with more training, they said they may have been able to draw more information from them. A second reason for higher hesitancy with simulated echolocation is the potential limitation imposed by its update speed. The click rate (every 1.75s) placed an upper bound on the speed at which participants received an up-to-date acoustic snapshot of the environment. It is possible that with a higher click frequency, participants would have been able to move more quickly, however, this could come with the risk of them losing the ability to temporally discriminate pops if they scattered too quickly.

Across both environments a unifying strategy for simulated echolocation was hypothesised to be one in which the “quietest” path or the path of least resistance was followed. In other words, if no obstacles were directly in front of a participant, then the returning pop sounds were few and so the soundscape relatively quiet. In this way, a path could be weaved between obstacles. Consistent with this, there was a trend towards fewer collisions in the echolocation condition compared to humming (by 46%, p = 0.067).

Distance-dependent hum volume modulation. Participant task performance indicated that encoding objects’ spatial distance using volume-modulated humming was an intuitive sonification method, more so than the simulated echolocation. In both the maze and obstacle corridor, navigation using the humming sonification was faster (higher mean velocity and lower time to reach the goal) compared to the simulated echolocation along equivalent path lengths. This suggests higher confidence levels with the humming technique. Looking at the more challenging task of obstacle avoidance in the corridor environment, with humming, the walking speeds were on average 27% slower than those in the maze. Although the humming obstacle corridor velocities were faster than those in the echolocating obstacle corridor by 0.7m/s, the drop in speed from humming-based navigation in the simple maze to the more complex obstacle corridor was larger than the corresponding performance drop using echolocation-based navigation.

These results together suggest that of the two sonification methods, humming was quicker and easier to learn than echolocation. Humming, however, is inherently limited by the amount of information it can represent. A hum, based on its volume level conveys distance to an obstacle, and based on its stereo components conveys information about its spatial position. On the other hand, a hum offers no information about object shape. Furthermore, absolute hum volume, necessary for participants to ‘calibrate’ their volume-distance correspondence, may be difficult to immediately discern. Critically too, the presence of multiple obstacles, resulting in the overlapping of humming zones, was reported to make it challenging to distinguish space between obstacles or clear paths ahead. We believe, therefore, that simulated echolocation has greater potential for representing more complex and detailed environments for the task of navigation. Pointing toward this, participants rated echolocation as easier than humming in the more complex corridor environment in their follow-up qualitative feedback, despite their poorer navigational performance here. One possible explanation for this difference could be that participants are positively projecting their capacity to learn a richer sonification for more complex environments in the future (i.e. perceiving the method as easier in the present because of their predicted ability to improve or master the method in the future).

To enable the estimation of distance with a humming approach, future work could include modulating the hums in ways that would make judging absolute distance easier for the human auditory system. Hums could be pulsed with a frequency proportional to obstacle distance, or modulated in pitch or with a filter envelope as a participant approaches an obstacle. The presence of multiple objects as overlapping sources of hums also poses a challenge, as it is easy to overwhelm a participant with a cacophony of sound. Here, an intelligent way of identifying the most mobility-relevant objects in the scene might be useful so that sounds from other less relevant objects or objects further away may be suppressed.

4.3 Observed learning effects in stereosonic navigation

For each sonification condition, participants received minimal instruction prior to the trials and no feedback during the trials. As part of the minimal instruction, participants underwent a training period with the device—this took place in a drastically simplified version of the obstacle corridor (with only one obstacle) and comprised only a small number of trials. The training period was primarily intended to familiarise participants with the headset and experimental procedure rather than the sonification methods themselves, and the number of training trials did not differ between conditions. The brevity of the instruction and training period was also intended to provide a fair baseline for comparison to other visual-to-audio SSDs in navigational proficiency.

With this in mind, it is therefore impressive that learning effects were observed for the humming sonification over only 6 trials. The clearest learning effect was seen in the decrease in time taken to reach the goal: improvements of 24.2s and 28.1s were achieved in the maze and the obstacle corridor, respectively. Furthermore, in the obstacle corridor environment, obstacle deviation distances improved by ∼0.5m using the humming sonification. No significant effect was seen for echolocation, though the direction of the effects was consistent with improvement.

While the sonification performance does not reach performance when sight is available, the results speak to the possibility that with further training this performance difference could be reduced. More experience with each sonification method as well as a more detailed understanding of the sonifications’ formulations (in particular simulated echolocation which shows promise in representing more complex environments) may allow participants to narrow this performance gap with practice.

4.4 Considerations for real-life application

From the current study it is too bold to claim that the two sonification methods explored here encode visual spaces in sufficient detail for seamless spatial navigation. Our results, however, do suggest that participants were able to obtain a 3D spatial awareness of their virtual surroundings. The fact that participants were able to spatially place themselves and manoeuvre through two distinct virtual environments, avoiding walls and simple obstacles, and also improve their navigational performance over trials is indicative of this. We believe that it is, therefore, interesting to consider the potential for these strategies as a real-life sensory substitution device for visually impaired people.

To this effect, it will be necessary to investigate how blind or visually impaired participants specifically adapt to these different sensory substitution approaches, and how the approaches complement or interact with their existing navigation strategies. On one hand, blind or visually impaired participants may in fact outperform our normally-sighted participants, as it has been shown that early-blind individuals in particular may display superior performance in auditory tasks like pitch discrimination [93] and spatial sound localisation [94–97]. In addition, they will be more experienced in using such spatial properties of audio signals to navigate and detect obstacles in their environment [32, 33]. With regard to our hypothesised audio-based navigational strategies, blind or visually impaired individuals might in fact be more inclined than our sighted participants to use head rotation for echolocation as they will know from experience that this helps to sample the auditory environment.

Conversely, in particular those visually impaired participants who have been blind from birth or a very young age may have altered representations of locomotor space that could interfere with performance [30, 98]. This may reflect a greater tendency to rely on an egocentric rather than allocentric spatial reference frames [99], which is thought to be particularly pronounced for larger-scale spatial representations such as those relevant to locomotor activity [100]. There is evidence that such altered spatial representations may negate the advantages of superior auditory discrimination in sound localisation in larger spaces [101]. How well the performance of our sighted participants correlates to the performance of blind and visually impaired individuals is therefore an important question to address in future work.

Moreover, in order to translate our sonification mappings to the real world, it will be necessary to implement the current virtual world sonification in real 3D space, and ultimately transfer the mappings onto a stand-alone wearable SSD. By pairing an RGB camera with a depth sensor and applying computer vision algorithms to the incoming video streams, it is possible to extract 3D spatial information of the real-world environment. On the incoming RGB images, object detection methods [102] will localise the presence/absence of objects, and object recognition methods [103, 104] will identify the types or classes of these objects (for example, desks, chairs, people, but also more abstract categories like walls, floors and ceilings). The incoming depth maps (from the depth sensor) will provide real-world distance estimates from the camera to densely scattered points in the environment. If the camera were mounted on the user’s head, these two streams of information would allow for the building and updating of a 3D egocentric representation of the user’s environment which could subsequently be converted into its corresponding soundscape. For simulated echolocation, the dense distances provided by the depth maps are synonymous with the distances the projected particles travel. In a similar way, these distances could be used to modulate the volume/pulse frequency of the humming sounds. Central to this process will be the selection of or emphasis on objects which are relevant to mobility so as not to overwhelm the user with irrelevant audio cues.

Extending this, recent years have seen the proliferation of methods for the fusion of global 3D maps of environments [65, 66, 105–107]. These methods allow for 3D maps of whole environments to be built and updated on-the-fly as the user moves within them. Access to such a map would offer the ability to integrate global-level information of an environment (which may not be available from the user’s local camera view) into the sonification methods—for example, sonifying objects behind the user, objects that dynamically move in and out of the user’s field of view, or objects at far distances or in occluded positions.

Implementing these real-world sonification methods into a wearable SSD presents several challenges. The system must be able to capture, build a representation of the user’s environment, and sonify the relevant parts of the scene all in near real-time. Furthermore, the device must be light in both weight and power consumption, placing significant limits on the computer vision algorithms running on-board the platform. Real-time object detection and recognition methods already exist [104], however, and we can expect increasingly reliable and light-weight solutions to follow in the near future following the rise of household robotic systems and autonomous vehicles.

A crucial consideration in building a wearable audio-based SSD for blind and visually impaired people is the fact that the resultant soundscapes must be delivered in a way that does not mask or impede the ambient sounds on which the user may already heavily rely to perceive their surroundings and remain oriented. Bone-conduction headphones, which transmit sounds through vibrating pads placed on the jaw bones, have been shown to allow equivalent performance in the spatial localisation of audio cues when compared with standard stereo headphones [108], and have been successfully used for auditory spatial navigation through VR environments [74]. This suggests that they do have the potential to accurately represent 3D spatial audio for navigation within a real-world visual-to-audio SSD.

Finally, it is worth noting that another successful real-life implementation of the sonification algorithms presented in this study may yet be entirely virtual. Since the advent of VR environments, an important question has been how to make such environments accessible to the blind and visually impaired community [109]. The focus of VR system development has often been toward a better visual experience, but there have been notable exceptions focusing on the non-sighted population [84, 110, 111]. Our auditory VR navigation paradigm can add to the available methods for making virtual worlds more accessible to those without sight.