2.1.1 Recognition region.

The neural network’s inputs are the FFT values from a rectangular region of the spectrogram covering a predefined range of frequency values F (for example, 1–8 kHz) at some number of the most recent frames n_fft. Any time t falls within frame τ(t), and t − t_fft falls within the previous frame, so the recognition region that the neural network receives as input consists of the spectrogram values over F at τ(t) and those from the contiguous set of recent frames: T = { τ(t), τ(t − t_fft), τ(t − 2t_fft)… τ(t − n_fft t_fft)}. Time windows of 30–50 ms—the latter will yield n_fft = |T| = ⌊50 ms/t_fft⌉ frames—of frequencies spanning 1–8 kHz generally work well.

Six examples of chosen target moments in the song, with recognition regions F = 1–8 kHz and T = 30 ms, are shown in Fig 1.

2.1.2 Building the training set.

The training set is created in a fashion typical for neural networks: at each time frame t the rectangular |F| × |T| recognition region in the spectrogram as of time t is reshaped into a vector ξ_t, which will have length |F||T| and contain the spectra in F taken at all of the times in the set T: from τ(t − n_fftt_fft) to τ(t). These vectors are placed into a training matrix, Ξ, such that each column ξ_t holds the contents of the recognition region—containing multiple frames from the spectrogram—as of one value of t.

Training targets y_t are vectors with one element for each desired detection syllable. That element is, roughly, 1 if the input vector matches the target syllable (t = t*), 0 otherwise, for each target syllable (of which there may be any number, although they increase training time, and in our implementations the number of distinct output pulses is constrained by hardware). Since the song alignment may not be perfect, and due to sample aliasing, a strict binary target may ask the network to learn that, of two practically identical frames, one should be a match and the other not. Thus it is preferable to spread the target in time, such that at the target moment it is 1, and at neighbouring moments it is nonzero. We found that a Gaussian smoothing kernel around the target time with a standard deviation of 2 ms serves well.

With inputs well outside the space on which a neural network has been trained, its outputs will be essentially random. In order to reduce the false positive rate it is necessary to provide negative training examples that include silence, cage noise, wing flapping, non-song vocalisations, and perhaps songs from other birds. Although it will depend on the makeup of the non-song data, we have found that training with as low as a 1:1 ratio of non-song to song—or roughly 10 minutes of non-song audio—yields excellent results on most birds.

2.1.3 Normalisation.

In order to present consistent and meaningful inputs to the neural network and to maximise the effectiveness of the neural network’s training algorithm, we normalise the incoming data stream so that changes in the song structure of the sound are emphasised over changes in volume.

The first normalisation step is designed to eliminate differences in amplitude due to changes in the bird’s location and other variations in recording. Each recognition region vector ξ_t—during training, each column of the training matrix Ξ—is normalised using MATLAB’s zscore function , so that the content of each window has mean and standard deviation .

The second step is designed to ensure that the neural network’s inputs have a range and distribution for which the training function can easily converge. Each element i of is normalised across the entire training set—each row of Ξ—in the same way: , so that the values of that point across the whole training set have mean and standard deviation . This is accomplished during training by setting MATLAB’s neural network toolbox normalisation scheme to mapstd, and the scaling transform is saved as part of the network object used by the realtime detector so that it may be applied to unseen data at runtime.

These two normalisation steps provide a set of inputs that are more robust to outliers and less likely to produce false positives during silence than other normalisation schemes, such as linear or L₂ normalisation.

2.1.4 Neural networks.

While any learned classifier might suffice, we chose a two-layer feedforward neural network. In brief, our network takes an input vector ξ_t—as described above—and produces an output vector y_t, and when any element of y_t is above a threshold (described below), the detector reports a detection event. The network uses two matrices of weights, W₀ and W₁, and two vectors of biases, b₀ and b₁. The first takes the input ξ_t to an intermediate stage—the “hidden layer” vector. To each element of this vector is applied a nonlinear squashing function such as tanh, and then the second weight matrix W₁ is applied. This produces output y_t: During the network’s training phase, the elements of the matrices and bias vectors are learned by back-propagation of errors over a training set. A more detailed explanation of neural networks may be found in [14].

Essentially, after training, the network is available in the form of the two matrices W₀ and W₁ and the two vectors b₀ and b₁, and running the network consists of two matrix multiplications, two vector additions and the application of the squashing function.

2.1.5 Training the network.

The network is trained using MATLAB’s neural network toolbox, with Scaled Conjugate Gradient (trainscg). We tried a variety of feedforward neural network geometries, from simple 1-layer perceptrons to geometries with many hidden nodes, as well as autoencoders. Perhaps surprisingly, even the former yields excellent results on many syllables, but a 2-layer perceptron with a very small hidden layer—with a unit count 2-4 times the number of target syllables—was a good compromise between accuracy and training speed. For more variable songs, deep structure-preserving networks may be more appropriate, but they are slow to train and unnecessary for zebra finch song.

2.1.6 Computing optimal output thresholds.

After the network is trained, outputs of the network for any input are now available, and will be in (or, due to noisy inputs and imperfect training, close to) the interval (0, 1). We must choose a threshold above which the output is considered a positive detection. Finding the optimal threshold requires two choices. The first is the relative cost of false negatives to false positives, C_n. The second is the acceptable time interval: if the true event occurs at time t, and the detector triggers at any time t ± Δt, then it is considered a correct detection. Then the optimal detection threshold is the number that minimises [false positives] + C_n ⋅ [false negatives] over the training set, using the definitions of false positives and negatives given in Section 2.3.1. Since large portions of the cost function are flat, we use a brute-force linear search, which requires fractions of a second. For the results presented here, we have used Δt = 10 ms, and arbitrarily set C_n = 1.

2.1.7 De-bouncing.

During runtime, the network may produce above-threshold responses to nearby frames. Thus, after the first response, subsequent responses are suppressed for 100 ms. However, for the accuracy measurements presented here, we used the un-de-bounced network response.

2.1.8 Our parameter choices.

We used an FFT of size 256; a Hamming window; and chose a target spectrogram frame interval of t_fft = 1.5 milliseconds, resulting in a true frame interval of t_fft = ⌊1.5 ⋅ 44.1⌉/44.1 ≈ 1.4966 ms. We set the network’s input space to 50 ms long, and to span frequencies from 1–8 kHz, which contains the fundamentals and several overtones of most zebra finch vocalisations.

We found these parameters to work well across a variety of target syllables, but various other parameter sets yield results similar to those presented here. Some of the parameters trade off detection accuracy or temporal precision vs. training time. For example, decreasing the frame interval generally decreases both latency and jitter, but also increases training time. Sometimes the effects are syllable-specific: for example, using a 30-ms time window rather than 50 ms speeds training while usually having a minimal effect on detector performance (as is the case, for example, for all of the detection points for the bird “lny64”), but occasionally a syllable will be seen for which extending the window to 80 ms is helpful.

2.2.1 Swift.

This detector uses the Swift programming language and Core Audio interface included in Apple’s Mac OS X operating systems.

The Core Audio frameworks provide an adjustable hardware buffer size for reading from and writing to audio hardware (different from our two circular buffers). Tuning this buffer size provides a tradeoff between the jitter in the detection and the processor usage needed to run the detector. We used buffer sizes ranging from 8 samples (0.18 ms at 44.1 kHz) to 32 samples (0.7 ms at 44.1 kHz) depending on the frame size used by the detector.

Vector operations—applying the Hamming window, the FFT, input normalisation, matrix multiplication, and the neural network’s transfer functions—are performed using the Accelerate framework (vDSP and vecLib), which use modern vector-oriented processor instructions to perform calculations.

When the neural network detects a match, it instructs the computer to generate a TTL pulse that can be used to trigger downstream hardware. This pulse can be either written to the computer’s audio output buffer (again, in 8- to 32-sample chunks) or sent to a microcontroller (Arduino) via a USB serial interface. Sending the trigger pulse via the serial interface and microcontroller is noticeably faster (2.2 ms lower latency), likely due to the fact that the audio buffer goes through hardware mixing and filtering prior to output.

The above code can be run on multiple channels of audio on consumer hardware (such as a 2014 Mac Mini) with little impact on CPU usage (<15%). Depending on the experimental needs, latency can potentially be further decreased (at the expense of processor usage) by adjusting the audio buffer sizes.

We ran the Swift detector on a Late 2014 Mac Mini with a Intel Core i5 processor at 2.6GHz with 16 gigabytes of RAM, running Mac OS X 10.11.

2.2.2 LabVIEW.

This implementation requires special software and hardware: LabVIEW from National Instruments—we used 2014 service pack 1—and a data acquisition card—we use the National Instruments PCI-6251 card on a PC with an Intel Core i5-4590 processor at 3.7GHz (a relatively low-end machine) with 24 gigabytes of RAM, running Microsoft Windows 8.1 Pro and Windows 10.

This implementation has several drawbacks: it requires expensive hardware and software from National Instruments (a data acquisition card and LabVIEW), Windows (we use hardware features of LabVIEW that are unavailable on MacOS or Linux), and due to the programming language it is difficult to modify and debug—indeed, a persistent bug in our implementation currently renders it substantially less accurate than the other detector implementations on some syllables. However, our test configuration proved itself capable of excellent performance, and further gains should be possible if the implementation were retargeted onto field-programmable gate array (FPGA) hardware—which would have the additional benefit of providing deterministic “hard realtime” guarantees—or just run on a faster desktop system.

2.2.3 MATLAB.

This detector uses the built-in audio input and output hardware on a compatible computer. We tested on a 2014 Mac Mini (the same machine used for the Swift detector described above) and 2015 Mac Pro. The Mac Pro does not have an audio input jack, so input was through an M-Audio MobilePre external USB audio interface. Despite the faster processor, the latter system did not achieve higher performance than the former, due to USB data transfer overhead.

Because of how MATLAB’s DSP toolbox interfaces with audio hardware, there is a double buffer both reading from and writing to audio hardware. As a result, much of the code focuses on a lightweight audio hardware interface, in order to have the smallest achievable audio buffer. To help achieve this, the MATLAB implementation spreads data acquisition and processing across two threads, due to the higher computational overhead of the interpreted programming language.

The most versatile implementation, MATLAB runs on a variety of hardware and operating systems, and is perhaps the easiest to modify. While it did not perform as well as the other implementations, the convenience may outweigh the timing performance penalty for some experiments. Key to minimising jitter is the size of the audio buffer: on a 2014 Mac Mini running MATLAB 2015b the smallest buffer size that did not result in read overruns was about 4 ms.

As with the Swift detector, it is also possible to modify the MATLAB implementation to generate the TTL pulse from a microcontroller (Arduino) controlled via a serial over USB interface. This eliminates the double buffer required when sending triggers via the audio interface, reducing latency by close to half.

2.3.1 Accuracy.

We define the accuracy of the network based on its classification performance per frame. In order to avoid the apparent problem of every non-detected non-matching frame counting as a true negative, we also tried defining accuracy on a per-song basis, such that a song-length sample without the target syllable counted as a single true negative. Computing the optimal output thresholds on a per-frame rather than a per-song basis resulted in higher thresholds and thus a slight reduction in both the false-positive and true-positive rates (reversible by increasing C_n), while also providing a valid definition of the false-positive rate for data streams that had not been segmented into song-sized chunks and thus allowing easy interpretation of false-positive rates from calls, cage noise, etc.

The accuracy as defined above is used for computing the optimal thresholds above which the network’s output should be interpreted as a match on the training data as described in Section 2.1.6, for evaluation of the detectors on the training songs, and while live.

2.3.2 Timing.

We evaluate the time taken from the complete presentation of each instance of the target syllable to the firing of the detector’s TTL pulse. For example, if the target trigger point is at 200 ms with respect to the corpus of aligned songs, and if the detection region is 30 ms long, then when the detector has seen the region from 170–200 ms, it should recognise the syllable. If for a given presentation of the song the detector fires at 203 ms, we take the latency on that trial to be 3 ms.

While playing the audio test file from any audio playback device, we used the TTL output from the ground-truth channel of the audio output as the trigger pulse for an oscilloscope, and compared it to the TTL pulse produced by the detector, which sees only the birdsong channel of the audio file. For this purpose we used a pulse generator (Philips PM 5715, with a listed latency of 50 ns, jitter of ≤ 0.1% or 50 ps, whichever is greater) to widen the detector’s output spike to a number much larger than the jitter (∼100 ms). This obviates pulse length variability in the output device by essentially discarding the falling edge of the output pulse. The oscilloscope is then set to averaging mode (128-trigger average) in order to collect timing data. The canonical signal is the trigger at t = 0, and the average of the detector’s detection events will be seen as a low-to-high transition with form approximating the cumulative probability distribution function (CDF) of the detector’s output in response to the chosen song event, with individual detection events visible as steps in this curve.

We define latency as the time between the training target (during our tests, this is indicated by the canonical signal in the test file) and the corresponding detection event. It is a helpful number, but not a critical one; due to the stereotyped song of the zebra finch, a detector with high but constant latency can often be trained to trigger at a point somewhat before the true moment of interest (for example, if triggering should occur at the very beginning of a syllable, the detector may be trained to recognise the previous syllable and the gap thereafter). Usually the variability in latency is the more important number. We define jitter as the standard deviation of detection latency, measured over the test songs.

When measuring timing, it is useful to compare against a theoretical optimal, in order to control for any imprecision in the song extraction and alignment of our real data inherent in the method we use (described in [11]). We propose two measures thereof:

First we test the “ideal” latency and jitter in the time shift used in calculating new columns of the spectrogram. By passing a recorded audio sequence into the detector offline, and assuming no input or output latency, we compare how many additional audio samples beyond the syllable are needed before the spectrogram can match the inputs needed to trigger the neural network. This latency reflects the FFT size used for calculating the spectrogram, the FFT time shift between columns in the spectrogram, and the width of the Gaussian smoothing kernel applied to the ground truth data when training the neural network, but ignores computation times, audio buffers, and other operating system overhead.

Next we use a “δ-syllable” consisting of a δ function in the time domain, and train the network to trigger 5 ms after this pulse. This song is fed into the live detector. The results for this measurement show the latency and jitter inherent in the complete detector including audio read and write buffers, classifier computation, and FFT time shift aliasing, but excluding imprecision in the song alignment as well as detection timing effects due to differences between instances of the bird’s song.

Finally, we measure latency on real bird song aligned as in [11]. This extracts and aligns the centres of the sampled songs, and the canonical signal is given with respect to that alignment. These timing measurements reflect not only detector performance, but also the variability of the bird’s song.

3.2.1 FFT frame interval.

Detector latency and jitter depend on the FFT frame rate. Our 1.5-ms–frame default is a compromise: shorter frames increase the precision of timing, but also increase computational requirements both during training and at runtime. Fig 4 shows how these numbers vary over a useful range of frame rates on our ideal detector, the Swift detector with serial output, and for the LabVIEW detector, for both the δ-syllable and for lny64’s song at .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Timing varies as the FFT frame interval changes.

Here we show results for the ideal detector and the LabVIEW and Swift+serial implementations, for the constructed δ-syllable and for trigger of lny64’s song. The lines show latency; error bars are standard deviation (jitter). Points have been shifted horizontally slightly for clarity; original positions are [0.5 1 1.5 2 4] ms.

https://doi.org/10.1371/journal.pone.0181992.g004

3.2.2 Syllable choice.

Syllable choice impacts detector performance, but despite the variety of syllables shown here, performance was stable across syllables. Fig 5 shows measured timing data for lny64’s test syllables for the Swift+serial detector compared to the ideal, with t_fft = 1.5 ms. Given the variety of the test syllables, the variability is surprisingly low, and is summarised in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Latency and jitter variability (95% confidence) for lny64’s six test syllables.

https://doi.org/10.1371/journal.pone.0181992.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Timing data for lny64’s 6 test syllables, for the ideal and the Swift+serial detectors, with an FFT frame rate of 1.5 ms.

Point centres show latency; error bars show jitter.

https://doi.org/10.1371/journal.pone.0181992.g005

The negative latency is due to the way in which the network responds to the song through time: as the recognition region looks increasingly similar to the trained match, the network’s evaluation of similarity rises, and will generally cross the triggering threshold before it reaches its maximum value. A heuristic as simple as triggering at an apparent turning point after crossing the threshold might improve timing consistency at the expense of latency, but we did not test this.

3.2.3 Detector implementations.

We compared latency and jitter across our different detector implementations for the δ-syllable and lny64’s , again with t_fft = 1.5 ms. Results are shown in Table 4 and Fig 6. Fig 7 gives a more detailed view of what the timing curves look like for the five implementations of our detector on lny64’s .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Latency and jitter for each of our detector implementations on the synthetic δ-syllable and on syllable from lny64.

https://doi.org/10.1371/journal.pone.0181992.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The different detectors for the constructed δ-syllable and for lny64’s song at .

Point centres show latency; error bars show jitter.

https://doi.org/10.1371/journal.pone.0181992.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Raw timing curves for all detectors measured during detection of lny64’s using 1.5-ms frames.

We extract the trigger events from each curve, from which we obtain the mean—latency—and standard deviation—jitter.

https://doi.org/10.1371/journal.pone.0181992.g007