Lab 3: The Synthetic Psychology of Sound Localization

Return to the lab overview, or move on to task 6.

Task 5: Learn about neurons, coincidence detection, and some math!

Now that you have a working artificial auditory system on your robot, your next step is to make it localize sounds. As you now know, the two environmental features or "cues" available to you (and your robot) are inter-aural intensity differences (IIDs, i.e., volume differences between the ears) or inter-aural timing differences (ITDs, i.e., differences in the precise time sound arrives at your ears). Since your robot doesn't have a big squishy head and brain, it casts less of an acoustic shadow. As a result, IIDs are less likely to be useful. In addition, from an engineering stand point, small differences in measured voltage might be due to your circuit (components aren't perfect and always have some error). Psychophysicsts might call that "sensory noise." The noise might confuse your interpretation of the voltage differences. Thus, we are going to try to utilize ITDs to localize sound.

This might seem quite a challenge, but it's actually a nice situation to be in. We have before us a straightforward computational question: How does one compute timing differences arriving at the two microphones? Perhaps before we dive into this too deeply, let's talk a little about how brains and neurons perform this task.

The Medial Superior Olive and its coincidently amazing coincidence neurons

First, lets take a quick look at the anatomy of your brain just so you have a sense of what we are taking about. This figure shows the overall organization of the auditory system (ascending pathway). Of particular interest to us right now is the small node called the Superior olivary nucleus which is down near the brainstem.

Certain neurons in the medial superior olive (MSO) (a sub-region with the superior olive) fire selectively to interaural timing differences of sounds between the ears. Here's a plot from David Heeger's lecture notes showing the response of a single MSO neuron as a function of the timing difference between the left and right ear:

This particular neuron responds strongly at around 75 microseconds of delay, but other neurons in that brain area respond strongly to other delay values (20 microseconds, etc...).

Cool! So problem solved, eh? We now know that neurons in the brain respond selectively to timing differences! We just need to get some of those and put them in our Ardunio code, right!? Well, sorry but it's not that simple. Just showing that there are neurons that do this doesn't really tell us much.

To see why, consider the following thought experiment about the engineering implications of this finding: Can you use this knowledge to help you build your robot? The answer is likely no. Knowing that neurons fire selectively to timing differences just tell us that someplace in the brain timing differences are represented. We can't use this information alone to help us build our robot! In fact, we already knew that brain cares about timing differences from our simple behavioral experiment in Step 1 of the lab. We did that without measuring any neurons directly just using a sheet of paper and a pair of headphones!

It'd be better if we knew how these neurons do this! Luckily Jeffress (1948) proposed good idea we should consider for a moment.

Jeffress' paper wasn't a empirical discovery but more a proposal about how the brain might do detect timing differences. Jeffress' idea went something like this:

According to the model, there is an array of coincidence detector neurons (again, those that fire most strongly when their inputs are similar or identical). In addition, input from the left and right ear travel along very long axons. Thus, neural spikes that start from the right auditory pathway (indicated on the right of the figure) take time to travel to the left. Likewise spike trains from the left ear (indicated on the left) take time to travel to the right. Various places along the axon the coincidence detectors receive input. If sound arrives first at the right ear, it will have more time to travel down the axon flowing right to left than will a sound arriving in the left ear (who's spike travel left to right). As a result, coincidence neurons further to the left of the figure will be more strongly activated. In contrast, if sound arrives first on the left, neurons towards the right of the array will be most strongly activated. Sounds that arrive simultaneously will activate the "middle" neuron in the figure.

According to this idea, which coincidence neuron is most strongly activated should tell use what the ITD was between the ears. (Jeffress' model included an additional set of coincidence "counters" that would sum up the coincidence spikes from the detectors to integrate the similarity of sounds over time).

Jeffress' model is a theoretical idea. For example, no one know for sure how many coincidence detectors there are in your brain. However, the idea exemplifies HOW certain neurons could be strongly activated by ITDs between ears. It provides and explanation of the neural firing data plotted above. In addition, it shows how this simple neural circuit might be able to code/tell what the ITD is between ears.

Now here's the really crazy thing. Jeffress theory seems to have been right! Later work by neuroanatomists and neurophysiologists examined closely the anatomy of the MSO and found a wiring diagram very similar to what was proposed by Jeffress! In fact, this idea is now probably the most widely accepted view of how the auditory system detects and encodes ITDs (although there are certainly many more complex issues).

Coincidence as correlation

Where does this leave us with our robot? Well, we can try to use a similar principle as the Jeffress model in our robot. Let's take this model apart. First, we need a way of detecting similarity between two sounds waves over time (the role that the coincidence neurons and the coincidence counters might play in the brain).

First let's take a look at two hypothetical sounds which have been converted into time-varying voltages and recorded by your Arduino (I know we haven't gotten here yet, but you will). These values will be "sampled" by the analog-to-digital converter inside the Arduino chip. Basically, this just means that the value of the continuously varying voltage in the circuit will be recorded at fixed points in time. Here is me playing some music to my Arduino robot:

PUT GRAPH OF SAMPLES WAVEFORM HERE

Notice this looks quite similar to what was displayed on the oscilloscope in step 4.

How similar are these wave forms? One way to calculate the similarity of data like this is to use a standard correlation coefficient. Remember from your stats class, correlation measures how much one measure relates to another. Correlations can be positive (indicating a strong relationship), zero (indicating no relationship), or negative (suggesting an inverse relationship).

To compute the correlation, we're just going to just make a scatter plot where the x-value of each point contains the value of the "right ear" signal and the y-value is the "left ear" signal. For the waveform above this looks something like this:

PUT GRAPH OF CORRELATION HERE

Notice that the correlation value of this set of data is X. Also plotted is an scatter plot from a situation where the left and right channels are quite different. Notice how the correlation is reduced.

Cross-correlation as a measure of timing delay

Ok, now we have a measure of "similarity" between the two signals received as the ear. How can we use it to detect the ITD? The basic idea was anticipated by Jeffress' model! We can just compute the correlation of the two signals at different "shifts" or "delays" from one another. Here's a visual example of a signal which has a true ITD (phase shift) between the left and right signal. In each graph, I shifted the signal a bit to the left or right and recomputed the correlation. The resulting plot at the bottom is known as the "cross-correlation" function.

The peak of the cross-correlation function tells you which amount of shift is required to make the signals most similar! In other words it provides the exact same information given by the coincidence neurons in the Jeffress model!

Return to the lab overview, or move on to task 6.
Copyright © 2013 Todd Gureckis, Diagrams and schematics of the Parallax robot come from the Parallax website, much excellent material was taken from David Heeger's course notes on sound localization