In sensory substitution, we replace information usually conveyed by one sense (e.g., vision) by another sense (e.g., hearing or touch). Sensory substition research thus has the potential to help people who lack a sensory modality (e.g., vision-impaired or blind people). It also allows investigating some fundamental questions about how the brain interprets visual information, first, because Interpreting visual information that is conveyed through another sense requires learning. Second, the brains of people who lack a sensory modality are different, because their brains have adapted and use other sources of information to gain the same knowledge that is usually provided by vision. The visual cortex in blind people is very active, even in people who have never used their eyes. However, it is still a mystery what the visual cortex is doing, and even, whether it is indeed still a 'visual cortex', or whether it is now used to process exclusively tactile or auditory information (Neuroplasticity). Sensory substition research allows us to unravel this mystery, thus providing new insights into vision and neuroplasticity.
Our sensory substitution device:
We have recently developed an electro-tactile sensory substitution device that conveys information from a video camera to a 32x32 tactile display that people can place on their tongue. The tongue display (TD) was developed by two engineers, Ernst Ditges and Nick Sibbald, and a post-graduate student (Dustin Venini) has developed some more equipment and algorithms to display the information. In our current set-up, the video camera is inlaid into a pair of blackened skiing goggles, and we convert the video image into a black-and-white image so that, for example, only white objects are displayed on the tongue. So if a white object moves from right to left in the visual field, you can feel an object with the same shape moving from right to left on your tongue (but you won't be able to see it because the skiing goggles are black).
32x32 tongue display 4 electrodes (magnified) The set-up Camera
The resolution of the tactile display is a critical factor, as a high resolution is important for correctly displaying objects -- especially complex objects such as faces, but also simple features like diagonal lines. The standard resolution for tactile displays is currently 20x20, whereas our tactile display is currently consists of a 32x32 matrix.
The graphs show a cross and Dustin's face at a 20x20 resolution (middle image) and our 32x32 resolution (right image). The white pixels correspond to electrodes that would be activated and felt on the tongue.
What we have done so far:
Thanks to Hayley Jach, who joined the project in 2015, we were able to test our tactile display with blind and vision-impaired participants. Hayley wrote to diverse organisations and managed to attract more participants to our first study than we could accomodate. The opportunity to work with blind and vision-impaired people has provided us with valuable information and experience, and we are currently working with 6 blind or vision-impaired participants and 2 sighted (blindfolded) participants.
The tongue is very sensitive to tactile stimulation; therefore we need only low currents (below 20V) to create tactile images. The tip of the tongue is usually the most sensitive part, and sensitivity drops further towards the back of the tongue. Prior to the experiment, participants have to adjust the voltage for each tongue section until the activation feels equally strong across the entire tongue. In our normalisation procedure, we present a bar that is 32 pixels long and 4 pixels wide successively on 8 positions on the tongue, going from the front to the back, and participants tell us how we should adjust the voltage for each of the 8 sections until the activation feels equal everywhere. The graph below shows the median voltage for 8 participants in each of the 8 tongue sections. The violin plots below show that the voltage had to be increased towards the back of the tongue for all participants, and that some participants required very large increases towards the back of the tongue whereas others needed only small increments.
2. Moving Bar Experiment
To see how well participants could sense objects with the tactile display when they first encountered it, we moved a white bar in front of the camera, so that it covered an entire row or column of pixels. The bar could move either up, down, right, or left. The results showed that all participants were very accurate in detecting vertical or horizontal movements of the bar (75-100% correct), but there was some initial confusion about whether the tip or the back of the tongue represented the top of the visual world. So while the left-right mapping was apparently very intuitive, the top/bottom mapping had to be learnt.
3. Touch Screen Experiment
In this experiment, we displayed a 6cm white dot on one out of 32 positions of a touch screen and asked participants to localise the dot and touch it with their index finger using the tongue display (with the blackened skiing goggles). The picture below shows how the dot and the hand were represented on the tactile display, and the video shows the progression of how the dot is first localised and then touched with the index finger of the participant's hand when participants have little experience with the task.
The target dot and hand of the participant as they appeared on the tactile display.
The graphs below show the pointing behaviour of 6 participants, with the red circle indicating the true location of the dot. As shown in the graph, participants mostly localised the dot accurately. Some participants showed a slight bias to touch a position shifted to the right and down from the true location of the dot, which was in line with Dustin's observation that the finger often dropped just before it touched the screen.
In the meantime, Dustin has run several more of these experiments to explore whether the pointing error can be reduced by rendering the participant's hand visible or invisible, or by providing an outer reference together with the target dot (the monitor frame). These changes did not result in a significant improvement, hence we now think that we need to provide better feedback. Participants were provided only with verbal feedback after each trial (i.e., Dustin telling the participant where the target dot was in relation to the chosen position), which was probably not ideal. So in further experiments, we will ask participants to place an object (e.g., a candle) on a white cut-out 'plate' on the table. In this task, the objects are all tangible, so our participants can directly feel how accurate they were after each trial.
Other Tasks: Stacking Cups
When participants finish an experiment early, we often informally test some new tasks that are in the planning stage, to receive participant's feedback about how difficult the task is, where exactly the difficulties are and how we could improve it. On this note, I would like to highlight that our participants so far have been absolutely fantastic in providing us with feedback. A big thank you to all our participants, and especially those from the vision-impaired population, for helping us with this project. We really could not have wished for better participants and we hope that you'll continue helping us in the future.
In the video below, you can watch one of our blind participants stacking plastic cups with the tactile display.
Enjoy the video!
Other Vision Research
A long-standing research interest of mine is visual selective attention, that is, how the visual system selects information from cluttered visual scenes. Specifically, I am trying to gain a deeper understanding about how much control we have over attention and eye movements, what the limitations are, and the factors and mechanisms underlying visual selection.
My work has been greatly influenced by Anne Treisman's Feature Integration Theory, Jeremy Wolfe's Guided Search model, various saliency-based models of visual search, and the Contingent Capture Account of Chip Folk and Roger Remington. My own work shows that we indeed have a large amount of control over visual selective attention, as we can tune attention to sought-after objects which then quickly attract the gaze when they are present. There are however also bottom-up limitations to this goal-driven selection process that can completely frustrate our attempts to find an object.
For my work, I'm using a variety of different methods, including eye-tracking, EEG and fMRI.
Eye tracking (coloured contacts) EEG MRI scanner
1. The Relational Account
One of my most important findings so far was that we don't tune attention to specific feature values such as red/green, or a particular size or shape (as predicted by current models of visual search). Rather, attention is tuned to relative features of objects such as redder, greener, yellower, darker/brighter and larger/smaller. So attention operates in a context-dependent manner: The visual system computes how a target object would differ from all other objects in a given situation, so for example, it computes whether an orange target would be yellower or redder than the surround. Then, depending on how the target differs from the surround, attention is directed either to the reddest or yellowest item in the display. This account of attention has become known as the 'relational account'.
How do we select the goal keeper in these two pictures? According to the relational account, attention is biased to redder item when searching for the goal-keeper in the yellow team, and to yellower in the red team.
2. Top-down control or bottom-up stimulation?
One question that has kept vision researchers busy for quite a long time is the question how much attention is under voluntary control versus determined by attributes of the stimuli. This question has its problems, as visual processing is largely automated, including processes that occur when we have a goal (such as to find an orange ball). Although our goals are certainly under voluntary control, not all aspects of visual processing that occur while we're trying to find this ball are voluntary or perfectly controllable (I discuss this in a bit more detail in Becker, 2007, JEP-HPP and some of the feature priming work). So along with some other researchers, I have re-defined this question and now try to separate processes that occur because participants are following a certain goal (task-driven processes) from processes that occur because of the stimuli have certain characteristics (stimulus-driven processes).
A widely-held view is that objects with a high contrast can automatically attract attention and our gaze. For example, a red item among all-green items has a higher feature contrast than a green-blue item among all-green items, and consequently the red object would attract attention more strongly. In turn, my own research so far has shown that attention and eye movements are very 'programmable', in that such stimulus-driven effects are quickly overridden by our search goals.
There are some automatic processes, such as carry-over effects which come into play when the attributes of the search target change from trial to trial. Apparently, selecting the target with a particular color or shape primes us to select similar-looking items on the next trial, even when we know that the target could have a different colour or shape then. However, this priming effect has nothing to do with the attributes of stimuli in themselves (e.g., it has nothing to do with their feature contrast), but are driven by our memory. Moreover, these priming effects depend also on the task (and, by the way, priming also operates on relative colors, shapes, etc. rather than absolute features; e.g., Becker, 2013, AP&P).
So overall, we are quite good at ignoring objects that are not relevant to our current goals or tasks. This however does not mean that stimulus-driven factors don't exist -- only, that these effects can be difficult to find. Together with Gernot Horstmann, I have investigated whether feature contrast, intensity and other stimulus characteristics can attract attention when they are unexpected, and we found some evidence to support this view (see our work on "Surprise Capture"). However, once such a salient stimulus is expected and known to be irrelevant, it seems that our search goals largely override and neutralise its effects. Overall, it seems that stimulus-driven factors act like a passive limitation for visual selection rather than an active driving force -- except when they are unexpected and/or potentially informative -- but the jury is certainly still out on this.
3. Emotional expressions
A third main topic in my research concerns facial expressions and especially emotional facial expressions. Together with Gernot Horstmann and Roger Remington, I investigated whether and to what extent emotional expressions can attract attention and our gaze. While emotions are very possibly the most important driving factor in our decisions and actions, our research so far suggests that attention and eye movements are more strongly influenced by perceptual factors than emotional factors. For example, in 'normal' angry and happy schematic faces (left images), the angry face is found faster than the happy face. However, by changing the contour of the face, the results pattern reverses, so that happy faces can be found faster than angry faces.
Changing the contour does not change the emotional expressions. Hence, our explanation is that the 'normal' angry faces can be found faster because the happy faces have a better Gestalt and can therefore be grouped and rejected more easily when they are the distractors. So faster search for angry faces is not driven by the angry target face, but rather by the happy distractors that constitute the background.
One way to assess whether distractors can be grouped more easily is to limit the visible area in visual search to a small versus large region around the fixation (moving window technique). This allows assessing how many distractors are grouped in each instance (angry vs. happy faces), and the video below shows an example of that technique. (The red dot showing the actual eye position was not visible in the experiment.)
The fact that attention and eye movements are strongly influenced by perceptual factors does not mean that emotional expressions or our own emotional states will exert no effects on attention -- only, that these effects may be difficult to find. In more recent studies, we have kept the stimuli the same and actively manipulated the participants' mood to study how emotional factors influence attention and eye movements. Our results so far are quite encouraging.