Audio feature: Oh, I See

Seeing with your ears.

An audio feature I produced over the summer for Pod Academy, exploring the development of the vOICe technology and it’s impact on blind users. The vOICe is a computer program developed by dutch engineer Dr Peter Meijer which essentially converts images into sound. Through training and experience blind users can learn to interpret these sounds as a sort of ‘synthetic vision’. The piece explores the technology from the perspective of blind user Pat Fletcher, and uncovers some of the science and technology behind its use with it’s creator Dr Peter Meijer and cognitive psychologist Dr Michael Proulx (University of Bath).

It was my thought that technology and the computer would be my way out of blindness.

-Pat Fletcher, vOICe user

Download it HERE

Pat Fletcher
Blind user Pat Fletcher uses the vOICe with video glasses, stereo headphones and a portable notebook computer (Image: Barbara Schweizer )

Essentially, the software takes spatial information captured by a camera and converts this into a coded soundscape. Users can then learn how to decode this auditory signal into a visual one thanks to a process known as ‘sensory substitution’, where information from one sense is fed to the brain via another. Fundamentally what the vOICe is doing is re-routing information usually obtained by the eyes and delivering it through another sense organ, the ears.

Although the neuroscience and psychology behind the technology is still largely unknown, it is thought that the visual cortex is eventually recruited to process the incoming auditory information and through experience, is able to decode it as spatial / visual information. There’s a great article over at New Scientist that goes into greater depth about the neuroscience behind it  – including a useful diagram depicting how the technology works.

The software is currently freely available and can be used with virtually any imaging device, from webcams to camera-mounted glasses – there’s even an android version available for mobile devices! With the increasing prevalence of mobile computing, the vOICe technology is liberating users from their blindness, allowing them to step outside and experience the world through a completely new visual perspective.

For more information visit: http://www.seeingwithsound.com/ where you can experiment with the vOICe for youself and learn more about how it works. I’ve also prepared a page with a collection of images as heard through the vOICe software, including some featured within the piece above.

Music

  1. Hypermagic – Start Again Start
  2. Ed Prosser – Untitled
  3. – – b31
  4. No Color – L’Aube
  5. Hpermagic – Pico Bisco
  6. Ed Prosser – Untitled
  7. Marcel Pequel – Four

Freesound Credits (freesoundarchive.com)

  1. Alarm Clock – 14262__xyzr-kx__alarm-clock
  2. Camera Shutter – 16071__heigh-hoo__nikonf4
  3. Data sound – 3647__suonho__futuretrocomputing-10-suonho
Advertisements

3 thoughts on “Audio feature: Oh, I See

  1. I agree with everything in the script. I myself like to begin a demonstration, in a darkened room, facing the laptop screen camera.. There is sufficient glow while The vOICe is running to count the fingers held up , and , importantly, with white paper or pencil, to demonstrate how position, size and slope are delivered. If a cheapo webcam is plugged in, party tricks like picking up a dropped sugar lump-behind the chair -without- looking can raise a smile. Kids love to show how easy it is to pick up the rules.
    This deliberate, calculating,aimed approach delivers data and rules to the brain. Later, the full-screen soundfield experience will somehow segue the process onwards. I suppose it must be like learning musical scales before conducting the orchestra.
    Give it a try, you can`t lose by the experience.
    Incidentally, the PC hardware platform is overdue an overhaul. The Raspberry-pi board is surely the emulator platform to choose. Anybody interested?

  2. Well, now in November 2015… The Raspberry- pi A and B have been programmed (as Linux platforms) and have barely shown capable of working The vOICe.
    The ODROID-C is an exciting little board, using the ANDROID platform. The vOICe for Android loads nicely, boots and runs well. However there are problems with distortion of the sound output. This is explained as a result of the very intense processor activity during a small part of the cycle, and the bottlenecks created by multiple USB connections. However, as a proof of concept demonstrator of the principles of portability and economy it has been very effective.
    The CHINAVASION mini-sports 2.4 inch screen cellphone uses Android and delivers The vOICe reliably.
    However, it has to rely upon touch screen operation , no real keyboard… But when worn on a headband it delivers good performance with only earbuds, no extra cables. Again, a reliable and cheap one-piece demonstrator.
    The Colour sensing facility can deliver surprising impacts where the user , having had and lost sight, regains perception of the colours of fabrics, individual threads, can maybe analyse family photographs… you name it. Great scope for experiment and development.
    Any ANDROID phone will run with The vOICe. However their size, value and lack of external camera reduce their appeal for rough usage.
    THE RASPBERRY-PI 2 has arrived. This board will definitely deliver The vOICe in full. Currently great progress is being made in improving the speed and stability, and a 3D printed case with simple knob-and-button controls has been prototyped.
    The battery packs now available offer hours of operation.
    At the time of posting.. the Raspberry-pi ZERO ultra cheap board has arrived.
    Back to the drawing board, perhaps?
    Sites of interest… Raspberry-VI….www.raspberryvi.org
    The vOICe for Android – Google Groups

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s