ACOUSTIC VISION –
Acoustic Perception Based On Real Time
Video Acquisition for Navigation Assistance
Supreeth K Rao#, Arpitha Prasad B*, Anushree R Shetty&, Chinmai$ , Rajeshwari Hegde@
#,*,&,,

Department of Telecommunication Engineering, @ Guide and faculty
BMS College of Engineering,
Bangalore, India
#

supreethkrao@gmail.com arpithaprasad@gmail.com & anushree.shetty12@gmail.com $ cpchinmai@gmail.com *

Abstract—
A smart navigation system based on an object detection mechanism has been designed to detect the presence of obstacles that immediately impede the path, by means of real time video processing. This paper is discussed keeping in mind the navigation of the visually impaired.
A video camera feeds images of the surroundings to a
Da-Vinci Digital Media Processor, DM642, which works on the video, frame by frame. The processor carries out image processing techniques whose result contains information about the object in terms of image pixels. The algorithm aims to select that object, among all others, that poses maximum threat to the navigation. A database containing a total of three sounds is constructed. Hence, each image translates to a beep, where every beep informs the navigator of the obstacles directly in front of him. This paper implements a more efficient algorithm compared to its predecessor, NAVI.
Keywords— Navigation, Edge Detection, Flood Function,
Object Detection, DM642, Acoustic Transformation

I. INTRODUCTION
Assistance for the blind or visually impaired can range from simple measures, such as a white cane or a guide dog, to a very sophisticated computer technology (enhanced imaging, synthetic speech, optical character recognition, etc.). Many of those who are visually impaired can maintain their current employment or be trained for new work with the help of such aids.
This paper deals with a vision substitution system that is based on an image to sound conversion concept. This finds particular applications for the

navigation of the visually impaired and even in the case of autonomous intelligent rovers. The output of this system can either be fed as an actuation to a smart control system, or converted to an audio signal which is fed to the blind person‟s earphones. The paper aims at creating a portable system that allows visually impaired individuals to travel through familiar and unfamiliar environments without the assistance of guides.

II. PROPOSED ALGORITHM
A vision acquisition device such as a video camera captures the information about the system surroundings. Image frames are procured from the video and subjected to a series of image processing techniques. Figure 1 shows the block diagram describing an overview of the algorithm used for the system implementation. The algorithm entrusts significant weightage to the location and size of the objects in the image under consideration. A flood function has been designed to calculate the same.
These parameters are then used to assign priorities to the objects based on proximity and size. The object that has gained the highest priority is selected and depending upon its proximity and size, acoustic transformation is performed resulting in one out of two sounds. This sound is what gives the ultimate intended information about the surroundings that helps the user to have a collision free navigation.
Object recognition has been implemented to inform the user about what obstacles are present before him, rather than just their presence.

VIDEO ACQUISITION

the resolution). The above has been selected keeping in mind the data loss in resizing.

IMAGE EXTRACTION

B. Edge detection:

RESIZING [32x32]

RGB COMPONENT EXTRACTION
RED

GREEN

BLUE

EDGE DETECTION (CANNY)

COMBINE EDGES [LOGICAL OR]

DILATION

FILL

EROSION

FLOOD FUNCTION

The objects present in an image are recognized by their boundaries. Edge detection is a technique that extracts the edges of the objects. There are various types of edge detection: sobel, canny, prewitt, laplacian to name a few, and the algorithm is friendly to both canny and sobel. The image that is acquired is a colour image consisting of red, green and blue
(RGB) components [1]. Colour image segmentation offers greater accuracy when compared to edge detection in grayscale images [2]. Hence, we extract the RGB colour components of the resized image and perform edge detection on each. The results are combined by performing a logical OR operation on the three edge detected images to obtain a single binary image with clearly defined objects whose outline is in white.
C. Dilation
Morphological processing involves operations that process images based on shapes. They apply a structuring element to the input image that suitably altering it. Dilation and erosion are the two morphological operations that the algorithm uses.
Dilation has been used to connect broken edges in the edge detected image. A structural element of 2x3
„one‟s is appropriately chosen to dilate every white pixel in the binary image. This leads to thickening of the edges, hence connecting minor breaks.

OBJECT PREFERENCE

D. Fill:
ACOUSTIC TRANSFORMATION

AUDIO SIGNAL

Fig. 1: Proposed Algorithm
III.

METHODOLOGY

The proposed methodology makes use of the following standard preprocessing techniques:
A. Resizing
The image captured is resized to 32x32. This is mainly to achieve the real time processing prerequisite of small computational time, but it also provides flexibility of changing the camera (and thus

As the size of the object has to be calculated, the area within each object must be obtained i.e., the number of pixels constituting the object. The image as of now consists of objects outlined in white. The fill function, when applied to the above image, fills in the black area within objects with white pixels. It can thus be seen that the object size is the number of white pixels it contains.
E. Erosion:
Dilation adds pixels to the boundaries of objects in an image, while erosion removes pixels on object boundaries. While this sharpens the dilated objects, it also removes unwanted white specks, which would otherwise be viewed as small objects themselves.
The structural element here is a disk with radius one.

F. Proximal Area
The location of the objects is an important aspect for collision free navigation. The presence of objects in certain parts of the image poses greater danger to the navigator than their presence in other parts. To understand this, let us divide an image into four parts, left, right, centre, back, as shown in the figure(2) below.
Back

L e f t C

R

to the navigator. Therefore, we can say that this region of the image should be given higher preference when compared to the other regions in the image. To classify the objects based on their location, the frame has been divided into three regions (Fig. 4), two of which constitute the high priority region –
Proximal area consisting of A1 & A2 (Fig.2).
Within the Proximal area, objects present in A1 are assigned the highest priority, while those present in
A2 are assigned a lower priority.

32x32

16x16

Fig. 2: Analysis of the image frame
A2

Consider the following cases in Fig. 3 with the assumption that the user is walking straight ahead:

8x16
A1

Fig. 4: Proximal Area

Fig. 3: Selection of Proximal Area

Case (a): An object is in the left(L) portion of the image and it is apparent that this object does not cause any obstruction to the navigator.
Case (b): The object in the right(R) portion of the image. This again will not obstruct the path of the person. Case (c): Here, the object is at the back portion of the image. As we can see, the user can walk a short distance before he encounters this object.
Case (d): In this figure, however, where the object is present in the central portion of the image, we can see that the object causes an immediate obstruction

This paper assigns preferences to A1 and A2 through the use of masks M1 and M2 consisting of ones. M1
A1
is a mask of dimension 24x16 highlighting the objects in the areas A1 and A2. M2 is a mask of dimension 8x16 highlighting the objects in A1only.
Using these masks, the following operations are performed: M1 = Image & M1; (objects in A1 and A2 are highlighted) M2 = Image & M2; (objects in A1 alone are highlighted) Image = Image + M1 + M2;
This results in the image consisting of ones, twos and threes in the regions outside the proximal area, in A2 and in A1 respectively.

G. Flood Function
The objects in the image consist of ones, twos and threes; hence, calculating the object size would be to calculate the total number of ones, twos and threes.
Also, the concentration of objects (which is the number of pixels) in A1 and A2 have to be calculated. This would mean counting the number of twos to calculate object concentration in A2, and threes to calculate the object concentration in A1.
To calculate the size of the objects present in the frame, and its concentration in A1 and A2, a flood function is designed.

The image is first searched for a one, two or three, and the flood function is called when these pixel intensities are encountered. The function called at a pixel spreads to its neighbours if its pixel intensities are not zeros. The function has been defined for a connectivity of eight, i.e, all eight neighbours of each pixel is checked for either a one, two or three. In effect, when a flood function is called at a pixel of an object, then it spreads to cover the entire object till it reaches the object‟s boundaries. During flooding, pixel count can be incremented (which would be the object size) and also, the number of twos and threes can be counted (which would mean the object concentration in A2 and A1 respectively). In order to prevent the function from flooding into already explored pixels, the intensities of those pixels where the function has been called are changed to zero.
Therefore, not only does the function count the required values, it also shrinks the object into inexistence. This has no consequences as all necessary information has already been gathered.
The scanning continues, and the procedure is repeated as and when other objects are encountered.
A database containing object sizes and concentrations is thus created.
H. Priority Assignment

Objects falling in the Proximal Area should be given high priority. Not only should this been done, but importance should be given to the object size.
Consider the following conflicting cases:
Case
1: If there are two objects lying in A2, the larger object should be given higher priority. Case 2: If there is a large object in A2 and a smaller object in
A1, as the user encounters the object in A1 first, objects in A1 should be given more priority than those in A2, regardless of the difference in size.
Case 3: If a small percentage of a huge object lies within A1and A2, and an object of smaller size lies only within A2, then that object whose concentration within A1 or
A2 is more gains higher priority.

user continues on his path and (iii) no sound, which informs that there is no object yet that can pose a threat. IV.

HARDWARE IMPLEMENTATION

The system has used the Digital Media Processor
TMS320DM642 (Version 3), which belongs to the
Da Vinci family of Texas Instruments‟ C6000 series.
The DM642 Evaluation Module (EVM) is a lowcost, high performance video & imaging development platform designed to jump-start application development and evaluation of multichannel, multi-format digital and other future proof applications. DM642 has been specially designed for real time video and audio processing, with dedicated video encoders and a decoder. Leveraging the high performance of the TMS320C64x DSP core, this development platform supports TI‟s
TMS320DM642, DM641 & DM640 digital media processors. The TMS320C64x™ DSPs (including the TMS320DM642 device) are the highestperformance fixed-point DSP generation in the
TMS320C6000™
DSP platform. The
TMS320DM642 device is based on the secondgeneration high-performance, advanced VelociTI™ very-long-instruction-word (VLIW) architecture
(VelociTI.2™) developed by Texas Instruments (TI), making these DSPs an excellent choice for digital media applications. DM642 offers a speed of
720MHz, 4 MBytes Flash, 32 MB of 133 MHz
SDRAM and 256 kbit I2C EEPROM[3].The JTAG emulator used to communicate with the processer is
XDS510USB Plus. A PAL (Phase Alternating Line) camera has been utilised to acquire the input (at 30 frames per second) to the system and a set of 3.5 mm jack ear-phones are used to provide the output of the system to the user.

I. Acoustic Transformation

Each object by now would have gained a level of priority. Among all objects, that object which has gained the highest priority must have its presence conveyed to the blind user. Thus, the algorithm translates the object into an audio signal, resulting in a beep. All priority levels are categorized into three sounds: (i) a sound of high pitch indicating the presence of an object posing greatest threat and implying that the user must immediately change his direction (ii) a sound of medium pitch indicating that an object would soon pose the greatest threat if the

Fig. 5: Hardware Implementation using DM642
V.

RESULTS

This section shows the results for a sample image: Fig. 6: Image Extraction & Resizing

assignments then take place to categorize detected objects based on the amount of threat they pose; and finally, an acoustic transformation is performed to translate the visual information to a meaningful sound. This is executed by the Da Vinci Digital
Media Processor, DM642, in half a second, and the real time system‟s execution dives again to the first step and the procedure is carried out repeatedly.
Neuroscience and psychology research indicate recruitment of relevant brain areas in seeing with sound, as well as functional improvement through training. However, the extent to which cortical plasticity allows for functionally relevant rewiring or remapping of the human brain is still largely unknown and is being investigated in an open collaboration with research partners around the world[4]. In effect, this paper is translating one sense organ‟s experience to be understood by another which eventually gets used to the user, owing to the neural plasticity of the human brain. The visually impaired people can now “see” by listening to the output of this algorithm.

VII.
Fig. 7: Image processing

A1:

9

0

0

0

0

A2:

105

0

0

0

0

Objsize:

114

9

0

0

0

Priority:

115

0

0

0

0

Highest priority: 115
Fig. 8: End results considering a maximum of five objects VI.

CONCLUSION

Acoustic vision is a sensory substitution system
(vision) that acquires, processes, analyses, and understands images from the real world and ultimately aims to provide a synthetic vision through sound, relevant to obstacle detection based navigation. Here, image extraction from a video input is carried out; boundaries of objects present in the image are identified and the objects‟ sizes are calculated. Importance is given to the size and proximity of objects through well-defined iris areas.
The flood function counts the size of objects and their concentration in these iris areas. Priority

REFERENCES

NAVI: An Improved Object Identification for NAVI,
Nagarajan R, Sainarayanan G, Yacoob S, Porle R.R,
TENCON IEEE Region 10 Conference, Volume A
[1] A Color Edge Detection Algorithm in RGB Color
Space
Soumya Dutta, Bidyut B. Chaudhuri
[2] International Conference on Methodsand
Modelsin ComputerScience,2009
Comparative Study of Image Segmentation
Techniques and Object Matching using
Segmentation
S Sapna Varshney, Navin Rajpa and Ravindar
Purwar
[3] Texas Instruments DM64x Digital Media
Developer's Kit
[4] Sensory substitution and the Human – machine interface, Paul Bach-y-Rita and Stephen W. Kercel
[5] TMS320DM642 Evaluation Module with TVP
Video Decoders – Technical Reference, 2004
Spectrum Digital.
[6] Driver Examples on the DM642 EVM – Texas
Instruments- Application Report, SPRA932 – August
2003.