Synaesthsia

Synaesthesia Course: Sensors in Mobile Systems

Student Name : Calum James BeckMatriculation Number : 40099589Edinburgh Napier University

Title : Synaesthesia

OVERVIEW /ABSTRACT

Synaesthesia is a system created to facilitate the traversal of small journeys for those of hard of sight to work in conjunction with a White Cane or Seeing Eye Dog. This system should allow users to avoid obstacles of waist height and alert them to moving obstacles within a distance of 3m.The system works by reading values from three ultrasonic sensors connected to an Arduino board and sends the values to an Android device to interpret the distance and play corresponding tones of varying speed and volume. This is then sent to a Bluetooth headset to allow for stereoscopic sound to alert the user as to where the objects are approximately located.This project endeavours to deliver a reliable solution to these objectives in a portable environment. A physical circuit incorporating an embedded system and an Android app are part of this solution. These are described herein.

Edinburgh Napier University Matric: 40099589


TABLE OF CONTENTSTABLE OF FIGURES.............................................................................................................................3

INTRODUCTION....................................................................................................................................4BACKGROUND/ CONTEXT.................................................................................................................................4AIMS & DELIVERABLES....................................................................................................................................5

CHAPTER 1.........................................................................................................................................6

DESIGN DELIBERATIONS AND COMPONENT ANALYSIS............................................................................6

1 INTRODUCTION..............................................................................................................................61.1 COMPONENT COMPARISONS..................................................................................................................6

1.1.1 Distance Detection: Infrared versus Ultrasonic............................................................................61.1.2 Object notification: Servo Motors Versus Audio...........................................................................71.1.3 Wireless Communication: Bluetooth.............................................................................................7

CHAPTER 2.........................................................................................................................................8

DESIGN OF THE SYNAESTHESIA...........................................................................................................8

2 INTRODUCTION..............................................................................................................................82.1 THE SYSTEM COMPONENTS AND THEIR FUNCTIONS................................................................................8

CHAPTER 3.........................................................................................................................................9

IMPLEMENTATION OF THE PROJECT......................................................................................................9

3 INTRODUCTION..............................................................................................................................93.1 SOFTWARE INTERFACE........................................................................................................................13

3.1.1 Code Listing................................................................................................................................153.1.1.1 Arduino Code.................................................................................................................................... 153.1.1.2 Android Code.................................................................................................................................... 15

3.2 INTERFACE TEST PLAN........................................................................................................................17

RESULTS..........................................................................................................................................25

CONCLUSIONS AND FUTURE WORK....................................................................................................25

EVALUATION OF ACHIEVEMENT..........................................................................................................26

REFERENCES....................................................................................................................................27

APPENDICES.....................................................................................................................................28APPENDIX A - INITIAL PROJECT PROPOSAL...................................................................................................28



BACKGROUND AND RATIONAL............................................................................................................29

DELIVERABLES..................................................................................................................................29



TABLE OF FIGURES

FIGURE 1 – CONCEPTUAL ILLUSTRATION OF IDEA 4

FIGURE 2 - SIMPLIFIED COMPARISON OF EMISSION BETWEEN IR AND ULTRASONIC7

FIGURE 1 - THE HARDWARE INTERFACE BLOCK SCHEMATIC 8

FIGURE 2 - INITIAL TESTING ENVIRONMENT 9

FIGURE 5 - WIRING DIAGRAM OF ARDUINO BOARD AND BREADBOARD 10

FIGURE 6 - FINAL WIRING IMPLEMENTATION 11

FIGURE 7 - SYSTEM IN PROTECTIVE BOX 12

FIGURE 8 - FULLY PORTABLE SYSTEM 12

FIGURE 9 - DEBUGGING AND FINAL APPLICATIONS 13-14

FIGURE 10 – HAND GUIDANCE SYSTEM IDEA 27



INTRODUCTION

BACKGROUND/ CONTEXT

In everyday commuting, it is easy to take our sight for granted when getting from A to B. With busy town centres in particular, there are always obstacles we move around without any thought and moving in and out of crowds is reliant on intuition rather than contemplating each move. For those hard of sight, this is not as simple.

‘Synaesthesia’ aims to help those who find themselves in this situation without being too bulky or intrusive to everyday travel. It is designed in order to help those in this situation and not to completely combat the problem, hence, it should be used in conjunction with a White cane but should allow pre-emptive movement by indicating obstacles within a 3m radius.

Figure 3 - Conceptual illustration of idea.

The system should allow those of hard sight to hear where objects are at waist height whilst still being vigilant of that on the ground thanks to the White cane. This system is also good for those with such conditions such as macular degeneration where the sight comes in pin hole views and it is hard for the mind to interpret all objects seen; ‘Synaesthesia’ provides a digital horizontal field of vision to aid those in this circumstance.



AIMS & DELIVERABLES

This project aims to build and document a system which will allow the user to perceive objects via sound in a stereoscopic environment from multi-directional origins in order to allow the user to prioritise direction and paths based on sound alerts. The main project will focus on building an environment to cater the conception, implementing a system to facilitate collecting the necessary data and a practical solution for processing this data.

As such, the final product will feature the appropriate equipment to enable implementation and will allow users to traverse paths and obstacles without the use of sight. It will feature a system to allow stereo sound to be manipulated to show proof of concept that an audio based system can allow those hard of sight to commute with a higher degree of ease, to create a feeling of sight through sound; Synaesthesia.



CHAPTER 1

DESIGN DELIBERATIONS AND COMPONENT ANALYSIS

1 INTRODUCTION In elaborating on the main concept behind the project, key areas were targeted and highlighted in order to create an adequate system fit for purpose. The driving forces behind this were data collection for distance, communication between devices and physical alerts for the user. All of this had to be facilitated in a portable environment as well so no wired instruments could be used independently without a method of wireless communication.

Hence, research was needed for a point of reference to compare and contrast the growing idea. On researching the subject of digital sight aids, other projects similar to the concept of this project. With similar traits, a project revolving around the white cane was created, allowing those with hard of sight to use their walking stick (white cane) to extend their range. With sensors fitted to the shaft and motors vibrating in different places, the user could feel the direction that obstacles lay. Dr. G. Prasanthi (2015) theorised that this would allow the user to interact with their surroundings with the least impact to their daily lives. From the standpoint of this project however, this approach would make the white cane much more bulky and would require modifiying users personal belongings [1]. This was felt to be a good idea, but outwith the scope of the project. It is important at this point to mention that the scope of the this project was to incorporate the use of sensors in mobile systems and to apply these to an idea rather than take existing objects and modify them..

In search for visual impairment aids there were many other applications that used a camera on a phone to track lines and help the user navigate a indoors where there is no GPS [2] [3]. This however uses advanced tracking and did not allow for a hands free experience seeing as one hand would be occupied by a White cane. Another project fitted ultrasonic sensors to a cap and projected from the head. This was good, but the range would surely be limited pointing at the ground. [4]

At this point, it was concluded that the system should be used around waist height using a proximity sensor with either vibration of sound as an indication of distance. This could all be inclusive as a standalone system or incorporate a larger processing power on the premise it would all still be portable.

1.1 COMPONENT COMPARISONS

1.1.1 DISTANCE DETECTION: INFRARED VERSUS ULTRASONIC

Both Infrared (IR) and Ultrasonic sensor are readily available for the build environment being used and are well documented on how to implement. When choosing to build a system which is designed to help those wishing to traverse obstacles relying on data procured by the device, then the accuracy of the device is incredibly important. Both Infrared (IR) and Ultrasonic sensors are readily available for the build environment being used so a comparison was made between both in order to attain the best option for this project.



Mustpha, Zayegh and Begg (2013) conducted a study into the uses of IR and ultrasound in an project with similar aim to this where the distance detection tools were interchanged to ascertain the ramifications of using each. Of the main points brought up, in order to use IR efficiently “knowledge of the surface properties must be known beforehand. In other words, the nature in which a surface reflects and absorbs infrared energy is needed to interpret the sensor output as distance measure” . [5] This restricts the ability to use the system in everyday circumstances but it should be mentioned that Ultrasonic is noted in the same article as being affected by mirrored surfaces as well as weather elements.

Despite Ultrasonic waves being affected by humidity and wind however, IR is severely hampered by harsh sunlight and can only be used at dusk and night outdoors. However, the preceding factor was the scope of the beams. Infrared only returns object in a precise direction whilst Ultrasonic spreads as it travels capturing a larger area.

Figure 4 - Simplified comparison of Emission between IR and Ultrasonic

1.1.2 OBJECT NOTIFICATION: SERVO MOTORS VERSUS AUDIO

In order for the user to receive feedback, a method of converting the digital data into physical feedback was needed. Two choices were narrowed down to a vibration effect against sound. Using servos means only having to build a small embedded system with no outside help whilst Audio would require a higher processing power. At this point, stereo was the more favourable choice for the developer purely to further their knowledge and build an android application to handle sound manipulation and audio streaming.

1.1.3 WIRELESS COMMUNICATION: BLUETOOTH

Using Bluetooth meant an easy connection procedure for the user and a large amount of readily available headsets to take advantage of. Connecting using a node would be simple and Bluetooth does not use heavy encryption which is ideal for



devices such as Arduino which have low memory capacities and processing power. This keeps processing overheads down resulting in less power consumption. As well as this, transferring sequential data from Ultrasonic sensors would be idea for Bluetooth, with the developer not expecting data transfer to exceed 1Mb/s.

From these conclusions, a basic design was created using ultrasonic sensors as the data source, a Bluetooth module as communication and an Android app to facilitate audio generation via a Bluetooth headset for the user.



CHAPTER 2 DESIGN OF THE SYNAESTHESIA

2 INTRODUCTION As previously discussed, the components selection was crucial to a working system. The component chosen are displayed in the block diagram below with the addition of an Arduino Uno board to connect needed sensors.

Figure 5 - The Hardware Interface Block Schematic

2.1 THE SYSTEM COMPONENTS AND THEIR FUNCTIONS

The idea was for the Ultrasonic sensors to collect data and pass it to the Arduino board which would convert the data to a ASCII value and broadcasts a serialised stream of the data using a Bluetooth (BT) Module. This part of the system would all be powered using a 12v battery.

The broadcast is then picked up by the Bluetooth receiver in the Android device. This would then in turn, create or play a sound that would correspond to the values the Ultrasonic sensors are producing (i.e. in relation to the distance). This sound



would then be played through the BT headset and, depending on which sensor sent the data, played through the correct stereo channel.

With this design, Synaesthesia conforms to the constraints placed by the conceptual design and was very close to the solution initially outlined in the Initial Project Proposal (documented as appendix C).

CHAPTER 3

IMPLEMENTATION OF THE PROJECT

3 INTRODUCTION The main components driving this system are the Arduino Uno embedded code and the Android device receiving the data. This meant a supplementary application was needed to process the data coming in. This started out only using two Ultrasonic sensors and the Bluetooth module.

Figure 4 - Initial Testing Environment

In the final environment, each ultrasonic sensor is connected to a ‘tiny breadboard’ to access 5v power pin and connect to the GND pin. They are then directly connected to pin on the Arduino board above the first two pins (pins 0 and 1). The Bluetooth module sits on the breadboard and is connected to the 3v pin and the other GND. This can be seen in the diagram below.



Figure 5 - Wiring Diagram of Arduino Board and Breadboard

The way the system works is as so: The data read in from the ultrasonic sensors and the Id of that sensor is added to the value (i.e. if the sensor has an Id value of 1, then a reading of 1cm is changed to 11). This is then sent over Bluetooth where the app is looking for a BT signal and establishes a connection with only specific devices corresponding to name of the BT module.

From here the app attempts to create an input stream and takes the current byte packets by Arduino and converts it into a string value. An if else statement is then run checking the value of the of the string. It then picks the first digit of the string and plays the corresponding tone depending on the distance from the Ultrasonic sensor. The initial number read corresponds to the channel output for the sound (e.g. left = left channel). The sound is then streamed to a Bluetooth headset to be heard by the user.



Figure 6 - Final Wiring implementation

Once the System was complete, it was compacted into a small box to protect components and then mounted in a portable bag for the user to carry it in. In order to avoid colliding waves from the sensors, they were mounted at angles of 20 degrees away from each other, the middle pointing forward.

Figure 7 - System in protective box



Figure 8 - Fully Portable System



3.1 SOFTWARE INTERFACE

The main software composes of an Android App that loads data from the Arduino board. Throughout development, a development app was used in conjunction with the Logcat console to debug the code as the project progressed.



Figure 9 - Debugging and Final Applications

The buttons correspond to Opening a connection to Bluetooth, Sending a signal to the Arduino Serial console, closing the Bluetooth connection and, in the debugger, clearing the white space below. The white space was a Text field which updated with the serial data being streamed. The ‘Serial Data’ label in the corner also displayed incoming data in both versions.



3.1.1 CODE LISTING

Full Code listings can be obtained on request. Some has been left in order to convey some aspects of how the system worked overall.

3.1.1.1 ARDUINO CODE

Pin setup example//Sonar 1

int echoPin1 =11;

int initPin1 =12;

int distance1 =0;

Setup codevoid setup()

{

pinMode(initPin1, OUTPUT);

pinMode(echoPin1, INPUT);





/*



*/

//Setup usb serial connection to computer

Serial.begin(9600);

// initialize all the readings to 0:

for (int thisReading = 0; thisReading < numReadings; thisReading++)

readings[thisReading] = 0;

//Setup Bluetooth serial connection to android

bluetooth.begin(115200);

bluetooth.print("$$$");

delay(100);

bluetooth.println("U,9600,N");

bluetooth.begin(9600);

}

Get Distance functionint getDistance (int initPin, int echoPin){

digitalWrite(initPin, HIGH);

delayMicroseconds(10);

digitalWrite(initPin, LOW);

unsigned long pulseTime = pulseIn(echoPin, HIGH);

int distance = pulseTime/58;



return distance;

}

3.1.1.2 ANDROID CODE

Setting up SoundPools for each stream //Set up streams for each feed

sp = new SoundPool(1, AudioManager.STREAM_MUSIC, 0);

sp2 = new SoundPool(1, AudioManager.STREAM_MUSIC, 0);

sp3 = new SoundPool(1, AudioManager.STREAM_MUSIC, 0);

//Add tones to Array for retrieval

spMap = new SparseArray<Integer>();

//left ear/Ultrasonic sensor

spMap.put(closest, sp.load(this, R.raw.closest, 1));

spMap.put(closer, sp.load(this, R.raw.closer, 1));

spMap.put(close, sp.load(this, R.raw.close, 1));

spMap.put(away, sp.load(this, R.raw.away, 1));

spMap.put(far_away, sp.load(this, R.raw.far_away, 1));

//middle ear/Ultrasonic sensor

spMap.put(closest_quiet, sp2.load(this, R.raw.closest_quiet, 1));

Listener for data stream workerThread = new Thread(new Runnable()

{

public void run()

{

while(!Thread.currentThread().isInterrupted() && !stopWorker) try {

int bytesAvailable = mmInputStream.available();

if (bytesAvailable > 0) {

byte[] packetBytes = new byte[bytesAvailable];

//Read packets

mmInputStream.read(packetBytes);

for (int i = 0; i < bytesAvailable; i++) {

byte b = packetBytes[i];

if (b == delimiter) {

byte[] encodedBytes = new byte[readBufferPosition];

System.arraycopy(readBuffer, 0, encodedBytes, 0, encodedBytes.length);

//Data received saved as a string

final String data = new String(encodedBytes, "US-ASCII");

//String converted to an integer value

final int numData = Integer.parseInt(data.replaceAll("[\\D]",""));

Play sound function public void playSoundLeft(int playTone) {

//Left Ultra

//Sets playtone so same value is not played twice

currToneLeft = playTone;



//Clears soundpool so new tone can play

clear(sp);

/*Gets value on method call as sets as playtone which corresponds

to value in spMap, left stream value (1 is full volume, 0 is off),

right stream value, priority value, repeat value, frequency of play

*/

sp.play(spMap.get(playTone), 1, 0, 1, 0, 1f);

}



3.2 INTERFACE TEST PLAN

A test driven development cycle was adopted throughout the whole project as it was classified as experimental to the developer, not having worked in a multi-platform environment before. Here is a log of the methodical process taken to develop the system.

Test 1 : Get distance from an Ultrasonic sensor and print in Serial Monitor in Arduino Console

Result:

(Pass/Fail)

PurposeTo understand the values from the sensors in order to use them in later development and test wiring was correct

MethodUse supplied example code from Arduino software to ensure the values of the sensors is coming through.

Steps include setting up Trigger & Echo pins and pinging device

Project Files Used : Test Filename(s) : ultrasonic_sensor.ino

Expected Result Output in Serial Monitor to reflect proximity to sensor

Actual Result Expected

Pass

Date of Test:

10/03/2015



Test 2 : Get distance from many Ultrasonic sensors and print in Serial Monitor in Arduino Console

Result:

(Pass/Fail)

PurposeAscertain the possibility of using multiple Ultrasonic sensors in the one system.

MethodAdapt supplied example code from Arduino software to allow more than one input source

Steps include adding another set of Trigger & Echo pins and pinging devices

Project Files Used : Test Filename(s) : mult-ultrasound.ino

Expected Result Output in Serial Monitor to reflect proximity on multiple sensors


Pass

Date of Test:

10/03/2015

Test 3 : Create method for feedback from different Sensors

Result:

(Pass/Fail)

PurposeAssociate sound with proximity to a sensor

MethodAdd piezo sound effects to readings less than 10 cm

Steps include adding piezo device to board and pin definitions as well as implementing if statement

Project Files Used : Test Filename(s) : mult-ultrasound.ino

Expected Result When output in Serial Monitor drops below 10cm, reflect proximity using sound effect


Pass

Date of Test:

10/03/2015



Test 4.1 : Connect bt module to ‘tiny breadboard’ Result:

(Pass/Fail)

PurposeWire BT module to board with Ultrasonic sensors connected

MethodAdd BT module to board and show power. Connected to pin 0

Expected Result BT module is auto configured so should show up on other devices scan

Actual Result Error: avrdude: stk500_getsync(): not in sync: resp=0x00

Cannot upload code to Arduino

Fail

Date of Test:

12/03/2015

Test 4.2 : Connect bt module to ‘tiny breadboard’ Result:

(Pass/Fail)

PurposeWire BT module to board with Ultrasonic sensors connected

MethodAdd BT module to board and show power. Changed pin connections

Expected Result BT module is auto configured so should show up on other devices scan


Pass

Date of Test:

12/03/2015



Test 5 Use Bluetooth Module to connect to Bluetooth Headset

Result:

(Pass/Fail)

PurposeStart a connection Headset to send rudimentary sounds

MethodPair Bluetooth module with headset and send system beep to headset

Project Files Used : Test Filename(s) : btConnect.ino

Expected Result Sound played

Actual Result Could not send sound using serial bluetooth module

Fail

Date of Test:

20/03/2015

Test 6 : Communicate via Bluetooth module to Android device and vice versa

Result:

(Pass/Fail)



PurposeStart a connection with Android device to ensure data can be transferred by sending data from Arduino monitor.

MethodSend a message from Arduino to Android and vice versa

Project Files Used : Test Filename(s) : btConnect.javaBT_and_Ultra.ino

Expected Result ‘Hello’ displayed in text field and in serial monitor


Pass

Date of Test:

22/03/2015

Test 7 : Stream Serial data from UltraSonic sensors to Android Device

Result:

(Pass/Fail)



PurposeAssure that sending multiple values is possible from

MethodPrint to Bluetooth stream rather than Serial monitor


Expected Result Stream of numbers appearing on Android text field


Pass

Date of Test:

22/03/2015

Test 8.1 : Trigger Sound on reading change in Android using AudioManager

Result:

(Pass/Fail)

PurposePlay sound based on data from Ultrasonic sensors.

MethodUse if statement to play different method depending on data in


Expected Result Sound played changes depending on data in from sensors

Actual Result Sound repeatedly plays closest association sound

Fail

Date of Test:

02/04/2015



Test 8.2 : Re attempt Trigger Sound on reading change in Android using Soundpool

Result:

(Pass/Fail)

PurposePlay sound based on data from Ultrasonic sensors.

MethodSame as before but preload sounds into Soundpool instance


Expected Result Sound played changes depending on data from sensors

Actual Result Sound played once for each range identified

Note: Changed array to Sparse array for faster read times

Pass

Date of Test:

02/04/2015

Test 9 : Implement audio structure to read from Array, loop sounds and manipulate frequency and volume to create constant alert

Result:

(Pass/Fail)



PurposeChange sound depending on distance

MethodUsed sound pool parameters to gain effect

Project Files Used : Test Filename(s) : ‘Arduino_App’ Main_Activity.java

BT_and_Ultra.ino

Expected Result Sound played changes depending on data in from sensors and manipulated used Android processing

Actual Result Sound clashed and loaded to slowly whilst repeatedly playing over each other

Fail

Date of Test:

05/04/2015

Test 10 : Stop audio stream collisions in looping sounds by pausing audio stream

Result:

(Pass/Fail)

PurposeUse AudioManager to control Soundpool and pause when needed


BT_and_Ultra.ino

Expected Result Sound played changes depending on data in from sensors and manipulated used Android processing and would pause to allow change

Actual Result Sound clashed less than before but data was still lost when streaming to Bluetooth headset

Pass

Date of Test:

10/04/2015

Test 11: Increase delay on Arduino board when reading Ultrasonic sensors to allow Android to keep up with

Result:

(Pass/Fail)



changing events

PurposeLoops were loading logic to slow to cope with data coming in

MethodUse messages in logcat to ascertain the frequency at which the audio streams were being played as audiostreams were different from audio id

Expected Result The value of streams would increment once per loop.


Pass

Date of Test:

10/04/2015

Test 12 : Manipulate .ogg files and read data incoming sequentially to reduce collisions and play sounds at different intervals for easier directional recognition

Result:

(Pass/Fail)

PurposeTo speed up the process, do not use Android function to change the tone, pitch and frequency, but load in pre-set sounds to speed up overall system

MethodA program called ‘Audacity’ was used to engineer tones of different volume, tone and frequency

Pass

Date of Test:

17/04/2015

Test 13 : Play two sounds at once using Soundpool Result:

(Pass/Fail)



PurposePlay all different tones through head set to create stereo sound depending on input

MethodUsed sound pool parameters, change the channel depending on input and change clip which played depending on distance


BT_and_Ultra.ino

Expected Result Sound played changes depending on data in from sensors and is played through correct ear piece

Actual Result Sounds came out in correct ear and correct files were played but frequency was off and volume needs calibration.

Pass

Date of Test:

18/04/2015

Iterative Test : User Testing Result:

(Pass/Fail)

PurposeTo ascertain whether the system is fit for purpose

MethodUse system in everyday use

Expected Result Stereo sound would yield an intuitive solution to object awareness

Actual Result Sound did help, but was deemed slightly confusing.

Fail

Date of Start:

10/04/2015

Date of Completion:

24/04/2015



RESULTSFrom the results of implementation and iterations through the design, the concept is sound as far as the idea goes. Using ultrasonic sensors to gauge distance is a viable solution for those making small systems in conjunction with Arduino. The main implementation of the physical components was also easily understandable with a basic knowledge in electrical circuitry and pin handling.

Streaming serialised data from digital sources over Bluetooth works well and is efficiently fast, with precautions taken in order to manage the speed the data is read in. It was found during the late testing stages that (if incorrectly logically driven), the sound files saved on the android would rapidly try to replay with the data coming in twice as fast at the loop could handle.

Once understood though, it can be concluded that the audio features built into Android are fit to play multiple tones at once through different channels but more research is needed on the subject. With a more in depth development of the audio files, a system could be built providing a more intuitive reaction to obstacles.

CONCLUSIONS AND FUTURE WORK

After many amendments and tests, it was concluded that although the structure technically works and demonstrates the possibility of a successful system, it was unfortunately not fit for purpose. The ultrasonic sensors worked reliably in short distances and the sound cues are understandable when the user expects the result produced, but using the system in the initial practical sense it was designed for is not intuitive enough for the general user.

This stems from the sound cues and the multiple streams being delivered at once. This was the initial idea of the system (a stereo feed leading to a depth of sound) but in testing, it was discovered that directly using 3 sources of sound was not sufficient enough. It only creates a pseudo-surround sound and is very much a stereo feed in the most literal sense. This was the initial goal of the project but humans learn to sense sound in a fully 3-D environment so hearing sounds from only 3 sources is slightly confusing. If more ultrasonic sensors were used (reaching around the back of the user as well as the front), a better effect could be realised, but significant audio manipulation would have to take place dynamically, fading sounds into one another to reach the desired effect. In the initial concept of the system (see Appendix C), an additional Android device was not considered but after implementing the system, it would be favourable for a much longer Android development window.

As stated, the current system is very good for short distances and on a much smaller scale than that of the outside world. As such, using an ultrasonic sensor to aid picking up objects or navigating small rooms could prove much more successful. One idea would be to limit the amount of Ultrasonic sensors to one and use it as a hand navigator and may be good for those who adjusting to sight loss or in recovery of sight.



Figure 10 - Hand Guidance system idea

As stated, using one sound is much easier to understand as well as implement and debug. This idea could also be applied to those who use their phones a lot whilst walking.



EVALUATION OF ACHIEVEMENT

As far as this project goes, there are positive aspects and negative; the final product being of the latter. However, in terms of learning, this project has yielded many good results. I believe that I underestimated the amount of work that would go into software development leading to a less than perfect outcome in my eyes. Making an Android application to work in conjunction with the system was never in the original scope but was a great learning experience nonetheless.

Building the systems physical components was simple enough and fairly straight forward. It was an enjoyable and immensely rewarding experience for myself when components worked and communicated with each other, and certainly has intrigued me in terms of future development. This was ultimately a bad aspect to the project however as it created a false sense of security in terms of the difficulty of the implementation of the Android side. In hindsight, the application is not difficult to develop however, there are certainly components that just need to be known in order to progress (such as Bluetooth methods and audio streaming) and information found on sound producing methods for Android were scarce and, at times, conflicting. I have attempted a Bluetooth App in the past and did not succeed to have it working so I was very happy when saw the fruition of my work over the past couple of years, This all contributed the learning experience and have been beneficial for future endeavours.

Although it was forewarned when entering this project that time management would be critical, I believe I underestimated the overall implementation time. I would also change the final product next time into a soldered version and if it is a system I wish to use for myself and not just for proof of concept, I would invest in a smaller board, Having the wireless breadboard was great for development but is not as stable as soldered components.

Ultimately, I am very happy with what I learned and the direction the project took towards the end. I would say to get the product I had envisioned, a lot more time would have to be applied over months to give it a professional feel. I believe I attempted too much for a first project with a lack of understanding of the work involved. Despite this, I am glad I went ahead with the project and am happy with the outcome despite it not creating a wave of synesthetes.



REFERENCES

[5] A. Z. R. K. B. Baharuddin Mustapha, First International Conference on Artificial Intelligence, Modelling & Simulation, 2013. [Online]. Available: http://ijssst.info/Vol-15/No-2/data/3251a439.pdf. [Accessed 4 April 2015].

[3] arXiv, “MIT Technology Review,” 8 January 2014. [Online]. Available: http://www.technologyreview.com/view/523401/app-turns-smartphone-into-virtual-cane-for-the-blind/. [Accessed 25 March 2015].

[1] P. Dr G. Prasanthi, “Sensor assisted stick for the blind people,” Transactions of Engineering and Sciences, 2015.

[4] Reuters - Innovations Video Online, “Wearable sensor 'illuminates' surroundings for the blind,” 10 December 2014. [Online]. Available: http://www.sciencedaily.com/videos/40068be4cc3ee850f30614d96c7431b8.htm.

[2] S. H. S. M. Lisa Ran, “DrishtiL An integrated indoor/Outdoor Blind Navigation System Service,” Computer & Information Science & Engineering Department University of Florida, Gainesville, 2005.



{ new page}

APPENDICES

APPENDIX A - INITIAL PROJECT PROPOSAL

An Initial Project Overview of a system which helps people with sight problems preemptively identify risks in their path through the use for Stereo sound



BACKGROUND AND RATIONAL Whilst working with Ultrasonic sensors, I started to think of the ramifications of using such a technology to help those that did not have the innate ability to perceive distance. Whilst my grandfather was in his later years, he suffered from Macular Degeneration leading to a loss of sight and visual awareness. At the time, I had thought to create a device that could help him commute with more confidence but in a discrete manner.

With this in mind, I aim to create a system of Ultra sonic sensors connected to an Arduino board using a 9v battery to power the system; to allow for portable usage. The aim will be to connect to a Bluetooth head set to the system and use the sensors to gauge distance and create stereo sounds to alert those with sight difficulties how far an object lies ahead in their path.

In researching the project, it would appear that I not the first to attempt such a system. Whilst searching I found a system with the same idea at heart. Instead of having sensors on the person, this group decided to modify a walking stick to vibrate and move in the direction needed (Dr G. Prasanthi, 2015). The problem I could see with this is that it was too bulky and apparent. Other ideas included a belt, a similar system with audible sounds created by a speaker system (Reuters - Innovations Video Online, 2014) and a system that recorded GPS signals to relay voice commands through a headset (Lisa Ran, 2005).

The system I intend to create attempts to make the aid discreet whilst creating a fully functional guidance system for standalone use. This way, those that need help will be able to use a system without feeling like they are some sort of attraction to others walking around them. By making a stereoscopic sound effect, I intend to allow those with sight difficulties a better grasp of their surroundings as a whole, rather than have a system which gives commands as to where permanent hazards are, but not temporary or moving.

DELIVERABLES I intend to create a system which helps those with visual problems identify risks in their path or at hand. This is going to be accomplished using stereo sound via a Bluetooth headset.

The system will first send out ultrasonic waves which are then used to record the distance travelled until returned. The system will then convert that distance into a level of sound or frequency of a sound and transmit that to a Bluetooth headset. With multiple sensors pointing in different directions worn at the chest area at the front, a stereo sound should be formed for the user to allow a pseudo 3-D area in which to understand. This is the main directive of the project.

Another recommendation is to allow for another single senor which will be attached to the hand or wrist (not unlike the USTRAAP system1). Using a button on the Bluetooth headset, the user will be able to switch to this one sensor system and survey objects in close proximity (i.e. picking items off a table). This can be a system on its own but would be highly beneficial to include in this build.

The system will be predominantly built in C++ to cater for the Arduino environment. Testing in the system will include the range sensors (Ultrasound), latency issues with sound, multiple input on same frequency and user testing to ensure the system is helpful for those it’s aimed at. The additional hardware requirements are listed at the end of the Method section.

This system must not interfere too much with the sound around the person so must be discreet and unobtrusive. In order to achieve this, many different sounds, sound levels and frequencies will be applied during the build. The system must be

1 https://nfb.org/blog/atblog/ustraap-system

http://www.fastcompany.com/company/ustraapEdinburgh Napier University Matric: 40099589

https://nfb.org/blog/atblog/ustraap-system


portable and easy to use, so once the hardware and software have been distinguished and proven to be sound, then designing the system to be confortable worn will be important.

It is to be noted that this system is intended to be used as a mediator to allow preemptive movement and not to be completely relied upon for unassisted travel. A cane or walking aid should be used so as not to miss lower lying objects.



METHOD In order to complete this project and gauge how much can be achieved within the give timeframe, a time line of milestones has been created in order to depict what aspects should be completed by what time. This is documented below;

Week 1 o Create relational sound to distance using ultrasound and Bluetooth headset

Create a way to set frequency of sound, or sound level depending on distance input Distinguish Bluetooth protocol needed for Bluetooth Audio Module 2

Create system which works with one Ultrasonic Sensor Establish list of additional components not noted in IPD needed for full design

Week 2 o Create basic build with Arduino for multiple Ultrasound sensors and battery operation

Review angles for Ultrasound devices so as not to interfere with one another Begin programming for stereo sound based on multiple inputs and Bluetooth connectivity

Week 3 o Update system to use more than one Ultrasound device to gauge three distances at onceo Attempt to mount system on person

Review wiring and board positioning Week 4

o Create algorithm which uses relational sound from three sensors to create a stereo sound or 3d at best, 1 week

Continuous o Testing – Iterative

Extra – Program on and off button and additional sensors for handso Use Bluetooth buttons to change between hand and walking systems

Preliminary work can be conducted in the lab such as programming distance input and programming needs. However, for the final product, a new system will be built comprising of the following

Arduino , Bluetooth audio module, Bluetooth headset, ultrasound proximity sensors (x4), 9v battery adapter, soldering iron, breadboard, cables, small case for board

Although many piece of equipment are standard, a picture of the Bluetooth headset is placed below to give a better idea. In this example, the cord between the headphones works especially well so the user can remove the set but not misplace. The buttons on the side are hardcoded to give a standard signal to phones and mp3 players to move songs, but for this research project, can be used to turn the sounds on and off in the system.

2 Research had already started on components at this time - http://www.electrodragon.com/product/pkb-pcba-bluetooth-stereo-audio-module-for-music-speaker/


Synaesthsia

Devices & Hardware

Transcript of Synaesthsia