Term Paper On

The Heidelberg tactile vision substitution system

Submitted in the partial fulfillment of the requirement for the award of degree of

Bachelors of Technology In Electronics & Communication Engineering

Submitted by: PANKAJ SAHU Roll no A27 Reg. Number: 10907659

Department of Electronics & Comm. Engg Lovely Professional University

ACKNOWLEDGMENT

History of all great works is to witness that no great work was ever done without either active or passive support of a person surrounding and ones close quarters. Thus is it not hard to conclude how active assistance from senior could positively impact the execution of a project .I am highly thankful to our learned faculty Mr. Parambir Singh for his active guidance throughout the completion of project Last but not least , I would also want to extend my appreciation to those who could not be mentioned here but have well played their role to inspire me behind the curtain.

Contents

Introduction VTD - Virtual Tactile Display Data input and output Tactile Vision Substitution Systems Tactile Perception The PTD Brain Plasticity and Sensory Substitution in Human Blindness SenSub (QLG3-CT2000-01797) Result Reference

1.Introduction
Since the pioneering work performed by Bach-y-Rita et al. several different technologies have been explored to construct devices which are able to provide blind subjects with a substitution for vision making use of their tactile perception. Such devices are called tactile vision substitution systems (TVSS). In general three classes of stimuli, mechanotactile, electrotactile and thermotactile, have been exploited. A comprehensive overview including also applications of tactile displays for sighted people in the context of virtual reality can be found in . A common problem for all tactile displays is the necessity to generate tactile patterns over a rather large geometrical surface. This turns out to be a challenge for mechanical design, electrical power dissipation and portability of such devices. The attempt described in this paper uses a concept calledvirtual tactile display (VTD). Here, a relatively small active area of 48 tactile pixel elements (taxels) is moved by the user across a large surface (164 mm159 mm). The tactile sensation is received with the fingertips. Only the small active area requires mechanical elements, supply of electrical power and computer control. The system is therefore simple and can be produced at low cost. In the following we describe the technical design of the VTD, the different options of connecting data sources and first results obtained with the system comparing groups of sighted and blind subjects.

2. The Virtual Tactile Display (VTD)

A common problem for all tactile displays is the necessity to generate tactile patterns over a rather large geometrical surface. This turns out to be a challenge for mechanical design, electrical power dissipation and portability of such devices. Our group has developed a concept called 'virtual tactile display' (VTD). Here, a relatively small active area of 48 tactile pixel elements (taxels) is moved by the user across a large surface (164 mm x 159 mm). The tactile sensation is received with the fingertips. Only the small active area requires mechanical elements, supply of electrical power and computer control. The system is therefore simple and can be produced at low cost.

The tactile output device of the project is the VTD (Virtual Tactile Display) which has been developed and tested in the context of a thesis. The preprocessed data of the vision chip or image data of external sources such as graphics, internet, etc. can be presented in form of an elevated pattern on a virtual area, 2880 pixels in size, by means of the VTD. Braille modules, commonly used for reading at personal computers for blind persons are used to form the tactile pattern. Only that part of the image that is currently underneath the fingers of the observer is displayed in order to achieve a high resolution.

In a comparative examination with 6 blind and 4 seeing subjects it has been investigated by means of geometric patterns to what extent it is possible to recognize the presented pattern by tactile perception. We have recorded the movements during palpation and analyzed them in order to find out whether there are any similarities in pattern recognition between tactile and visual sensory channels. By comparing the two groups one will be able to see whether the loss of the visual channel causes pattern recognition to be transfered to tactile perception.

A photograph of the VTD used in the Heidelberg TVSS is shown in figure 1. An aluminium base unit holds two parallel linear guiding rails with carriages moving on ball bearings (Type KUME 09 manufactured by INA Wlzlager,Herzogenaurach, Germany). A third identical rail is mounted perpendicular on the two parallel ones so that the carriage on the single rail can be moved with very little friction in x and y directions independently. The carriage on the single rail holds the tactile output unit which consists of 6 standard piezoelectric Braille display elements. The Braille elements (manufactured by METEC GmbH, Stuttgart, Germany) have 8 movable plastic actuators arranged in a 24 matrix. Each individual actuator is mounted on a pair of piezoelectric levers providing vertical movements of up to 0.7 mm. The piezocrystals require an operation voltage of 200 Volts which is generated locally in the desktop base unit by a DC/DC converter. The spacings between actuators are3.21 mm and 2.45 mm in x and y-direction

respectively. The tactile output field consists in total of 48 actuators distributed in an area of 43 mm 16 mm which can be conveniently

Figure 1 : Photograph of the VTD. The 3 rails and the tactile output unit are visible. The base unit is equipped with a laptop computer for control tasks. The photograph also shows a CMOS vision chip camera used together with the VTD to form a complete TVSS. covered with 3 fingertips . The absolute position of the tactile output unit is recorded with an optical sensor taken from a standard optical mouse (manufactured by Mouse Systems Corp., Fermont, USA). The tactile output unit can slide across an area of 164 mm 159 mm providing access to 2600 virtual taxels (see definition above). The active surface of the tactile output unit corresponds to only 2% of the full accessible area. Table 1 summarises the mainproperties of the VTD. Number of active taxels 6 8 Number of virtual taxels 2880 Active surface area 43 mm x 16 mm Virtual surface area 164 mm 159 mm Distance between active taxels 3.21 mm (x) and 2.45 mm (y) Vertical taxel movement 0.7 mm Weight without computer 2.25 kg Mechanical properties of the virtual tactile display (VTD)

3.Data input and output

The VTD receives image data to be displayed and sends position data of the tactile output unit. The position data are of twofold importance for the VTD. They provide the information to the laptop computer to calculate the correct pattern to be displayed in the active taxel field and they can be recorded on mass storage. The second feature is used for offline analysis of the detection strategies developed by subjects using the device. The image data are sent froma laptop computer via a standard 24 bit digital I/O PCMCIA card (Type DIO-24 manufacturedby National Instruments). The card is also used to control additional external equipment like cameras. The position data recorded by the optical mouse system as well as the state of the three mouse buttons are sent to the serial port of the computer. A software package has been developed to control data input/output and to perform image processing calculations on the laptop computer. Image data can be provided by an external camera, via a TCP/IP link by another computer (e.g. Internet) or by bitmap images stored on the local computer. The bitmap images can also be assembled to form a video movie in order to simulate motion of objects. The software package includes a set of standard image processing filters in order to match the image information to the capabilities of the VTD and of the human user. The maximum update rate of images transferred to the VTD is approximately 60 Hz. This number is reduced when external devices (cameras) are connected or extensive image processing is performed on the laptop computer. The VTD is part of the complete Heidelberg TVSS which includes CMOS cameras for real world image recording [3 - 7] as well as analog image pre-processing circuits in VLSI implemention . The VLSI chips for image acquisition and control are dedicated developments in the framework of the TVSS development.

4.Tactile Vision Substitution Systems

Since the pioneering work performed by Bach-y-Rita et al. [Nature 221 (1969)] several different technologies have been explored to construct devices which are able to provide blind subjects with a substitution for vision making use of their tactile perception. Such devices are called tactile vision substitution systems (TVSS). In general three classes of stimuli, mechanotactile, electrotactile and thermotactile, have been exploited. A comprehensive overview including also applications of tactile displays for sighted people in the context of virtual reality can be found in the nice book by G.C. Burdea : Force and touch feedback for virtual reality, John Wiley & Sons, Inc., New York (1996). Modern microelectronics offers the fascinating opportunity to develop intelligent vision chips with properties adapted especially to the specific application. The image processing can be done by means of commercially available fast digital computers or analog networks. Nature applies the analog way in the retina. The reduction of pixelinformation to relevant parameters (e.g. edges) happens before the information is transfered to the brain. The Heidelberg TVSS project has focused on the development of vision chips and analog network chips. In order to optimize the necessary image processing an improved understanding of tactile perception is required. There are virtually no flexible and programmable output devices for tactile patterns available on the market. The Heidelberg group has therefore developed two tactile displays. One system, the VTD, has been successfully tested with students of the school for blind persons in Ilvesheim. Another one is being used in cooperation with the group of Prof. Sampaio of the Universit Louis Pasteur in Strasbourg for basic research in the field of tactile sensory perception. This pneumatic display (PTD) is operated with air pressure and can thus be used in the strong field of an fMRI magnet.

5.Tactile Perception
Functional Magnetic Resonance Imaging (fMRI) is a method of brain imaging. Briefly, a local measurement of the oxygen consumption gives information about activities in certain areas of the cortex (BOLD effect). The spatial resolution of fMRI measurements amounts to a few millimeters. Applying an ultrafast fMRI method known as 'echo-planar imaging' (EPI) acquires one image of the brain (containing more then 20 slices) in 2 seconds. The measurements take place in a strong static magnetic field (~2 Tesla) with an alternating electromagnetic field superimposed on it. It is therefore difficult to supply sensory stimulation. The operation of electromechanical and electronic systems is not possible in a magnetic field of this strength as such systems would also influence image acquisition. The Heidelberg group has developed an especially designed pneumatic tactile display (PTD), which can project tactile patterns on the skin of a proband. The patterns are generated with a dedicated software which is easy to control. The PTD has a modular structure so that different surface can be set up. Each module consists of 4 taxel (tactile elements).

Properties of the PTD:

Max. Size: 8x8 taxels Modular (E.g. flat or concav) Pitch of two taxels: 10mm Diameter of taxel: 4mm Max. frequency: 6Hz

structure

The PTD is currently being used for tactile stimulation in an fMRI environment as well as in a psychologists' lab for training purposes. The fMRI system is located in the Institute de Physique Biologique at the Universit Louis Pasteur in Strasbourg. Investigations are focussed on the processing of tactile perception.

The actual studies aim in the exploration of brainplastisity by sensory substitution. For this aim a studies with blind subjects and a sighted control group are carry out. The paradigms are designed to presented "visual like" tactile stimuli with the PTD. Three fMRI enquiries should demonstrate the variances in perception during training. To compare the results with the above study and for a deeper matter in the kind of perception with the PTD a study with 12 native sighted subject is carry out. During this event-related fMRI experiment the subject has to distinguish between patten with differen properties. Both study are done in the framework of the SenSub-project. moving versus static

random versus directed

difficult versus easy

One finished experiment focus on the examination of the shape of the tactile hemodynamic response in the human brain. Furthermore, a second experiment probes for dependencies on vibratory tactile stimulation in the frequency range of 2 to 6Hz. In previous experiments the activation induced from stimulation of index finger, arm and foot has been localized in the primary and secondary somatosensory cortex. Mapping of the somatotopic representation in the cerebral cortex which is called sensory homunculus has been performed in both primary and secondary cortex. For these investigations a smaller version of the PTD, consisting of one module (4 taxels), is mounted on the appropriate spot of the proband. We chose a walking pattern which alternates with rest phase. Each phase consists of 10 images and takes 40 seconds (one image of the brain takes here 4 seconds). All in all 200 images are recorded for each experiment.

6.The PTD

The PTD establishes tactile contact with the skin by a compressed air-driven piston. Thus, only air-tubes reach into the magnet of the scanner where the non-electronic tactile display unit is

located during the experiments. An electro-pneumatic interface (EPI), located several meters away from the center of the magnet, regulates the pressure timing . A trigger-unit synchronizes the timing of the display unit with the fMRI trigger. A serial bus based on a CAN -protocol connects the EPI and the trigger-unit to a control computer outside the Faraday cage. Via the control computer the stimuli can be manipulated interactively or a prepared script can be launched by the dedicated tactile vision substitution system control software (TVSSCS). The display unit requires a working pressure of 3-6 bar, which is provided by an external compressor and a reduction valve to reduce the primary pressure. The system set-up is shown schematically in Figure.

The PTD is able to generate highly reproducible tactile stimuli on a 2-dimensional matrix and can therefore be utilized for vision substitution by tactile perception, in instances where only low resolution is necessary, e.g. for edge detection or simple motion processing. It is fully automated and can be controlled by dedicated software which allows the generation of spatial and temporal complex patterns

Owing to its modular structure the display unit can be mounted onto most parts of the human body, e.g. abdomen, arm or foot.

7.Brain Plasticity and Sensory Substitution in Human Blindness SenSub (QLG3-CT-2000-01797)

A major question in the domain of sensory substitution is to understand the kind of percept generated by sensory substitution devices: is it similar to the one generated by a natural sensory modality (visual, tacile, auditory) or is it a totally new kind of perception? According to the sensory substitution theory, perceptual processing related to these substitutive stimuli should result in a visual-like perception. The main tasks of this project constists in analyzing: (a) the ongoing brain changes and (b) the links between sensory and congnitive brain representations, due to the utilization of a perceptual modality that is totally artificial and new to the subjects. To attain this objective, experimental psychology, neuroimaging and biomedical engineering techniques for sensory processing will be implemented.

Results
First tests of the VTD have been carried out with a group of 6 blind and 4 sighted subjects. All subjects have been informed about the nature of the device and the test procedure and haven given their free consent to participate in the study. The sighted subjects were in the age range between 21 and 30 with an average of 28 years. Two of them were females and two were males. The sighted subjects were blindfolded during the tests. The blind subjects were all males and in the age range between 16 and 25 with an average of 18 years. Five of the blind subjects had no remnant visual capabilities. One of them was able to recognise large intensity variations. Two of the blind subject lost their vision during the first 5 years of their lives. The other four blind subjects were congenitally blind. After an introduction to the VTD technology and operational principles the subjects were given the task to recognise 21 simple geometrical images stored in the data base of the computer. The images represented simple geometrical patterns like squares, triangles, circles and crosses with different sizes and orientations. No time limitation was given for the recognition of the images. All users were able to recognise all images correctly. The individual movement atterns of the tactile output unit was recorded and captured on mass storage for later offline analysis. Details of the test procedure as well as all test images are described in [10]. A first analysis of the offline data has demonstrated the functionality of the VTD and beyond this revealed some remarkable differences in performance and strategy of blind and sighted VTD users. The recording of space-time coordinates allows to quantitatively evaluate the users performance. The most obvious parameters in this aspect are recognition time and average moving velocity. Table 2 summarises the performance numbers for the two groups of blind and sighted subjects.

Roughly speaking, the blind subjects were able to recognise the images more than twice as fast as the sighted subjects. Their moving speed is almost a factor 2 higher than that of sighted subjects. This fact can also be seen in the correlation plot between recognition time and moving velocity shown in figure . The individual data points line up on a hyperbola in the time versus velocity diagram. This demonstrates the fact that both groups use the same effective scan path but with half the recognition time for blind subjects. In the language of information theory this means that the bandwidth for using the VTD is twice as high for blind subjects compared to sighted subjects. First studies have also been performed to better understand the recognition strategies whenusing the VTD. A single example is shown in figure 4 were the scan paths for a triangle areplotted for a blind and a sighted subject.

Figure : Pattern recognition strategy with the VTD comparing a blind and a sighted subject.

It is evident that the sighted subject follows the line pattern very carefully whereas the blind subject tends to make movements which are weakly correlated with the pattern to be recognised. The data recorded in this study have the potential to understand this feature in more detail by not only looking at the average performance but by analysing individual recognition strategies differentially in space and time. It is hoped that these studies will provide new insights into the pattern recognition capabilities of the blind, which in turn can be used to further improve the VTD concept or other tactile displays.

Reference [1] Bach-y-Rita, P., Collins, C.C., Saunders, F., White, B., Scadden, L. Vision substitution by tactile image projection Nature 221 (1969) [2] G.C. Burdea Force and touch feedback for virtual reality John Wiley & Sons, Inc., New York (1996). [3] M. Loose, K. Meier, J. Schemmel Self-calibrating logarithmic CMOS image sensor with single chip camera functionality IEEE Workshop on CCDs and Advanced Image Sensors, Karuizawa, Japan, R27 (1999). [4] M. Loose, K. Meier, J. Schemmel CMOS image sensor with logarithmic response and self calibrating fixed pattern noise correction International Symposium on Electronic Image Capture and Publishing Advanced Focal Plane Arrays and Electronic Cameras, T. M. Bernard, Editor, Proc. SPIE 3410 (1998). [5] M.Loose, K.Meier, J.Schemmel Entwicklung einer Kamera mit adaptiven Photorezeptoren in analoger CMOS Technologie Informatik aktuell, Mustererkennung 1996 B.Jhne et al., Editors, Proc. 18.DAGM Symposium, Springer Verlag, Berlin/Heidelberg (1996). [6] M. Loose, K. Meier, J. Schemmel A Camera with adaptive Photoreceptors in analog CMOS Technology International Symposium on Advanced Imaging and Network Technologies - Advanced Focal Plane Arrays and Electronic Cameras, T. Bernard, Editor, Proc. SPIE 2950 (1996). [7] M. Loose, K. Meier, J. Schemmel