Multimedia Systems 2011/2012 Perception Prof. Dr. Paul Müller University of Kaiserslautern Department of Computer Science Integrated Communication Systems ICSY http://www.icsy.de
Outline Multimedia and Sense Enabling Multimedia Sense of Vision Sense of Hearing Other Senses 2
Sitemap 3
Perception without Multimedia Systems 4
Senses and their Artificial Replacements Vision Audition Olfaction Gustation Vestibular 5
Multimedia and Sense Sense of Mode of Sense Display vision hear smell taste balance "5-sense" visual auditory olfactory gustatory vestibular optical acoustic - - - pressure vibration cold warmth pain skin touch tactile haptic position power proprioception kinesthetic 6
Perception with Multimedia Systems 7
Enabling Multimedia Not all signals of the real environment have to be reproduced. only those signals which cause the same perception in the sense organs as the original signals must be reproduced. Limited resolution of the human sensory system artificial signals only have to be transmitted in a proper resolution. Limitation of intensity Stimuli exceeding a certain limit of intensity cause damage The system of senses is widely modular. For example, text only needs pictures; a speaker needs voice and picture 8
Limitations of Sense Organs All human sensory systems have a limited spectrum of perception. Signals beyond this spectrum cannot be perceived or lead to damage of the sense organ. Examples: Audio: sound between 20 and 20.000 Hertz Visual: electromagnetic waves between 380 and 720 nm Vibration: oscillations between 10 and 500 Hertz The precondition to realize multimedia systems is digitization of the environmental signals 9
Example: digitization of sound tone = physical phenomena vibration of material compression spread out around the material approaching the ear = perceiving tone pattern of compression is a wave waves reiterate in regularly intervals = period (T) amplitude = absolute value of the highest intensity 10
Example: digitization of sound A A = Amplitude T =period 11
Example: digitization of sound periodic tones = sound of wind non-periodic tones = sound of water sound analog signal = infinite number of values continuous curve of tone direct digital representation signal digitization => limited number of values 12
Example: digitization of sound steps to digitize: sampling, quantization, encoding measurement of the amplitude per time step = sample capturing of continuous signal in a discrete interval 13
Example: digitization of sound (a) time-continuous signal (b) sequence of 256 samples 14
Example: digitization of sound inverse value of period = frequency frequency = number of periods per second (Hz) measurement of sampling in Hertz Nyquist-Shannon theorem: A signal with a maximum frequency f max has to be sampled with a minimal frequency of f a =2*f max to allow an accurate reconstruction of the original signal. signal must be sampled twice as often as highest frequency 15
Nyquist-Shannon-Theorem maximum frequency f max = 150 Hz sample with minimal frequency of f a =2*150 Hz = 300 Hz Amplitude 1 ------ s 100 Time 16
Nyquist-Shannon-Theorem enables correct reconstruction violation of the theorem: Oversampling Disadvantages: no quality gain, more storage space & higher data rates are required Undersampling Signal cannot be reconstructed 17
Nyquist-Shannon-Theorem quantization: Example: The voice signal in the telephone system is limited to 4.000 Hertz. With a sampling rate of 8.000Hz (i.e. one sample each 125µs) and a quantization of 8 bit we get 64 kbit/s: 8.000 (1/s) x 8 bit = 64.000 bit/s = 64 kbit/s 18
Quantization continuous values into discrete values 19
Sense of Vision interior surface of eyeball - approx. 65% - retina photosensitive cells in retina - rods and cones - photoreceptors convert incident light energy into signals signals send via optic nerve to brain middle of the retina = blind spot 20
Rods more numerous (110-125 million) - more sensitive than cones responsible for dark-adapted or scotopic vision scotopic vision no color vision light response peak in blue, minimum response in red 21
Cones 6-7 million less sensitive to light responsible for high resolution vision color vision - photopic vision blue (~2%) highest sensitive cones located outside blind spot red (~66%) and green (~32 %) sensitive cones mainly located in the blind spot 22
Cones blue red and green sensitive cones at about 445nm, 535nm, and 575nm. 23
Spectral Sensitivity of Brightness photopic vision = rod + cones vision scotopic vision = rod vision 24
Dynamic of See Sense Candela per square meter 25
Range of Visibility 26
Chromatic Aberration distinct vision = focusing light in blind spot blue light refractive index different from red and green blue = short wavelength red = long wavelength => blue lightly out of focus rays with different wavelengths focused at different positions at the retina 27
Chromatic Aberration maximum of energy of the red and blue phosphor of a screen are relatively apart from each other optical depiction error by blue-red contours error is called chromatic aberration amount of chromatic aberration depends on dispersion of glass (or eye) 28
Chromatic Aberration 29
Chromatic Aberration refraction power of the eye wavelength (nm) color description refraction difference 687 dark red +0.34 <-- 656 medium red +0.25 cannot be reference point --> 589 yellow 0 focused 527 yellow-green -0.30 simultaneously 486 blue -0.58 431 blue-violet -1.07 <-- 30
Red/Green-System: Search for Food Find the fruit faster than your competitor View of a blue/yellow-being. 31
Red/Green-System: Search for Food Find the fruit faster than your competitor View of a blue/yellow/red/green-being. 32
Invariant of Lighting The light check in the shadow is the same gray as the dark checks outside the shadow. 33
Invariant of Lighting 34
Knowledge versus Shadow Videoclip Source: Max Planck Institute for Biological Cybernetics 35
Artificial Retina http://www.seeingwithsound.com/etumble.htm 36
Sense of Hearing eardrum receives vibrations traveling up the auditory canal and transfers them through the tiny ossicle to oval window eardrum fifteen times larger than oval window of inner ear The tympanic membrane is very thin, about 0.1 mm, but it is resilient and strong. 37
Sense of Hearing 16.000 receptors (hair cells) in a human inner ear 3.500 inner cells 12.000 20.000 outer hair cells hearing range 20 Hz to 20.000 Hz number of fibers in auditory nerve 28.000 detection threshold 10 12 watts/m 2 Dynamic range 120 db (12 orders of magnitude) 38
Human Aural Properties Human ear can detect sounds between ~20Hz and 20kHz: Total audio range: 5Hz - 50kHz 39
Hearing Range 40
Decibel decibel (db) is a common unit of measurement for the relative loudness of a sound or, in electronics, for the relative difference between two power levels. decibels measure ratio of given intensity I to threshold of hearing intensity I 0, so that this threshold takes the value 0 decibels (0 db). 41
Decibel In sound, decibels measure a scale from the threshold of human hearing, 0 db, upward towards the threshold of pain, about 120-140 db. 42
Audio-Frequency Limitations 43
Dynamic of Hearing rocket launching 180 wind channel 160 jet take off 140 threshold of pain 150 rifle fire 130 propeller-aircraft take off 120 subway 100 110 permanent damages Niagara falls 90 damages old vacuum cleaner 80 traffic noise 70 chat between two persons 60 quiet restaurant 50 residential district by night 40 empty cinema 30 sound of leaves in the wind 20 breath 10 threshold of hearing 0 Values in db 44
The Physics of Acoustics Sound can be considered in one of two ways: Fourier: in the time domain in the frequency domain transformation is accomplished by a Fourier transform Any waveform can be created by a series of sine waves summed together 45
Other Senses Sense of smell e.g. "telesmell", see article in "Spiegel" about smell in movie theaters Electronic Nose Analysis of smells Synthesis of smells 46
Other Senses Sense of Touch mechanics of electronic touch devices are neither purely electronic nor simply mechanical, they're both. they translate digital information into physical sensations push on a mouse or a joystick, the device pushes back using magnetic actuators and sensors built into the device technically, this process is called force feedback resistance is only one of the hundreds of sensations like springs, liquids, textures, vibrations... 47
Other Senses Sense of taste Sense of balance (e.g. balance board) Vestibulum in the inner ear is sensor for balance, register changes of position in 3D, stimulated by electric signals One can simulate senses, as long as it can be translated into a mathematical equation 48
Perception with Multimedia Systems 49
Interesting Links (1) MIT encyclopedia of cognitive science http://cognet.mit.edu/ Mark Newbold's Animated Necker Cube http://dogfeathers.com/java/necker.html Perception http://psych.la.psu.edu/clip/perception.htm Artificial Retina Development http://www.nsf.gov/news/frontiers_archive/7-97/7retina.jsp 50
Interesting Links (2) The Human Body's Senses: Hearing Theme Page http://www.cln.org/themes/hearing.html The Human Body's Senses: Sight Theme Page http://www.cln.org/themes/sight.html The Human Body's Senses: Smell Theme Page http://www.cln.org/themes/smell.html The Human Body's Senses: Taste Theme Page http://www.cln.org/themes/taste.html The Human Body's Senses: Touch Theme Page http://www.cln.org/themes/touch.html 51
Interesting Links (3) That's Tasty http://faculty.washington.edu/chudler/tasty.html Touching... http://sln.fi.edu/qa97/me10/me10.html 52
Interesting Links (4) Retina reference http://retina.anatomy.upenn.edu/~lance/retina/retina.html Rotating mask and other visual illusions http://www.kyb.tuebingen.mpg.de/bu/demo/index.html Optical illusions and visual phenomena http://www.michaelbach.de/ot/index.html 53
Prof. Dr. Paul Müller Integrated Communication Systems ICSY University of Kaiserslautern Department of Computer Science P.O. Box 3049 D-67653 Kaiserslautern Phone: +49(0) 631 205-2263 Email: pmueller@informatik.uni-kl.de