Multimedia Systems 2011/2012

Transcription

1 Multimedia Systems 2011/2012 Perception Prof. Dr. Paul Müller University of Kaiserslautern Department of Computer Science Integrated Communication Systems ICSY

2 Outline Multimedia and Sense Enabling Multimedia Sense of Vision Sense of Hearing Other Senses 2

3 Sitemap 3

4 Perception without Multimedia Systems 4

5 Senses and their Artificial Replacements Vision Audition Olfaction Gustation Vestibular 5

6 Multimedia and Sense Sense of Mode of Sense Display vision hear smell taste balance "5-sense" visual auditory olfactory gustatory vestibular optical acoustic pressure vibration cold warmth pain skin touch tactile haptic position power proprioception kinesthetic 6

7 Perception with Multimedia Systems 7

8 Enabling Multimedia Not all signals of the real environment have to be reproduced. only those signals which cause the same perception in the sense organs as the original signals must be reproduced. Limited resolution of the human sensory system artificial signals only have to be transmitted in a proper resolution. Limitation of intensity Stimuli exceeding a certain limit of intensity cause damage The system of senses is widely modular. For example, text only needs pictures; a speaker needs voice and picture 8

9 Limitations of Sense Organs All human sensory systems have a limited spectrum of perception. Signals beyond this spectrum cannot be perceived or lead to damage of the sense organ. Examples: Audio: sound between 20 and Hertz Visual: electromagnetic waves between 380 and 720 nm Vibration: oscillations between 10 and 500 Hertz The precondition to realize multimedia systems is digitization of the environmental signals 9

10 Example: digitization of sound tone = physical phenomena vibration of material compression spread out around the material approaching the ear = perceiving tone pattern of compression is a wave waves reiterate in regularly intervals = period (T) amplitude = absolute value of the highest intensity 10

11 Example: digitization of sound A A = Amplitude T =period 11

12 Example: digitization of sound periodic tones = sound of wind non-periodic tones = sound of water sound analog signal = infinite number of values continuous curve of tone direct digital representation signal digitization => limited number of values 12

13 Example: digitization of sound steps to digitize: sampling, quantization, encoding measurement of the amplitude per time step = sample capturing of continuous signal in a discrete interval 13

14 Example: digitization of sound (a) time-continuous signal (b) sequence of 256 samples 14

15 Example: digitization of sound inverse value of period = frequency frequency = number of periods per second (Hz) measurement of sampling in Hertz Nyquist-Shannon theorem: A signal with a maximum frequency f max has to be sampled with a minimal frequency of f a =2*f max to allow an accurate reconstruction of the original signal. signal must be sampled twice as often as highest frequency 15

16 Nyquist-Shannon-Theorem maximum frequency f max = 150 Hz sample with minimal frequency of f a =2*150 Hz = 300 Hz Amplitude s 100 Time 16

17 Nyquist-Shannon-Theorem enables correct reconstruction violation of the theorem: Oversampling Disadvantages: no quality gain, more storage space & higher data rates are required Undersampling Signal cannot be reconstructed 17

18 Nyquist-Shannon-Theorem quantization: Example: The voice signal in the telephone system is limited to Hertz. With a sampling rate of 8.000Hz (i.e. one sample each 125µs) and a quantization of 8 bit we get 64 kbit/s: (1/s) x 8 bit = bit/s = 64 kbit/s 18

19 Quantization continuous values into discrete values 19

20 Sense of Vision interior surface of eyeball - approx. 65% - retina photosensitive cells in retina - rods and cones - photoreceptors convert incident light energy into signals signals send via optic nerve to brain middle of the retina = blind spot 20

21 Rods more numerous ( million) - more sensitive than cones responsible for dark-adapted or scotopic vision scotopic vision no color vision light response peak in blue, minimum response in red 21

22 Cones 6-7 million less sensitive to light responsible for high resolution vision color vision - photopic vision blue (~2%) highest sensitive cones located outside blind spot red (~66%) and green (~32 %) sensitive cones mainly located in the blind spot 22

23 Cones blue red and green sensitive cones at about 445nm, 535nm, and 575nm. 23

24 Spectral Sensitivity of Brightness photopic vision = rod + cones vision scotopic vision = rod vision 24

25 Dynamic of See Sense Candela per square meter 25

26 Range of Visibility 26

27 Chromatic Aberration distinct vision = focusing light in blind spot blue light refractive index different from red and green blue = short wavelength red = long wavelength => blue lightly out of focus rays with different wavelengths focused at different positions at the retina 27

28 Chromatic Aberration maximum of energy of the red and blue phosphor of a screen are relatively apart from each other optical depiction error by blue-red contours error is called chromatic aberration amount of chromatic aberration depends on dispersion of glass (or eye) 28

29 Chromatic Aberration 29

30 Chromatic Aberration refraction power of the eye wavelength (nm) color description refraction difference 687 dark red < medium red cannot be reference point --> 589 yellow 0 focused 527 yellow-green simultaneously 486 blue blue-violet <-- 30

31 Red/Green-System: Search for Food Find the fruit faster than your competitor View of a blue/yellow-being. 31

32 Red/Green-System: Search for Food Find the fruit faster than your competitor View of a blue/yellow/red/green-being. 32

33 Invariant of Lighting The light check in the shadow is the same gray as the dark checks outside the shadow. 33

34 Invariant of Lighting 34

35 Knowledge versus Shadow Videoclip Source: Max Planck Institute for Biological Cybernetics 35

36 Artificial Retina 36

37 Sense of Hearing eardrum receives vibrations traveling up the auditory canal and transfers them through the tiny ossicle to oval window eardrum fifteen times larger than oval window of inner ear The tympanic membrane is very thin, about 0.1 mm, but it is resilient and strong. 37

38 Sense of Hearing receptors (hair cells) in a human inner ear inner cells outer hair cells hearing range 20 Hz to Hz number of fibers in auditory nerve detection threshold watts/m 2 Dynamic range 120 db (12 orders of magnitude) 38

39 Human Aural Properties Human ear can detect sounds between ~20Hz and 20kHz: Total audio range: 5Hz - 50kHz 39

40 Hearing Range 40

41 Decibel decibel (db) is a common unit of measurement for the relative loudness of a sound or, in electronics, for the relative difference between two power levels. decibels measure ratio of given intensity I to threshold of hearing intensity I 0, so that this threshold takes the value 0 decibels (0 db). 41

42 Decibel In sound, decibels measure a scale from the threshold of human hearing, 0 db, upward towards the threshold of pain, about db. 42

43 Audio-Frequency Limitations 43

44 Dynamic of Hearing rocket launching 180 wind channel 160 jet take off 140 threshold of pain 150 rifle fire 130 propeller-aircraft take off 120 subway permanent damages Niagara falls 90 damages old vacuum cleaner 80 traffic noise 70 chat between two persons 60 quiet restaurant 50 residential district by night 40 empty cinema 30 sound of leaves in the wind 20 breath 10 threshold of hearing 0 Values in db 44

45 The Physics of Acoustics Sound can be considered in one of two ways: Fourier: in the time domain in the frequency domain transformation is accomplished by a Fourier transform Any waveform can be created by a series of sine waves summed together 45

46 Other Senses Sense of smell e.g. "telesmell", see article in "Spiegel" about smell in movie theaters Electronic Nose Analysis of smells Synthesis of smells 46

47 Other Senses Sense of Touch mechanics of electronic touch devices are neither purely electronic nor simply mechanical, they're both. they translate digital information into physical sensations push on a mouse or a joystick, the device pushes back using magnetic actuators and sensors built into the device technically, this process is called force feedback resistance is only one of the hundreds of sensations like springs, liquids, textures, vibrations... 47

48 Other Senses Sense of taste Sense of balance (e.g. balance board) Vestibulum in the inner ear is sensor for balance, register changes of position in 3D, stimulated by electric signals One can simulate senses, as long as it can be translated into a mathematical equation 48

49 Perception with Multimedia Systems 49

50 Interesting Links (1) MIT encyclopedia of cognitive science Mark Newbold's Animated Necker Cube Perception Artificial Retina Development 50

51 Interesting Links (2) The Human Body's Senses: Hearing Theme Page The Human Body's Senses: Sight Theme Page The Human Body's Senses: Smell Theme Page The Human Body's Senses: Taste Theme Page The Human Body's Senses: Touch Theme Page 51

52 Interesting Links (3) That's Tasty Touching

53 Interesting Links (4) Retina reference Rotating mask and other visual illusions Optical illusions and visual phenomena 53

54 Prof. Dr. Paul Müller Integrated Communication Systems ICSY University of Kaiserslautern Department of Computer Science P.O. Box 3049 D Kaiserslautern Phone: +49(0)