Anda di halaman 1dari 59

ELEC 4170

Digital Media and Multimedia Applications


Lecture 1 Overview and the Start of our Media Journey
Prof. James She (james.she@ust.hk) 1

About me
Positions: ! Assistant Professor, ECE, HKUST ! Founding Director, HKUST-NIE Social Media Lab., ECE, HKUST. ! Research Fellow, Computer Laboratory, U. of Cambridge, UK (onleave) Research interests: 1) Cyber-physical Social Media Systems/ Networks 2) Analytics for Viral Media Delivery in Digital-Human Social Networks

Projects:
[Digital Social Networks] ! interests predictions in social networks ! human location and mobility prediction in social networks ! tipping point of viral media in social networks [Cyber-physical Social Systems/Networks] ! cyber-physical media ! interactive and tangle media ! big data systems
and more on http://www.youtube.com/watch?v=FumoTBHG39k

Outcomes from this lecture


1.! Course overview 2.! Sound 3.! Digital Audio and editing

What is ELEC 4170?

ELEC 4170

Focus on getting you enough backgrounds


! about digital media technologies, multimedia applications development, and multimedia systems ! about hands-on experiences to: ! editing, produce, process and production, ! design and develop exciting multimedia systems, mobile apps and interactive games. ! Weekly laboratory and several assignments about some challenging and exciting programming, design and production issues. ! Some interesting assessments to challenge you to apply what they learn for yourselves, the societies, and the world.
6

Intended Learning Outcomes


You could become the next-generation engineers: ! with a good senses of sciences, engineering, arts and businesses using digital media ! dare to dream and create things that could change the world; ! who believe the imaginations could potentially achieved through engineering. enGINEERING + IMAGINing + ! IMAGINEERs This course will also let you to: ! Think and present your works creatively with originality; ! work independently but also effectively as a team; ! recognize the impacts of digital media to promote effective messages and social values for profit/not-profit organizations, or for yourself.

www.youtube.com/watch?v=2b97KIlJuYs
7

Why ELEC4170 for your career


More than just an electronic or software engineer!
! ! ! ! ! ! ! Apps or software UI designer web-based and mobile gaming developer video producer and media engineer sound engineer theater effect engineer performance/stage effect engineer cyber-physical actors or artists!

Whats Digital Media?


Formal and conventional definition
! Digital media is a form of electronic media where data is stored in digital (as opposed to analog) form. ! What else?

What is Digital Media (/ Multimedia)?


Various new digital media forms along with emerging display, mobile, communication technologies and applications: ! ! ! ! ! ! ! ! ! Text, Web, Hypertext (links) Audio (Surround sounds) Image Animation Video (3D) Email, Internet, Social Media and Networks Mobility and Location (from your smart phones) Gesture? Smells?
10

What else in ELEC 4170


More than an engineering course 1.! Design concepts and principles with the uses of trendy tools 2.! Business and entertainment applications Marketing, Advertising, Filming, Gaming, etc. 3.! Creating meanings and impacts to soceity - Service Learning 4.! New Cool Projects (producing an mobile interactive games with various media companies) 5.! New facilities
11

Unique in ELEC 4170


New software for learning

12

Unique in ELEC 4170


Apps development ! apps graphic/ UI design

mobile app development

mobile app sound effect


13

Unique in ELEC 4170


Animation and apps with HTML5

Web

http://haodevelop.no-ip.org/test/draw/ ! Gaming

14

Unique in ELEC 4170


Video Production Facilities in Cyberport

! Chroma key
http://www.youtube.com/watch?v=a21_WMiTAVE

15

Unique in ELEC 4170


New meanings for learning in final project Services Learning 1.! Appling what you learnt on meaningful projects for communities in needed.

http://www.youtube.com/watch? v=0530uixzdK0

http://www.youtube.com/watch? v=FYHOurpYFTo

16

And what is more!...

17

Unique in ELEC 4170

Testing your production on NEW SENG Cyber-physical Social Space

1.! Sound effect designs 2.! Background designs

18

Unique in ELEC 4170


Scraping and visualizing social data

19

Course Facebook Page

20

We will all on Facebook for teaching and submissions


Connect us on Facebook today for 1% of out-class activity one!

21

Our teaching team

22

Assessment Scheme
1. In/Out-of-class activities: to give away marks related to the topics, skills and activities for this course

6%
2. Labs: labs with various hand-on learning experiences

24% 24%

3. Assignments: Meaningful assignment to apply the skills learned 4. Group Project: Related to an issue for a community partner

36% 10%

5. Mid-term Exam: open-book, open-notes, open-internet, open everything

23

Assessment Scheme
what else we expected to get you As ! to be imagineers! ! to be engineers mastering the technical theories with the tools! ! to be artists with very creative mindset to design! ! to be actors with impactful but funny presentation skills

24

Scoring Rubric For the Labs


max. 4 (100% ) out of 4 4 (100%) Exceeds Standard On top of the standard deliverables, students showed unusual insight, complexity, originality, or creativity with his/ her message or choice of an art form or medium to express it. Work shows rich evidence that student researched his/her OWN ideas, art form, and the techniques of your art form. 2 (50%) Meets Standard Students completed ALL required message and reproduced an art form or medium that expected. 1 (25%) Approaches Standard Students partially completed the required message and reproduced an art form or medium that expected. 0 (0%) Fails Students FAILED to complete any require message and reproduce any art form or medium that expected. The deliverables prepared and presented by the students from the Lab

Work shows full evidence that students completed the required ideas, art form, and the techniques of art form expected.

Work shows partial evidence that students completed some required ideas, art form, and the techniques of art form expected.

Work shows NO evidence that students completed any required ideas, art form, and the techniques of art form expected.

25

This is the ELEC 4170 for Spring 2013.

26

Questions?

27

Outcomes from this lecture


1.! Course overview

2.!Sound
3.! Digital Audio and editing

28

Sound and Sound Effect


why bother?
Lets compare a nice media production with and without sound http://www.youtube.com/watch? v=2pIWzUB79cQ&feature=youtube_gdata_player

29

Sound

30

How ear works?

z..z!Z
Let us utilizes the multimedia! http://www.youtube.com/watch?v=ahCbGjasm_E

31

Sound as a wave form


What is a wave?
-! a wave as "a disturbance or variation that transfers energy progressively from point to point in a medium and that may take the form of an elastic deformation or of a variation of pressure, electric or magnetic intensity, electric potential, or temperature. by Webster's dictionary 1.! Sound is a series of longitudinal or compression waves that move through air or other materials. 2.! Sound does not travel in a vacuum. air particles

32

Basic physics of waveform


What is a wave?

-! Amplitude ! Loudness

33

Human Perception: Loudness (Amplitude)


How to measure loudness?
Sound Pressure Level (SPL) meter: -! Unit is Pascal (Pa) or Newton/m2 -! Loudness limits of ears: 2x10-5Pa 120Pa Our ears response to sound pressure in logarithm scale: -! Decibel (dB): a better measure to describe how loud a sound is. -! dB = 20 log (P/P0) where P0=20 Pa, is the threshold of hearing
Approximate dB-SPL level of common sounds. (Information from S. S. Stevens, F. Warshofsky, and the Editors of TimeLife Books, Sound and Hearing, Life Science Library, TimeLife Books, Alexandria, VA, 1965, p. 173.)

34

Human Perception: (Pitch) - Frequency


Frequency is perceived as pitch
! Audible frequency range: 20Hz -> 20KHz ! Max. sensitivity at 1 to 5kHz ! Relatively insensitivity at low and high frequencies e.g., a barely audible 30Hz tone is in fact 60 dB louder than a barely audible 4kHz tone

35

Human Perception: (Pitch) - Frequency


Frequency is perceived as pitch
! Simple sine waves with different frequencies can be combined to form complicated waveforms by superposition

! Perception of pitch is related mainly to the fundamental frequency of the sound being played 36

Digital Audio

37

Capture and playback of digital audio


Air pressure variations

Digital to Analogue Converter

Captured via microphone

Converts back into voltage Signal is converted into binary (discrete form) 0101001101 0110101111

DAC

Analogue to Digital Converter

ADC

Air pressure variations


38

!!Sampling

rate (frequency) !!Resolution: number of bits per sample

Sampling Rate (Frequency)


Nyquist sampling theorem: In order to perfectly capture and record a signal, it must be sampled at a sampling rate (fS) greater than or equal to twice its highest frequency component (fc) i.e., fS > 2fC

Sampling at 1 time per cycle

39

Sampling Rate
Sampling at 1.5 times per cycle

Sampling at 2 times per cycle

Sampling > 2 times per cycle


sample at many time per cycle 40

Sampling Resolution - 1
Quantization and Quantization error

!!

!!

The smaller the Least Significant Bit (LSB) ! the more accurate the conversion and the smaller the quantization error

41

Sampling Resolution - 2
Quantization error
1.! Some info. will be lost during the quantization 2.! More samples/cycle (i.e., higher sampling rate) and more resolution per sample will increase file size

3. Many file formats are available and being researched with various improved 42 quantization and compression performances.

Relationship between quality and file size, etc.


Can we make good sound quality with small costs?
! Human ear can perceive 10-20 KHz ! Full range used in music ! CD quality audio: ! sample rate of 44,100 samples/sec ! sample size of 16-bits ! 60 min x 60 secs/min x 44,100 samp/sec ! x 2 bytes/samples x 2 channels (stereo) ! = 635,040,000, about 600 Mbytes (typical CD) ! Can use compression to reduce ! mp3, RealAudio
43

Audio Processing

44

Audio Processing
Types of processing
Amplitude control: Volume control, Panning, Dynamic range control, Noise gating, Down-mixing, etc. Time delay: Echo, Reverberation, Chorus and Flanging, Phasing Time domain control: Time stretch, re-sampling Frequency response: Equalization

45

Non-uniform Volume Control


Transitions
Abrupt change in a continuous audio should be avoided

Transitions are introduced in volume change


1.0

Amplitude multiplier
0

Attack

Decay

Sustain

Audio waveform multiplier Release

Start of sound file

End of sound file

time

46

Delay/Echo
!Simple echo
Input Delay Output Decay

original

Produced by adding a time-delayed signal

!Multiple echoes
Input Delay Output Decay
47

Delay/Echo
Special effects from echos
Simple echo V ! ery short delay (say 15ms) and long decay time results in metallic or
robotic sound

Multiple echoes ! Multiple delay with Pre-delay/echo give the effect of traveling back in
time in science fiction

48

Time stretch/shrink
Change the duration of audio sample
Important for many applications !Change the tempo of a musical sample !Synchronizing sound to video in film production !Delaying a persons speech if he/she talk too fast Not a easy task !Simply stretching/shrinking the time axis may not work as it will modify the pitch/quality as well

stretching

shrinking
49

Questions?

50

So, can you make people listening to feel better now?

- End of Lecture 1 -

51

Sound Effect
What is Sound Effect?
It can be created by
1.! 2.! 3.! Recording the sounds naturally produced from the sources in nature, environments , objects, animals, devices, etc. Synthesizing from the scratch through software Combining above and applying various audio processing techniques (through software) on a set of audio samples in a pre-recorded sound clip ( http://www.youtube.com/watch?v=IMgovfQjxUA) (00:18:00 -00:40:00)

http://www.youtube.com/watch?v=fbDFAaPfl28

Why?
1. Complement and enhance artistic presentations such as podcast, animations, video, etc. 2. Professional and creative sound effects make your digital presentation professional and impactful
52

Sound Effects
Copyrights
1.! May be illegal to use a sound effect that you dont have a proper right to use it 2.! Pay for a sound effect that is not economical or possible for you to produce or other reasons 3.! Better to use royalty-free sound effects 4.! Create (produce/modify) your own sound effect, especially more creative, personalized or unique ones.
53

For those motivated !


For those interested in music, take a look at
http://en.wikipedia.org/wiki/Normalizing_the_musical_scale http://en.wikipedia.org/wiki/Grand_piano http://en.wikipedia.org/wiki/Harmonic http://en.wikipedia.org/wiki/Middle_C http://en.wikipedia.org/wiki/Perfect_fifth

54

Human Perception: (Timbre) - Harmonics


Affecting Sound Quality
! A harmonic of a sine wave is any sine wave which has an integer multiple of the fundamental frequency

55

Human Perception: (Timbre) - Harmonics


The proper word for sound quality
!Same note on a guitar has different timbre (tone quality, tone color) when played on a piano !Timbre: a subjective attribute of sound related mainly to the harmonic components (called partials), but not the fundamental frequency !The relative relationship of the partials denote the signature for the sound !The fundamental can be deleted and the same pitch can be heard
56

Digital Representation of Audio


Recall: Sound as a waveform
1. How our ear works? Analog waveform (in air) ! electrical signals ! our brain 2. How the microphone works? Similar to our ears: analog waveform ! electrical signals (capacitance or current value) ! digital info. 3. How the speaker works? Reverse: digital info. (from a computer) !electrical signal ! analog waveform (in air)
57

Digital Representation of Audio


! Pressure level and frequency at a given time is represented by numbers ! A sequence of numbers is recorded to represent (pressure; frequency) separated by a fixed time-interval

Waveform! Electrical signals ! digital info. (10011)

t
time domain representation

58

Digital Representation of Audio


Waveform !Electrical signals ! digital info. (011) ! Pressure level and frequency at a given time is represented by numbers ! A sequence of numbers is recorded to represent (pressure; frequency) separated by a fixed time-interval

59