Anda di halaman 1dari 59

ELEC 4170

Digital Media and Multimedia Applications

Lecture 1 Overview and the Start of our Media Journey
Prof. James She ( 1

About me
Positions: ! Assistant Professor, ECE, HKUST ! Founding Director, HKUST-NIE Social Media Lab., ECE, HKUST. ! Research Fellow, Computer Laboratory, U. of Cambridge, UK (onleave) Research interests: 1) Cyber-physical Social Media Systems/ Networks 2) Analytics for Viral Media Delivery in Digital-Human Social Networks

[Digital Social Networks] ! interests predictions in social networks ! human location and mobility prediction in social networks ! tipping point of viral media in social networks [Cyber-physical Social Systems/Networks] ! cyber-physical media ! interactive and tangle media ! big data systems
and more on

Outcomes from this lecture

1.! Course overview 2.! Sound 3.! Digital Audio and editing

What is ELEC 4170?

ELEC 4170

Focus on getting you enough backgrounds

! about digital media technologies, multimedia applications development, and multimedia systems ! about hands-on experiences to: ! editing, produce, process and production, ! design and develop exciting multimedia systems, mobile apps and interactive games. ! Weekly laboratory and several assignments about some challenging and exciting programming, design and production issues. ! Some interesting assessments to challenge you to apply what they learn for yourselves, the societies, and the world.

Intended Learning Outcomes

You could become the next-generation engineers: ! with a good senses of sciences, engineering, arts and businesses using digital media ! dare to dream and create things that could change the world; ! who believe the imaginations could potentially achieved through engineering. enGINEERING + IMAGINing + ! IMAGINEERs This course will also let you to: ! Think and present your works creatively with originality; ! work independently but also effectively as a team; ! recognize the impacts of digital media to promote effective messages and social values for profit/not-profit organizations, or for yourself.

Why ELEC4170 for your career

More than just an electronic or software engineer!
! ! ! ! ! ! ! Apps or software UI designer web-based and mobile gaming developer video producer and media engineer sound engineer theater effect engineer performance/stage effect engineer cyber-physical actors or artists!

Whats Digital Media?

Formal and conventional definition
! Digital media is a form of electronic media where data is stored in digital (as opposed to analog) form. ! What else?

What is Digital Media (/ Multimedia)?

Various new digital media forms along with emerging display, mobile, communication technologies and applications: ! ! ! ! ! ! ! ! ! Text, Web, Hypertext (links) Audio (Surround sounds) Image Animation Video (3D) Email, Internet, Social Media and Networks Mobility and Location (from your smart phones) Gesture? Smells?

What else in ELEC 4170

More than an engineering course 1.! Design concepts and principles with the uses of trendy tools 2.! Business and entertainment applications Marketing, Advertising, Filming, Gaming, etc. 3.! Creating meanings and impacts to soceity - Service Learning 4.! New Cool Projects (producing an mobile interactive games with various media companies) 5.! New facilities

Unique in ELEC 4170

New software for learning


Unique in ELEC 4170

Apps development ! apps graphic/ UI design

mobile app development

mobile app sound effect


Unique in ELEC 4170

Animation and apps with HTML5

Web ! Gaming


Unique in ELEC 4170

Video Production Facilities in Cyberport

! Chroma key


Unique in ELEC 4170

New meanings for learning in final project Services Learning 1.! Appling what you learnt on meaningful projects for communities in needed. v=0530uixzdK0 v=FYHOurpYFTo


And what is more!...


Unique in ELEC 4170

Testing your production on NEW SENG Cyber-physical Social Space

1.! Sound effect designs 2.! Background designs


Unique in ELEC 4170

Scraping and visualizing social data


Course Facebook Page


We will all on Facebook for teaching and submissions

Connect us on Facebook today for 1% of out-class activity one!


Our teaching team


Assessment Scheme
1. In/Out-of-class activities: to give away marks related to the topics, skills and activities for this course

2. Labs: labs with various hand-on learning experiences

24% 24%

3. Assignments: Meaningful assignment to apply the skills learned 4. Group Project: Related to an issue for a community partner

36% 10%

5. Mid-term Exam: open-book, open-notes, open-internet, open everything


Assessment Scheme
what else we expected to get you As ! to be imagineers! ! to be engineers mastering the technical theories with the tools! ! to be artists with very creative mindset to design! ! to be actors with impactful but funny presentation skills


Scoring Rubric For the Labs

max. 4 (100% ) out of 4 4 (100%) Exceeds Standard On top of the standard deliverables, students showed unusual insight, complexity, originality, or creativity with his/ her message or choice of an art form or medium to express it. Work shows rich evidence that student researched his/her OWN ideas, art form, and the techniques of your art form. 2 (50%) Meets Standard Students completed ALL required message and reproduced an art form or medium that expected. 1 (25%) Approaches Standard Students partially completed the required message and reproduced an art form or medium that expected. 0 (0%) Fails Students FAILED to complete any require message and reproduce any art form or medium that expected. The deliverables prepared and presented by the students from the Lab

Work shows full evidence that students completed the required ideas, art form, and the techniques of art form expected.

Work shows partial evidence that students completed some required ideas, art form, and the techniques of art form expected.

Work shows NO evidence that students completed any required ideas, art form, and the techniques of art form expected.


This is the ELEC 4170 for Spring 2013.




Outcomes from this lecture

1.! Course overview

3.! Digital Audio and editing


Sound and Sound Effect

why bother?
Lets compare a nice media production with and without sound v=2pIWzUB79cQ&feature=youtube_gdata_player




How ear works?

Let us utilizes the multimedia!


Sound as a wave form

What is a wave?
-! a wave as "a disturbance or variation that transfers energy progressively from point to point in a medium and that may take the form of an elastic deformation or of a variation of pressure, electric or magnetic intensity, electric potential, or temperature. by Webster's dictionary 1.! Sound is a series of longitudinal or compression waves that move through air or other materials. 2.! Sound does not travel in a vacuum. air particles


Basic physics of waveform

What is a wave?

-! Amplitude ! Loudness


Human Perception: Loudness (Amplitude)

How to measure loudness?
Sound Pressure Level (SPL) meter: -! Unit is Pascal (Pa) or Newton/m2 -! Loudness limits of ears: 2x10-5Pa 120Pa Our ears response to sound pressure in logarithm scale: -! Decibel (dB): a better measure to describe how loud a sound is. -! dB = 20 log (P/P0) where P0=20 Pa, is the threshold of hearing
Approximate dB-SPL level of common sounds. (Information from S. S. Stevens, F. Warshofsky, and the Editors of TimeLife Books, Sound and Hearing, Life Science Library, TimeLife Books, Alexandria, VA, 1965, p. 173.)


Human Perception: (Pitch) - Frequency

Frequency is perceived as pitch
! Audible frequency range: 20Hz -> 20KHz ! Max. sensitivity at 1 to 5kHz ! Relatively insensitivity at low and high frequencies e.g., a barely audible 30Hz tone is in fact 60 dB louder than a barely audible 4kHz tone


Human Perception: (Pitch) - Frequency

Frequency is perceived as pitch
! Simple sine waves with different frequencies can be combined to form complicated waveforms by superposition

! Perception of pitch is related mainly to the fundamental frequency of the sound being played 36

Digital Audio


Capture and playback of digital audio

Air pressure variations

Digital to Analogue Converter

Captured via microphone

Converts back into voltage Signal is converted into binary (discrete form) 0101001101 0110101111


Analogue to Digital Converter


Air pressure variations



rate (frequency) !!Resolution: number of bits per sample

Sampling Rate (Frequency)

Nyquist sampling theorem: In order to perfectly capture and record a signal, it must be sampled at a sampling rate (fS) greater than or equal to twice its highest frequency component (fc) i.e., fS > 2fC

Sampling at 1 time per cycle


Sampling Rate
Sampling at 1.5 times per cycle

Sampling at 2 times per cycle

Sampling > 2 times per cycle

sample at many time per cycle 40

Sampling Resolution - 1
Quantization and Quantization error



The smaller the Least Significant Bit (LSB) ! the more accurate the conversion and the smaller the quantization error


Sampling Resolution - 2
Quantization error
1.! Some info. will be lost during the quantization 2.! More samples/cycle (i.e., higher sampling rate) and more resolution per sample will increase file size

3. Many file formats are available and being researched with various improved 42 quantization and compression performances.

Relationship between quality and file size, etc.

Can we make good sound quality with small costs?
! Human ear can perceive 10-20 KHz ! Full range used in music ! CD quality audio: ! sample rate of 44,100 samples/sec ! sample size of 16-bits ! 60 min x 60 secs/min x 44,100 samp/sec ! x 2 bytes/samples x 2 channels (stereo) ! = 635,040,000, about 600 Mbytes (typical CD) ! Can use compression to reduce ! mp3, RealAudio

Audio Processing


Audio Processing
Types of processing
Amplitude control: Volume control, Panning, Dynamic range control, Noise gating, Down-mixing, etc. Time delay: Echo, Reverberation, Chorus and Flanging, Phasing Time domain control: Time stretch, re-sampling Frequency response: Equalization


Non-uniform Volume Control

Abrupt change in a continuous audio should be avoided

Transitions are introduced in volume change


Amplitude multiplier




Audio waveform multiplier Release

Start of sound file

End of sound file



!Simple echo
Input Delay Output Decay


Produced by adding a time-delayed signal

!Multiple echoes
Input Delay Output Decay

Special effects from echos
Simple echo V ! ery short delay (say 15ms) and long decay time results in metallic or
robotic sound

Multiple echoes ! Multiple delay with Pre-delay/echo give the effect of traveling back in
time in science fiction


Time stretch/shrink
Change the duration of audio sample
Important for many applications !Change the tempo of a musical sample !Synchronizing sound to video in film production !Delaying a persons speech if he/she talk too fast Not a easy task !Simply stretching/shrinking the time axis may not work as it will modify the pitch/quality as well





So, can you make people listening to feel better now?

- End of Lecture 1 -


Sound Effect
What is Sound Effect?
It can be created by
1.! 2.! 3.! Recording the sounds naturally produced from the sources in nature, environments , objects, animals, devices, etc. Synthesizing from the scratch through software Combining above and applying various audio processing techniques (through software) on a set of audio samples in a pre-recorded sound clip ( (00:18:00 -00:40:00)

1. Complement and enhance artistic presentations such as podcast, animations, video, etc. 2. Professional and creative sound effects make your digital presentation professional and impactful

Sound Effects
1.! May be illegal to use a sound effect that you dont have a proper right to use it 2.! Pay for a sound effect that is not economical or possible for you to produce or other reasons 3.! Better to use royalty-free sound effects 4.! Create (produce/modify) your own sound effect, especially more creative, personalized or unique ones.

For those motivated !

For those interested in music, take a look at


Human Perception: (Timbre) - Harmonics

Affecting Sound Quality
! A harmonic of a sine wave is any sine wave which has an integer multiple of the fundamental frequency


Human Perception: (Timbre) - Harmonics

The proper word for sound quality
!Same note on a guitar has different timbre (tone quality, tone color) when played on a piano !Timbre: a subjective attribute of sound related mainly to the harmonic components (called partials), but not the fundamental frequency !The relative relationship of the partials denote the signature for the sound !The fundamental can be deleted and the same pitch can be heard

Digital Representation of Audio

Recall: Sound as a waveform
1. How our ear works? Analog waveform (in air) ! electrical signals ! our brain 2. How the microphone works? Similar to our ears: analog waveform ! electrical signals (capacitance or current value) ! digital info. 3. How the speaker works? Reverse: digital info. (from a computer) !electrical signal ! analog waveform (in air)

Digital Representation of Audio

! Pressure level and frequency at a given time is represented by numbers ! A sequence of numbers is recorded to represent (pressure; frequency) separated by a fixed time-interval

Waveform! Electrical signals ! digital info. (10011)

time domain representation


Digital Representation of Audio

Waveform !Electrical signals ! digital info. (011) ! Pressure level and frequency at a given time is represented by numbers ! A sequence of numbers is recorded to represent (pressure; frequency) separated by a fixed time-interval