ELE 774 Adaptive Signal Processing

Dr. Cenk Toker Block F3, Room: 3304/A

www.ee.hacettepe.edu.tr/~toker/ELE774-Homepage.html

ELE 774 - Adaptive Signal Processing

Course Content (Tentative)

1.

Textbook: S.Haykin, Adaptive Filter Theory, 4th Ed., Prentice Hall, 2002

Introduction BACKGROUND REVIEW 2. Discrete-time and random signals 3. Mathematical Tools OPTIMAL LINEAR FILTERS 4. Wiener Filters 5. Linear Prediction ADAPTIVE FILTERING 6. Stochastic Gradient Descent Algorithms 7. Family of LMS Algorithms 8. Method of Least Squares 9. Recursive Least Squares (RLS) Algorithm 10. Square-Root Algorithms 11. LMS and RLS Algorithms: Practical Issues 12. Kalman Filtering (?) APPLICATIONS 13. Spectrum Estimation 14. Array Processing ELE 774 - Adaptive Signal Processing

Definition of filtering

A filter is commonly used to refer to a system that is designed to extract information about a prescribed quantity of interest from noisy data.

ELE 774 - Adaptive Signal Processing

Applications

Communications; radar, sonar, Control Systems; navigation, Speech/Image Processing; echo and noise cancellation, biomedical engineering Others; seismology, financial engineering, etc.

ELE 774 - Adaptive Signal Processing

Applications

!!! Noise and errors are statistical in nature !!! We will use statistical tools.
ELE 774 - Adaptive Signal Processing

Basic Kinds of Estimation

Filtering (real-time operation)

Smoothing (off-line operation)

Prediction (real-time operation)

ELE 774 - Adaptive Signal Processing

Filter
Linear
A filter is linear if the filtered, smoothed or predicted quantity at the output of the filter is a linear function of the observations applied to the filter input. u(t) u(n) Filter y(t) y(n) Non-linear or

Non-Linear
Otherwise, it is non-linear.

Linear

or

!!! Non-linear filters may be hard to Analyse, if not impossible !!!

ELE 774 - Adaptive Signal Processing

Optimum Filter

Definition: Solution of an optimization problem wrt. a certain criterion with statistical parameters.
Nonlinear: Maximum Likelihood (ML) sense (very difficult to implement) Linear: Minimum Mean Square Error (MMSE) sense

Wiener filters, (Stationary environment) Kalman filters, (Non-stationary environment)

Etc.

(Any other criterion, e.g. ZF)

ELE 774 - Adaptive Signal Processing 8

Adaptive Filters

Wiener Filter requires -a priori information of several statistics

-estimation (knowledge of the system) is needed before filtering -inversion of a huge matrix
-computationally inefficient!

Adaptive filtering can overcome these disadvantages!

Recursive algorithm No complete a priori information required

Algorithm develops this information with increasing # of iterations.

If the environment is stationary converges to the Wiener soln. non-stationary tracks the changes.
ELE 774 - Adaptive Signal Processing 9

Analysis of Adaptive Filters

Rate of convergence (to the optimum Wiener soln.) Misadjustment (deviation from the optimum Wiener soln.) Tracking (the variations in a non-stationary environment) Robustness(to disturbances of small energy) Computational Requirements/Cost Numerical Properties (Numerical stability & accuracy)

ELE 774 - Adaptive Signal Processing

10

Linear Filter Structure

Structure: FIR (Finite-duration Impulse Response)

Transversal Filter (Tapped-delay line) Lattice Systolic array

IIR

(Infinite-duration Impulse Response)

ELE 774 - Adaptive Signal Processing

11

Transversal Filter
unit-delay element Convolution Sum

multiplier

adder
ELE 774 - Adaptive Signal Processing 12

Transversal Filter

xH: Hermitian transpose xH=(xT)*

ELE 774 - Adaptive Signal Processing

13

Lattice Predictor

ELE 774 - Adaptive Signal Processing

14

Lattice Predictor

Predictor order (# of stages): M Forward prediction error

Backward prediction error m: the mth reflection coefficient Input seq. u(n) is correlated, backward prediction error b(n) is uncorrelated Together with m, b(n) approximates d(n) (innovations process).
ELE 774 - Adaptive Signal Processing 15

Systolic Array

Boundary cell

Internal cell
s

y3 ELE 774 - Adaptive Signal Processing

16

Systolic Array

Represents a parallel computing network Used for efficient pipelined operation

Matrix multiplication Triangularisation Back substitution (Matrix eqn. solving)

Example 3x3 matrix/3x1 vector

ELE 774 - Adaptive Signal Processing

17

IIR Filter
May have stability problems, We will prefer FIR filters for Adaptive filtering.

ELE 774 - Adaptive Signal Processing

18

Adaptive Filtering Algorithms

Error
Difference

between the filter output a desired response Mean Square Error

Weighted

Filter

y(n) d(n)

(n) : Error + -

Error Squares

ELE 774 - Adaptive Signal Processing

19

Adaptive Filtering Algorithms

Stochastic Gradient Approach

Cost Function depends on Mean-Square Error criterion (!!! Stochastic, depends on second order statistics !!!) Solution of Wiener-Hopf Equations

Results in Wiener soln. but with an iterative approach Requires Expectations E{.}

Based on Method of Steepest Descent

updated value old value learning rate gradient of tap - weight = of tap - weight parameter of (step size) cost function vector vector

Use instantaneous values instead of expectations (LMS)

updated value old value learning rate tap - error of tap - weight = of tap - weight parameter input signal (step size) vector vector vector
ELE 774 - Adaptive Signal Processing 20

Adaptive Filtering Algorithms

Least-Squares Estimation

Cost Function depends on sum of weighted error squares Low computational complexity due to recursive operation

Three

categories
Relies on Matrix Inversion Lemma Numerically unstable, high computational complexity Based on QR-decomposition Numerically stable Exploits certain matrix structures to reduce complexity.
ELE 774 - Adaptive Signal Processing 21

Standard RLS

Square-root RLS algorithm

Fast RLS algorithm

Applications

Four Classes

Identification
system identification layered earth modeling deconvolution adaptive and blind equalisation linear predictive coding adaptive differential PCM spectrum analysis signal detection noise canceling echo cancellation adaptive beamforming
22

Inverse modeling

Prediction

Interference cancellation

ELE 774 - Adaptive Signal Processing

System Identification

Observing the output of a plant(system), given the input signal, tries to estimate the IR of the plant. Filter coefficient are found by an adaptive algorithm.

ELE 774 - Adaptive Signal Processing

23

Adaptive Equalization

Removes intersymbol interference (ISI). Filter coefficient are found by an adaptive algorithm.

ELE 774 - Adaptive Signal Processing

24

Adaptive Spectrum Estimation

Parametric (AR) model Linear AR filter

input: white noise output: observed signal aim: find the model parameters by an adaptive algorithm.

ELE 774 - Adaptive Signal Processing

25

Adaptive Noise Cancellation

Electrocardiography (ECG) Acoustic noise in speech Active noise cancellation (headphones)

ELE 774 - Adaptive Signal Processing

26

Echo Cancellation

Coupling due to imperfect balancing in hybrid transformer creates an echo in analog telephone lines. Echo signal can be estimated by an adaptive filter and the subtracted out.

ELE 774 - Adaptive Signal Processing

27

Adaptive Beamforming

Multiple sensors (antenna, microphone, etc) used to steer the beam to a specific position. Radar, sonar Commun. systems, Astrophysical exploration, Biomedical signal processing, etc.

ELE 774 - Adaptive Signal Processing

28

Historical Notes

To understand a science it is necessary to know its history. Auguste Comte (1798-1857)

Linear Estimation Theory

Method of least squares, Gauss, 1795 Minimum mean square error estimation, late 1930s Discrete-time Wiener-Hopf equation, Levinson, 1947 Kalman filter, Swerling, 1958 and Kalman, 1960 Stochastic approximation, Robins and Monro, 1951 LMS algorithm, Widrow and Hoff, 1959 Gradient adaptive lattice (GAL) algorithm, Griffiths, 1977-8
29

Stochastic gradient algorithms, late 1950s

ELE 774 - Adaptive Signal Processing

Historical Notes

Recursive Least Squares Algorithm

Kalman filter, Godard Algorithm, Godard, 1974 Relationship between RLS and Kalman, Sayed and Kailath, 1994 QR decomposition based systolic array, Gentleman & Kung, 1981 Fast RLS algorithm, 1970s, Morf Logical calculus for neural networks, McCulloch and Pitts, 1943 Perceptron, Rosenblatt, 1958 Back-propagation algorithm, Rumelhart, et al., 1986 Radial basis function network, Broomhead and Lowe, 1988

Neural Networks

ELE 774 - Adaptive Signal Processing

30

Applications

Adaptive Equalisation, 1960s

Zero-forcing equaliser, Lucky, 1965 MMSE equaliser, Gersho, 1969, Proakis&Miller, 1969 Godard Algorithm, Godard, 1974 Fractionally Spaced Equaliser (FSE), Brady, 1970 Decision Feedback Equaliser (DFE), Austin 1967, MMSE, Monsen, 1971. Maximum Likelihood speech prediction, Saito and Itakura, 1966 Linear Predictive Coding (LPC), Atal and Hanauer 1970-1 Adaptive Lattice Predictor, Nakhoul and Cossell, 1981

Speech Coding

ELE 774 - Adaptive Signal Processing

31

Applications

Spectrum Analysis, early 1900s

Maximum entropy method, Burg, 1967 Method of multiple windows, Thomson, 1982

Adaptive noise cancellation, started at 1965 Adaptive Beamforming

Intermediate Frequency (IF) sidelobe canceller, Howells, 1950 Control law for adaptive array antenna, Applebaum, 1966 Application of LMS, Widrow et al., 1967 Minimum Variance Distortionless Response (MVDR) beamformer, Capon, 1969
32

ELE 774 - Adaptive Signal Processing