Stability Analysis of Fast Recursive Least Squares Algorithm: Application to Adaptive Filtering

Farid Ykhlef, M. Arezki, A. Guessoum and D. Berkani

LATSI laboratory, Department of Electronics, Faculty of Engineering Sciences, University of Blida, Algeria. f_ykhlef@yahoo.fr
Abstract- In this paper, we introduce a new numerically stable version for the Fast Recursive Least Squares (FRLS) algorithm. As an additional contribution, we present an analysis of the FRLS algorithm instability problems. An experimental study is conducted in order to determine the origin of the numerical instability. The originality of our investigation lies mainly in finding the relation between the stability of the FRLS algorithm and the positions of its (forward/backward) prediction part zeros. Keywords: Fast Recursive Least Squares, stability, prediction.

I.

INTRODUCTION

signal xt over a length of N points, nt is the additional noise, yt is the desired signal, N ,t is the a priori error ~ and the quantity N ,t K N ,t denotes the adaptation gain (or Kalman gain) calculated, independently of the filtering part, by a Fast Recursive Least Squares (FRLS) algorithm using linear forward/backward prediction analysis on the input signal xt [8]. Here is the first difficulty for FRLS algorithms when applied to the identification of (Moving Average) MA models: the adaptation gain calculated on the input signal xt is more sensitive to this non-stationary signal than to the system to be identified [7] [8].
nt xt
Unknown System

In the field of adaptive signal processing, it is well known that FRLS algorithms can produce a good trade-off between convergence speed and computational complexity. But, it is also well-known that the FRLS algorithm suffers from numerical instability when operating under the effects of finite precision arithmetic [1]. For this reason, it is not widely used because of its undesirable tendency to diverge when operating in finite precision arithmetic [2]. To compensate, modifications to the algorithm have been introduced that are either occasional (performed when a predefined condition(s) is violated) or structured as part of the normal update iteration. To improve on the stability of the FRLS algorithm, a number of methods of that kind have been proposed [1]-[5]. In the same trend, we introduce a new numerically stable version for the stationary case of this algorithm. In the present work, we investigate in detail the instability problem of the FRLS algorithm, in the context of systems identification application. A. Adaptive Identification The identification is done by modelling the unknown system with an adaptive finite impulse response (FIR) filter and subtracting the desired signal yt from the t [6]. A basic schema of identification by output signal y an adaptive algorithm is shown in Fig. 1. The coefficients of the adaptive filter WN,t 1 are adjusted by a FRLS algorithm, according to the following equations: t (1) N ,t = yt y ~ WN ,t =WN ,t 1 N ,t N ,t K N ,t (2)

+
t y

yt

Adaptive Filter

N ,t

Figure 1. System identification structure.

with
T t = WN y ,t 1 X N ,t
T

(3)

where exponent (.) denotes matrix or vector transposed, N is the length of the filter, t is the discrete time index, X N ,t denotes a vector which summarizes the past of the

FRLS Algorithm B. For the following, we consider the so-called Fast Transversal Filters (FTF) version of the FRLS algorithm [3]. The FRLS algorithm is divided into two parts: a Prediction part and a Filtering part (Fig. 2). The prediction part provide to the filtering part an adaptation gain (or Kalman gain) vector to identify the unknown system. Moreover, the FRLS algorithm is a result of taking advantage of redundancies arising from the solution of four transversal filtering problems, each related through their use of the same input data: a filter for each of the one-step forward and backward prediction problems ( AN ,t / B N ,t ), a filter defining the gain vector (Kalman gain) of the Recursive Least Squares (RLS) algorithm ~ ( N ,t K N ,t ), and a filter to provide the weight vector corresponding to the desired problem being solved ( W N,t ). Combined in a special way, these four transversal filters provide the exact solution to the RLS problem at all times and define the FRLS algorithm [2]. The filtering part of the FRLS algorithm has the distinct advantage of robustness to numerical errors. Knowing the good numerical properties of the filtering part and the non influence of this last on the prediction part, our study is entirely based around the prediction part.

FRLS algorithm
xt
Prediction Part

TABLE I NS-FRLS ALGORITHM.

N ,t K N ,t

Filtering Part

t y

- Prediction part:
T eN ,t = xt AN X N ,t 1 , t 1

N ,t
Figure 2. The FRLS algorithm.

N ,t = N ,t 1 +

N ,t 1

2 eN ,t

II. NEW NUMERICALLY STABLE VERSION OF THE FRLS ALGORITHM Several numerical solutions of stabilization, with stationary signals, are proposed in the literature [1], [3], [4], [5] and [9]. On the other hand, the FRLS algorithm is notoriously unstable, but it is possible to maintain stability by using a few equations. Here, we follow the method made in [5], to propose a new numerically stable version of the FRLS algorithm. This method is based on a first order model of the propagation of the numerical errors. The general principle is to modify the numerical properties of the algorithm without modifying its theoretical behavior [5].In this case, by using some known relationships between the different backward a priori prediction errors, we define a control variable N ,t , theoretically null, given by [10]:
N ,t =
c rN ,t

0 1 e ~ K N +1,t = ~ + N ,t K N ,t 1 N ,t 1 AN ,t 1 ~ AN ,t = AN ,t 1 eN ,t N ,t 1 K N ,t 1
T rNc,t = xt N BN ,t 1 X N , t ~ N +1 f0 rN ,t = N ,t 1 K N +1,t

~ N +1 rNf,t1 = N +1 N ,t 1 N ,t 1 K N +1,t

N ,t = rNc,t [(1 s ) rNf,t0 + s rNf,1 ] t

rNs,t = rNc, t + N , t

N +1,t =

N ,t 1 N ,t N ,t 1 N +1,t N ,t = ~ N +1 1 + N +1,t rNs,t K N +1,t

(old

version)

N ,t =

[(1 s )

rNf,0 t

+ s rNf,1 t]

(4) (5.a) (5.b) (5.c)

with
c T rN ,t = xt N B N ,t 1 X N ,t ~ N +1 rNf ,0 t = N ,t 1 K N +1,t

~ K ~ ~ N +1 BN ,t 1 N ,t = K N +1,t + K N +1,t 1 0 ~ BN ,t = BN ,t 1 rNs,t N ,t K N ,t

N ,t

N ,t 1 ~ N +1 N ,t 1 (new version) + N ,t 1 rNs,t K N +1,t

N +1

~ N +1 N +1 rNf ,1 N ,t 1 N ,t 1 K N +1,t t =

N ,t = N ,t 1 + N ,t (rNs,t ) 2
- Filtering part:

where relations (5) represent the backward a priori prediction errors, theoretically equal, calculated differently. The scale parameter ( 0 s 1 ) controls the propagation of the numerical errors in the algorithm [9], [10]. In practice, the variable N ,t is never null on account of the machines precision. To stabilize the algorithm, we use the variable N ,t to calculate a backward a priori prediction error:
s c rN ,t = rN ,t + N ,t

~ WN ,t = WN ,t 1 N ,t N ,t K N ,t

N ,t = yt WNT,t 1 X N ,t

III. STABILITY ANALYSIS OF THE FRLS ALGORITHM It is well-known that the reduction in complexity of the FRLS algorithm is paid by a significant degradation of its numerical stability properties. The numerical errors are propagated in a way not limited in time, which leads to an unstable solution [11], [12]. The performances of convergence speed and tracking capabilities are degraded very quickly for a forgetting factor close to 1. With this method of numerical stabilization in stationary case, the NS-FRLS algorithm is still unstable for some particular cases, when we must choose a forgetting factor very close to the stability condition (7). Thereafter, tests show these particular cases. The analysis of numerical errors propagation proposed in [1] was conducted for the NSFRLS algorithm. The equations of numerical errors propagation for the recursive variables of the NS-FRLS algorithm according to the state linear model are [1], [8]: Z t = F(t)Z t 1 (8) with

(6)

Based on this method of stabilization, various versions of the FRLS algorithm were developed. For adaptive filtering, the new version of FRLS (NS-FRLS) is summarized on table I along with an old one [9]. This new version does not use the intermediate variable N +1,t . According to the simulations, the best results are given for s = 0.25. To simplify and to ensure the numerical stability of this version in the stationary case, it is necessary to choose a forgetting factor [10]: (7) > 1 1 pN where the parameter p is a real number higher than 2 to ensure the numerical stability of the algorithm.

at ~ , a = AN ,t , c = K N ,t , b = BN ,t c Z t = t t t t N ,t N ,t N ,t b t (9) where Vt stands for the first order approximation of the error on the theoretical variable Vt . The matrix F(t) is an approximate transition matrix for errors propagation [1], [5]. For the NS-FRLS algorithm, the F(t) matrix is given by:
F11(t) F12(t) F13(t) (10) F(t) = F21(t) F22(t) F23(t) F (t) F (t) F (t) 32 33 31 The algorithm is stable when the eigen-values of the F(t) matrix are close to 1 in magnitude. In general, the stability analysis of the F(t) matrix is very difficult, due to the complexity of some of its components and to its dependency on the input signal [1], [5]. In this work, the instability study of FRLS algorithm is conducted entirely around the submatrix F22 (t ) . This sub-matrix contains a c companion matrix M (t ) [10]. The roots of the backward predictor vector B N ,t are equal to the eigen-values of the M c (t ) matrix [13]: 1 0 0 L 0 BN ,t 1 2 1 0 L 0 BN ,t 1 3 (11) M c (t ) = 0 1 L 0 BN ,t 1 M M M O M 0 0 L 1 B N N ,t 1

AR5f (poles far from the unit circle): 0.75 e j2 ( 0.45) , 0.6 e j2 ( 0.33) , 0.5. AR5c (some poles close to the unit circle): 0.98 e j2 ( 0.45 ) , 0.6 e j2 ( 0.33) , 0.5.
TABLE II STATE OF STABILITY FOR VARIOUS ORDERS N F C 0.9167 stable unstable 4 0.9375 stable stable 0.9333 stable unstable 5 0.95 stable stable 0.9524 stable unstable 7 0.9643 stable stable

F : AR5f signal ; C : AR5c signal.

Table II shows the stability state of NS-FRLS for various orders, and with the two different input signals. We note that the NS-FRLS algorithm is unstable for the AR5c signal and it is stable for the AR5f signal, for a forgetting factor very close to the stability condition. Table II presents also the stable case of the NS-FRLS algorithm for the AR5c signal; this stability is only assured if we increase the value of the forgetting factor . Figs. 3 and 4 show some results for the two synthetic AR signals with an order equal to 5. Referring to this result, we see that if one zero of the backward predictor exceeds the value 1, a divergence of the algorithm is observed (see Fig. 4); that means that one eigen-value of the sub-matrix F22 (t ) is greater than 1.
Forward predictor 1 Imaginary Part 0.5 0 -0.5 -1 Imaginary Part 1 0.5 0 -0.5 -1 -1 -0.5 0 0.5 Real Part 1 -1 -0.5 0 0.5 Real Part 1 Backward predictor

Tests on AR signals are realized to determine the origin of instability for this particular case. IV. RESULTS A. Test with AR Signals We assume that the input signal to the adaptive filter can be modeled as an autoregressive (AR) process whose order is the same as the adaptive filter length. To test the numerical behavior of the prediction part of the numerically stable FRLS algorithm, the input signals used for this simulation are AR signals (autoregressive). Then, for a number of selected input parameters of the NS-FRLS algorithm, we observe the evolution of the zeros of the forward/backward predictors variables. And we will observe also the convergence of these variables, for an order of the predictor: 1) Equal to the AR order; 2) Higher than the AR order (over-estimated order); 3) Lower than the AR order (underestimated order). The goal of these simulations is to analyze the position of the zeros of the forward/backward predictors in the zplane (unit circle) in order to verify the numerical stability. In these simulations, we will choose various values for the poles of the synthesized AR processes: 1) Poles far from the unit circle; 2) Poles close to the unit circle. We will use fifth-order AR processes characterized by the following poles:

Predicted pole Real pole Figure 3. Position of the estimated zeros of the forward/backward predictors in steady-state for the AR5f signal: poles far from the unit circle (=0.9333(p=3), N=5).
Forward predictor 1 Imaginary Part 0.5 0 -0.5 -1 -1 -0.5 0 0.5 Real Part 1 Imaginary Part 1 0.5 0 -0.5 -1 -1 -0.5 0 0.5 Real Part 1 Backward predictor

Predicted pole Real pole Figure 4. Position of the estimated zeros of the forward/backward predictors just before the divergence for the AR5c signal: poles close to the unit circle (=0.9333(p=3), N=5).

Therefore the errors propagation sub-system corresponding to the sub-matrix F22 (t ) is unstable. Various experiments have been performed to assess the stability of the NS-FRLS algorithm. These experiments have been carried out on several signals of the same kind as the AR5f signal and the AR5c signal. The results reveal that for the signals of the same kind as the AR5f signal, the choice of a forgetting factor very close to the stability condition (7) allows to maintain the stability of

the algorithm NS-FRLS. But, for the signals of the AR5c kind, stability is only assured when we choose a forgetting factor extremely higher than the condition. Moreover, for the AR5c signal, the stability of the algorithm is also observed in the largely over-estimated case. Application to Systems Identification In this part, we have identified a system with the setup of Fig. 1. The adaptive filter is based on the new numerically stable version of the FRLS algorithm and the reference system has N = 32. A zero-mean Gaussian noise nt with a SNR set to 30 dB was added to the output of the filter. For the first experiment, the input signal is a white noise and the desired signal was filtered by (FIR) obtained synthetically. The system mismatch:
mis (t ) = Wopt WN ,t Wopt
2 2

These algorithms are stable for a suitable choice of the forgetting factor; its minimal limit depends in fact on the nature of the input signal. With the new proposed version of NS-FRLS, simulation results were presented in this work for the systems identification application. REFERENCES
[1] D. T. M. Slock, and T. Kailath, Numerically Stable Fast Transversal Filters for Recursive Least Squares Adaptive Filtering, IEEE Trans. on Signal Processing, vol. 39, No. 1, pp. 92-114, January 1991. J. R. Bunch, R. C. LeBorne, and I. K. Proudler, Measuring and maintaining consistency: a hybrid FTF algorithm, Int. Journal of Applied Math and Comp. Sci., vol. 11, No. 5, pp. 1203-1216, 2001. J. M. Cioffi, and T. Kailath, Fast Recursive Least Squares Transversal Filters for Adaptive Filtering, IEEE Trans. Acoustic, Speech, Signal Processing, vol. Assp-32, No. 4, pp. 304 -337, April 1984. J.L. Botto, Stabilization of fast recursive least-squares transversal filters for adaptive filtering, in Proc. ICASSP 87 Conf., (Dallas, TX), pp. 403-406, April 1987. A. Benallal, and A. Gilloire, A New Method to Stabilize Fast RLS Algorithms Based on a First-Order Model of the Propagation of Numerical Errors, in Proc. ICASSP 88 Conf., (New York, N.Y.), pp. 1373-1376, April 1988. R. Morgan, J. Benesty, and M. Sondhi, On the evaluation of estimated impulse response, IEEE Signal processing letters, vol. 5, No. 7, pp. 174-176, July 1998. A. Benallal, A study of the transversal fast recursive least squares algorithms and application to the identification of acoustic impulse responses, (in French) Ph. D. dissertation, University of Rennes I, Rennes France, December 1988. J. K. Soh, and S. C. Douglas, Analysis of the Stabilized FTF Algorithm with Leakage Correction, Proc. 30th Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, vol. 2, pp. 1088-1092, November 1996. M. Arezki, A. Benallal, F. Ykhlef, A. Guessoum, and D. Berkani, New method of comparison NS-FRLS and NLMS Algorithms for the acoustic echo cancellation, Proceedings of the 4th international symposium on CSNDSP 2004, University of Newcastle, U.K, pp. 528-532, July 2004. F. Ykhlef, A study of the stability of the fast recursive least squares algorithms with predictable signals and application to the speech signal, (in French) Master thesis, University of Blida, Algeria, February 2002. D. T. M. Slock, Backward consistency concept and round-off error propagation dynamics in recursive least-squares algorithms, Opt. Engr., vol. 31, No. 6, pp. 1153-1169, June 1992. P. Regalia, Numerical stability issues in fast least-squares adaptation algorithms, Opt. Engr., vol. 31, No. 6, pp. 1144-1152, June 1992. P. Stoica, and A. Nehorai, On stability and root location of linear prediction models, IEEE Trans. Acoustic, Speech, Signal Processing, vol. Assp-35, No. 4, pp. 582 -584, April 1987.

B.

[2] [3]

(12)

[4] [5]

where Wopt denote the optimal system, is computed as a function of time and plotted in Fig. 5. A stationary 10th-order autoregressive input signal is chosen for a second experiment. The poles of the input signal are located at 0.98 e j2 ( 0.45 ) , 0.98 e j2 ( 0.2 ) , 0.6 e j2 ( 0.33) , 0.5e j2 ( 0.25 ) and 0.25e j2 ( 0.4 ) on the z-plane. Fig. 6 shows also the evolution of the system mismatch for the AR10 input signal. The stability condition of the algorithm is always respected for all experiments.
System mismatch 0 Amplitude (dB) -10 -20 -30 -40 0 2000 4000 Samples 6000 8000 10000

[6] [7]

[8]

[9]

[10]

Figure 5. System identification (input signal: white noise, N=32 and =0.9895833(p=3)).
System mismatch 0 Amplitude (dB) -10 -20 -30 0 2000 4000 Samples 6000 8000 10000

[11] [12] [13]

Figure 6. System identification (input signal: AR10 signal, N=32 and =0.9921875 (p=4)).

V. CONCLUSION In this paper a new numerically stable version of the FRLS algorithm was proposed for the stationary input signal cases. Initially, we tested the stability of numerically stabilized FRLS algorithm with synthetic autoregressive (AR) input signals. The analysis of the propagation of numerical errors was conducted for the NS-FRLS algorithm. A sub-matrix of the propagation errors matrix was tested for a particular case.