An introduction to CG and BI CG Method

Wei-Len Lee
Department of Applied Mathematics, National University of Kaohsiung,
Kaohsiung811, Taiwan. E-mail: a0934133@nuk.edu.tw
15May 2007
Abstract
1.ConjugateGradient Method (CG)
The Conjugate Gradient method is an eective method for symmetric positive
de!nite systems.It is the oldest and best known of the nonstationary methods
disscussed here.The method proceed by generating vector sequences of iterates
(i.e.,successive approximations to the solution),residuals corresponding to the it-
erates,and search directions used in updating the iteratesand residuals. Although
the length of these sequences can become large, only a small number of vectors
needs to be kept in memory .In every iteration of themethod ,two inner products
are performed in order to compute update scalars that are de!ned to make the
sequences satisfy certain orthogonality conditions. On a symmetric positive de!-
nitelinear systemthese conditions imply that this distance to the true solution is
minimized in some norm. 2.BiConjugate Gradient (BiCG) The Conjugate Gradi-
ent methid is not suitable for nonsymmetric systems because the residual vectors
cannot bemadeorthogonal with short recurres (for proof of this seeVoevodin [213]
or Faber and Manteuel [96]). The GMRES method retains orthogonality of the
residuals by using long recurrences, at the cost of a large storage demand. The
BiConjugateGradient method takesanother approach, replacingtheorthogonal se-
quenceof residual by two mutually orthogonal seguences, at thepriceof no longer
providing a minimization.
1. Conjugate Gradient Method (CG)
Theiterationx
()
areupdatedineachiterationbyamultiple(

) of thseaarch
direction vector !
()
"
()
="
(1)
+

!
()
Correspondingly theresiduals #
()
=$ %"
()
areupdated as
Preprint submitted to Elsevier Science 15May 2007
PD F !"# "pdfFactory Pro" $#%&'( w w w .ahasoft.com .tw /FinePrint/
#
()
=#
(1)
&
()
where &
()
=%!
()
(1)
The choice =

= #
(1)

#
(1)
'!
()

%!
()
minimizes #
()!
%
1
#
()
over all
possiblechoices for in eqution (1)
Thesearch directions areupdated usingtheresiduals
!
()
=#
()
+(
1
!
(1)
) (2)
where the choice (

= #
()

#
()
'#
(1)

#
(1)
ensures that !
()
and %!
(1)
|
or equivalently, #
()
and #
(1)
| are orthogonal.In fact,one can show that
this choice of (

makes !
()
and #
()
orthogonal to all previous %!
(")
and #
(")
respectively.
1.1.theory
Theconjugategradient method constructs theith iterate"
()
as an element
of "
(0)
+*!+,f #
(0)
) ---) %
1
#
(0)
gso that ("
()
b ")
!
%("
()
b ") is minimized,
where b " is theexact solution of %" =$-this minimumis guaranteed to exist
in general only if % is symmetric positivedenite.
The above minimization of the error is equivalent to the residuals #
()
=
$ %"
()
.Since for symmetric % an orthogonal basis for Krylov subspace
*!+,f #
(0)
) ---) %
1
#
(0)
g can be constructed with only three-termrecurrences,
such a recurrence also su!ces for generating the residuals. In the Conjugate
Gradient methodtwocoupledtwo-termrecurrencesareused; onethat updates
residualsusingasearchdirectionvector, andoneupdatingthesearchdirection
with a newly computed resiual.This makes the Conjugate Gradient method
quiteattractivecomputationally.
Thereis a closerelationship between theConjugateGradient method and
theLanczosmethodfor determiningeigensystems,sincebotharebased onthe
contruction of an orthogonal basis for theKrylov subspace, and a similarlity
transformation of the coe!cient matrix to tridiagonal form. The coe!cient
computed duringtheCG iteration then arisefromtheLU factrization of this
tridiagonal matrix. Formthe CG iteration one can reconstruct the Lanczos
process, andviceversa;seePaigeandSaunders[168] andGolubandVanLoan
[109,x10.2.6]. This relationship can beexploited to obtain about theeigensys-
temof thematrix %-
1.2.Convergence
2
PD F !"# "pdfFactory Pro" $#%&'( w w w .ahasoft.com .tw /FinePrint/
For theConjugateGradient method, theerror canbeboundedintermsof the
spectral conditionnumber .
2
of thematrix %-(Recall that if /
max
and/
min
are
the largest and smalllest eigenvalues of a symmetric positive denite matrix
0, then thespectral condition number of 0 is .
2
(0) =/
max
(0)'/
min
(0)).
If b " is the exact solution of the linear system%" = $, with symmetric
poistivedenitematrix %, then for CG, it can beshown that
jj"
()
b "jj
#
! 2
()
jj"
(0)
b "jj
#
where =(
p
.
2
1)'(
p
.
2
+1) (seeGolub and Van Loan [109,x10.2.8], and
Kaniel [131],and jj1jj
2
#
" (1) %1)). Fromthis relation weseethat thenumber
of iterations to reach a relative reduction of 2 in the error is proportional to
p
.
2
.
1.3.Algorithm
step1 initial guess "
(0)
=0
step2 #
(0)
=$ %"
(0)
step3 3
(1)
=#
(0)
step4 For 4 =1) ---) ,

$
=
%&
(!1)
' &
(!1)
(
%)
(!)
' #)
(!)
(
step5 "
($)
="
($1)
+
$
!
($)
step6 #
($)
=$ %"
($)
=#
($1)

$
%!
($)
step7
*
$
=
%&
(!)
' &
(!)
(
%&
(!1)
' &
(!1)
(
step8 !
($+1)
=#
($)
+*
$
!
($)
step9 check convergence;continueif necessary
step10 End for
3
PD F !"# "pdfFactory Pro" $#%&'( w w w .ahasoft.com .tw /FinePrint/
2.BiConjugate Gradient (BiCG)
2.1.theory
Theupdaterelationsfor residualsintheConjugateGradient methodareaug-
mentedintheBiConjugateGradient method by relations that aresimilar but
based on %
!
instead of %. Thus weupdatetwo sequences of residuals
#
()
=#
(1)

%!
()
) e #
()
=e #
(1)

%
!
e !
()
-
and two sequences of search directions
!
()
=#
(1)
+(
1
!
(1)
) e !
()
= e #
(1)
+(
1
e !
(1)
-
Thechoices

=
%&
("1)
'e&
("1)
(
%#)
(")
'e)
(")
(
) (

=
%&
(")
'e&
(")
(
%&
("1)
'e&
("1)
(
-
ensurethebi-orthogonality relations
5 #
(")
) e #
()
6=5 %!
(")
) e !
()
6=0-
2.3.Convergence
For theoretical results areknown about theconvergenceof BiCG.For sym-
metric positivedenitesystems themethod delivers thesameresults as CG,
but at twice the cost per iteration. For nonsymmetric matrices it has been
shown that in phases of theprocess wherethereis signicant reduction of the
normof theresidual, themethod is moreor less comparableto full GMRES
(intermsof numbersof iterations)(seeFreundandNachtigal[102]). Inpractice
this is often conrmed, but it is also observed that theconvergencebehavior
maybequiteirregular, andthemethodmayevenbreak down. Thebreakdown
situation due to the possible event that 5 #
(1)
) e #
(1)
6# 0-can be circum-
vented by so-call look-ahead strategies(see Parlett,Taylor and Liu[172]).This
leads to complicated codes and is beyond the scope of this book. The other
breakdown situation,5 %!
()
) e !
()
6# 0)occurs when the LU-decomposition
fails, and canberepaired by usinganother decomposition. This is doneinthe
version of QMR.
2.4.Algorithm
step1 initial guess "
(0)
=0
4
PD F !"# "pdfFactory Pro" $#%&'( w w w .ahasoft.com .tw /FinePrint/
step2 e #
(0)
=#
(0)
step3 !
(1)
=#
(0)
step4 e !
(1)
=#
(0)
step5 For 4 =2) 3) ---) ,
(
$1
=
%e&
(!1)
'&
(!1)
(
%e&
(!2)
'&
(!2)
(
step6 !
($)
=#
($1)
+(
($1)
!
($1)
step7 e !
($)
= e #
($1)
+(
($1)
e !
($1)
step8

$
=
%e&
(!1)
' &
(!1)
(
%#)
(!)
' e)
(!)
(
step9 "
($)
="
($1)
+
$
!
($)
step10#
($)
=#
($1)

$
%!
($)
step11 e #
($)
= e #
($1)

$
%e !
($)
step12check convergence;continueif necessary
step13End for
References
[1] RichardL.Burden, J . DouglasFaires, Numerical analysis, 8thed., CA:Brooks/ Cole,2005.
[2]Wikipedia: The Free Encyclopedia, http:/ / en,wikipedia.org/ wiki/ Electro
statics
[3]Zhaojun Bai ...[etal],Templates for thesolution of Linear Systems:Building
Blocks for IterativeMethods.
5
PD F !"# "pdfFactory Pro" $#%&'( w w w .ahasoft.com .tw /FinePrint/