9/20/2016

LeastSquares

LeastSquares
DefinitionandDerivations
Wehavealreadyspentmuchtimefindingsolutionsto

Ax=b
Ifthereisn'tasolution,weattempttoseekthexthatgetsclosesttobeingasolution.Theclosestsuchvector
willbethexsuchthat

Ax=projWb
whereWisthecolumnspaceofA.

NoticethatbprojWbisintheorthogonalcomplementofWhenceinthenullspaceofAT.Henceifxisathis
closestvector,then

AT(bAx)=0ATAx=ATb
NowweneedtoshowthatATAnonsingularsothatwecansolveforx.
Lemma

IfAisanmxnmatrixofrankn,thenATAisnonsingular.

Proof
WewanttoshowthatthenullspaceofATAiszero.If

0=ATAx
thenmultiplyingbothsidesbyxT,weget

0=xTATAx=(Ax)TAx=Ax.Ax=||Ax||2
Ifthemagnitudeofavectoriszero,thenthevectoriszero,hence
http://ltcconline.net/greenl/courses/203/MatrixOnVectors/leastSquares.htm

1/6

9/20/2016

LeastSquares

Ax=0
Since

rank(A)=n
wecanconcludethat

x=0
Wecannowstatethemaintheorem.

Theorem
LetAbeanmxnmatrixorrankn,thenthesystem
Ax=b
hastheuniqueleastsquaressolution
x=(ATA)1ATb
Examples
Example
Findtheleastsquaressolutionto

Ax=b
with

Solution
WecanquicklycheckthatAhasrank2(thefirsttworowsarenotmultiplesofeachother).Hencewecan
compute

Noticethat

http://ltcconline.net/greenl/courses/203/MatrixOnVectors/leastSquares.htm

2/6

9/20/2016

LeastSquares

notexactlyb,butascloseaswearegoingtoget.

LeastSquaresRegressionLine
Offundamentalimportanceinstatisticalanalysisisfindingtheleastsquaresregressionline.
Example
Anengineeristrackingthefrictionindexovermileageofabreakingsystemofavehicle.Sheexpectsthatthe
mileagefrictionrelationshipisapproximatelylinear.Shecollectsfivedatapointsthatareshowinthetable
below.

Mileage
FrictionIndex

2000
20

6000
18

20,000
10

30,000 40,000
6
2

Thegraphbelowshowsthesepoints

Weareinterestedinthelinethatbestfitsthedata.Morespecifically,ifbisthevectoroffrictionindexdata
valuesandyisthevectorconsistingofyvalueswhenwepluginthemileagedataforxandfindybythe
equationoftheline,thenwewantthelinethatminimizesthedistancebetweenbandy.Iftheequationofthe
lineis

ax+b=y
thenwegetthefiveequations

2a+b=20
6a+b=18
20a+b=10
http://ltcconline.net/greenl/courses/203/MatrixOnVectors/leastSquares.htm

3/6

9/20/2016

LeastSquares

30a+b=6
40a+b=2
Thecorrespondingmatrixequationis

Ax=b
or

Althoughthisdoesnothaveanexactsolution,itdoeshaveaclosestsolution.Wehave

Wecanconcludethattheequationoftheregressionlineis

y=0.48x+20.6
BestFittingCurves
Often,alineisnotthebestmodelforthedata.Fortunatelythesametechniqueworksifwewanttouseother
nonlinearcurvestofitthedata.Herewewillexplainhowtofindtheleastsquarescubic.Theprocessforother
polynomialsissimilar.

Example
Abioengineerisstudyingthegrowthofageneticallyengineeredbacteriacultureandsuspectsthatisit
approximatelyfollowsacubicmodel.Hecollectssixdatapointslistedbelow

Timein
Days
Grams

2.1

3.5

4.2

3.1

4.4

6.8

Heassumestheequationhastheform

ax3+bx2+cx+d=y
Thisgivessixequationswithfourunknowns

a+b+c+d=2.1
8a+4b+2c+d=3.5
http://ltcconline.net/greenl/courses/203/MatrixOnVectors/leastSquares.htm

4/6

9/20/2016

LeastSquares

27a+9b+3c+d=4.2
64a+16b+4c+d=3.1
125a+25b+5c+d=4.4
216a+36b+6c+d=6.8
Thecorrespondingmatrixequationis

Wecanusetheleastsquaresequationtofindthebestsolution

Sothatthebestfittingcubicis

y=0.2x32.0x2+6.1x2.3
Thegraphisshownbelow

BacktotheMatricesandVectorsHomePage
BacktotheLinearAlgebraHomePage
http://ltcconline.net/greenl/courses/203/MatrixOnVectors/leastSquares.htm

5/6

9/20/2016

LeastSquares

BacktotheMathDepartmentHomePage
emailQuestionsandSuggestions

http://ltcconline.net/greenl/courses/203/MatrixOnVectors/leastSquares.htm

6/6