SIFT,track,OpenCV,ImageProcessing
EricYuan'sBlog
PerstandoetPraestando
Home
AboutMe
SIFT
ByERIC |Published:OCTOBER9,2013
SIFT(Scaleinvariantfeaturetransform)isoneofpopularfeaturematching
algorithms,itisgoodbecauseofitsseveralattributes.
Search
Tosearch,typeandhitenter
TagCloud
Algorithmalphabeta
activecontour
Therearekindsofprimitivewaystodoimagematching,forsomeimages,even
comparethegrayscalevaluepixelbypixelworkswell.However,whatiffeatures
inimagesareindifferentscales?Whatiffeaturesareindifferentorientations?
Whatiftheilluminationofeachimageisdifferentfromeachother?Whatifpart
ofthefeatureisshelteredbysomething?
Firstletslookatthisexample(picturestolenfromDavidG.Lowespaper):
ArmadilloBilateralFilterBlur
pruning
C++camShiftClusterCNNComputer
VisionConvolutionalNeuralNetworks
DeepLearningdeepnetwork
DynamicProgrammingGradient
DescentimageblendingImageProcessing
LeetCode
imagepyramid
MachineLearning
Matlab
MNISTneural
networksNLPOctaveOpenCV
meanShift
PanoramaPCAPoissonblendingPooling
pyramidblendingRANSACRecurrentNeural
NetworksRNNsseamcarvingSIFTsnake
SoftmaxSparseAutoencoder
UFLDLUnsupervisedLearning
track
ZCA
Pages
AboutMe
Categories
Algorithm
DeepLearning
Sayifwewannafindthetoytrainandfrog,asleftpictures,intheimageonthe
right,alloftheproblemsaboveappear,forexample,thefrog,morethanhalfof
itsbodyisshelteredbythatblackobject.
DynamicProgramming
Graphics
LeetCode
MachineLearning
MachineReadingComprehension
BySiftmethod,wegotthisresult:
Maths
NLP
OpenCV
Somethingelse
Twaddle
Uncategorized
Vision
Archives
July2016
August2015
June2015
April2015
http://ericyuan.me/sift/
1/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
March2015
October2014
September2014
July2014
June2014
May2014
April2014
March2014
February2014
January2014
November2013
October2013
September2013
Reallyeffectivemethod,andOMG!Ididntnoticethetrainingreenrectangle.
Friends
Nowletstalkabouthowthismethodworks.
JekyCui'sblog
QiCai'sBlog
Ingeneral,themethodincludestheseparts:
Imagepyramid
Keypointsfinding
Refiningkeypoints
Keypointsorientationassignment
Localimagedescriptor
IMAGEPYRAMID
ThereisonearticleaboutImagePyramidinmyblog,soIllnotgivemuchdetails
aboutwhatisimagepyramidandhowtogenerateithere.
HereweareusingLaplacianpyramid,i.e.differencebetweendifferentlevelof
Gaussianpyramid,thatisbecause,inLaplacianpyramid,wecaneasilyfindhigh
frequencyinformationoftheimage,andfeaturesofanimage,aremostlyinthese
highfrequencypartsofimage.
Infact,DifferenceofGaussian(DoG)isanapproximationofLaplacianof
Gaussian(LoG).
Hereweactuallyneedtobuildseveralpyramids,fromdifferentscaleofimage,
andeachpyramid,wecallitanoctave:
http://ericyuan.me/sift/
2/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
Foreachoctaveofscalespace,theinitialimageisrepeatedlyconvolvedwith
Gaussianstoproducethesetofscalespaceimagesshownontheleft.Adjacent
GaussianimagesaresubtractedtoproducethedifferenceofGaussianimageson
theright.Aftereachoctave,theGaussianimageisdownsampledbyafactorof
2,andtheprocessrepeated.
Whyweneedtheseoctavepyramids?Thatisforscaleinvariant,bydoingthis,
wecanfindkeypointsindifferentscales.
Thenumberofoctavesandscalesdependonthesizeofyouroriginalimage
(Lowesuggestsmaking4octavespyramidsand5blurlevelsforeachpyramid).
Furthermore,Lowesuggeststhatwecandoublethesizeoftheoriginalimage,
thatisforproducingmorekeypoints.
KEYPOINTSFINDING
Thisstepiseasy,wejustgottheDoGpyramids,andnowwejustfind
maxima/minimaincurrentscale,andfindmaxima/minimainadjacentscales.
FirstineachlevelsinDoGpyramid,wefindpixelswhichisbigger(or
smaller)thantheeightadjacentpixels,andforallthesemaxima/minimapoints,
compareitsvaluewiththevalueofitsadjacentpixelsinlevelsaboveandbelowit.
Andthemaximaandminimapointswefinallyget,havegreaterorleastvalue
thanits9+8+9=26neighbors.
http://ericyuan.me/sift/
3/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
Forthefirstscaleandlastscaleofeachoctave,therearenoenoughadjacent
pixelstocalculate,sowesimplynotconsiderthesescaleswhenfindinglocal
maxima/minima.
REFININGKEYPOINTS
Afterfindingmaximaandminima,thenextstepistoperformadetailedfittothe
nearbydataforlocation,scale,andratioofprincipalcurvatures.This
informationallowspointstoberejectedthathavelowcontrast(andaretherefore
sensitivetonoise)orarepoorlylocalizedalonganedge.
FirstusingTaylorexpansionofscalespacefunction,bythisfunctionandits
derivatives,wecanfindlocationoftheextremum,andthefunctionvalueatthe
extremum,isusefulforrejectingunstableextremawithlowcontrast.
Secondly,wemusteliminateedgeresponses,becausefeaturesonedgesarenot
goodfeatures,whatweneediscorners,thisisaboutHarriscornerdetector,
andmaybeIllcoverthatinfuturearticles.
Infact,ifwewanttofindacorner,wedonothingjustcalculategradientsof
twodifferentorientationofanimage,ifbothofthesegradientsarebig,itisa
cornerelseifonlyonegradientisbigandtheotherissmall,thispointisonan
edgeelsethispointisinaflatregion.
KEYPOINTSORIENTATIONASSIGNMENT
Byassigningaconsistentorientationtoeachkeypointbasedonlocalimage
properties,thekeypointdescriptor(willbecoveredinnextstep)canbe
representedrelativetothisorientationandthereforeachieveinvariancetoimage
rotation.
Assumingwearegoingtoassignorientationtothebelowkeypoint:
Wechoosearegionwhichthekeypointisinthecenterofit,andthesizeofthis
regionisuptothescalewhichthatkeypointisin.
http://ericyuan.me/sift/
4/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
Byusingtheaboveformula,wecancalculatethegradientmagnitudemand
orientationthetaofeachpixelintheregionwechose:
andanorientationhistogramisformedwiththeseorientationsand
magnitudes.Theorientationhistogramhas36binscoveringthe360degree
rangeoforientations.Eachsampleaddedtothehistogramisweightedbyits
gradientmagnitudeandbyaGaussianweightedcircularwindowwithathatis
1.5timesthatofthescaleofthekeypoint.
Sayaboveisthehistogramwejustcreated,wecanseethepeakisat2029
degrees,andwesimplyassignthekeypointorientation3(thethirdbinof
histogram)furthermore,wecanseeanotherpeakwhichat300309degrees,
andthevalueofitisover80%ofthethirdbinvalue,underthiscircumstance,we
createanewkeypointwhichisassignedbyorientation31(whichmeansthe31st
binofhistogram).
LOCALIMAGEDESCRIPTOR
Thepreviousoperationshaveassignedanimagelocation,scale,andorientation
toeachkeypoint.Thenextstepistocomputeadescriptorforthelocalimage
regionthatishighlydistinctiveyetisasinvariantaspossibletoremaining
variations.
Fordoingthis,wefirstcreatea16*16pixelwindowwhichhavegradientvaluesof
pixelsaroundkeypointinit,andwebreakthiswindowintosixteen4*4pixel
windows,ofcourse,inordertoachieveorientationinvariance,thecoordinatesof
http://ericyuan.me/sift/
5/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
thedescriptorandthegradientorientationsarerotatedrelativetothekeypoint
orientation.
Sincewealreadyknowthegradientmagnitudesandorientationsofthesepixels,
justlikethepreviousstep,weputorientationsofthesepixelsintoa8bin
histogram(45degreesperbin).
Additionally,aGaussianweightingfunctionwithequaltoonehalfthewidthof
thedescriptorwindowisusedtoassignaweighttothemagnitudeofeachsample
point.ThepurposeofthisGaussianwindowistoavoidsuddenchangesinthe
descriptorwithsmallchangesinthepositionofthewindow,andtogiveless
emphasistogradientsthatarefarfromthecenterofthedescriptor,astheseare
mostaffectedbymisregistrationerrors.Thisprocessisshowedinthepicture
below,andinthepicture,assumingweuseda8*8pixelswindow,andthe
windowwasbrokenintofour4*4pixelwindows.
Bynow,wehavea4*4arrayofhistogramswith8orientationbinsineach,that
is,a4*4*8=128elementfeaturevectorforeachkeypoint.
Finally,thefeaturevectorismodifiedtoreducetheeffectsofilluminationchange
bynormalizingvectorslength.
Ifwedidallaboveprocesstotwoimages,saywegotk1keypointsinimage1,and
k2keypointsinimage2,sowehavek1*128vectorsinimage1,andk2*128vectors
inimage2,whatwedoissimplycompareeachkeypointsvectorsinimage1with
keypointsvectorsinimage2,andfindiftherearepairsofkeypoints.
http://ericyuan.me/sift/
6/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
OPENCVCODE
ItissupereasytodothisinOpenCV,theSIFTclassalreadydideverythingfor
you,sojustuseit!
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#include<iostream>
#include<opencv2/opencv.hpp>
#include<opencv2/nonfree/nonfree.hpp>
#include"opencv2/legacy/legacy.hpp"
usingnamespacecv;
usingnamespacestd;
intmain(intargc,char**argv)
{
//readimages
Matimg_1c=imread("img1.jpg");
Matimg_2c=imread("img2.jpg");
Matimg_1,img_2;
//transformimagesintograyscale
cvtColor(img_1c,img_1,CV_BGR2GRAY);
cvtColor(img_2c,img_2,CV_BGR2GRAY);
SIFTsift(50,5);
vector<KeyPoint>key_points_1,key_points_2;
Matdetector;
//dosift,findkeypoints
sift(img_1,Mat(),key_points_1,detector);
sift(img_2,Mat(),key_points_2,detector);
SiftDescriptorExtractorextractor;
Matdescriptors_1,descriptors_2;
//computedescriptors
extractor.compute(img_1,key_points_1,descriptors_1);
extractor.compute(img_2,key_points_2,descriptors_2);
//useburteforcemethodtomatchvectors
BruteForceMatcher<L2<float>>matcher;
vector<DMatch>matches;
matcher.match(descriptors_1,descriptors_2,matches);
//drawresults
Matimg_matches;
drawMatches(img_1c,key_points_1,img_2c,key_points_2,ma
tches,img_matches);
imshow("sift_Matches",img_matches);
waitKey(0);
return0;
}
YoucanalsochangevalueswhencreatingSIFT,by:
1 SIFT::SIFT(intnfeatures=0,intnOctaveLayers=3,doubleco
ntrastThreshold=0.04,doubleedgeThreshold=10,doublesigm
a=1.6);
TESTRESULT
(using50keypointsand5octavelayers)
http://ericyuan.me/sift/
7/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
ThisentrywaspostedinAlgorithm,OpenCVandtaggedImageProcessing,OpenCV,SIFT,
track.Bookmarkthepermalink.Postacommentorleaveatrackback:TrackbackURL.
BilateralFiltering
PoissonBlending
OneComment
Susannah
PostedFebruary4,2016at1:07pm|Permalink
Imalittleconfusedbythecodeinlines2529.IfImreadingthedocumentation
correctly,theSIFToperator(lines2223ofyourcode)alreadycomputesthedescriptors.
Soitappearstomethatyourcodeiscomputingthedescriptorstwiceisthatcorrect?
Reply
OneTrackback
BySURF,SIFT,OpenCV,ComputerVisiononNovember13,2013at11:30am
PostaComment
Youremailisneverpublishednorshared.Requiredfieldsaremarked*
Comment
http://ericyuan.me/sift/
8/9
9/9/2016
SIFT,track,OpenCV,ImageProcessing
YoumayusetheseHTMLtagsandattributes<ahref=""title=""><abbrtitle=""><acronym
title=""><b><blockquotecite=""><cite><code><deldatetime=""><em><i><qcite="">
<s><strike><strong>
Name*
Email*
Website
PostComment
PoweredbyWordPress.BuiltontheThematicThemeFramework.
http://ericyuan.me/sift/
9/9