Anda di halaman 1dari 10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

ABriefHistoryofMachineLearning
PublishedonNovember2,2016

Dr.JaideepGangulyFollow

DirectorofSoftwareDevelopment,Amazon

305

16

63

ThebuzzaroundMachineLearningandDeepLearningpromptedmetotracethehistoryofArtificial
IntelligenceatMITandelsewhereandtakestockofthecurrentstateofMachineLearning.Beforeweget
started,aquickoverviewofsometermsLearningisacquisitionofknowledge,discoveryistheobservationof
anewphenomenaandinventionistheprocessofmakingsomethingnew.Learningisnecessaryforinvention
butisnotasufficientconditionforinnovation.MachineLearningasitstandstoday,doesnotinventbutit
doesdiscoverpatternsinlargequantitiesofdata.Inparticular,DeepNeuralNetworks,haveattractedthe
imaginationofmanybecauseofsomeinterestingsolutionsitoffersinallthreechannelstext,speechand
images.Incidentally,mostdeepneuralnetsareratherwideandareusuallynotmorethantenlayersdeep.So
thenameshouldhavereallybeen"wideneuralnets"buttheword"deep"hasstuck.

"Thequestionofwhetheracomputercanthinkisnomoreinterestingthanthequestionofwhethera
submarinecanswim",saidDijkstra.ItismoreinterestingtounderstandtheevolutionofMachineLearning

howdiditstart,wherearewetodayandwheredowegofromhere.Thehumanbrainisaremarkablething
ithasenabledusunderstandscienceandadvancemankind.Theideaofmimickingthehumanbrainoreven
improvingthehumancognitivefunctionsisanalluringoneandisanobjectiveofArtificialIntelligence
research.Butwearenotevencloseinspiteofacenturyofresearch.However,itcontinuestohaveamajor
holdonourimaginationgiventhepotentialoftherewards.

Itseemsthatabout50,000yearsago,afterwehavebeenaroundforaboutahundredthousandyearsorso,
palaeontologistsbelievethatsomeofus,possiblyjustafewthousand,wereabletodealwithsymbols.Thisis
amajorstepinevolution.NoamChomskythinkswewerethenabletocreateanewconceptfrom2existing
ideasorconceptswithoutdamagingorlimitingtheexistingconcepts.Around350
BC,Aristotledevisedsyllogisticlogic,thefirstformaldeductivereasoningsystemtomodelthewayhumans
thinkabouttheirworldandreasonwithit.2,000yearslater,BertrandRusselandAlfred
WhiteheadpublishedPrincipiaMathematicathatlaiddownthefoundationsforaformalrepresentationof
Mathematics.JohnMcCarthy,whochampionedthecauseofmathematicallogicinAI,wasAristotleofhis
day.In1942,AlanTuringshowedthatanyformofmathematicalreasoningcouldbeprocessedbyamachine.
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

1/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

By1967,MarvinMinskydeclaredthat"withinageneration,theproblemofcreatingArtificial
Intelligencewouldsubstantiallybesolved".Clearlywearenotthereyet,attemptstobuildsystemswith

firstorderlogicasdescribedbyearlyphilosophersfailedbecauseoflackofcomputingpower,inabilityto
dealwithuncertaintyandlackoflargeamountsofdata.

In1961,Minskypublished"StepstowardsArtificialIntelligence"wherehetalkedaboutsearch,matching,
probability,learninganditwasquitevisionary.Turingtoldusthatitwaspossibletomakeamachine
intelligentandMinskytoldushow.In1986,Minskywrotethehighlyinfluentialbook"TheSocietyofMind",
24centuriesafterPlatowrote"Politeia"MinskywasPlatoofhisdays.Minskytaughtustothinkabout
heuristicprogramming,McCarthywantedustouselogictotheextreme,Newelwantedtobuildcognitive
modelsofproblemsolvingandSimonbelievedthatwhenweseesomethingthatiscomplicatedinbehavior,it
ismoreofaconsequenceofacomplexenvironmentratherthanbecauseofacomplexthinker.Thereafter,a
numberofmodelbackedsystemswerebuilt.TerryWinogradbuiltamodelbackedsystemfordialog
understanding,PatrickWinstonbuiltanothermodelbackedsystemforlearningandGeraldSussmanbuilta
modelbackedsystemforunderstandingblocks.DuringthesameeraRogerSchankbelievedthat
understandingstoriesisthekeytomodelinghumanintelligence.DavidMarr,whoisbestknownforhiswork
onvision,treatedvisionasaninformationprocessingsystem.Marr'strilevelhypothesisincognitivescience
comprisedofacomputationallevelwhatdoesthesystemdo,analgorithmiclevelhowdoesthesystemdo
andaphysicallevelhowisthesystemphysicallyrealized,e.g.,inthecaseofbiologicalvision,whatneural
structuresandneuronalactivitiesimplementthevisualsystem.

Inthe1980s,theexpertsystemswereofgreatinterestandfocusedonknowledgeandinferencemechanisms.
Whilethesesystemsdidaprettygoodjobintheirdomains,theywerenarrowinspecializationandwere
difficulttoscale.ThefieldofAIwasdefinedascomputersperformingtasksthatwerespecificallythoughtof
assomethingonlyhumanscando.However,oncethesesystemsworked,theywerenolongerconsideredto
beAI!Forexample,todaythebestchessplayersareroutinelydefeatedbycomputersbutchessplayingisno
longerreallyconsideredasAI!McCarthyreferredtoasthe"AIeffect".IBM'sWatsonisaprogramatalevel
suchasthatofahumanexpertbutitisnotcertainlynotthefirstone.FiftyyearsagoJimSlagle'ssymbolic
integrationprogramatMITwasatremendousachievement.Nevertheless,itisveryhardtobuildaprogram
thathas"commonsense"andnotjustnarrowdomainsofknowledge.

Today,atthecoreisthedebatebetweenlogicinspiredandneuralnetworkinspiredparadigmsfor
cognition.LeCun,BengioandHintonstatethatsuccinctlyinareviewpaperinNature,dated28thMay2015,
as"Theissueofrepresentationliesattheheartofthedebatebetweenthelogicinspiredandthe
neuralnetworkinspiredparadigmsforcognition.Inthelogicinspiredparadigm,aninstanceofa
symbolissomethingforwhichtheonlypropertyisthatitiseitheridenticalornonidenticaltoother
symbolinstances.Ithasnointernalstructurethatisrelevanttoitsuseandtoreasonwithsymbols,
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

2/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

theymustbeboundtothevariablesinjudiciouslychosenrulesofinference.Bycontrast,neural
networksjustusebigactivityvectors,bigweightmatricesandscalarnonlinearitiestoperformthe
typeoffastintuitiveinferencethatunderpinseffortlesscommonsensereasoning".

RosenblattiscreditedwiththeconceptofPerceptrons,amachinewhichsenses,recognizes,remembers,
andrespondslikethehumanmindasearlyasin1957butinacriticalbookwrittenin1969byMarvin

MinskyandSeymourPapertshowedthatRosenblattsoriginalsystemwaspainfullylimited,literallyblindto
somesimplelogicalfunctionslikeXOR.Inthebooktheysaid:"...ourintuitivejudgmentthatthe
extension(tomultilayersystems)issterile".ThisintuitionwasincorrectandthefieldofNeural

Networksprettymuchdisappeared!GeoffHintonbuiltmorecomplexnetworksofvirtualneuronsthat
allowedanewgenerationofnetworkstolearnmorecomplicatedfunctions(liketheexclusiveorthathad
bedeviledtheoriginalPerceptron).Eventhenewmodelshadseriousproblemsthough.Theylearnedslowly
andinefficientlyandcouldntmasterevensomeofthebasicthingsthatchildrendo.Bythelate1990s,neural
networkshadagainbeguntofalloutoffavor.In2006,Hintondevelopedanewtechniquethathedubbed
deeplearning,whichextendsearlierimportantworkbyYannLeCun.Deeplearningsimportantinnovationis
tohavemodelslearncategoriesincrementally,attemptingtonaildownlowerlevelcategories(likeletters)
beforeattemptingtoacquirehigherlevelcategories(likewords).

InApril2000,inaseminalworkpublishedinNaturebyMrigankaSur,et.al,atMIT'slaboratoryforbrain
andcognitivesciences,theauthorswereabletosuccessfullyrewire"brainsinveryyoungmammals,inputs
fromtheeyeweredirectedtobrainstructuresthatnormallyprocesshearing.Theanimal'sauditorycortex
successfullyinterpretedinputfromitseyes.Butitdidn'tdothejobaswellastheprimaryvisualcortexwould
have,suggestingthatwhilethebrain'splasticity,orabilitytoadapt,isenormous,itislimitedbygenetic
preprogramming.Environmentalinput,whilekeytothedevelopmentofbrainfunction,doesnot"writeona
blankslate".Thisaddressesanageoldquestionisthebrainisgeneticallyprogrammedorshapedby

environment?Itisadramaticevidenceoftheabilityofthedevelopingbraintoadapttochangesinthe
externalenvironment,andspeakstotheenormouspotentialandplasticityofthecerebralcortextheseatof
ourhighestabilities.Thisprovidedsometheoreticalunderpinningtotheneuralnetcomputationtheory.

DeepneuralnetsspawnedasubsetknownasRecurrentNeuralNetswhichwereanattempttomodel
sequentialevents.SupportVectorMachines,logisticregression,feedforwardnetworkshaveprovedvery
usefulwithoutexplicitlymodelingtime.Buttheassumptionofindependenceprecludesmodelinglongrange
dependencies.DNNswerealsohelpedbytheemergenceofGPUswhichenabledparallelismasmuchof
computationinDNNisintrinsicallyparallelinnature.RNNsareconnectionistmodelswiththeabilityto
selectivelypassinformationacrosssequencestepswhileprocessingsequentialdataoneelementatatime.
Theycanmodelinputand/oroutputconsistingofsequencesofelementsthatarenotindependent.However,
learningwithrecurrentnetworksisdifficult.Forstandardfeedforwardnetworks,theoptimizationtaskisNP
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

3/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

complete.Learningwithrecurrentnetworksischallengingduetothedifficultyoflearninglongrange
dependencies.Problemsofvanishingandexplodinggradientsoccurwhenbackpropagatingerrorsacross
manytimesteps.In1997,HochreiterandSchmidhuberintroducedtheLongShortTermMemory(LSTM)
modeltoovercomevanishinggradients.LSTMshavebeenproventoberemarkableinspeechand
handwritingrecognition.Similarly,anothervariationoftheDeepNetModelistheConvolutionNeural
Network(CNN)thathasbeenverysuccessfulinclassifyingimages.

Inconclusion,wehavecomealongway.DeepNetsappeartobeverypromisinginsomeareasalthoughthey
arecomputationallyveryexpensive.However,deeplearningisonlypartofthelargerchallengeofbuilding
intelligentmachines.Itlackswaysofrepresentingcausalrelationships,havenoobviouswaysofperforming
logicalinferences,andisstillalongwayfromintegratingabstractknowledge,suchasinformationaboutwhat
objectsare,whattheyarefor,andhowtheyaretypicallyused.ThemostpowerfulA.I.systems,likeWatson
usetechniqueslikedeeplearningasjustoneelementinaverycomplicatedensembleoftechniques,ranging
fromthestatisticaltechniqueofBayesianinferencetodeductivereasoning.

Thename"MachineLearning"isindicativeofthepotentialsthatitcanpossiblyachieveinthefuture.Inthe
nextarticle,Iwilltalkaboutwhatproblemsintheindustrythatcanbesolvedwiththecurrentstateof
technologyanditsevolutioninthenextcoupleofyears.

Reportthis

Dr.JaideepGanguly

DirectorofSoftwareDevelopment,Amazon

Follow

1post

16comments
Recommended
Leaveyourthoughtshere

https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

4/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

3d

Sabyasachi"Sky"Basu
ThoughtLeaderofDigitalInnovation
Jaideep,agreatintroductiontoMachineLearning.

"However,oncethesesystemsworked,theywerenolongerconsideredtobeAI!"

Inmid80sImetthegranddameofCognitiveScienceandAIMargaretBoden
(https://en.wikipedia.org/wiki/Margaret_Boden)inaconference.Iaskedheraverybasic
question"WhatisAI?".Andheranswerwas(Iamparaphrasing)

"Innaturalscience,whenwedonotknowhowanaturalphenomenonworkswecallit
'metaphysics',whenweknowwecallit'physics','chemistry','biology'.SimilarlyinComputer
Sciencewhenwedon'tknowhowtosolveaproblemwecallit'AI',andwhenwecansolveit
fairlywellwethencallit'database','computergraphics','networking'."

Ithoughtthenandstillthinkthereisawonderfulwisdominthatexplanation.
Like

Reply

12

2d

Dr.JaideepGanguly
DirectorofSoftwareDevelopment,Amazon
Wellsaid.AndthatiswhyAIgetsthrownunderthebusfromtimetotime,manyofthe
thingsthatwetakeforgrantedtodayarebecauseoftheworkdoneinthisfield.
Like

Reply

4d

KPMDas
Director,CybersecurityandTrust,IndiaatCiscoSystems
Brilliantprimerforastatustakepartone.....recallhavingmadelittleprogresswith"rules"based
reasoninginthelateeightiesonPROLOGandLISPimplementations,thiswaveofMLseems
prodigious...............lookforwardtoparttwo

https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

5/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

Like

Reply

4d

Dr.JaideepGanguly
DirectorofSoftwareDevelopment,Amazon
ThanksKPM!
Like

Reply

2d

RonKaplan
VicePresidentandChiefScientistatA9.com(Amazon)
HiJaideep,
Verynicesummary.Iwouldpointoutthatthedefinitionof"learning"asacquiring"knowledge"
begsadeeperquestion:whatisknowledge?

Ononeview,knowledgeisinformationthatcanbedemonstratedorputtomultipleuses,
inspectedinthecourseofmakinginferences,andtransmittedtootherssothattheycanalso
makeuseofit.Onthatview,learninginthesenseofacquiringknowledgeisdifferentfrom
learninginthesenseofacquiringaskill.IfIhavelearnedthelocationofaparticularbusiness,I
candemonstratethatIhavethatknowledgebygoingthere,butIcanalsotellsomeoneelsehow
togothere,estimatethedistancefromotherlocations,etc.ButwhenIhavelearnedtheskill,
say,ofridingabike.Ican'tcommunicatewhatIhavelearnedtosomeoneelseinawaythatwill
enablethemtoalsorideabike.Theyhavetolearnitbythemselves,presumablybylotsoftheir
ownpractice.

Atleastintheircurrentstateofdevelopment,Ithinkwhatdeeplearningsystemsacquireand
representintheirnetworkmodelsismorelikeaskillthantransmissibleandinspectable
knowledge.Icandeliveramodeltosomebodyelsesothattheycanexecuteit(whichisagood
thing),butthereisn'tmuchelsethatIortheycandobeyondthat.IfIhaveagoodspeech
recognitionmodel,IcanrecognizespeechbutIcan'tsaymuchifanythingabouthowthatis
accomplished(whichmightbeusefulforotherapplications).

Theknowledge/skill(orhabit)distinctionharkensbacktothecognitivereactiontothebehaviorist
traditioninpsychology,theargument(byChomskyandothers)thatitisimportanttorecognize
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

6/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

thedifferencebetweenacquiringknowledgeintheformofinternal,manipulablemental
representationsandacquiringtheabilitytomapfrominputstocorrelatedoutputs(stimulito
responses,acousticwavestowordsequences,imagestoclasses,etc.)Andsincewebothhave
nowmentionedChomsky,ofrelevancetothiskindofdiscussionmightbeChomsky'sreviewof
Skinner'sVerbalBehavior,writteninthe50's.Thetechnologyisnew,buttheissueshavebeen
aroundforawhile.

Best,Ron
Like

Reply

1d

StphanePisani
tudiantchercheur|Enrdactiondemmoire
Ilikethis'genealogicway'ofdescribingmachinelearning.Ihavenowabetterunderstandingof
allconceptsrelatedtomachinelearning.Thetracingoftheirlineagesandhistorycouldbeeasily
capturedinamindmapformatforfuturereference.ThanksalotJaideep!
Like

Reply

2d

RajaBoddu.Ph.D
Principal&Head,R&DatLenoraCollegeofEngineering
Goodone
Like

Reply

2d

SubbuMandiga
ManagerIII,SoftwareDevelopmentatAmazon
Awesomeread...lookingforwardforthefollowuparticle
Like

Reply

3d

AnandIyer
SeniorTestArchitect
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

7/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

Whenyoutalkedaboutthesocalled'debatebetweenlogicinspiredandneuralnetworkinspired
paradigmsforcognition',itclearedupalongstandingquestiononmymind.and,thatwasthis
howdoIdistinguishbetweenAIandasoftwareprogram,oristhereadifferenceatall?
especially,giventhattheformeralsorequiressoftwareprogrammingatsomelevel.

Atleast,nowIknoweventheexpertsaresplitbetweenthetwo!

Thanksfortheenlightenment,Dr.JaideepGanguly.Lookingforwardtothenextpart!
Like

Reply

3d

VaibhavMittal
CompensationandBenefitsSpecialistGlobalCompetencyCenter
atXerox
PriyalMittal
Like

Reply

2d

RavindraPrasad
HeadofEngineeringandTechnologyatTesco
GreatinsightsintoMachinelearning.BestarticleIreadsofaronMachinelearning.Thankyou
Jaideep.
Like

Reply

1d

Dr.RaviVadlamani
Professor&Head,CenterofExcellenceinAnalytics,Institutefor
DevelopmentandResearchin
Verywellcaptured.However,onemustnotforgetIvakhnenko'sworkonGroupMethodofData
Handling(GMDH)networkof1965,whichisthefirst'deeplearning'neuralnetwork,eventhough
hedidnotcointhatword.
Like

Reply

4d
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

8/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

SharmilaGanguly
DesignConsultant
Excellentresearch
Like

Reply

3d

SouvikKar
SoftwareDevelopmentEngineerIIatAmazon
Awesomeread
Like

Reply
3d

SanjayHora
Founder,TechArda.com
nicebackgroundinformationonML.cantwaittoseethenextpost:)
Like

Reply
3d

MethilSreekumar
consultant,strategictelecommanagement,generaladministration
Averyinterestingarticle,chronologicallylaidouttopickuplostthreads.Atthistime,Ihaveonly
onetocomment,isevolutionofAIpromptingordrivinghumanintelligencetopursue
unimaginablefeats/heightsandmaintainstheproverbialgapwiththeformerlaggingbehind.
Lookingforwardforthenextsequeltoflow.
MethilSreekumar
Like

Reply
2d

ShreedharTorgal
Technologist|EnterpriseArchitect|ITStrategy|cognitive
Computing|Programmanagement|CxO
Goodone..
Like

Reply

https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

9/10

11/7/2016

ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn

2h

RaviVenkatesan
ResearchDirectoratSystemsResearchCorporation
Thanksforsharing.
Like

Reply

https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment

10/10

Anda mungkin juga menyukai