ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
ABriefHistoryofMachineLearning
PublishedonNovember2,2016
Dr.JaideepGangulyFollow
DirectorofSoftwareDevelopment,Amazon
305
16
63
ThebuzzaroundMachineLearningandDeepLearningpromptedmetotracethehistoryofArtificial
IntelligenceatMITandelsewhereandtakestockofthecurrentstateofMachineLearning.Beforeweget
started,aquickoverviewofsometermsLearningisacquisitionofknowledge,discoveryistheobservationof
anewphenomenaandinventionistheprocessofmakingsomethingnew.Learningisnecessaryforinvention
butisnotasufficientconditionforinnovation.MachineLearningasitstandstoday,doesnotinventbutit
doesdiscoverpatternsinlargequantitiesofdata.Inparticular,DeepNeuralNetworks,haveattractedthe
imaginationofmanybecauseofsomeinterestingsolutionsitoffersinallthreechannelstext,speechand
images.Incidentally,mostdeepneuralnetsareratherwideandareusuallynotmorethantenlayersdeep.So
thenameshouldhavereallybeen"wideneuralnets"buttheword"deep"hasstuck.
"Thequestionofwhetheracomputercanthinkisnomoreinterestingthanthequestionofwhethera
submarinecanswim",saidDijkstra.ItismoreinterestingtounderstandtheevolutionofMachineLearning
howdiditstart,wherearewetodayandwheredowegofromhere.Thehumanbrainisaremarkablething
ithasenabledusunderstandscienceandadvancemankind.Theideaofmimickingthehumanbrainoreven
improvingthehumancognitivefunctionsisanalluringoneandisanobjectiveofArtificialIntelligence
research.Butwearenotevencloseinspiteofacenturyofresearch.However,itcontinuestohaveamajor
holdonourimaginationgiventhepotentialoftherewards.
Itseemsthatabout50,000yearsago,afterwehavebeenaroundforaboutahundredthousandyearsorso,
palaeontologistsbelievethatsomeofus,possiblyjustafewthousand,wereabletodealwithsymbols.Thisis
amajorstepinevolution.NoamChomskythinkswewerethenabletocreateanewconceptfrom2existing
ideasorconceptswithoutdamagingorlimitingtheexistingconcepts.Around350
BC,Aristotledevisedsyllogisticlogic,thefirstformaldeductivereasoningsystemtomodelthewayhumans
thinkabouttheirworldandreasonwithit.2,000yearslater,BertrandRusselandAlfred
WhiteheadpublishedPrincipiaMathematicathatlaiddownthefoundationsforaformalrepresentationof
Mathematics.JohnMcCarthy,whochampionedthecauseofmathematicallogicinAI,wasAristotleofhis
day.In1942,AlanTuringshowedthatanyformofmathematicalreasoningcouldbeprocessedbyamachine.
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
1/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
By1967,MarvinMinskydeclaredthat"withinageneration,theproblemofcreatingArtificial
Intelligencewouldsubstantiallybesolved".Clearlywearenotthereyet,attemptstobuildsystemswith
firstorderlogicasdescribedbyearlyphilosophersfailedbecauseoflackofcomputingpower,inabilityto
dealwithuncertaintyandlackoflargeamountsofdata.
In1961,Minskypublished"StepstowardsArtificialIntelligence"wherehetalkedaboutsearch,matching,
probability,learninganditwasquitevisionary.Turingtoldusthatitwaspossibletomakeamachine
intelligentandMinskytoldushow.In1986,Minskywrotethehighlyinfluentialbook"TheSocietyofMind",
24centuriesafterPlatowrote"Politeia"MinskywasPlatoofhisdays.Minskytaughtustothinkabout
heuristicprogramming,McCarthywantedustouselogictotheextreme,Newelwantedtobuildcognitive
modelsofproblemsolvingandSimonbelievedthatwhenweseesomethingthatiscomplicatedinbehavior,it
ismoreofaconsequenceofacomplexenvironmentratherthanbecauseofacomplexthinker.Thereafter,a
numberofmodelbackedsystemswerebuilt.TerryWinogradbuiltamodelbackedsystemfordialog
understanding,PatrickWinstonbuiltanothermodelbackedsystemforlearningandGeraldSussmanbuilta
modelbackedsystemforunderstandingblocks.DuringthesameeraRogerSchankbelievedthat
understandingstoriesisthekeytomodelinghumanintelligence.DavidMarr,whoisbestknownforhiswork
onvision,treatedvisionasaninformationprocessingsystem.Marr'strilevelhypothesisincognitivescience
comprisedofacomputationallevelwhatdoesthesystemdo,analgorithmiclevelhowdoesthesystemdo
andaphysicallevelhowisthesystemphysicallyrealized,e.g.,inthecaseofbiologicalvision,whatneural
structuresandneuronalactivitiesimplementthevisualsystem.
Inthe1980s,theexpertsystemswereofgreatinterestandfocusedonknowledgeandinferencemechanisms.
Whilethesesystemsdidaprettygoodjobintheirdomains,theywerenarrowinspecializationandwere
difficulttoscale.ThefieldofAIwasdefinedascomputersperformingtasksthatwerespecificallythoughtof
assomethingonlyhumanscando.However,oncethesesystemsworked,theywerenolongerconsideredto
beAI!Forexample,todaythebestchessplayersareroutinelydefeatedbycomputersbutchessplayingisno
longerreallyconsideredasAI!McCarthyreferredtoasthe"AIeffect".IBM'sWatsonisaprogramatalevel
suchasthatofahumanexpertbutitisnotcertainlynotthefirstone.FiftyyearsagoJimSlagle'ssymbolic
integrationprogramatMITwasatremendousachievement.Nevertheless,itisveryhardtobuildaprogram
thathas"commonsense"andnotjustnarrowdomainsofknowledge.
Today,atthecoreisthedebatebetweenlogicinspiredandneuralnetworkinspiredparadigmsfor
cognition.LeCun,BengioandHintonstatethatsuccinctlyinareviewpaperinNature,dated28thMay2015,
as"Theissueofrepresentationliesattheheartofthedebatebetweenthelogicinspiredandthe
neuralnetworkinspiredparadigmsforcognition.Inthelogicinspiredparadigm,aninstanceofa
symbolissomethingforwhichtheonlypropertyisthatitiseitheridenticalornonidenticaltoother
symbolinstances.Ithasnointernalstructurethatisrelevanttoitsuseandtoreasonwithsymbols,
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
2/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
theymustbeboundtothevariablesinjudiciouslychosenrulesofinference.Bycontrast,neural
networksjustusebigactivityvectors,bigweightmatricesandscalarnonlinearitiestoperformthe
typeoffastintuitiveinferencethatunderpinseffortlesscommonsensereasoning".
RosenblattiscreditedwiththeconceptofPerceptrons,amachinewhichsenses,recognizes,remembers,
andrespondslikethehumanmindasearlyasin1957butinacriticalbookwrittenin1969byMarvin
MinskyandSeymourPapertshowedthatRosenblattsoriginalsystemwaspainfullylimited,literallyblindto
somesimplelogicalfunctionslikeXOR.Inthebooktheysaid:"...ourintuitivejudgmentthatthe
extension(tomultilayersystems)issterile".ThisintuitionwasincorrectandthefieldofNeural
Networksprettymuchdisappeared!GeoffHintonbuiltmorecomplexnetworksofvirtualneuronsthat
allowedanewgenerationofnetworkstolearnmorecomplicatedfunctions(liketheexclusiveorthathad
bedeviledtheoriginalPerceptron).Eventhenewmodelshadseriousproblemsthough.Theylearnedslowly
andinefficientlyandcouldntmasterevensomeofthebasicthingsthatchildrendo.Bythelate1990s,neural
networkshadagainbeguntofalloutoffavor.In2006,Hintondevelopedanewtechniquethathedubbed
deeplearning,whichextendsearlierimportantworkbyYannLeCun.Deeplearningsimportantinnovationis
tohavemodelslearncategoriesincrementally,attemptingtonaildownlowerlevelcategories(likeletters)
beforeattemptingtoacquirehigherlevelcategories(likewords).
InApril2000,inaseminalworkpublishedinNaturebyMrigankaSur,et.al,atMIT'slaboratoryforbrain
andcognitivesciences,theauthorswereabletosuccessfullyrewire"brainsinveryyoungmammals,inputs
fromtheeyeweredirectedtobrainstructuresthatnormallyprocesshearing.Theanimal'sauditorycortex
successfullyinterpretedinputfromitseyes.Butitdidn'tdothejobaswellastheprimaryvisualcortexwould
have,suggestingthatwhilethebrain'splasticity,orabilitytoadapt,isenormous,itislimitedbygenetic
preprogramming.Environmentalinput,whilekeytothedevelopmentofbrainfunction,doesnot"writeona
blankslate".Thisaddressesanageoldquestionisthebrainisgeneticallyprogrammedorshapedby
environment?Itisadramaticevidenceoftheabilityofthedevelopingbraintoadapttochangesinthe
externalenvironment,andspeakstotheenormouspotentialandplasticityofthecerebralcortextheseatof
ourhighestabilities.Thisprovidedsometheoreticalunderpinningtotheneuralnetcomputationtheory.
DeepneuralnetsspawnedasubsetknownasRecurrentNeuralNetswhichwereanattempttomodel
sequentialevents.SupportVectorMachines,logisticregression,feedforwardnetworkshaveprovedvery
usefulwithoutexplicitlymodelingtime.Buttheassumptionofindependenceprecludesmodelinglongrange
dependencies.DNNswerealsohelpedbytheemergenceofGPUswhichenabledparallelismasmuchof
computationinDNNisintrinsicallyparallelinnature.RNNsareconnectionistmodelswiththeabilityto
selectivelypassinformationacrosssequencestepswhileprocessingsequentialdataoneelementatatime.
Theycanmodelinputand/oroutputconsistingofsequencesofelementsthatarenotindependent.However,
learningwithrecurrentnetworksisdifficult.Forstandardfeedforwardnetworks,theoptimizationtaskisNP
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
3/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
complete.Learningwithrecurrentnetworksischallengingduetothedifficultyoflearninglongrange
dependencies.Problemsofvanishingandexplodinggradientsoccurwhenbackpropagatingerrorsacross
manytimesteps.In1997,HochreiterandSchmidhuberintroducedtheLongShortTermMemory(LSTM)
modeltoovercomevanishinggradients.LSTMshavebeenproventoberemarkableinspeechand
handwritingrecognition.Similarly,anothervariationoftheDeepNetModelistheConvolutionNeural
Network(CNN)thathasbeenverysuccessfulinclassifyingimages.
Inconclusion,wehavecomealongway.DeepNetsappeartobeverypromisinginsomeareasalthoughthey
arecomputationallyveryexpensive.However,deeplearningisonlypartofthelargerchallengeofbuilding
intelligentmachines.Itlackswaysofrepresentingcausalrelationships,havenoobviouswaysofperforming
logicalinferences,andisstillalongwayfromintegratingabstractknowledge,suchasinformationaboutwhat
objectsare,whattheyarefor,andhowtheyaretypicallyused.ThemostpowerfulA.I.systems,likeWatson
usetechniqueslikedeeplearningasjustoneelementinaverycomplicatedensembleoftechniques,ranging
fromthestatisticaltechniqueofBayesianinferencetodeductivereasoning.
Thename"MachineLearning"isindicativeofthepotentialsthatitcanpossiblyachieveinthefuture.Inthe
nextarticle,Iwilltalkaboutwhatproblemsintheindustrythatcanbesolvedwiththecurrentstateof
technologyanditsevolutioninthenextcoupleofyears.
Reportthis
Dr.JaideepGanguly
DirectorofSoftwareDevelopment,Amazon
Follow
1post
16comments
Recommended
Leaveyourthoughtshere
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
4/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
3d
Sabyasachi"Sky"Basu
ThoughtLeaderofDigitalInnovation
Jaideep,agreatintroductiontoMachineLearning.
"However,oncethesesystemsworked,theywerenolongerconsideredtobeAI!"
Inmid80sImetthegranddameofCognitiveScienceandAIMargaretBoden
(https://en.wikipedia.org/wiki/Margaret_Boden)inaconference.Iaskedheraverybasic
question"WhatisAI?".Andheranswerwas(Iamparaphrasing)
"Innaturalscience,whenwedonotknowhowanaturalphenomenonworkswecallit
'metaphysics',whenweknowwecallit'physics','chemistry','biology'.SimilarlyinComputer
Sciencewhenwedon'tknowhowtosolveaproblemwecallit'AI',andwhenwecansolveit
fairlywellwethencallit'database','computergraphics','networking'."
Ithoughtthenandstillthinkthereisawonderfulwisdominthatexplanation.
Like
Reply
12
2d
Dr.JaideepGanguly
DirectorofSoftwareDevelopment,Amazon
Wellsaid.AndthatiswhyAIgetsthrownunderthebusfromtimetotime,manyofthe
thingsthatwetakeforgrantedtodayarebecauseoftheworkdoneinthisfield.
Like
Reply
4d
KPMDas
Director,CybersecurityandTrust,IndiaatCiscoSystems
Brilliantprimerforastatustakepartone.....recallhavingmadelittleprogresswith"rules"based
reasoninginthelateeightiesonPROLOGandLISPimplementations,thiswaveofMLseems
prodigious...............lookforwardtoparttwo
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
5/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
Like
Reply
4d
Dr.JaideepGanguly
DirectorofSoftwareDevelopment,Amazon
ThanksKPM!
Like
Reply
2d
RonKaplan
VicePresidentandChiefScientistatA9.com(Amazon)
HiJaideep,
Verynicesummary.Iwouldpointoutthatthedefinitionof"learning"asacquiring"knowledge"
begsadeeperquestion:whatisknowledge?
Ononeview,knowledgeisinformationthatcanbedemonstratedorputtomultipleuses,
inspectedinthecourseofmakinginferences,andtransmittedtootherssothattheycanalso
makeuseofit.Onthatview,learninginthesenseofacquiringknowledgeisdifferentfrom
learninginthesenseofacquiringaskill.IfIhavelearnedthelocationofaparticularbusiness,I
candemonstratethatIhavethatknowledgebygoingthere,butIcanalsotellsomeoneelsehow
togothere,estimatethedistancefromotherlocations,etc.ButwhenIhavelearnedtheskill,
say,ofridingabike.Ican'tcommunicatewhatIhavelearnedtosomeoneelseinawaythatwill
enablethemtoalsorideabike.Theyhavetolearnitbythemselves,presumablybylotsoftheir
ownpractice.
Atleastintheircurrentstateofdevelopment,Ithinkwhatdeeplearningsystemsacquireand
representintheirnetworkmodelsismorelikeaskillthantransmissibleandinspectable
knowledge.Icandeliveramodeltosomebodyelsesothattheycanexecuteit(whichisagood
thing),butthereisn'tmuchelsethatIortheycandobeyondthat.IfIhaveagoodspeech
recognitionmodel,IcanrecognizespeechbutIcan'tsaymuchifanythingabouthowthatis
accomplished(whichmightbeusefulforotherapplications).
Theknowledge/skill(orhabit)distinctionharkensbacktothecognitivereactiontothebehaviorist
traditioninpsychology,theargument(byChomskyandothers)thatitisimportanttorecognize
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
6/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
thedifferencebetweenacquiringknowledgeintheformofinternal,manipulablemental
representationsandacquiringtheabilitytomapfrominputstocorrelatedoutputs(stimulito
responses,acousticwavestowordsequences,imagestoclasses,etc.)Andsincewebothhave
nowmentionedChomsky,ofrelevancetothiskindofdiscussionmightbeChomsky'sreviewof
Skinner'sVerbalBehavior,writteninthe50's.Thetechnologyisnew,buttheissueshavebeen
aroundforawhile.
Best,Ron
Like
Reply
1d
StphanePisani
tudiantchercheur|Enrdactiondemmoire
Ilikethis'genealogicway'ofdescribingmachinelearning.Ihavenowabetterunderstandingof
allconceptsrelatedtomachinelearning.Thetracingoftheirlineagesandhistorycouldbeeasily
capturedinamindmapformatforfuturereference.ThanksalotJaideep!
Like
Reply
2d
RajaBoddu.Ph.D
Principal&Head,R&DatLenoraCollegeofEngineering
Goodone
Like
Reply
2d
SubbuMandiga
ManagerIII,SoftwareDevelopmentatAmazon
Awesomeread...lookingforwardforthefollowuparticle
Like
Reply
3d
AnandIyer
SeniorTestArchitect
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
7/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
Whenyoutalkedaboutthesocalled'debatebetweenlogicinspiredandneuralnetworkinspired
paradigmsforcognition',itclearedupalongstandingquestiononmymind.and,thatwasthis
howdoIdistinguishbetweenAIandasoftwareprogram,oristhereadifferenceatall?
especially,giventhattheformeralsorequiressoftwareprogrammingatsomelevel.
Atleast,nowIknoweventheexpertsaresplitbetweenthetwo!
Thanksfortheenlightenment,Dr.JaideepGanguly.Lookingforwardtothenextpart!
Like
Reply
3d
VaibhavMittal
CompensationandBenefitsSpecialistGlobalCompetencyCenter
atXerox
PriyalMittal
Like
Reply
2d
RavindraPrasad
HeadofEngineeringandTechnologyatTesco
GreatinsightsintoMachinelearning.BestarticleIreadsofaronMachinelearning.Thankyou
Jaideep.
Like
Reply
1d
Dr.RaviVadlamani
Professor&Head,CenterofExcellenceinAnalytics,Institutefor
DevelopmentandResearchin
Verywellcaptured.However,onemustnotforgetIvakhnenko'sworkonGroupMethodofData
Handling(GMDH)networkof1965,whichisthefirst'deeplearning'neuralnetwork,eventhough
hedidnotcointhatword.
Like
Reply
4d
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
8/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
SharmilaGanguly
DesignConsultant
Excellentresearch
Like
Reply
3d
SouvikKar
SoftwareDevelopmentEngineerIIatAmazon
Awesomeread
Like
Reply
3d
SanjayHora
Founder,TechArda.com
nicebackgroundinformationonML.cantwaittoseethenextpost:)
Like
Reply
3d
MethilSreekumar
consultant,strategictelecommanagement,generaladministration
Averyinterestingarticle,chronologicallylaidouttopickuplostthreads.Atthistime,Ihaveonly
onetocomment,isevolutionofAIpromptingordrivinghumanintelligencetopursue
unimaginablefeats/heightsandmaintainstheproverbialgapwiththeformerlaggingbehind.
Lookingforwardforthenextsequeltoflow.
MethilSreekumar
Like
Reply
2d
ShreedharTorgal
Technologist|EnterpriseArchitect|ITStrategy|cognitive
Computing|Programmanagement|CxO
Goodone..
Like
Reply
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
9/10
11/7/2016
ABriefHistoryofMachineLearning|Dr.JaideepGanguly|Pulse|LinkedIn
2h
RaviVenkatesan
ResearchDirectoratSystemsResearchCorporation
Thanksforsharing.
Like
Reply
https://www.linkedin.com/pulse/briefhistorymachinelearningdrjaideepganguly?trk=hpfeedarticletitlecomment
10/10