Performing Lookups and Transformations With Talend - Appirio

10/03/2016 PerformingLookupsandTransformationswithTalendAppirio
Services
Technology
Company
Customers
Partners
Blogs&Resources
ContactUs
Search
Search Go
CloudPoweredBusinessBlog
TechBlog
GreyMatterBlog
TopcoderBlog
Subscribeto
theBlogs
Blog: Select...
Email:
SignMeUp!
TechBlog
PerformingLookupsandTransformationswith
Talend
12
ByAppirio@appirio|December22,2009
WardLoving
Inthisarticle,IdliketoexploretheworkhorseoftheTalendOpenStudiothetMapcomponent.As
youdexpect,thiscomponentallowsyoutovisuallymapfieldsfromaninputsourcetofieldsonan
outputstream.Italsogivesyoutheabilitytodolookupstootherinputfilesordatabaseswhile
http://appirio.com/category/techblog/2009/12/performinglookupsandtransformationswithtalend/ 1/18
performingtransformationsonyourdatawithbasicjavaexpressions.Weregoingtomanipulatesome
textfilestogiveyouafeelforthiscomponentscapabilities,butbeforewediveinletsdiscussa
scenarioortwowhereyoumightneedtousethetMapcomponent:
UseCaseA:Youreonaspacewalkdoingsomeroutinemaintenanceandyoulookupandseethatyour
toolboxisslowlydriftingoutofreachintogeosynchronousorbit.
UseCaseB:Youhavesomedatathatyouneedtomovefromonesystemtoanother.Threeindividual
addressfieldsinthesourcedataneedtobeconsolidatedintoasingletextareafieldinthedestination
system.Youalsohavearequirementtoensurethatonlyvalidemailaddressesaretobeincludedthe
outputfiles,butyoudontwanttoloseawholerecordifanemailaddressisinvalid.
Sadly,datamigrationskillsareprobablynotgoingtohelpyoumuchwithUseCaseA.Youcanskipthis
article.But,withUseCaseB,youreinbusiness.WellstartwhereweleftoffintheTESTTALEND
projectthatIcreatedinthepreviousarticleinJune.Wewontbereusingthecodefromtheprevious
examplebutyoumightwanttowalkthroughthestepsofsettinguptherepositoryandcreatinganew
projectifyouhaventdonethatbefore.YoucanfindanddownloadthelatestTalendOpenStudioby
goingtohttp://www.talend.com.
OpenuptheTalendOpenStudioandselectaproject.RightclickontheJobDesignsfolderinthe
RepositorytabandselectCreateJob.GoaheadandnamethejobIcalledmineSecondJob.
Fig1CreateJob
ImamworkingquiteabitwithdatamigrationstoSalesforce.comthesedaysandalthoughyoucanpull
datadirectlyfromSalesforceusingTalend,IoftenfindmyselfusingTalendtoworkwithfilesrather
thanusingittopulldataoutofSalesforce.comdirectly.Talenddoesntallowus(yet),toadjustthe
numberrecordsexportedineachbatchsoitismuchquickertousetheSalesforce.comDataLoaderto
pullrecordsoutofSalesforce.com.Thedifferenceinperformancebetweenthetwotoolsisnotabigdeal
forafewthousandrowsbutwhenyouretalkinghundredsofthousandsormillionsofrowsofdata,
everyoptimizationishelpful.ForthisexampleletsuseTalendstFileInputDelimitedComponentwhich
canbeusedtoprocessthelinguafrancaofSaleforce.comDataMigrationsthecsvfile.Youcanfind
thisunderFile>InputinthecomponentslistoryoujustcanjusttypetFileInputintothePaletteSearch
field.
Fig2SearchPalette
DragthetFileInputComponentontotheworkspace.
Fig3Workspace
WeregoingtopointthistFileInputComponenttoafile.Youcansimplycutandpastethefollowingdata
intoafilecalledsample_input_data.csv:
FirstName,LastName,Email,Address1,Address2,Address3,GenderCode
John,Smith,jsmith@yahoo.com,6665thAve,19thFloor,Suite15,M
Jane,Doe,jdoe@gmail.com,4448thAve,SnyderPlaza,Building7,F
Ralph,Emerson,invalid@email!com,108Street,,Apt3B,M
Edgar,Poe,epoe@scarystory.com,1000BostonStreet,,,M
Forthisexample,IveputthefileinmyC:\datadirectory.Nowifyoudoubleclickonthe
tFileInputComponentintheworkspaceitwillbringuptheBasicSettingsoftheComponenttabatthe
bottomoftheworkspace.
Fig4ComponentTabBasicSettings
IvehighlightedthefollowingsettingsinFig4abovewhichneedtobeupdated:
SelecttheFileNameforthesample_input_datafile.
Selectacomma,ratherthanasemicolonasthefieldseparator
Selectthecsvoptioncheckbox
Put1intheHeaderfieldtodesignatethatyourinputfilehascolumnheaders.
Wellalsoneedtoinputtheschemaforourcsvfile,soclickEditSchemaandenterthecolumnheaders
foryourfileasshowninthefigurebelow.Clickingthegreenplussignaddsacolumnentryandclicking
theredcrossremovesone.Therearealsosomehandycontrolsheretoimportandexportyourschema
entries.Thiscansaveyoualotoftimewhenyouareworkingwiththesameschemaacrossmultiplefiles.
ClickOKwhenyouvecompletedtheschemaentries.
Fig5SchemaDialog
NowswitchtotheAdvancedSettingspanelandclicktheTrimAllColumnscheckbox.AsfarasIcan
tell,thereisalmostnoreasonwhyyouwouldntwanttotrimthefieldsofyourcsvfiles.Makingita
habittocheckthiswillsaveyoualotofheadachesperformingmatchingagainstyourlookupfiles.
Fig6ComponentTabAdvancedSettings
YoucanalsochangethenameofyourcomponentintheworkspaceontheViewpanel.Ivechangedthe
labelformytFileInputComponenttoSampleInputFilebyupdatingtheLabelFormatfield.
Fig7ComponentTabView
NextwellcreateasimplelookupfiletoprocessourGenderCodes.Cutandpastethefollowingdatainto
anotherfilecalledgender_lookup.csv:
Code,Value
M,Male
F,Female
WellneedtorepeatthestepsshowninFigures37forthesecondtFileInputComponentwhichis
pointingtothegenderlookupfile:
DragthetFileInputComponentontotheworkspace
UpdatetheBasicSettingsFileName,Separator,CSVOption,andHeader
CreatetheschemafortheLookupfile.Itwillhavetwocolumns:CodeandValue.
ChecktheTrimallcolumnscheckboxontheAdvancedTab
Updatethecomponenttoamoreuserfriendlyname(Optional).IvenamedmineGender
Lookup
Nowinadditiontoinput,ourmigrationprocessisalsogoingtohaveoutput.Letsstickwiththe
delimitedfileformatfornowandcreateanothercsvfilefortheoutputdocumentusingthe
tFileOutputDelimitedcomponent.ItcanbefoundunderFile>OutputsectionoftheComponentPalette.
GoaheadanddragthetFileOutputDelimitedcomponentoutintotheworkspace.
Fig8ComponentTabBasicSettingstFileOutputComponent
ThestepsheretoupdateatFileOutputComponentarentthatdifferentfromthetFileInputComponent:
SelecttheFileNameforthesample_output_datafile.
Selectacomma,ratherthanasemicolonasthefieldseparator
SelecttheIncludeHeadercheckboxtodesignatethatyoudlikeyouroutputfiletohavecolumn
headers.
Wellalsoneedtoupdatetheschemafortheoutputcomponent.Rememberwehavearequirementto
consolidatetheaddressfieldsfromthreeseparatefieldsdowntoone,soourschemaisgoingtochangea
littlefromtheinputfileversion.
Fig9tFileOutputSchemaDialog
Clickokwhenyouvefinishedenteringyourschema.Nowwellgototheadvancedsettingspaneland
clicktheCSVOptionscheckboxtoensurethatanywanderingcommasareproperlyescaped.
Fig10ComponentTabAdvancedSettingstFileOutputComponent
SwitchtotheViewpaneltoupdatethenameofthetFileOutputComponent.Iveupdatedmineto
SampleOutputFile.Yourworkspaceshouldlooksomethinglikethis.
Fig11WorkspacewithInputandOutputComponents
Alloureffortshavebeeninpreparationforthismomentsodontblowthisbigguy.Ifyoucantakethe
pressure,andIknowthatyoucan,selectthetMapcomponentfromthecomponentpaletteunderthe
Processingheaderanddragitontotheworkspace.
Fig12tMap
Thecrowdroars.Thatwasntsobadnow,wasit?
Thenextstepistocreatetheflowforourjob.RightclicktheSampleInputFilecomponentandselect
Row>Mainasshownbelow.
Fig13CreateFlow
ConnectthelittleplugtoyourtMapcomponent.YoualsoneedtodothesamethingwithyourGender
LookuptFileInputComponent.WhenyourightclickandselectRow>MainfromthetMaptotheSample
DataOutputcomponentthefollowingdialogwillpopup.IvenamedtheflowtoOutputFileasshown
below.Itispossibletocreatemultipleoutputsforeachjobbutwejustneedoneforthisexample.
Fig14OutputNameDialog
Aseconddialogwillcomeupaskingifyouwanttogettheschemaofthetargetcomponent.Youdo.
Whenyouvedoneallthisyourjobshouldlooksomethinglikethis:
Fig15WorkspacewithConnectedComponents
DoubleclickonthetMapcomponenttobringuptheMapEditor.Yourinputfieldsareontheleftsideof
thescreenandyouroutputfieldsareontheright.Tomaponefieldtoanothersimplydragthefieldfrom
thelefttotherightsideofthescreenanddropitintotheExpressioncolumn.
Fig16MappingFields
Aftermatchingrow1.FirstNametotoOutputFile.FirstNameyoullseeayellowarrowmarkingthe
association.MatchuptheLastNamefieldaswell.
Youcanconcatenateallthreeaddressfieldstogetherbysimplydraggingeachofthemtothesingle
addressfieldontheoutputsideofthemapeditor.
Fig17AddressMapping
Unfortunately,thisturnsouttobealittlecrude.Ifoneoftheaddressfieldsisblank,thenyouendupwith
unsightlygapsinyouraddress.InSalesforce.commoststreetaddressfieldsaretextareas,soitsniceto
preservetheaddresslinesinthetextareawhicharecapturedinthesourcedata.Clickingthebutton
labelednexttothetoOutputFileAddressfieldinthetMapEditortobringsuptheExpressionEditor.
ThebuttonismarkedwitharedarrowinFigure17.
Fig18ExpressionEditor
Cutandpastethefollowingcodeintotheexpressioneditorformoreadultaddressprocessing:
(row1.Address1!=null&&row1.Address1.length()>0?row1.Address1:"")+
(row1.Address2!=null&&row1.Address2.length()>0?"\n"+row1.Address2:"")+(row1.Address3
!=null&&row1.Address3.length()>0?"\n"+row1.Address3:"")
Thiscodesimplycheckstoseeifavalueispresentbeforeaddingalinefeedtotheaddresslineentry.
ClickOktoupdatetheexpressionforyourAddressfieldandreturntotheMapEditor.
TofilteroutinvalidemailaddresswecanaddaregularexpressiontotheExpressionEditor.Clickthe
buttonnexttotheEmailfieldandenterthisexpression:
(row1.Email!=null&&
row1.Email.matches("^[\\w\\.]+@([\\w\\]+\\.)+[AZaz]{2,4}$")?row1.Email.toLowerCase():"")
Thisregularexpressionisntaperfectemailvalidator,butitkeepsoutmostoftheriffraff.Ifanemail
doesntmatchtheregularexpression,itprobablyisntavalidaddressandwerejustreplacingtheinvalid
textwithanemptystring.AnEmailtypefieldinSalesforce.comwillactuallybouncerecordswhichhave
aninvalidemailformatwhenyouloadthemtothesystem,whichmaynotbethedesiredbehavior.
WeveonlyscratchedthesurfaceofthecapabilitiesoftheTOSExpressionEditor.Ithasallkindsof
string,numerical,anddateprocessingfunctionsavailabletoyou.ThemostfrequentidiomthatIuseisa
simplejavaternaryoperator.Infact,wecouldusethisidiomtoprocessourgendercodesifwewanted
to:
(row1.GenderCode!=null&&row1.GenderCode.equals("M")?"Male":
(row1.GenderCode!=null&&row1.GenderCode.equals("F"):""))
ThiscodesimplyresolvestheGenderCodetoMaleorFemaleifthecodeequalsMorFand
returnsanemptystringifitdoesntmatcheitherone.Butletsresolveourgendercodesinmorestylish
fashionusingalookup.Thesimplicityofthisexamplebeliesthepowerofthiscapability.Itispossibleto
resolveforeignkeyrelationshipsinthiswaybetweenfileswithhundredsofthousandsorevenmillions
ofrecords.TalendslookupsaremorepowerfulthanVLookupsinExcelasawaytocreateforeignkey
relationshipsfordatamigrationsandaremuchmorerepeatable.Thereisnolimittothenumberof
lookupsyoucaninclude.Mypersonalrecordiseleveninasinglejob.AlthoughTalendhasthecapability
ofwritingtempfilestodiskforlargelookups(thatisthebuttonthatlookslikeadiskdriveontherow2
headerintheMapEditor),Ihavefoundthatyoucanrunintomemoryissuesifyourdatainputhas
severalmillionrowsandyourlookupfilehasseveralmillionrowsaswell.Theresonlysomuchheapto
goaround.ThewayIveworkedaroundthislimitationistodividetheinputfileintopartsusinga
separateTalendjob,andtheniteratethroughthepartsusingatFileListcomponent.Youcancheckthe
TalendcomponentdocumentationtogetmoreinformationabouttFileListconfiguration.
TocreatealookuprelationshipbetweentheGenderCodeandtheGenderLookupfile,dragthe
row1.GenderCodefielddowntotheExpr.Keycolumnoftherow2.Codefield.Youshouldseeapurple
arrowmarkingtheassociation,orisitmauve?Youknow,thesearethingsthatkeepmeupatnight.
Fig19LookupMapping
Next,dragtherow2.ValuefieldtothetoOutputFile.Genderfieldontherightsideoftheeditorwindow.
Thatsallthereistoit.Yourcompletemappingshouldlooklikethefollowing.
Fig20CompletedMapping
IfyouclicktheInnerJoincheckboxontheheaderoftherow2editor,showninFig19above,onlyrow1
recordsthathavematchinggendercodewillbewrittentoyouroutputfile.ClickOktoclosetheMap
Editorandletsrunthisbadboy.SelecttheRuntabatthebottomofthescreenandpresstheRunbutton.
Fig21RuntheSecondJob
Checkyouroutputfiletoseetheresults.Itshouldlooksomethinglikethis:
FirstName,LastName,Email,Address,Gender
"John","Smith","jsmith@yahoo.com","6665thAve
19thFloor
Suite15","Male"
"Jane","Doe","jdoe@gmail.com","4448thAve
SnyderPlaza
Building7","Female"
"Ralph","Emerson","","108Street
Apt3B","Male"
"Edgar","Poe","epoe@scarystory.com","1000BostonStreet","Male"
Wellyouhaventretrieveda$100Kgreasegunfromtheabyss,butyouhaveaddedacoupleoftricksto
youdatamigrationtoolkit.IencourageyoutodownloadtheOpenStudioandtryitout.
Tags:Salesforce.com,Talend,WardLoving
1 4 2 Google+ 1 3
AboutAppirio@appirio
AppirioisaglobalservicescompanythathelpscustomerscompeteandwinintheCloudEconomy.As
theleaderinenterprisecrowdsourcing,wedeliverunmatchedcustomervaluebycombiningtheworlds
toptalentwithauniquecloudservicesplatform.Ourplatformharnessesthepowerofoneoftheworlds
largestcrowdsourcingcommunities,thousandsofreadytodeployassets,andleadingcloud,industryand
technicalexperts,tohelpourcustomersdrivecompetitiveadvantage.Appirioisatrustedpartnertosome
oftheworldslargestbrands,includingCardinalHealth,CocaCola,eBay,Facebook,GeorgiaPacific,
HomeDepot,Moen,andSonyPlaystation.
FOLLOW:
Twitter
Facebook
Linkedin
GooglePlus
12ResponsestoPerformingLookupsandTransformationswithTalend
January14,2010at11:11am
SergeiTvorogovsays:
Greatwork!
Thankyouverymuch!
Talendisreallyamazingthing)
Reply
March22,2010at9:58am
Nimishsays:
Excellentstepbystepinstructions.ThanksalotWard.
Reply
May24,2010at2:15pm
Raghusays:
Awesomepost.showsthepowerofTalend
Reply
May25,2010at2:34pm
Raghusays:
HeyWard,
Canusuggestanydocsfordatamigration/dataintegrationthatinvolveslookupinbothinputandoutput
sidesofsalesforce
Reply
June28,2010at12:46am
RameshBhatsays:
Thanksalotman
Reply
August21,2011at3:49pm
Anonymoussays:
Justtrytoperformrelationshipqueriescantbedone.Oh,youcansimulateitin*some*cases,with
separateinputsperSFDCobjectandtMapbutnotallsuchasthecriticalActivitiesHistoryquery.
YouCANNOTquerythatobjectdirectlybyitselfyouMUSTuseaparentchildrelationshipquery
whichthetSalesforceInputcomponentdoesnotsupport.
Reply
November22,2011at4:31am
Anonymoussays:
ImusingTalendtoolformigratingoneDBtoSFDC.
1)IinsertedACCOUNTrecordssuccessfuly.
2)NowIwanttoinsertCASErecords,eachcaserelatedtoAccount,IneedtoAccountrecordid.
HowtoperformParentchildrelationshiphere.
OtherDBworkswithPrimary/foreignkey,howeverinSFDCeachrecordidentifiedthroughrecordid.
Reply
November27,2011at6:30pm
Wardsays:
SotypicallywithatoollikeTalendyouddoyouremigrationoftheAccountandCasesintwoseparate
jobs.ItsoundslikeyouvealreadypushedyouraccountsintoSFDCsuccessfully.Thatisjobone.The
secondjobistogetyourCasedatainserted.YourprimaryflowwillbetheCasedataitself.Butyoull
alsoneedtodoalookuptotheAccounttabletopulldowntheaccountidstocreatetheforeignkey
relationships.YoucancreatetheforeignkeysbyextractingtheAccountswithatSalesforceInput
component.AreyoumovingthedatafromSalesforce.comorgtoorgorfromanexternalsystem?
Reply
December28,2011at9:24pm
Herbsays:
Excellentpost!Thisissuperhelpful,andquitesimple.
Reply
August30,2012at12:22am
BasantKumarVermasays:
HiWard,
IhavetomigrateAccountsfromanexternalsystemtoSFDCwhereIalsohavetomigrateCreatedById
andlastModifiedByIdfromExternalsystem.
IusedthesamewayyouexplainedhereforlookupUsers,buthereyoushownonlyOneLookupandin
mycaseIneedtolookuponUsersfortwofieldsCreatedById&lastModifiedByIdwhichitdoesnot
allowing.
RightnowIamfetchingUserstwiceforthis,butitsdecreasingmyjobsperformance.
Canwedoitbyasinglefetchofusers.???
Hereyoucanseemymappingsscreenshot
1.WithoneFetchwhichwasthrowingexceptionhttp://screencast.com/t/vTXzLGMLIn1s
2.WithtwofetchwhichIamusingcurrentlybutIdontwanttouseit.
http://screencast.com/t/36Nz7CkrV
Reply
August30,2012at11:31am
Wardsays:
Basant:
IvelookedatyourscreencastandIthinkyourapproachofusingtwolookupsisvalidandwillworkbut
thereseemtobeperformanceimplicationsthatareunacceptableinyourcase.Ithinktheproblemwith
thesinglelookupisthatwiththecurrentsyntaxTalendisgoingtoconcatenatetheCreatedByIdand
LastModifiedByIdtogetherandtrytomakeamatchwithyourUserLookuptablewhichisntgoingto
work.Maybeyoucouldtryanorseparatorbetweenthetwoidslikethis(||)andseeifthathelps
haventtriedthismyselfbutitmightwork
March27,2015at11:59am
Raghavsays:
IamNewtoTalendTOS,justineedtoknowwhichcomponentineedtousetoviewthedatacoming
fromcomponents.
forexample:wewilladdDataVeiwertotheDataflowCOmponentsinSSIStoknowwhatkindofdatais
movingfromonecomponenttoothercomponent.
Pleaseletmeknowhowtoveiwthedataintalendjobsnotinforfinalresultset,buttoveiwthedata
whichflowsfromcomponenttocomponent.
(1)tmssqlinput1>tmap>tmssqlrow1>tmap>tmssqlrow2
nowineedtoveiwthedatatmapandtmssqlrow1,sotoveiwthisdataisthereanycomponent.
ThanksinAdvance
Reply
LeaveaReply
Enteryourcommenthere...
Heressomemorepostsyoumightlike
HowtoGrowaQualityEmailList
5CustomerExperienceTrendsYouNeedtoKnowin2016
Services
Strategy
Implementation
Integration
Development
Crowdsourcing
Technology
AppirioServicesPlatform
CloudProducts
Company
CompanyOverview
Leadership
News,Awards&Events
CorporateResponsibility
CareersAtAppirio
AppirioJapan
Customers
FeaturedCustomers
CrowdsourcingProjects
Partners
StrategicPartners
SolutionPartners
TopcoderPartners
Blogs&Resources
AppirioBlogs
Resources
CustomerSupport
ContactUs
ConnectwithUs
Twitter
Facebook
Linkedin
GooglePlus
Youtube
Email
2016Appirio.AllRightsReserved|PrivacyPolicy

Performing Lookups and Transformations With Talend - Appirio

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Performing Lookups and Transformations With Talend - Appirio

Diunggah oleh

Hak Cipta:

Format Tersedia

10/03/2016 PerformingLookupsandTransformationswithTalendAppirio

Anda mungkin juga menyukai