CredosoftwhitepaperV0.

Eight key steps which help ensure a

successful data migration project: A white
paper for inspection management
professionals
Data migration defined
Datamigrationistheselection,preparation,extraction,transformationandtransferofdata
whichisofthecorrectquality.

The challenge
Inspectiondepartmentsoftenholdlargeamountsofinformation,ofvariablequalityandinvarious
formats.Whenconsideringtheprocurementofanewinspectionmanagementsolution,
inspectionprofessionalsareusuallykeentounderstandhowaprospectivevendorwillmanage
thetransferoftheirdatatothevendorssystem.
ThiswhitepaperwilloutlineanapproachthatCredosoftrecommendtoensureasuccessful
datamigrationproject.Pleasenotethatthisisnotanexhaustiveordetailedlist..

1. Ensure the client understand his role in data

migration
Theclientknowsbest!

Whilsttechnologywillbeutilisedtomanagethephysicalextraction,transformationandloading
ofyourdata,toooftendatamigrationisviewedasapurelytechnicalexercise.Therealityisthat
onlytheownersandconsumersofthedataunderstandthemeaningandvalueofthedata.This
Page1of5

CredosoftwhitepaperV0.1

valuemustberetainedandenhancedduringdatamigration.Emphasisingtechnologyover
businessknowledgeshouldbeavoided.
Datamigrationisabusinessissue,notatechnicalissue
Theinspectionteamaretheexpertsinwhatthedepartmentdoes.Theyhavebeenrunningthe
legacysystemsandthesystemshavebeenrunningtheinspectionactivities.Finallythe
inspectionteamhavetheexpertisetomakejudgementsaboutthequalityandusefulnessofthe
data.Thatbeingsaidthattheclientwillrightlyexpectguidanceonhowtomakethedatafitthe
newinspectionsystem.

Thedatamigrationanalystcannotknowmoreaboutthebusinessrulesthantheinspection
team

2. Data profiling
Theinspectionteamknowwhereallthedatasourcesthatrunthedepartmentarelocated.Early
identificationofallpossibledatasourcesisvitaltoensureaccurateestimatingoftimescales.
Departmentsoftenhavevariousspreadsheetsorminidatabaseswhichsupporttheirdaily
inspectionactivities.Somemaybeunacknowledgedhowevercouldbecriticaltothe
departmentsprocessessometimesaddressinggapsinintheofficialsystem.Eachwillbein
differentformatsandvaryingqualityandtherewilloftenbedifficultiesinlinkingthemtogether.
Onceallthelegacydatastoreshavebeenidentified,thisisthepointwhereyoumayagreethe
scopeofwork
Atthisstagewediscoverandcatalogallthedatastoresandtheirrelationshiptooneanother.
Welookinsidethedatastorestounderstandhowtheywork,whatdatatheyholdandwhat
challengestheymayhold.
Finally,weneedtoagreeonwhatdatayouwanttomigrate.Inanidealworldyouwouldmigrate
everythinghowevertherealityisthattherewillbedatawhichmaynotaddvalueormaybe
coveredbyadifferentsource.

Page2of5

CredosoftwhitepaperV0.1

3. Gap Analysis and mapping

Gap Analysis
Thisiswhereweanalyseandquantifydatagaps.Theremaybeissuesinthelegacysystems
someissueswillbeknowntotheendusers(whichinpartmayhavecausedaproliferationof
inhousedevelopedsolutions).Someissueswillonlybediscoveredwhenthedatabaseis
examinedformigration.Itisimportantthattheseissuesareflushedoutasearlyaspossibleso
thatappropriateactioncanbetaken.
Realitychecks

Wheredomainexpertscheckthedatatoensureitmatches
businessreality

Internalgaps

Wherethedatadoesnotconformtointernalrulesforexample
referentialintegritydoesnotexist

Migrationmodelgaps

Thisoccurswhenthereisdifferencesinthedatafromdifferent
legacydatastores

Targetmodelgaps

Thisoccurswhentherearedifferencesindataanddata
structuresbetweenthesourceandtargetsystems

Mapping
Thisisthelinkingoffieldsinthelegacydatastorestofieldsinthetarget.Mappingrequiresclose
collaborationbetweentheclientandthevendor.Theuseofaspreadsheetisperfectly
acceptableforthisactivity.

4. Define Data Quality Rules

Dataqualityrulesareusedtomeasurethequalityofthedataandtohelpfixormitigateanydata
qualityissues.Everydataqualityissueshouldhaveadataqualityrule.
ACTION

DESCRIPTION

Ignore

Theseareoftenlowleveldataissuesthatdontcauseaproblem.
Itslikelythatthemajorityofissueswillfallintothiscategory.In
somecasesyoumaybecontenttocarrytheissueintothenew
systembecauseyouhavebiggerissuestoresolve..

Fixinflight

Inthiscaseyouwillperformsomekindoftransformationonthe
datatocorrectit.Forexample,convertlookupcodestoupper

Page3of5

CredosoftwhitepaperV0.1

case.Thisactivityiscarriedoutasthedatainloadedintothe
target
Fixinsource

Inthiscaseyoumayfixthedatainthesourcedatasystem.This
maybethesimplestsolution.

Fixintarget

Inthiscaseyoumayfixthedataafterithasbeenloadedintothe
targetsystem.Howeverifthedataissuesbreaksvalidationrulesin
thetarget,youmayhavetotemporarilydisablethevalidationrules
toallthedatatopassintothetarget.Thisisperhapstheriskiest
optionandinmostcaseswouldbeunnecessary..

5. Create a system retirement plan

(de-commissioning)
Asswitchingoffthelegacydatastoresisoneofthegoalsofanydatamigrationproject,itis
importanttohavetheseconversationsearlyon.Forexampletheendusersshouldbemade
awarethatonxdate,thelegacysystemwillnotbeavailabletothem.Atthispointweneedto
ensuretheuseriscomfortablewithdecommissioning.Startingthisconversationalsohelps
makesrealtheeventualityofadatamigration.
Inadditiontheremaybelegalrequirementsgoverningthestorageandretentionofthedata.

6. Create a migration plan

Althoughdatamigrationisakeyactivityintheoverallprojectplan,themainmigrationactivities
needtobemappedoutandtimescalesagreedforeach.

Agree the implementation approach

BigBang
Allthedataismovedinonego.Thisisthemostcommonapproach

Phased
Thedataismovedinseparateparts,perhapsbybusinessorgeographiclocation

Parallel
Thedataismovedtothetarget,howeverthelegacydatacontinuestobeusedand
changesaresynchronisedtothetarget.
Page4of5

CredosoftwhitepaperV0.1

7. Refine the migration scripts

Refiningdatamigrationscriptsisaniterativeprocesswhichcancontinueuntilclosetogolive.
Throughoutthisperiodthevendorshouldengagetheclientinreviewsofthescriptsandthe
output,makingadjustmentswherenecessary.Ideallytheclientshouldbeabletoviewandtest
thedatainthetargetsystem.

8. Load
Amigrationcontroller,oftenknownastheextract,transformandloadtoolisused(ETL)toload
thedataintothetargetsystem.Thevendormayuseasinhouseorathirdpartytool.Thesetools
oftendomorethanjustthesethreetasksandthefollowingarekeyfeaturesyoushouldexpect:

Readthedatafromthelegacydatastore
Validatethedata
Reformatthedataandmergefrommultiplesources
Startandstoptheprocess
Writethedatatothetargetdatabase
Managedataerrors
Providereportsonexecution
Provideanaudittrail
Synchroniseintheeventthatthesourcedatamustbeusedduringthemigration,
changesinthesourcecanbeupdatedwiththetarget

Page5of5