Acceptance Test Engineering Guide Vol I RC1 Full 102609 PDF

AcceptanceTest
EngineeringGuide
VolumeI:ThinkingaboutAcceptance
HowtoDecideifSoftwareisReadyforYou
andYourCustomers

GrigoriMelnik,GerardMeszaros,JonBach

ForewordbyKentBeck
ReleaseCandidate1October26,2009
COMMUNITYPREVIEWLICENSE
Thisdocumentisapreliminaryreleasethatmaybechangedsubstantiallypriortofinalcommercial
release.ThisdocumentisprovidedforinformationalpurposesonlyandMicrosoftmakesnowarranties,
eitherexpressorimplied,inthisdocument.Informationinthisdocument,includingURLandother
InternetWebsitereferences,issubjecttochangewithoutnotice.Theentireriskoftheuseorthe
resultsfromtheuseofthisdocumentremainswiththeuser.Unlessotherwisenoted,thecompanies,
organizations,products,domainnames,emailaddresses,logos,people,places,andeventsdepictedin
exampleshereinarefictitious.Noassociationwithanyrealcompany,organization,product,domain
name,emailaddress,logo,person,place,oreventisintendedorshouldbeinferred.Complyingwithall
applicablecopyrightlawsistheresponsibilityoftheuser.Withoutlimitingtherightsundercopyright,
nopartofthisdocumentmaybereproduced,storedinorintroducedintoaretrievalsystem,or
transmittedinanyformorbyanymeans(electronic,mechanical,photocopying,recording,or
otherwise),orforanypurpose,withouttheexpresswrittenpermissionofMicrosoftCorporation.
Microsoftmayhavepatents,patentapplications,trademarks,copyrights,orotherintellectualproperty
rightscoveringsubjectmatterinthisdocument.Exceptasexpresslyprovidedinanywrittenlicense
agreementfromMicrosoft,thefurnishingofthisdocumentdoesnotgiveyouanylicensetothese
patents,trademarks,copyrights,orotherintellectualproperty.
2009MicrosoftCorporation.Allrightsreserved.
MicrosoftaretrademarksoftheMicrosoftgroupofcompanies.
Allothertrademarksarepropertyoftheirrespectiveowners.
TableofContents
ForewordbyKentBeck 1
Preface:AcceptanceTestEngineeringGuide 3
Introduction 10
ACautionaryTale 20
PARTI.ThinkingAboutAcceptance
Chapter1 TheAcceptanceProcess 27
Chapter2 ElaboratingontheAcceptanceProcess 53
Chapter3 DecisionMakingModel 71
Chapter4 ProjectContextModel 82
Chapter5 SystemsRequirementsModel 91
Chapter6 RiskModel 99
Chapter7 DonenessModel 104
PARTII.PerspectivesonAcceptance
Chapter8 BusinessLeadsPerspective 119
Chapter9 ProductManagersPerspective 131
Chapter10 TestManagersPerspective 138
Chapter11 DevelopmentManagersPerspective 143
Chapter12 UserExperienceSpecialistsPerspective 148
Chapter13 OperationsManagersPerspective 153
Chapter14 SolutionArchitectsPerspective 156
Chapter15 EnterpriseArchitectsPerspective 159
Chapter16 LegalPerspective 161
PARTIII.AcceptingSoftware
Chapter17 PlanningforAcceptance 166
Chapter18 AssessingSoftware 190
Chapter19 ManagingtheAcceptanceProcess 207
Chapter20 FinetuningtheAcceptanceProcess 219
Appendices
AppendixA OrganizationStereotypesandReaderPersonas 231
AppendixB KeyPoints 237
Foreword
Ifigureit'smyjobinaforewordtotellyouwhyIthinkabookisworthreading.Oftenthisinvolves
settingcontext,placingthebookwithrespecttohistoricalorsocialortechnicaltrends.SometimesI
pointouthowtheauthorshavebeentoohumbleintheirpresentation.SometimesItalkaboutwhatI've
learnedfromabook.InthiscaseIamabletouseallthreetechniques.
Technologistshavespentthelasthalfcenturyswingingonapendulumbetweenbeinglowlyserfs
obeyingthecommandsoftheirbusinessmastersandactingashighpriestsatthealtarofComputing.
Thelastdecadehasseenthefirsttentativestepstowardsamorerealisticandsustainablepositionof
partnershipwithbusinesssponsorsandusers.Thenewemphasisonacceptancetestingisevidencethat
thistrendcontinues.
Isay"emphasis"becauseacceptancetestingisnotnew.Eversincethefirstusertoldthefirst
programmer"That'snotquiteright,"acceptancetestinghasbeenpartofthedevelopmentworkflow.
Sincethosefocusedontechnologyandthosefocusedonbusinessbringdifferentvalues,expectations,
andvocabulary,somemisunderstandingisinevitable.What'sthatProfessorHeisenberg?Oh,yes,thanks
forthereminder.Theappearanceofasystemautomatically(itseems)changesitsrequirements.
Betweenthesefactors,acceptancetestingispartofeffectivedevelopment.
Legitimatelyshiftingrequirementshaveaprofoundeffectonthetiming,role,andtechniquesof
acceptancetesting.Oneofthetrendsindevelopmentoverthepastdecadeisthattestingtakesplace
soonerandmorefrequentlyindevelopment.Evenwiththebulkofacceptancetestingshiftedtojustin
advanceofdevelopment(AcceptanceTestDrivenDevelopment),someonestillneedstobelisteningfor
"Thanksforthat,butwhatIreallywantis..."
Oneimplicationofthegrowingfrequencyofacceptancetestingistheneedforautomationofrepetitive
testingtasks.Thisisnotajobfortestersalone.Toincreasetherateoffeedback,programmersmust
maketheirsystemsmoretestable.Automationfreesresourcesforhigherleveragedmanualtesting.
OnethingIappreciatedinthisbookwastheconsistentconnectionbetweenacceptancetestingandrisk
management.Acceptancetestingisworthwhileforthevalueitaddstodevelopment:increased
probabilityofcustomersatisfaction,reducedriskoferrors,andbettertrackingofprogress.Theimageof
riskmanagementasawaytobuyreactiontimewillstickwithme.Earlyincrementalacceptancetests
canbethescoutssentouttoinformthearmyoftroubletocome.
Oneofthestrengthsofthisseriesofbooksisthevarietyofformatsfortheinformation.Thefirst
volumepresentstheintellectualfoundationsofacceptancetesting.Remainingvolumespresent
concretepracticesthatworkwithinthatframework,amenuof"Here'stechniquesthathaveworked"as
wellasaworkedexampleforthoselookingtounderstandacceptancetestingincontext.
AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 1

I'llclosebypointingoutwhereIthinkGrigori,Gerard,andJonshumilitygotthebetterofthem.Whatif
softwaredevelopmentwasturnedonitshead?Whatifthewholeprocessworkedbackwardsfrom"The
customerisdelightedwiththesystem"?Beforeyoucouldexpecttohearastatementlikethat,you'd
needtospendalotoftimetestingthesysteminrealisticenvironments.You'dneedtomakesureallthe
easilyunderstoodfeaturesworkedasexpected.Acceptancetestingandacceptancetestdriven
developmentcanencouragethis.Novelfeaturesandinteractionsoffeatureswouldalsoneedtoworkin
asensibleway,anothergoalacceptancetestingcancontributeto.Whenthose"Great,butwhatI'd
reallyliketoseeis..."momentshappen,incrementalacceptancetestingcanhelpthemoccurearly
enoughthatthewholeteamcanrespond.
Astheconcretemanifestationofthedialogbetweenbusinessandtechnology,acceptancetestingcan
playacentralroleinsoftwaredevelopment.Hereinyourhandsisabookthatdistillsyearsof
experiencewithacceptancetesting,abagofideasyoucanreachintoagainandagainasyoutravel
towards:"Thecustomerisdelightedwiththesystem..."
KentBeck,June2009

Preface:AcceptanceTestEngineering
Guide
WhyWeWroteThisGuide
Acceptingsoftwareisoneofthemostimportantdecisionsforsoftwaretotakeflightandtostart
deliveringvalue.Yet,thereexistslittleguidanceaboutwhatthisdecisionshouldbebasedonandhowto
makeiteffectively.
Overmanyyearsinindustryandappliedresearch,itwasourobservationthatpeopleoften
misunderstoodthechallengesofacceptingsoftware,orworse,didnotevenhaveaclearnotionofwho
isinchargeofacceptance.Manyappearnottobeawareofthearsenalofpracticesandtoolsavailable
tothemtoaddressthosechallenges.Thisledtodisappointments,failedopportunities,brokenbusiness
relationships,andlitigations.Thereexistsarichbodyofknowledgeonsoftwaretesting(withnotable
worksbyBorisBezier,BertrandMeyer,GeraldWeinberg,CemKaner,JamesBach,BrettPettichord,Lee
Copeland,MarkFewster,DorothyGraham,RexBlack,BrianMarick,JamesWhittakertonameafew),
butsoftwareacceptanceingeneral,andacceptancetesting,inparticular,asaprocess,receivedno
serious,wellreasonedorcomprehensivetreatment.Wesetouttofillthisgapwithtwokeyobjectives:
devisementalmodelsforthinkingaboutsoftwareacceptanceandofferactionableguidanceon
achievingsoftwareacceptance.
WhoShouldReadThisGuide
Thisguideisintendedforanyonewhoisinvolvedintheprocessofdecidingthedegreetowhicha
softwareintensiveproductmeetstheacceptancecriteriaofthosewhocommissioneditsconstruction.
Specifically,thisguidewillhelpyouifanyofthefollowingapplytoyou:
Youareinvolvedindecidingwhethertoacceptthesoftwareasithasbeenbuilt.Thisisthe
acceptancedecision.
Youareinvolvedindecidingonwhatacceptancecriteriashallbeusedtomaketheacceptance
decision.
Youareinvolvedincollectingdatathatthepersonmakingtheacceptancedecisionrequiresto
makethatdecision.Thisisacceptancetesting.
Youareinvolvedindecidingwhethertheproductisreadytobeseenbythepeopleinvolvedin
theacceptancedecisionoracceptancetesting.Thisisthereadinessdecision.

Youareinvolvedincollectingdatathatthepersonmakingthereadinessdecisionrequiresto
makethatdecision.Thisisreadinessassessment.
Youareinvolvedindefiningtheexpectationsagainstwhichthereadinessassessmentor
acceptancetestingactivitieswillbeconducted.Thisisacombinationofrequirements
gathering,testplanningandtestdesign.
Youareinvolvedinmanaginganyoftheprecedingactivities.

Thisguidedescribesthepracticesusedbypeopleintheprecedingroles,regardlesswhatyourjobtitle
is.Ifanyofthemdescribesyourrole,youshouldfindsomethingofinterestinthisguide.Appendix
ReaderPersonasdescribesourtargetaudienceinmoredetail.Thesepersonasareusedto:
remindtheauthorsandreviewersofourprimarytargetaudience.
giveyou,thereaderafiguretoempathizeorassociateyourselfwith.Thisshouldhelpyoupick
thepersonathatmostcloselyresemblesyourownjobresponsibilities.
IfanyofthegoalsdepictedinFigure1areapplicabletoyou,theguideandtheonlineknowledgebase
companion(TestingGuidance.codeplex.com)areforyou.

Figure1.
Readergoalsandpathsthroughtheguide.
GuideStructure
Thisguideisstructuredasafourvolumeseries(Figure2):
VolumeIprovidesanoverviewoftheacceptanceprocessandhowacceptancetestingandother
keypracticesfitintotheprocess.Thisvolumeisintendedtobereadfrombeginningtoend.
Itissubdividedintothreemainparts:
VolumeIIcoversAcceptancePlanningandManagementpracticesandpatterns.
VolumeIIIcoversFunctionalAcceptanceTestpracticesandpatterns.

VolumeIVcoversOperationalAcceptanceTestpracticesandpatterns.Operationalacceptance
includesnotonlyperformance,recoverability,availability,securitybutalsotestingofother
nonfunctionalrequirementsandqualities,suchaslearnability,usability,supportability,
upgradeabilityetc.

Figure2.
GuideStructure.
VolumeIisintendedtobereadasanintroductiontotheconcepts,termsandpracticesdescribedin
VolumesIIIV.VolumesIIIVareintendedtobeusedasareference;mostreaderswillnotreadthem
frombeginningtoend.
VolumeIThinkingAboutAcceptanceconsistsof3parts:
PartIThinkingaboutAcceptanceexplainssixmentalmodelsthatareusefulwhen
thinkingabouttheacceptanceprocess.
PartIIPerspectivesonAcceptancedescribestheacceptanceprocessfromthe
perspectivesofkeystakeholdersintwodifferentkindsoforganizations:theInformation
TechnologyDepartmentinabusinessandtheProductDevelopmentCompany.Most

readersinvolvedintheacceptanceprocessshouldfindsomecommonalitywithatleast
oneoftherolesdescribes.
PartIIIAcceptingSoftwareintroducesthepracticesthatarenecessaryforplanningthe
acceptanceprocess,forperformingacceptancetestingandforimprovingthe
acceptanceprocess.

VolumesIIthroughIVcontainacollectionofthumbnailsandsamples:
AThumbnailisashortoverviewofapracticethatexplainswhatitis,whenyoumaywanttouse
it,therisksthatitmitigates,andanoverviewofhowtoperformthepractice.Thumbnails
alsoincludealistofreferencestopapers,books,andotherresourcesthatprovidemore
completedescriptionsofthepracticesinquestion.Themainpurposeofathumbnailisto
describeatopicwellenoughtoprovideanoverview,serveasamentalreminderfor
someonewhohasusedthepracticeonhowtodoit,andgivesomeoneunfamiliarwiththe
practiceenoughinformationaboutthepracticeanditsapplicabilitytodetermineifthey
wanttolearnmoreaboutit.Someofthesetopicsandpracticeshaveentirebookswritten
aboutthemthatdescribetheconceptsingreaterdetailanddepththanthisguidecould
possiblydo.
ASampleisaanartifactgeneratedbyapplyingdifferentpracticesinafictionalbutrealistic
situationforthefictionalcompanyGlobalBank.Theseartifactsareembeddedinaseriesof
casestudiesofwhattheGlobalBankteammayhaveproducedwhilebuildingthe
application.Thecasestudiesprovidesomecontexttotheindividualartifacts.Theyalso
providecrossreferencestothethumbnails.Theartifactsareintendedtobeusedaswayto
learnmoreabouthowtoperformapractice;theycanalsobeusedastemplatesforyour
ownartifacts.

HowtoReadThisGuide
Thewayyouapproachthisguidewilldependonwhatroleyouhaveandwhatyouwanttolearnabout
theprocessofacceptingsoftware.Dependingonwhatyouwanttodo,youwillwanttoapplydifferent
strategies.
GetanOverviewofAcceptancePracticesandProcesses
StartbyreadingVolumeIifyouwanttodoanyorallofthefollowing:
Learngeneralinformationaboutacceptancetesting.
Findacceptancetestingpractices.
Createaprojectplanthatincorporatessoftwareacceptance.
Justifyaprojectplanthatincorporatessoftwareacceptance.

Justifyanapproachusedforacceptancetesting.
Validatethatyouareontrackwithyouracceptancetestingstrategyorapproach.
Getyourprojectunstuckwhensoftwareisunacceptable.
Determinewheretheremaybegapsinyouracceptancetestingapproachorstrategy.

AfterreadingVolumeI,youmaywanttoskimparticularpracticesofinterestandthecorresponding
samplesinVolumesIIthroughIV.
DecideWhichAcceptancePracticestoUseonYourProject
StartbyreadingVolumeItogetanoverviewofpossiblepractices,andthenrefertothethumbnailsin
VolumesIIIVforspecificpracticesyouareconsidering.Eachthumbnailincludesasectiontitled"When
toUseIt,"whichincludesadviceaboutwhenthepracticeshouldbeused,andasectiontitled
"Limitations,"whichprovidesguidelinesaboutwhenthepracticeshouldnotbeapplied.
LearnHowtoPerformaSpecificAcceptancePractice
StartbyfindingathumbnailinVolumesIIIVifyouwanttodoanyofthefollowing:
Learnaspecificacceptancetestingpracticeorstrategy.
Teachaspecificacceptancetestingpracticeorstrategytosomeoneelse.
Reviewaspecificacceptancetestingpractice.
Findmoreinformationandrelatedresourcestoconsultaboutaparticularpractice.

Afteryoulocatethethumbnailforthespecificpracticeyouwanttolearnabout,readitandanyrelated
samples.Ifyouneedmoredetailedinformationaboutthepractice,seethe"References"sectioninthe
thumbnail.
GetaTemplateforaSpecificArtifact
StartbyfindingansampleinVolumesIIIVifyouwanttodoanyofthefollowing:
Findatemplateforaspecificartifact.
Learnhowtofillinaspecificartifact.

FindtheexampleyouwantinVolumesIIIV,removethesampleinformation,andpopulateit
appropriately.Ifyouneedtoreviewthepracticethatgeneratedtheexample,theexamplelistsallthe
appropriatethumbnailsinVolumesIIIV.
PlantheExecutionofthePracticesonYourProject
StartbyreadingVolumeItogetanoverviewofhowthepracticesfittogetherandsupporteachother.In
particular,thesectionsontheAcceptanceProcess,theDecisionMakingModel,andtheDoneness
Modelmaybeofparticularinterest.Afterthat,reviewthespecificthumbnailsinVolumesIIIV,paying

particularattentiontothesubsection,"TestLifeCycleApplicability"inthesection,"WhentoUseIt."
Eachsampleartifactisaccompaniedbyanotationthatindicatesatwhatpointinthehypothetical
projecttheartifactwasproduced.Notethatsomeartifactsappearatseveralpointsintheproject
timelinebecausetheyevolveovertime.Ifyoufindyouracceptanceprocesstakestoolongyoumayfind
theStreamliningAcceptanceProcesschapterofVolumeItobehelpful.
FindToolsforAcceptanceTesting
Ifyouarelookingfortoolstoperformacceptancetesting,youmayusetheguidetoexplorethe
availablespace.Althoughsomeofthecasestudiesillustrateusingspecifictools,rememberthatthe
primaryfocusofthisguideisondescribingpractices.
Thechoiceoftoolsusedwhileproducingthisguideshouldnotbeinterpretedasanendorsementofany
tool,norshoulditbeinterpretedasanindicationthatanytoolusedisthebestoneforthejob.Bythe
timeyoureadthisguide,thetoolsillustratedinthesamplesintheguidemayhavebeensupplantedby
newerandbetterones.

TheTeamWhoBroughtYouThisGuide
Mainauthors:
GrigoriMelnik,GerardMeszaros,JonBach
Contributors:
HakanErdogmus,MichaelPuleio,RohitSharma
AdvisoryBoardMembers:
MakAgashe,MohammadAlSabt,LoisPhilippeCarignan,SreekumarChandra,BartHsu,ChrisShaffer,
EngChongLim,DonnaMcLeod,KarlMetivier,TracyMonteith,NoelNyman,KenPerilman,BobSnead,
KeithStobie,JamesWhittaker
Reviewers:
JennittaAndrea,MarkAustin,DanBarnes,KentBeck,LarryBrader,HansBuwalda,JanetGregory,Sam
Guckenheimer,MihaKralj,JamesLyndsay,DanielPiessens,MaxPoliashenko,TomPoppendieck,BJ
Rollison,DonSmith,BasVodde,ThomasWoisetchlger
SpecialThanksto:
JamesBachforconstantremindertotestourtestingguidance
JeffPattonfortheProductContextchart(VolumeI,Ch.4)
BobBrumfieldfortheMultiplexingsidebarsuggestion(VolumeI,Ch.17)andmanyinsightful
conversations
patterns&practicesleadsforsupportingthisendeavor
Production:
RichardBurte,RoAnnCorbisier,DennisDeWitt,TinaBurdenMcGrayne,VeronicaRuiz
Community:
AttendeesatSTARWEST,patterns&practicessummitsandTechEdconferenceswhoprovidedinformal
feedbackonthis.

Introduction
Thisguideisaboutacceptingsoftware.Acceptingsoftwareinvolvesacceptancetesting,butitismuch
morethanthat.Theconceptofacceptancetestingmeansdifferentthingstodifferentpeople.Insimple
terms,acceptancetestingisthesetofactivitiesweperformtogathertheinformationweneedtomake
thedecisionIsthissoftwarereadyforme(ormycustomer)anddoesitfullfillmy(andmycustomers)
requirements?Thisdecisionisusuallycomposedofseveraldecisions,eachwithsupportingactivities.
Therefore,todefineacceptancetesting,itmaybeusefultounderstandtheprocessbywhichthe
decision(s)aremade.Thisprocessmayinvolveseveralorganizationalentities,eachwithoneormore
decisionmakers.Thesoftwareistypicallypassedbetweentheorganizationalentitiesforthemtodecide
whetherthesoftwareisreadytogothroughthenextstep.Thisprocessisintroducedinmoredetailin
thesection"AcceptanceProcessModel"andfollowedupwithamoredetaileddescriptionofthe
decisionmakingprocessinthesectioncalled"DecisionMakingModel."
SoftwareAcceptanceandAcceptanceTesting
Acceptancereferstotheactofdeterminingwhetherapieceofsoftwareorasystemmeetstheproduct
ownersexpectations.Itincludesboththefinaldecisiontoacceptthesoftwareandanyactivities,
includingacceptancetesting,requiredtocollectthedataonwhichtheacceptancedecisionisbased.
Boththeacceptancetestingandtheacceptancedecisioncanberelegatedtoaseparateacceptance
phaseoftheprojectortheycanbedonethroughouttheproject,whichisknownasIncremental
AcceptanceTesting.
MentalModelsforAcceptanceTesting
Whilewritingthisguide,westruggledtodetermineasuitabledefinitionofacceptancetestingthat
wouldmakesensetoabroadrangeofreaders.Itseemedlikethereweremanydifferentvocabulariesin
usebydifferentcommunitiessuchasconsumerproductcompanies,informationtechnology
departmentsoflargebusinesses,anddataprocessingdivisionsoftelecommunicationserviceproviders,
tonamejustafew.Toassistusindescribingacceptance,wecameupwithseveralmentalmodelsof
variousaspectsofacceptancetesting.Wetestedthemodelsagainstnumerousexamplesfromour
collectiveprojectexperiencesatMicrosoftpatterns&practices,telecommunicationproductcompanies,
ITdepartmentsandbeyond.ThenwetestedthemodelswiththepeopleontheAdvisoryBoardforthe
project.Thiswasaniterativeprocess.Wealsotestedthesethroughthepublicreviewprocessby
releasingearlydraftsofthisguidetothecommunityandsolicitingfeedback.
Itisimportanttonotethatourearlymodelsfailedtheiracceptancetests!Thatwasagreatlessonabout
theneedtogetfeedbackincrementally,apracticeweadvocateforacceptancetesting.Basedon
feedbackfromouradvisorswerefactoredthemodelsandcameupwithadditionalmodelstofillthe
gaps.ThekeybreakthroughwaswhenwecameupwiththeDecisionMakingModel,whichtiestogether

mostoftheconceptsaroundacceptingasystem.ItbuildsontheAcceptanceProcessModelwhich
describesthekeystepsandactivitiesasthesystemundertestmovesfromrequirements,through
developmentandintotestingandfinallyproduction;italsodescribeshowthedecisiontoacceptthe
systemismade.TheDecisionMakingmodeldescribeswhomakesthedecisionsandwhoprovidesthe
dataforthosedecisions.
Thedecisionsarenotmadeinavacuum;thereareanumberofinputs.Theseincludetheproject
context,thenatureofthesystembeingbuiltandtheprocessbeingusedtobuildit.Thelatteris
importantbecauseitaffectshowwedefinedone.
Figure1illustratestherelationshipsbetweenthekeymodels.

Figure1

TheKeyMentalModelsofAcceptance
ThesemodelsarethefocusofPartIThinkingaboutAcceptancebutheresashortintroductiontoeach
modeltogetusstarted:
TheAcceptanceProcessModel.Thismodeldefinestheoverallstagesofsoftwaredevelopment
andthe"gates"thatmustbepassedthroughonthejourneyfromconstructiontosoftware
inuse.
DecisionMakingModel.Thismodeldescribeshowtodecidewhethersoftwarecangothrougha
gatetothenextstageandwhomakesthedecision.Italsodefinesthesupportingrolesthat
mayhelpthedecisionmakergathertheinformationneededtomakethedecision.
ProjectContextModel.Thismodeldescribesthebusinessandprojectfactorsthatinfluencethe
decision,includingtimeframesanddeadlines,resourceavailability,budget,andanything
contributingtoprojectrisks.
SystemModel.Thismodeldescribestheattributesofthesoftwareintensivesystemthatmay
factorintothedecision.Thisincludesbothfunctionalandnonfunctionalattributes.The
systemmodelmayariseoutofrequirementsgatheringactivitiesoramoreformalproduct
designprocess.Techniquesforcapturingfunctionalrequirementsincludesimpleprose,use
casemodels,protocolspecificationsandfeaturelists.Thenonfunctionalrequirementsare
typicallycapturedindocumentsorchecklists.
RiskModel.Thismodelintroducestheconceptsofevents,likelihood/probability,and
consequence/impact.Ithelpseveryoneinvolvedunderstandwhatcouldgowrongand,
thereby,prioritizetheacceptancecriteriaandthekindsofinformationtogathertohelp
maketheacceptancedecision.Italsodescribesseveraldifferentriskmitigationstrategies,
includingthefollowing:
Dosomethingearliertobuyreactiontime.
Doadditionalactivitiestoreducelikelihoodofsomethingoccurring.
DonenessModel.Thismodeldescribesdifferentdimensionsofdonenessthatneedtobe
consideredinareleasedecisionandhowtheyareaffectedbytheprocessmodel.

ThechaptersinPartIIIAcceptingSoftwareintroduceothermodelsthatbuildonthiscoremodel:
TestLifecycleModel.Thisdescribesthestagesthatanindividualtestcasegoesthroughhowto
gatherinformationformakingreadinessandacceptancedecisions.
ConcernResolutionModel.Thisdescribeshowtohandleanyconcernsthatareraisedduringthe
acceptancetestingprocess.


DevelopmentProcesses
Thesoftwaredevelopmentprocesshasasignificantimpactonhowacceptancetestingisperformed.
ThroughouttherestofthisvolumeandtheotherstofollowwefoundourselvessayingOnsequential
projectsandOnagileprojectsbutmanypeoplehavetheirowndefinitionsofwhattheseterms
mean.Wewantedtomakesureallreadersunderstoodwhatwemeantbytheseterms.Wefeelthat
thesenamesrefertopointsonaprocesscontinuumwithotherlabelledpointspossible.Thissection
describestheprocesscontinuumwithtwodistinctprocessstereotypesontheoppositeendsofthescale.
SequentialProcesses
Sequentialprocessesorganizeworkbasedonthekindsofactivitiesinvolvedandtheinterdepencies
betweentheactivities.Theclassicwaterfallapproachinvolvesasinglereleasewhileincremental
waterfallprojectshavemultiplereleases.
ClassicWaterfall
Thewaterfallapproach(sonamedafterthediagramsusedinapaperbyWinstonRoyce[Royce])
involvesorganizingtheprojectintoaseriesofdistinctphases.Eachphasecontainsaspecifictypeof
work(suchasrequirementsanalysis)andhasspecificentryandexitcriteria.Intheclassicorpure
waterfallapproachthephasesdonotoverlap.Theentryandexitcriteriarequiretheoutcomeofa
previousphasetobecompleteandcorrect(validated)beforethenextphasecanstart.Thispushesthe
deliveryoftheproductsfunctionalitytotheend:theproductasawholeisdeployedinabigbang
approach.Figure2illustratesthemajorphasesofawaterfallproject.

Figure2
AClassicalWaterfallProject
Thisprocessisusuallyimplementedbybreakingdowneachphaseintohierarchicallyorganizedunitsof
workappropriatetothetypeofworkinvolved.Forexample,withintherequirementsphase,thework
maybedividedbetweenanalystsbyrequirementtopic,butduringtheconstructionphase,workmaybe
dividedamongthedevelopersbymodule.Thehandoffsbetweenphasesareusuallyintheformof
documents,exceptthatthehandofffromconstructiontotestingalsoinvolvesthecodebase.Readiness
assessmentisdonebythesupplierorganization,whichwerefertoastheProductDevelopmentTeam,
afteralltheconstructioniscompleted;acceptancetestingisperformedbytheproductownerafterthe
softwareisdeemedtobeready.

IncrementalWaterfall
Itiscommonlyacceptedthatthelongeraprojectgoesbeforedeliveringsoftware,thehigherthe
probabilityoffailure.Ifthecontextorrequirementschangebeforetheprojectiscompleted,thepure
waterfallcannotsucceed.Onewaytocombatthisistobreaktheprojectintoincrementsof
functionality.Theseincrementscanbetestedandinsomecasesevendeployed.Figure3aillustratesa
waterfallprojectwithtwoincrementsofindependentfunctionalityeachofwhichistestedand
deployed.Thistypeofprojectisalsocalledcheckpointedwaterfall).
time
Requirements
Dply
Test
Archiecture
High-Level
Construction
Functionality
& Design
Planning
Analysis
Dplyt
Test
Construction

Figure3a
Multireleasewaterfallwithindependentfunctionality
Inthisapproach,theplanningphase,requirementsanalysisphase,anddesignphaseareperformed
onceearlyintheprojectwhiletheconstructionphase,testphase,anddeploymentphasearerepeated
severaltimes.Theworkwithineachphaseisdecomposedthesamewayasforsinglereleaseprojects.If
thefunctionalitybuiltinthesecondreleaseoverlapsthefunctionalityinthefirstrelease,thetestingand
deploymentmustencompasstheentirefunctionality.Figure3billustratesmultiplereleaseswith
overlappingfunctionality.Notehowthetestactivitymustspanthefunctionalityofbothreleases.

Figure3b
Multireleasewaterfallwithoverlappingfunctionality
Inthemultireleasewaterfallprocess,sometimesthetestphaseoftheareleasemayoverlapwiththe
constructionphaseofthesubsequentreleaseiftheconstructionandtestingteamsareseparateand
featuresacrossthethetworeleasesaresufficientlyindependent.

AgileProcesses
Mostagilemethodsuseaniterativeandincrementalapproachtodevelopment.Afteraninitialplanning
period,theprojectdurationisbrokenintodevelopmentiterationsthatdeliverincrementsofworking
software.Figure4illustratesaprojectwithtwoiterations;mostprojectswouldhavemanymore
iterationsthanthis.

Figure4
Iterative&IncrementalDevelopment
Figure4illustratestwoiterations,eachofwhichstartswithaniterationplanningsessionandendswith
someacceptancetesting.Ineachiterationtheworkisbrokendownintofeaturesoruserstories,each
ofwhichindependentlygoesthroughtheentiresoftwaredevelopmentlifecycle.Theproductowner,
onsitecustomerorcustomerproxy,whoisreadilyaccessibletotheProductDevelopmentTeam,is
responsiblefordescribingthedetailsoftherequirementstothedevelopers.Itisalsotheproduct
ownersresponsibilitytodefinetheacceptancetestsforeachfeatureoruserstory.Theymayprepare
theteststhemselves,delegatethejobtorequirementsortestspecialistswithinthProductOwnerTeam
orpreparethemincollaborationwiththeProductDevelopmentTeam.Thetestsactasamoredetailed
versionoftherequirementsdescriptioninaprocessknownasAcceptanceTestDrivenDevelopment
orStorytestDrivenDevelopment.Thisallowsthedeveloperstoexecutetheacceptancetestsduring
thedevelopmentcycle.
Whenallthetestspassforaparticularfeatureoruserstory,thedevelopersturnoverthefunctionality
totheproductowner(orproxy)forimmediate"incrementalacceptancetesting."Iftheproductowner
findsanybugs,thedeveloperfixesitassoonaspossible.ItistheProductOwnersdiscretiontoinsist
thatitbefixedrightawayortoconsideritpartofanotherfeaturetobeimplementedinalater
iteration.Iftheywantitfixedandthedeveloperhasalreadystartedworkingonanotherfeature,the
developertypicallyputstheotherfeatureonholdwhiletheyaddresstheProductOwnersconcerns;the

featureisntconsidereddoneuntilalltheconcernsareaddressed.Theproductownerconcernsare
notstockpiledinabugdatabaseforfixingduringabugfixingphaseoftheproject.
MultiReleaseAgileProjects
Mostagilemethodsadvocate"deliverearly,deliveroften."Intheory,theresultofanydevelopment
iterationcouldbedetermined,afterthefact,tobesufficienttobeputintoproduction.Thiswouldlead
directlytothedeploymentactivities.Inpractice,mostagileprojectsplanonmorethanonereleaseto
productionandtheiterationsarethenplannedtodeliverthenecessaryfunctionality.Figure6Multi
ReleaseAgileProjectillustratesanagileprojectwithtworeleases.

Figure5
MultiReleaseAgileProject.
Notehowthereisatestingcycleforthesecondreleasewhichincludesregressiontestingofthe
functionalitydeliveredinthefirstrelease.Mostagilemethodsemphasizetestautomationsothe
regressiontestingcostisminimized.
KanbanbasedAgileProcess
Someagilemethodologiesdispensewithiterationsinfavourofallowingafixednumberoffeaturesin
progressatanytime.Thisisdesignedtoemphasizetheconceptofacontinuousflowofworkingcode
fortheonsiteproductowner(orproxy)toaccept.Fromanacceptancetestingperspective,these
Kanbanbasedmethods[Kanban]stilldoincrementalacceptancetestingatthefeatureleveland
formal/finalacceptancetestingbeforeeachrelease,butthereisnologicalpointatwhichtotriggerthe
interimacceptancetestingthatwouldhavebeendoneatiteration'sendiniterationbasedagile
methods.Figure6KanbanbasedAgileProjectillustratesthisstyleofdevelopment.Notethelackof
iterations.

Figure6
KanbanbasedAgileProcess
Intheexample,itisimportanttonoteherethattherearenevermorethanthreefeatures(inthis
example,oneudergoingdesign,asecondundergoingconstruction,andathirdundergoingtesting)in
progressatanyonetime.Inotherwords,thereareonlythreedevelopment"slots,"andaslotbecomes
availableforanotherfeatureonlyafterithasfinisheditsincrementalacceptancetesting.Thisissimilar
tohowKanbanareusedtocontroltheinventoryinleanfactoryproductionlines.Intheory,theproduct
ownercandecideatanytimethatthereisenoughfunctionalitytowarrantdeployingtheproduct
followingashortperiodofregressiontesting.
Kanbanbasedsoftwareproceses[Scrumban]areimplementationsofamoregeneralphilosophyknown
asLeansoftwaredevelopment[LSD].
ProcessasaContinuum
"Agile"and"waterfall"areexamplesoftwohighlevelprojectstreotypesconsistingofcertain
combinationsofcharacteristics.Itiseasytoimaginethedecisiononeachofthesecharacteristicsas
beingthesettingofaprocessslidercontrol.Forexample,theNumberofreleasesslidermighthave
stopsat1,2,3,andsoon.TheNumberofiterationsslidercouldhavevaluesof1andsoon,which
indicatewhetherthereareintermediatecheckpointswithinarelease.TheMaximumnumberof
featuresinprogressslidersimilarlymaytakeonvaluesdependingonthenumberofdevelopmentslots
availableinaKanbanbasedsystem.AnotherdimensionmightbeIntegrationfrequency,withsettings
ofBigBang,MajorMilestone,Quarterly,Monthly,Biweekly,Weekly,Daily,andContinuous.
Thefollowingtablesummarizesthepositionsoftheseslidersforwhatisconsideredtobeastereotypical
projectofeachkind.Thesepositionsarenotdefinitiveorcomplete,buttheychallengeyoutocreate
yourownslidersandsettingsforyourcontext.

Type of Process
Classic Incremental Agile (Iteration) Agile (Kanban)

Project Attributes Waterfall Waterfall
Number of releases 1 1 2 or more 2 or more
Number of iterations 1 26 4 or more 1
Iteration length Not applicable Many months 1-4 weeks Not applicable
Maximum number of No maximum No maximum 1 iterations worth Less than the

features in progress number of
team members
Integration frequency Big Bang Quarterly Daily or hourly Daily or hourly
Requirement-to-test Months or years Months Days Days

duration
Test timing Separate phase Separate phase Mostly incremental Mostly

incremental
Release criteria Scope-based Scope-based Time-boxed Time-boxed
Average Requirement task Person months Person months Person days Person days
effort
Average development task Person days or Person days or Person hours Person hours
effort weeks weeks
Culture Hierarchical Hierarchical Collaborative Collaborative
Skills Highly Highly Generalists Generalists or

specialized specialized Specialists
Determining progress Work Work completed Delivery of working Delivery of

completed relative to plan code working code
relative to plan
Work remaining Estimate Estimate Estimated time for Estimated time

duration of duration of remaining features for remaining
remaining tasks remaining tasks features


References
[Royce]WinstonW.Royce,ManagingtheDevelopmentofLargeSoftwareSystems,Proceedingsof
IEEEWESCON,Aughust1970,pp19.
[Kanban]KenjiHiranabe,KanbanAppliedtoSoftwareDevelopment:fromAgiletoLean,published
onlineat<http://www.infoq.com/articles/hiranabeleanagilekanban>,January14,2008
[LSD]Poppendieck,Mary&TomLeanSoftwareDevelopmentAddisonWesley(2003)ISBN:032
1150783
[Scrumban]CoreyLadas,Scrumban:EssaysonKanbanSystemsforLeanDevelopment,Modus
CooperandiPress,2008


ACautionaryTale
Thefollowingisthestoryofaprojectwherethepersoninchargeofdefiningandacceptingaproduct
failstorisetothechallenge.Itdescribeswhatgoeswrongandwhy.Italsoprovidesanalternate
outcomeofhowtheprojectcouldhaveworkedoutifproperacceptancepracticeshadbeenused.
BobKellyisamidlevelmanagerinthemarketingdepartmentatAcmeProducts.Heisinchargeofthe
MidSizedMarketsproductgroupandistheproductmanagerofitscoreproduct,theXCelRator.Hes
justfinishedabunchofmarketresearchandhascomeupwithagreatideaforanewmoduleforthe
productthathebelieveswilldoubletherevenueoftheproduct.TheProductDevelopmentdivisionhas
ateamthatisjustwindingdownitscurrentprojectandBobiskeentogettheteamstartedonbuilding
thenewmodule.HecallsameetingandrunsthroughthePowerPointslidedeckhesbeenusingwhen
talkingwithfocusgroupsandpotentialclients.Heconcludesthepresentationbylayingoutthekey
deliverydateswhichmustbemettoallowthecompanytoshowcasetheproductattheannualtrade
show.Heaskstheprojectmanagertodrawuptheprojectplanandgettheteamworkingonthenew
module.
TheDevManagermeetswithhisteamtocollectsomeinformationandthenhedefinestheprojectplan.
BasedonthedeliverydatehedefinesintermediatemilestonesforRequirementsComplete,Design
Complete,CodeCompleteandTestComplete.
Teamstartswritingrequirementsdocuments.AtRequirementsComplete,devmanagerdeclares
requirementscompleteontime.Theproductdevelopmentteam(devteam)knowsfullwellthatsome
areasoftherequirementsarestilltoovaguetoimplement.Theywillhavetogetclarificationonthe
detailsastheydothedesign.Hopefullytheproductmanager,Bob,willbemoreavailabletoanswer
questionsabouttherequirementsashislackofavailabilityisatleastpartiallytoblameforthe
requirementsstillbeingvague.
AtDesignComplete,ditto.Devteamknowsithasntdesignedsomepartsofthefunctionalityexceptata
verycursorylevel.Theywillhavetofillinthedetailsastheycode.Meanwhile,Bob,theproduct
managerthinkseverythingisproceedingaccordingtoplanandthathellhaveagreatproducttodemo
atthetradeshow.Hestartsthinkingabouthowhellspendthebonushessuretogetforcompleting
thisprojectontime.Hecallshisarchitectandaskshertostartdrawingupplansfortheextensiontohis
house,completewithindoor/outdoorswimmingpool.
Atcodecompletetheteamrequestsanextramonthtofinishcodingpartsofthefunctionalitythey
haventhadtimetodoyet.TheTestManageraskswhatpartsofthesoftwarearecompletesothat
testerscanstarttestingandtheteamrespondsNone,weareeachworkingonadifferentpartofthe
systemanditwillallbefinishedataboutthesametime.
Toensuretheproductisreadyforthetradeshow,Bobasksthetestmanagertomakeuptheschedule
bycompressingthedurationofthetestphase.Hereducesthenumberofplannedtestcyclestoreduce
theelapsedtimebasedonassurancesthatthecodewillbeingreatshapeduetotheextratimethedev
teamisbeinggiven.Bobstillbelievestheproductcanbereadyforthetradeshow(butnonethelesshe

asksthearchitecttoscalebacktheextensiontohishousebyremovingtheenclosureforthepool
thinkingtohimselfIllstillbeabletouseitmostoftheyearandIcanalwaysaddtheenclosurewithmy
release2bonus.)
AsthenewCodeCompletedeadlineapproaches,theteamasksforanothermonth.Bobreluctantly
givesthem2weeks.Twoweekslaterwillmakeittightforthetradeshowbutwecandemothebeta,
hehopes.
Thedevteamfinallydeliversthecodetwomonthslate.Thetestteamstartstestingbutfindssignificant
problemsthatpreventcompletionofthefirstpassofthetestcases.Testinghaltsafterlessthanaweek.
Thedevteamtakesamonthtofixallthemajorshowstopperbugsbeforehandingthecodebackto
test.Testteammakesitthroughthefulltestcyclethistimebutfindsseveralhundreddefects.
Developmentstartsfixingthebugsastheyarefound.
ThetestteamfinallyreceivesanewbuildthatthedevteamsayshasalltheSev1and2bugfixes.
AlmostimmediatelytheyfindseveralnewSev1regressionbugs.Developmentdisagreesthatsomeof
thebugsareevenvalidrequirements.TherequirementsneversaidanythingaboutandWhywould
anyoneeverdothat?theyexclaim.Bob,theproductmanager,hastobebroughtintosettlethe
dispute.Heagreesthatsomeofthebugsarentrealrequirementsbutmostarevalidscenariosthathe
neverconsideredbutwhichneedtobesupported.TesttellsBobthatthereisnowayinthatthe
originalschedulecanbemet.
After4test&fixcyclesthattook50%longerthantheoriginalschedule(letalonemakingupthe
schedule),mostoftheSev1and2bugsappeartohavebeenfixed.ThereisstillseveralhundredSev3
and4bugsandtestteamhasstoppedbotheringtologtheSev5s(poorlywordedmessages,fieldlabels
orbuttonnames,etc.).Testteamsaystheproductisnotreadytoshipandwillrequireatleast2more
test&fixcyclesbeforetheywillagreetoreleaseit.
Theproductisnowseveralmonthslateandwontbereadyforthetradeshoweveninalpharelease.
Wecanstillshowtheboxanddosomeverycontrolleddemos.Bobassureshisboss.Bobisseeinghis
bonusdiminishwitheachweekofreducedsalesinthisfiscalyear.Hedecidestoshiptheproduct,
overrulingtheTestdepartment.Herevisesthesalesforecastsbasedonthelatelaunchcausedbythe
underperformanceofthedevelopmentandtestteams.
ThemanageroftheoperationsdepartmenthearsaboutthistoolateandcomesstormingintoBobs
office.Ihearyouareplanningtoreleasethenewmodulewith300knownSeverity3defects.Doyou
haveanyideawhatthiswilldotooursupportcosts?Acounterproductiveargumentensuesbecause
thereisnoturningbackatthispoint;theannouncementshavebeenmadeandtochangeplansnow
wouldbehugelyembarrassingtothecompany.IsurehopethisworksoutOK.thinksBobtohimself;
atthispointhedoesntfeellikehesinchargeofhisowndestiny.
Bobsmarketingmachinehasbeeninhighgearforquiteawhileandhasgeneratedquiteabitofpent
updemand,especiallysincesomeuserswerehopingtobeusingtheproductmonthsago.Userstry
usingthenewproductindroves.Theyrunintoallmannerofproblemsandcallthehelplinewhichis
overwhelmedbythecallvolumes.Manyusersgetbusysignals;theluckyoneswaitlisteningtorecorded
announcementsforlongperiodsoftime.Thehelpdeskhastobringonextrastaffwhoneedtobe

trainedveryhastilyandthereforecannotprovideverygoodservicetothecustomers.Somecustomers
giveuptryingtousetheproductbecauseoflongwaitsorpoorsupport.
Manyoftheuserproblemsarerelatedtousabilityissuesthatwerenotdetectedduringtherushed
testingbecauseitwassofocusedontheSev1&2bugs(andtheusabilitystuffwasusuallyrated3or
below,manyofwhichwerentevenloggedduetothefocusonthe1sand2s.)Atpeakusagetimesthe
systemslowstoacrawl;thedevteamiscalledintofigureoutwhyitissoslowdistractingthemfrom
workingonsomeoftheimprovementsidentifiedduringtesting.Usershavetroubleimportingdatafrom
priorversionsofthesoftwareorfromcompetitorsproducts.
Alargepercentageoftheusersabandontheproductafterthefreetrialisover;theconversionrateis
lessthanhalfoftheprojectedrate.Revenuesarerunningat40%oftherevisedprojectionsandless
than20%oftheoriginalprojections.Supportcostsare50%overoriginalprojectionsduetothehiring
bingeintheusersupportcentre.
Thecapitalcostis35%overtheoriginalbudgetandhaseatenintotheplannedbudgetforasecond
modulewithadditionalmusthavefunctionality.Instead,the2ndreleasehastofocusonimprovingthe
quality.Thenewmodulewillhavetowaituntil2ndhalfofnextyear.Theproductmanagerrevisesthe
revenueprojectionsyetagain(andcallsthecontractortocanceltheadditiontohishouse).Shortlyafter
sendinginhismonthlystatusreporthisphonerings.ItistheVP,hisboss,requestinghecometohis
officeimmediately...
WhatWentWrong
1. ProductmanagerBobdecidedscope,timeframesandresourcestherebyleavingonlyqualityas
thederivablevariable.Thisresultedinthedevteambeingovercommittedattheproduct
managersinsistence.(Sometimesthedevteamwillovercommitoutofoptimismbutinthis
casetheproductmanagerdidittothem.)
2. Sequentialprocessisinherentlyopaquefromaprogressperspective.Thefirstmilestonethat
isnteasytofudgeisCodeComplete.Thefirstrealisticassessmentofprogressiswellintothe
testcycle.
3. Productmanagerwasntavailabletoclarifyvagueandmissingrequirements.Testerswerenot
involvedsoonenoughtoidentifythetestscenariosthatwouldhavehighlightedthemissing
requirements.ButnoonecouldprovetherequirementswereincompletesoRCwasdeclared
ontime.
4. Productdevelopmentteamcouldntprovedesignwasntdone(becauseitisamatterof
opinionastohowdetailedthedesignneedstobe)soDesignCompletewasdeclaredontime.
Thesequentialprocessdoesntgivethedevelopmentteamagoodwaytoestimateitstrue
velocityuntillateintheproject,whenthecodeismostlywrittenanddebugged.
5. Devteamcutcornerstomakethenew(alreadylate)CodeCompletedeadline.Thecodewas
writtenbutmuchofitwasntproperlyunittested.Theteamwasdeepinunittestdebt,they
knewit,andwouldhavetoldanyonewhoaskedbutnoonewantedtoheartheanswer.

6. ThequalitywasawfulwhendeliveredtoTest.Soithadtoberedone(weneverhavetimetodo
itrightbutwealwaysmaketimetodoitover!)
7. Testwasaskedtorecovertheschedule(typical!)buttestingtooklongerbecausethepoor
qualitycoderequiredmoretest&fixcyclestogetitintoshape.
8. Nocleardefinitionofdonesodecisionismadeontheflywhenemotionsgetinthewayof
clearthinking.Theproductmanagerlethisattachementtohiscommitmentsoverrideany
sensibilityaboutquality.
9. Theoperationsacceptancecriteriawereneversolicitedandbythetimetheywereknownitwas
toolatetoaddressthem.
10. Sequentialprocesshidtrueprogress(velocity)untilitwastoolatetorecover.Therewasnoway
toscalebackfunctionalitybythetimeitbecameundeniablethatitwontallbedoneontime.
Therewasnowaytoreducescopetofitthetimelinebecausethesequentialstyleapproachto
theprojectplan(RequirementsComplete,DesignComplete,CodeComplete,TestingComplete
milestones)causedallfeaturestobeatroughlythesamestageofdevelopment.Therefore
cuttinganyscopewouldresultinalargewasteofeffortandverylittlesavingsofelapsedtime.
(Onlytheeffortofbugfixingwouldhavebeensaved.)
11. DevelopmentwasrushedinfixingtheSev1problemssothattestingcouldgetthroughatleast
onefulltestcycle.Thiscausedthemtomakemistakesandintroduceregressionbugs.Ittook
severaltest&fixcyclesjusttofixtheoriginalSev1&2sandtheresultingregressionbugs.Inthe
meantimetheSev3spiledupandupandup.Thisresultedinseveralmoretest&fixcyclestofix
andeventhenmorethanhalfwerestilloutstanding.
12. Lackofplanningfortheusagephaseoftheprojectresultedinapoorcustomersupport
experiencewhichexacerbatedthepoorproductquality.
HowitCouldHaveGone
Bob,theproductmanagercomestoproductdevelopmentteamwithrequirementsthatare
representativeofthecustomersexpectations.
Devteamestimatesrequirementsasbeing50%overteamscapabilitybasedondemonstrated
developmentvelocity.Theproductmanagerisnothappyaboutthis.Devteamproposesincremental
development&acceptancetesting.Insteadof4sequentialmilestonesbasedonphasesofdevelopment
theysuggest4incrementalinternalreleasesoffunctionalitywhereeachincrementcanbetested
properly.Thiswillallowthemtodeterminetheirdevelopmentcapacity(orvelocity)whichwillhelp
theproductmanagerplansubsequentmilestonesmoreaccurately.
Bobselectsfirstincrementoffunctionalitytodevelop.Withthehelpofthedevteamhedefinesthe
usagemodelconsistingofuserpersonasandtasks.TheDevteamwhipsupsomesketchesorpaper
prototypesandhelpsBobrunsomeWizardofOztestsonthepaperprototypesthatrevealsome
usabilityissues.BobadjuststherequirementsanddevteamadjuststheUIdesign.Bob,theproduct

managerworkswithdevteamandthetesterstodefinetheacceptancetests.Thedevteamautomates
thetestssotheycanberunondemand.Theyalsobreakdownthefeaturesintouserstoriesthateach
takejustafewdaystodesignandtest.
Teamdesignssoftwareandwritescodeusingtestdrivendevelopment.Allcodeisproperlyunittested
asitiswritten.Allpreviouslydefinedautomatedtestsarererunseveraltimesadaytomakesureno
regressionbugsareintroducedasthenewfeatureisimplemented.Aseachfeatureisfinished,as
definedbythefeatureleveldonedonechecklist,thedeveloperdemostoproductmanagerand
testerwhocanpointoutanyobviouslymissingfunctionalitythatneedstobefixedbeforethesoftware
isconsideredreadyforacceptancetesting.
Aspartofincrementalacceptancetestingtheydoidentifyafewnewusagescenariosthatwerenotpart
oftherequirementsandprovidethesetoBobassuggestionsforpotentialinclusioninthesubsequent
increments.Bobadjuststhecontentofthenextincrementbyincludingafewofthemorecriticalitems
andremovinganequivalentamountoffunctionality.Healsocontactstheoperationsmanagerto
validatesomeoftheoperationalusagescenariosidentifiedbythedevteamandtesters.Theoperations
managersuggestsafewadditionaloperationaluserstorieswhichBob,theproductmanager,addsto
thefeaturebacklogforthenextincrement.
Attheendoffirstincrementoffunctionality(whichtook5iterationstodevelop,onemorethan
expected)devteamrunsthenonfunctionalqualitiesteststoverifythesoftwareperformsupto
expectationsevenwith110%oftheratednumbersofusers.Thefirsttestresultsindicateitcanonly
handleroughly50%oftheexpectedusersandgetsevenslowerasthedatabasestartstoaccumulate
records.Theyaddsomeworkitemstothescheduleforthesecondincrementtoaddresstheseissues
andwarntestingaboutthelimitationstoavoidtheirwastingtimestumblingontothem.Testingfinds
onlyafewminorbugsduringexecutionofthefunctionaltestscripts(themajorstuffwasallcaught
duringincrementalacceptancetesting.)Theymoveontodoingsomeexploratorytestingusingsoap
operasandscenariosastheirtestsessioncharters.ThesetestsidentifyseveralpotentialscenariosBob
neverthoughtof;headdsthemtothefeaturebacklog.
Bob,astheproductmanagerarrangestodosomeusabilitytestingofthefirstincrementwithsome
friendlyusersbasedonsomeoftheusagescenariosidentifiedintheusermodel.Theresultsofthe
testingidentifyseveralenhancementsthatwouldimprovetheusersatisfaction.Bobaddsthesetothe
thingstodointhenextincrementoffunctionality.
Bobcalculatesthatdemonstrateddevelopmentvelocityis25%lessthanoriginalestimates.Basedon
thisheadjustshisexpectationsforthefunctionalitytobedeliveredinthereleasebyremovingsomeof
thelesscriticalfeaturesandthinningsomeofthecriticalfeaturesbyremovingsomenicetohave
glitz.Bettertodelivergoodquality,ontimethantotrytocraminextrafunctionalityandrisk
everythinghethinkstohimself.
Thedevelopmentteamdeliversthe2ndincrementoffunctionalitywithsimilarresultsasthefirst.The
worktheydidtoimproveperformanceresultsintheperformancetestspassingwithflyingcolorswith
acceptableresponsetimesat120%ofratedcapacityandnodegradationasthedatabasefillsupwith
transactionaldata.Theyaddsometestsforpenetrationtestingandscheduleasecurityreviewwiththe

securitydepartment.Bobmakessomeadjustmentstothefunctionalityplannedforthe3rdincrementof
functionality.Heincludessomefunctionalitytoaddressoperationalrequirementssuchasmigrating
datafromearlierversionsofthesoftwareandimportingdatafromcompetitorsproducts;hewantsto
makeitrealeasyforuserstoadopthisproduct.Heshappywithhowtheprojectisprogressingand
confidentthattheywillbeabletodeliverexcellentqualityandonschedule.
Inthethirdincrement,thedevelopmentteamdelivers20%morefunctionalitythanoriginallyplanned.
Theproductmanagerhadtoscrambletoprovidetheacceptancetestsfortheextrafeaturesbrought
forwardfromthefourthincrement.Basedonthis,theproductmanagerisabletoplanforsomeextra
functionalityforthefourthincrement.Heconsidersrevivingsomeofthefunctionalitythathedcutafter
thefirstincrementbutdecidesitreallywasntthatuseful;amuchbetteruseofthedevteamsefforts
wouldbesomeoftheusabilityenhancementssuggestedbythelastroundofusabilitytesting.Healso
addsfunctionalitytomakeiteasytoupgradetothenext(yetunplanned)releasewithouthavingtotake
downtheserver.Thatwillhelpreducetheoperationalcostsofthesoftware.Yes,thisisgoingtoa
greatproduct!hesaystohimself.
AsthedevelopmentteamisworkingonIncrement4,theproductmanagerdiscussestheAcceptance
TestPhaseoftheproject.Wehadoriginallyplanned3fulltest&fixcycleseachof2weeksdurationwith
aweekforfixesinbetweenforatotalof8weeksoftesting.recountstheTestManager.Butbasedon
theresultsoftestingIncrements1,2and3Impredictingthatwellonlyneedonefulltestcycleof2
weeksplusa1weekminicycleofregressiontestingforanyfixesthatneedtobedone(andImnot
expectingmany.)Theautomatedregressiontestingthedevteamisdoingaspartofreadiness
assessmenthasbeenpreventingtheintroductionofmanyregressionbugsandtheautomatedstory
testswevebeencoauthoringwithyouandthedevteamhaspreventedmisunderstandingsofthe
requirements.ThisisthemostconfidentIveeverfeltaboutaproductatthispointintheproject!
Summary
Therearegoodwaystoeffectivelydefine,buildandacceptproductsandtherearewaysthatmayresult
indisappointment.Therestofthisguideisdedicatedtohelpingyouunderstandwhichiswhich.

PartIThinkingAboutAcceptance
Akeypartofunderstandinganysystem,whetherwearedesigningit,buildingit,testingit,acceptingor
usingitistobuildmentalmodelsabouthowweexpectittowork.Systemsthatareeasytousemakeit
easyforuserstodevelopsimplementalmodelsoratleastfamiliarmentalmodelsthatallowthemto
reuseknowledgeabouthowothersystemswork.Thispartoftheguidedescribessixsimplemodelsthat
helpusreasonabouttheprocessofacceptingsoftware.
Theultimategoalofanydevelopmentprocessistobuildtherightproductofhighqualityandtobuildit
withminimumwaste.Theprimarydecisionswemustmakearetheworkflowandtheamountofwork
thatwillbemovedthroughtheworkflowatatime.Theworkflowconsistsofthesetofactivitiesthe
teammustpursueandtheorderingofthoseactivities,thatis,whethertheactivitieswillbecarried
sequentiallyorasneeded.Ourorganizationschoicesonthesetwocriticaldimensionsaffecthowwewill
goaboutmakingtheacceptancedecisions.Inaddition,howaprojectchoosestomeasuretherightness
andthequalityoftheproductwillmakeadifferenceindecidinghowtheproductdevelopmentteam
andtheproductownerwillknowifitsdoneandreadytodeploy.Themodelshelpaddresstheimpact
ofthechoicesthatthestakeholdersmakeonthesedimensions.
Chapter1TheAcceptanceProcessintroducestheoverallacceptanceprocess.Itdescribesthedecisions
thatmustbemadeandhowacceptancerelatestotheoverallsoftwaredevelopmentlifecycle,the
processeswithstagesandgates,andtheclassicVmodelofsoftwaredevelopment.
Chapter2TheDecisionMakingModeldescribestheprocessformakingthedecisionsintroducedin
thepreviouschapter.Italsoprovidesguidanceonwho(whichroles)shouldmakethedecisionsandwho
shouldcollecttheinformationonwhichthedecisionsarebased.
Chapter3TheProjectContextModeldescribesthevariousprojectfactorsthatmightinfluencethe
acceptanceprocess.
Chapter4TheSystemRequirementsModeldescribesvariouswaysrequirementscanbepartitioned
anddescribedandhowthatrelatestotheacceptanceprocess.
Chapter5TheRiskModeldescribesasimplewayofthinkingaboutpotentiallyundesirableeventsand
howtheymightinfluencetheacceptanceprocess.
Chapter6TheDonenessModeldescribeswaystodescribetowhatdegreetheproductisfinished.It
givesuswaystoaskthequestionwhatdoesdonemean?asawaytodefineouracceptancecriteriafor
individualfeaturesandforentirereleasesofproducts.Italsointroducestheconceptofincremental
acceptance.

Chapter1 TheAcceptanceProcess
ThischapterdefinestheprocessbywhichsoftwareisdeemedacceptablebytheProductOwner.It
introducesthethreemajorconstituencieswhomustmakedecisionsatkeypointsintheprocessandhow
thesedecisionsrelateto,andinfluence,eachother.WestartbydrillingintotheAcceptphaseofthe
softwaredevelopmentlifecycletoexaminethekeydecisionsandhowreleasecandidatesflowthrough
theprocess.Thenweexaminehowmorecomplexscenariossuchasmultiplereleases,complex
organizationsandcomplexproductsinfluencetheprocess.Sidebarsexaminehowthedecisionprocess
relatestoStageGateprocessesandQualityGates.Wefinishthischapterwithtechniquesfor
streamliningtheacceptanceprocess.Subsequentchaptersdescribetherolesandresponsibilitiesof
variouspartiesinvolvedinmakingthedecisionsandhowthisprocessportsfromsequentialtohighly
iterativeandincrementalagileprojects.
1.1 AcceptanceasPartoftheProductDevelopmentLifecycle
Softwareintensivesystemsgothroughanumberofstagesontheirjourneyfromconcept,ahalfformed
ideainsomeonesmind,toprovidingvaluetoitsusers.ThisprocessisillustratedinFigure1ASequential
ProductDevelopmentLifecyclewiththestagesplacedintoswimlanes[OMGOnSwimlanes]basedon
whichpartyhasprimaryresponsibilityforthestage.

Fiigure1A
ASequentialP
ProductDevelo
opmentLifecyycle
Acommonscenarioisforabusinessspersontoh haveanideafforhowtosolveabusinessproblemwith
software;thisistheco
oncept.Thebusinessperso on(hereincallledtheProdu uctOwner)ellaborateonthhe
concepttoproduceasspecificationoftheproducctthatonceb builtwilldelivvervalue.TheeProduct
Developm mentTeambu uildssoftwareetosatisfytheespecificationeventhougghfocussingo onthe
specificationinsteadofcustomersaatisfactionmaayoftenleadtothewrong gproductbuiiltphenomen non.
TheProduuctOwnerthe enassesseshhowwellitsatisfiestheneedsoftheinttendedusersandifsatisfieed,
acceptsth
hesoftware.W Whenthesofftwareisdeemedacceptable,thesoftw wareismadeavailabletotthe
userswhoothenusethe esoftwareto
orealizetheinntendedvaluethatsolvestheoriginallyyidentified
businesspproblem.The eProductDevvelopmentTeeamisusuallyyobligatedtoprovidesupp portfora
warrantyperiod.
ductdevelopm
Agileprod mentprocesssblurrthelinesbetweenthetimingand dresponsibiliitiesofthe
Elaboratee,BuildandAccceptactivitieesoftheprod
ductdevelopmentprocesssasillustrated
dinFigure1B
B
AnAgileP
ProductDevelopmentLifeccycle.
AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 28

Fiigure1B
AnAgileProdu
uctDevelopmentLifecycle
Agileprodductdevelopm mentischaraacterizedbyfaacetofacecollaborationbetweentheProductOwn ner
andtheProductDevelopmentTeam m.Whileeach hhastheirowwnresponsibiilities,theydo onthandofff
artifactsb
butrathercollaborateonp producingtheem.Thedevelopmentteam mismorelikeelytobedirecctly
involvedintheelaboraationofthereequirementsandproductdesignandth heproductow wnerisavailaable
toassesstheproductaasitisbeingb
builtratherth
hanwaitingfo orafinalacceeptancephaseoftheprojeect.
Elaboratioon,BuildingaandAcceptanceisdoneinccrementallyin nsmallchunkksratherthan ninasinglep
pass.
Asaresult,testingtheproductforbbothacceptanceandqualityassuranceestartsearlyiintheprocesss,
therebyspreadingthetestingactiviitythroughou utthedevelop pmentlifecyccleratherpusshingittotheevery
end.Increementaldeve elopmentand dincremental,earlytestinggallowalotm moreopportu unityforlearnning
aboutwhatistrulynee ededandtrullypossibleanndtypicallyreesultsinprodu uctswithbetterfitnessfor
purpose.Seethesideb barUsingWhholeTeamsto oBuildProductsformoreo onthemotivaationbehindthis.
Thereareeseveraldiffe
erentwaystodescribetheroleoftestin
ngandaccepttanceinthed
development
lifecycleo
ofasoftwareintensivepro
oduct.Someo
ofthewellknnownmodelsare:
TheSttageGateTMProcess.
TheV
VModel.
didate1 29

Thefirstisaproductdevelopmentprocessthatisnotspecifictosoftwareintensivesystems.Itdescribes
howtransitionsbetweenthedifferentstagesoftheproductdevelopmentlifecyclecanbemanaged.The
latterisaclassicalmodelspecifictosoftwaredevelopment.Itemphasizesequallyboththefrontend
andbackendoftheproductdevelopmentlifecyclebyexplainingtherelationshipsbetweendifferent
kindsofstagesandartefactscommonlyassociatedwithfrontendofthelifecycleanddifferenttypesof
testingandvalidationactivitiesthattraditionallytakeplaceatitsbackend.
Sidebar:UsingWholeTeamstoBuildProducts
Oneofthekeyformsofwasteinsoftwaredevelopmentishandoffsbetweenhighlyspecializedroles.A
keypracticeinagileandleanproductdevelopmentistheeliminationofhandoffswheneverpossible
throughtheuseoftheWholeTeamapproach.TheWholeTeamapproachbringseveryskillneededto
developtheproductontoasingleProductDevelopmentTeamwhichworkswiththeProductOwnerto
designthemostsuitableproductbasedontheneedsandconstraints.Theteamcollectivelycommits
todeliveringincrementsofpotentiallyshippableproductfunctionality.Thiseliminatesdependencies
betweenteamsandallowstheProductOwnertoworkdirectlywiththeteamtodesignthebest
possiblesolutiontothebusinessneeds.
Asanexample,theScrummethodadvocatestheuseoftheWholeTeamapproach.Scrumproponents
claimtypicalimprovementinproductivityof5xto10x(500%to1000%)[PrimaveraOnScrum,
KnibergOnScrumFromTrenches].Includingeveryoneneededtodelivertheproductonasingleteam
allowstheteamtocontinuallyimproveitsprocessbyeliminatingwaste.Organizationalboundariesdo
notgetinthewayofthisprocessstreamliningbecauseeveryoneisonthesameteamstrivingtoreach
thesamegoal:deliveringvaluableproducttotheProductOwner.
ThereisanongoingdebateintheagilecommunitywhethertheProductOwnerispartoftheteamor
separatefromit.Thereisnosingleanswertothisquestionasitdependsonthenatureofthe
organization(s)andpeopleinvolved.Thepragmaticansweristhatthereareoftentwolevelsofteam
atplay:
1. TheWholeTeamincludingtheproductownerandanyonehelpingthemdefine,buildandverify
theproduct.
2. TheProductDevelopment(sub)TeamwhichbuildstheproductandtheProductOwner(sub)Team
whichacceptstheproduct.Thesetwosubteamsmustworkcloselytogetherandoftensitinthe
sameteamroom.
<maybeagraphicshowingthetwosubteamsandtheskillswhichmightliveineachandwherethey
overlap>
Theexactbreakdownofwhichskillsareoneachsubteamvariesfromcasetocase.Developersusually
belongtotheProductDevelopmentTeamandanalystsusuallybelongtotheProductOwnerTeam.
Testers,documentationwriters,interactiondesignsmayallbelongtoeithersubteam.Theimportant
thingisthatalltheskillsarepresentandaccountableforworkingtogethertodeliverthebestpossible
productwithfewifanyexternaldependencies.

1.1.1 Processesw
P withStage
esandGattes
Manyprooductdevelop pmentprojecttsgothrough htheirlifecycllebyadoptinganessentiaallysequential
processto
oevolveaniddeaintoadep ployedproduct.Ifthepro ocesssdistincctstagesaregguardedby
decisionsthatcontrolprogressionffromonestaggetoanotherr,theprocessscanberepreesentedasa
streamoffalternatingsstagesandgates,aspropo osedintheStaageGateProcess
[CooperOOnDoingItRigh ht].Thegatesareassociattedwithgo/n nogoorscoppechangedeccisionswith
associateddresourcecoommitmentim mplications.Inastrictlyseequentialpro
ocess,thestaggesasmutually
exclusive::theprojectccanonlybein
nonestageattatimealtho oughwithinaastagemanykindsofactivvities
mightbehappeningattthesametim me.:Figure1cASequenttialProcesswwithStagesan ndGatesdepicts
suchapro
ocess.

Figure1c
withStagesan
ASequentialProcessw ndGates
Stagegateprocessesn neednotbesstrictlysequenntial[CooperOnStageGatee]inthesenseeofeverystaage
andeveryygatebeingddistinct.Anexxampleofaniterativeproccesswithstaggesandgatessisthe
IncrementalFundingM Method(IFM)[DenneHuan ngOnIFM]whosestagesan ndgatesinvollvethesame
activitiesanddecisionssthatarerep
peated.Inconntrasttothestrictlysequeentialdepictio
oninFigure11c
andsimilaartotheIFM,,theacceptannceprocesstthatwedescrribeinthisguidehasthesaameactivitiessthat
areexecuutedacrossstagesanddeccisionsthatarremademanyytimesovertthelifetimeo ofaproject.FFor
example,theproductd developmenttteammaybuildmanyreleasecandidaatesfortestin ngbutonlyaffew
maybeacccepted.Theprojectcould dpartiallybeintheDeployystage(apreeviousreleaseehavingpasseed
didate1 31

throughthegateaftertheAcceptstageGate5inFigure1c)eventhoughtheproductdevelopment
teamisworkingonprovidinganotherreleasecandidateforacceptancetesting.Theresultofsuch
dynamicsistheintroductionofmanystaggeredparallelprocessstreamswiththeirownstagesand
gates,interactionsbetweentheseparallelstreams,andloopswithinandacrossthestreams(seethe
SidebartitledRecastingaSequentialProcessesthroughWorkflowScheduleIndependenceand
Parallelization).Thedivergencefromthetraditionalsinglesequentialflowisinlinewiththemore
modernandflexibleinterpretationsandadaptationsadvocatedby[CooperOnStageGate].Howeverthe
acceptanceprocesswillfeatureintermediatestagesthatdirectlyfeedintosubsequentstageswithout
explicitgatesinbetween.Inaddition,thegates,ratherthanexplicitlycontrollingfundingdecisionsfor
subsequentstages,primarilydictatecontrolflow.Thesevariationsmaybeconsideredasdepartures
fromtheoriginalStageGateProcess[CooperOnDoingItRight].Thestagegatelikemodelunderlying
theacceptanceprocessisexpoundedinanothersidebartitledRepresentingtheAcceptanceProcess
withStagesandGatesinChapter2.
Sidebar:RecastingaSequentialProcessesthroughWorkflowScheduleIndependenceand
Parallelization
CoreyLadas[LadasOnScrumban]providestwopowerfulinsightsonsoftwareprocessesbasedon
Kanbansystems[HiranabeOnKanban]andleandevelopment[PoppendiecksOnLean]thatallowsusto
seetheminanewlight.Thefirstinsightexplainshowworkflow,theorderandinterdependenceof
stepsinherentinasoftwareprocess,canbeseparatedfromhowtheworkthatflowsthroughthe
processisscheduled.Thisiscalledworkflowscheduleindependence.Thesecondinsightallows
reorganizingasequentialworkflowasparallelstreamswithmergingpointswhereworkfromthe
streamscanbeintegrated.Whenbothideasarecombined,softwareprocesses,regardlessofwhether
theiressentialworkflowissequentialcanbemadeiterative,incremental,andparallel.Forany
process,theessentialworkflowisthesequencingofsteps,frombeginningtoend,appliedtoaworking
system,toimplementanewimprovement,whetheranewpieceoffunctionalityoranewnon
functionalrequirement.
Thiskeyideabehindworkflowscheduleindependenceisindependenceofrequirements.
Independenceofrequirementsinturnresultsfromhowworkisdividedupinsmallerchunksinthe
beginningofaworkflow,oftenbyanelaborationorrequirementsanalysisactivity.Ifthechunks,the
resultinglowlevelrequirementsthattheproductdevelopmentteamtransformsintoworking
functionality,aresmallandcanindividuallybecompletedallthewaytointegration,acceptance
testing,anddeployment,theyareindependent.Theextenttowhichthisconditionissatisfieddictates
whetherthechunkscanbescheduledtoflowthroughtheprocessindividually,ingroupsofsmaller
batches,orasasinglebigbatch.Themoreindependentthechunksare,thesmallerthebatchescan
get,downtotheleveloftheindividualrequirements.Iftheyformindependentgroupsof
interdependentchunks,thenthegroupscanbescheduledindependently,butnotthechunks
themselves.Regardlessofthegranularityofthebatchesthatflowthoughtheprocess,theunderlying
essentialworkflowstaysthesame,buttheworkflowexecutedagainforeachbatch.

Thesecondinsight,whichLadasreferstoaspipelining,leveragesanyindependenceofthestepsofthe
essentialworkflowitselfinsteadofthechunksofworkthatflowthroughit.Thestepsthatdontshare
resourcescanbeparallelized.Theresultofsuchparallelizationisworkflowwithstaggeredbranches
andminimalunusedcapacity.Parellelizationincreasestheefficiencyoftheprocessprovidedthatthe
outputsofparallelstreamscanbeintegratedinsuchawaythecapacitygainintroducedbythe
parallelizationexceedstheextraoverheadofintegration.
Thusworkflowscheduleindependenceandparallelizationtogethermayallowaseeminglystrictly
sequentialprocesstobeexecutedinahighlyiterative,incremental,andparallelmanner,makingit
moreefficient,flexibleandresponsivetochange.Theyapplytovariousdegreestomanysoftware
processesandlifecyclemodelswithsequentialdepictions.Inparticularworkflowschedule
independenceandparallelizationexplainhowthevariousdevelopmentprocessesontheprocess
continuumdiscussedinIntroductioncanbederivedfromtheessentiallysequentialworkflow
underlyingtheclassicalwaterfallmodel.
Ladassinsightsalsoapplytotwoadditionalrelatedprocessmodelsthatwediscussinthischapter.
BothofthesemodelsprocessesexpressedintermsofstagesandgatesandtheVModelhave
sequentialdepictionsnottoodissimilartothatoftheclassicalwaterfallprocessdiscussedin
Introduction.Andbothofthesemodels,liketheclassicalwaterfallprocess,areamenabletorecasting
inaniterative,incremental,andparallelmannerbyapplyingworkflowscheduleindependenceand
parallelization.Suchrecastingmayoccuratdifferentgranularities:atthelevelofwholereleases,
iterationswithinreleases,featuresets,orindividualfeatures.

1.1.2 TheVModelofSoftwareDevelopment
InthedevelopmentofsoftwareintensivesystemstheVModeliscommonlyusedtodescribethe
relationshipsbetweenactivitiesfocusingonbuildingthesystemandactivitiesforsystemverification].A
variantoftheVModeladaptedfrom[FewsterGrahamOnTestAutomation]isshowninFigure1dTheV
ModelofSoftwareDevelopment.

Analysis Business Tests Acceptance

Requirements Testing
Architecting Product Tests System
Requirements Testing
Design Components
Tests Component
Testing
Coding Units Tests Unit

Testing
Build Verify
Figure1dTheVModelofSoftwareDevelopment
InFigure1d,activitiesandartefactsthatareassociatedwithbuildingthesystemareshownontheleft
sideoftheVshape.Testingactivitiesareassociatedwithsystemverification,andtheseareshownon
therightsideofthefigure.Testsspecifictoalayertietogethertheactivitiesandartefactsofthatlayer
shownontherightsideofthelayertotheverificationrelatedactivitiesofthesamelayershownonthe
left.Testsforeachlayercanbedefinedearlyduringtheactivitiesthatareassociatedwithbuildingthe
system,buttheycanbeexecutedonlyafterdevelopmenthasprogressedsufficientlytotherightsideof
theVshapeintothecorrespondingverificationactivity.NotethatthisinterpretationoftheVModelcan
accommodatebothsequentialortestlaststyleverificationandincrementalortestasyougostyle
verification.TheprinciplesdescribedintheSidebartitledRecastingaSequentialProcessesthrough
WorkflowScheduleIndependenceandParallelizationapplytotheVModel.Thereforetheunderlying
workflowcanbeexecutediterativelyatdifferentlevelsofgranularity,onafeaturebyfeature,iteration
byiteration,orreleasebyreleasebasis.
Aftercodinghasbegunandtheunittestsfortheimplementedpiecesareinplace,unittestingmay
start.Asacomponentsunitsareimplemented,certainAPIsofthecomponentmaybecomefunctional.
IfthecorrespondingcomponenttestsforthoseAPIsareinplacethencomponenttestingforthatmay
begin.Continuingthisprogression,weclimbuptherightsideoftheVshape:theunitsarerolledinto
componentsandultimatelycomponentsarecomposedintoproductfeaturessuchthatthebusiness
requirementsarereadytobeexercised,thusreachingtheacceptancetestingactivityatthepinnacleof
therightside.Theacceptanceprocessdescribeswhathappensatthisfinalactivityofthetoplayer.We
makethedecisiontoacceptorrejectthesoftwarebasedonhowwellitmeetsthebusiness
requirementsshownonthetopleftsideoftheVModel.
Whenappliedinastrictlysequentialmanner,weassociatetheAcceptingTestingactivitywithadistinct
AcceptanceTestPhase.ContrastedwithpracticessuchasAcceptanceTestDrivenDevelopmentand
IncrementalAcceptanceTestingadvocatedinagilesoftwaredevelopment,theAcceptanceTestPhase
undertakentowardtheendofthedevelopmentlifecyclerepresentsthetestlastapproachto
acceptancetesting..Thespecificpracticescommonlyusedintheagilesoftwaredevelopmentcontext

aredescribedlaterinthischapterinsectionTheAcceptanceProcessforHighlyIncremental
Development.
1.2 PartiesoftheAcceptanceProcess
Theacceptanceprocessdescribestheinteractionsbetweentwokeyparties:theProductOwnerandthe
ProductDevelopmentTeam.Eachhasveryspecificresponsibilitiesintheproductdevelopmentprocess.
Whilethenamesofthesepartiesmaycoincidewithnamesusedinspecificmethods(suchasScrum),we
provideourowndefinitionsforthepurposesofthisbook.
1.2.1 ResponsibilitiesoftheProductOwner
TheProductOwneristhepartywhohascommissionedtheconstructionoftheproductbutdoesnthave
thetimeorskillstobuilditthemselves.TheProductOwnerisresponsibleforclearlycommunicating
theirexpectationsoftheproducttotheProductDevelopmentTeamandfordecidingwhetherthose
expectationshavebeensatisfiedbytheproductdeliveredtothem.Thedetailedresponsibilitiesofthe
ProductOwnerinclude:
Understandthepotentialusersoftheproductoridea.
Determinewhatcapabilitiestheproductneedstohavetoaddresstheusersneeds.
Determinetheamountofresourcestheyarepreparedtoinvestintobuildingtheproduct.
Specifythedesignoftheproduct(notthedesignofthesoftwareinsidetheproduct).
HelptheProductDevelopmentTeamunderstandthepotentialusersandtheirneedsandhowthe
productwilladdressthem.
Definetheacceptancecriteriathatmustbemetbeforetheproductcanbeaccepted.Theseinclude:
Examplesofhowtheproductwillbeusedbyusersandhowtheproductshouldreactin
eachexample.
Criteriaregardingnonfunctionalpropertiesoftheproductincludingresponsetime,
availability,accessability,configurability,etc.
Defineanyconstraintsontheproductincludingtechnicalconstrainsts(whattechnologycanormust
beused).
AssessthecostanddeliverydateestimatesprovidedbytheProductDevelopmentTeam.
Makethedecisionastowhethertheproductisworthbuilding.
TheProductOwnermaycarryouttheseresponsibilitiesthemselvesortheymaydelegatetomembersof
theirteam(knownastheProductOwnerTeam.)Theymayalsoenlistthehelpofpeopleoutsidethe
ProductOwnerTeamtohelpthemcarryouttheirresponsibilities.ThisincludesaskingtheProduct
DevelopmentTeamtodosomeofthework.Butneitherdelegationnorcollaborationabsolvesthe
ProductOwnerfromultimateresponsibilityforthem.

AgoodProductOwnerhasaclearvisionofwhatdonemeansandsharesthatvisionwiththeProduct
DevelopmentTeamasearlyaspossible.AgoodProductOwneralsorecognizesthatcommunicationis
inherentlyflawedandthatyouneedtoregularlyverifythatwhatyouintendedtocommunicatehas
beenunderstood.Therefore,agoodProductOwnerispreparedtotryoutandprovidefeedbackonthe
softwareunderdevelopmentatfrequentintervals;agoodProductOwnerdoesntsimplythrowa
specificationoverthewallandsaydontbothermeuntilyouarereadyforacceptancetesting.They
takeanactiveinterestintheemergingsoftwareandencouragetheProductDevelopmentTeamto
providefrequentopportunitiesforfeedback.
1.2.2 ResponsibilitiesoftheProductDevelopmentTeam
TheProductDevelopmentTeamisthepersonorpersonswhohaveacceptedthetaskofbuildingthe
productcommissionedbytheProductOwner.TheProductDevelopmentTeamisresponsiblefor
deliveringaworkingproductthatmeetstheProductOwnersacceptancecriteria.Thedetailed
responsibilitiesoftheProductDevelopmentTeaminclude:
Determinehowtherequestedfunctionalityshouldbeachievedincluding:
Hardwarevs.firmwarevs.softwarepartitioning.
Determinewhattechnologieswillbeusedtobuildtheproduct(unlessthesewerepartof
therequirementsorconstraintsprovidedbytheProductOwner)
BuildtheproductandverifythatitmeetstheProductOwnersexpectationstothebestoftheir
ability.Theproductincludes:
Workingsoftwareandhardware
Whateveruserdocumentationisrequired(unlessthisisprovidedbytheProductOwneras
partoftherequirements.)
Whateverartifactsarerequiredtomaintainthesoftwareincludingsoftwareandhardware
designdocumentation,unitandcomponenttests,testtoolsrequiredtotestthe
product,etc.
TheProductDevelopmentTeammaycarryouttheseresponsibilitiesthemselvesortheymayenlistthe
helpofpeopleoutsidetheProductDevelopmentTeamtohelpthemcarryouttheirresponsibilities.But
neitherdelegationnorcollaborationabsolvestheProductDevelopmentTeamfromultimate
responsibilityforthem.
1.2.3 ResponsibilitiesofOtherSpecialities
Thenatureoftheproductbeingbuiltoftendictatesthekindsofspecialiststhatneedtobeinvolvedin
theproject.ThesespecialistsmaybeengagedbyeithertheProductOwnerTeamortheProduct
DevelopmentTeam.Regardlessofwhichpartytheyworkwith,theyneedtounderstandtheirownrole
indeliveringaproductthatsatisfiesthepotentialusers.

1.3 TheBasicAcceptanceProcess
ThebasicacceptanceprocessatitshighestlevelsubsumestheBuildAcceptUsecycledepictedin
Figures1Aand1B.EachofthehighlevelstagesshowninFigure1Aand1Binvolvesmultiplesubstages
oractivitiesandanexitdecision.
TheBuildstageiscomposedoftheproductsconstruction,itsreadinessassessment,andasubsequent
readinessdecision,allperformedbytheproductdevelopmentteam.
TheAcceptstageiscomposedofacceptancetestingoverseenbytheproductowner(andsometimes
performedincollaborationwiththeproductdevelopmentteam),andasubsequentacceptancedecision
madebytheproductowner.Normally,thereadinessdecisionandtheacceptancedecisionaremadeon
thesamesetofcriteria.Theprecedingactivities,readinessassessmentandacceptancetesting,may
involveoverlappingactivities,althoughreadinessassessmentmaybemoreinternallyfocusedthan
acceptancetesting.Howeverthetwodecisionsdifferintheirgoals.Readinessdecisiondetermines
whethertheproductisreadyfortheproductownertoevaluateandtheacceptancedecisiondetermines
whethertheproductmeetsallitsrequirementsandisreadytobeused.Thelaststage,theUsestage,
mayinvolveafinal,userfacingevaluativeactivityineitheraproductionorenduserenvironmentto
determinewhetheritisusableanddeployabletoitsintendedaudience.
Figure2TheAcceptanceProcessdrillsdownintoeachoftheBuild,Accept,andUsestagestoexpress
themintermsoftheirlowerlevelcomponents..Thefigureshowsonlythepathassociatedwiththe
positiveoutcomesoftheunderlyingdecisions.

Figure2
TheAcceptanceProcess
Intheidealworldthissequenceofactivitieswouldbedoneexactlyonceandtheassessmentcouldbe
doneentirelybytheProductOwner;inpractice,thiscouldbearecipefordisasterasuntestedsoftware
cancontainlargenumbersofbugsperthousandlinesofcode.Itusuallytakesmanytriestobuilda
productthattheProductOwnerfindsacceptablethereforetheacceptanceprocessistraversed,atleast
partially,manytimes.Toensurethataqualityproducthasbeenbuiltandtherebyminimizethenumber
oftimestheProductOwnerisaskedtoacceptthesameproduct,mostProductDevelopmentTeamswill
includesomelevelofselfassessmentofeachreleasecandidatebeforeprovidingthesoftwaretothe
ProductOwnerformakingtheacceptancedecision.Inthisbookwecallthetestingandother
verificationactivitiesthatareperformedpriortoaskingtheProductOwnertoacceptthesoftware
readinessassessmentandthedecisiontohandoffthesoftwareiscalledthereadinessdecision.The
testingdonebytheProductOwnerortheirproxyafterreceivingthesoftwarefromtheProduct
DevelopmentTeamiscalledacceptancetestingandthedecisionwhethertoacceptthesoftwareinits
currentstateiscalledtheacceptancedecision.Thedecisioneachusermakeswhetherornottoactually
usetheproductonceitisavailabletothemiscalledtheusagedecision.
Eachdecisioncanresultinapositiveornegativeoutcome.Onlythepositiveoutcomesareshownin
Figure2;thenegativeoutcomesareshowninFigure3PathsthroughtheAcceptanceProcess.
ThetransitionsfromtheBuildstagetotheAcceptstageandfromtheAcceptstagetotheUsestage
typicallyrequirethemovementofsoftware(orsoftwarecontainingproducts)fromoneenvironmentto
another.Forthepurposesofthisdiscussion,thespecificstepsinvolvedinmakingthesoftwareavailable,
thoughimportantforacceptance,arenotconsideredapartoftheacceptanceprocess.
1.3.1 AssessingReleaseCandidatesusingtheAcceptanceProcess
Theversionofthesoftwarethatisputthroughtheacceptanceprocessisoftenreferredtoasarelease
candidate.Itgoesthroughthereadinessdecisionandacceptancedecisionprocessesstepbystepand
decisionbydecisionuntilitmeetsoneofthefollowingrequirements:
Itpassesthroughallthedecisionsandisdeemedreadyforusebyusers.
Itisdeemedinsufficientinsomeway;atwhichpoint,itissentbacktoanearlierphase.

Anegativeacceptancedecisionorreadinessdecisioncancausethereleasecandidatesoftwaretobe
sentbacktoanearlierpointintheprocess.
Figure3PathsthroughtheAcceptanceProcessillustratesthepossiblepathsthroughtheacceptance
processwhenthereadinessoracceptancedecisionscannotbemadeandrequireadditionalcapabilities
orinformation.Inthefigure,QualityDatareferstotheinformationaboutthequalityofthesystem
obtainedfromreadinessassessmentoracceptancetesting.

Fiigure3
PaathsthroughtheAcceptan
nceProcess
Fromthereadinessdecision,areleaasecandidateecanbesentbacktoread dinessassessm mentifthequuality
dataobtainedfromthe ereadinessassessmentisinsufficienttomakeawellinformedreeadinessdeciision.
Ifthedataafromtheaccceptancetestingphaseissufficienttodetermineth hatcriticalfun
nctionalityis
missingortheproductthassevered deficiencies,tthereleasecaandidatecanbesentbacktodevelopment
forrework.
Fromtheacceptanced decision,arelleasecandidaatecanbesenntbacktoaccceptancetesttingifthequaality
macceptanceisinsufficientt.Iftheaccep
datafrom ptancedecisio
onalsodepen ndsondatacollecteddurin ng
readinessassessmentanditisdeem medtobeinsufficient,thereleasecand didatecanbesentbackto
readinessassessment.Ifcriticalfun missingorhasssufficientlyseveredeficieencies,thereelease
nctionalityism
candidateecanbesentbacktodevelopmentforrrework.
Normalpracticeistolooganybugsffoundinthebbugmanagem mentsystemssothatthePrroduct
Developm mentTeamcanstarttheprrocessofrem mediationbuttocontinuettestinguntilaallthetestshave
beenrunortestingisbblockedbyabbugthatprevventsfurthertestingfromoccurring.Attthispointteesting
stopsuntiilanewreleaasecandidateeisreceived.A ormorebugss,anewrelease
Afterremediaationofoneo
candidateeisbuiltandp
passedthrougghtheaccepttanceprocesss.Eachreleassecandidatesshouldbea
distinctlynamedversio onofthesofttware(andshhouldbetaggedassuchinthesourceco odemanagem ment
system.)
Whenalaargenumberofbugsisfouundduringtheacceptanceeprocess,theeProductOwnermayneed dto
decidewh whichrequirefurtherinvesstigationandwhichcanbeedeferredtoa
hichbugsmustbefixed,w
futurerelease,aproceessknownasBugTriage.B Bugtriaginginvolvesassiggningapriorittyandseverittyto
eachbug..Whilethisisacommonppractice,itisttypicallyasym
mptomofdeeeperissuesinnthe
organizationalcultureandstructureeoftheenterrprise.Ratherrthanfocusin ngonbecomingbetteratffixing
thebugs,mostorganizzationswoulddbebetterseervedbyundeerstandingtherootcausessoftheirbug
backlogandchangingh howtheybuildtheirprodu uctstoreducethenumberofbugsfoun nd.Thesectiion
TroubleshootingtheA AcceptanceProcessinChaapter20FineTuningtheAcceptanceProcessofferrs
didate1 39

somepossiblerootcausesofalargebugbacklogandpossibleavoidancestrategies.RefertotheSidebar
IncrementalAcceptanceandBugTrackingonAgileProjectsinChapter8.
Note:Althoughtheacceptanceprocessoutlinedhereappearstobesequentialwiththesoftware
movingfromonestagetoanother,thisisnotalwaysthecase.Onwellrunprojects,followingthe
practiceofincrementalacceptance,eachfeaturemaytraversethisprocessindividually.Therefore,itis
possibleforonefeaturetobeintheconceptstage,withthedevelopmentteamworkingonanother
setoffeaturesinthedevelopmentstage,andanothersetoffeatureswiththeProductOwnerinthe
acceptancestage.Thatisatleastthreeinstancesoftheprocessrunninginparallel.
Evenonclassicalwaterfallprojectswemayhavemorethanonereleasecandidategoingthroughthe
processatthesametime.Forexample,theAlphareleasemaybeInUse,theBetaisinAcceptance
Testinganddevelopersmaybeworkingonadditionalfunctionalityforthegeneralreleasewhilealso
fixingbugsinthenextreleasecandidateforthisrelease.Thatisfourinstancesoftheprocessrunning
inparallel.Suchparallelismhoweverisnotwithoutimplicationsduetoincreasedintegrationand
coordinationoverhead.Inparticular,sourcecodemanagementbecomescomplexduetobranching
andsubsequentmergingofsourcecodetrunks(forprovenbranchingstrategiesinTFS,see
[MSOnTFSBranchingGuide]).Bugtrackingbecomescomplexduetotheneedtocoordinatetheissues
fromthereleasesunderacceptancetestingwiththefixesandadditionsinthereleasesunder
development.
Amorecompletediscussionofoverlappingreleasesfollowslaterinthischapterundertheheadings
TheAcceptanceProcessforAlpha&BetaReleases.Forthegeneralprinciplesgoverningincrementality
andparallelism,refertotheSidebarRecastingaSequentialProcessesthroughWorkflowSchedule
IndependenceandParallelization.
WhySeparateReadinessAssessmentfromAcceptanceTesting?
Theprocessofverifyingandvalidatingthesoftwareintensivesystemagainsttherequirementsand
expectedusagecanbeviewedasasinglemonolithicactivityorasveryfinegrainedsteps.Theprimary
reasonforgroupingthesefinegrainedstepsintotwomajorbucketsistoseparatetheactivitiesthat
shouldbecarriedoutbytheProductDevelopmentTeambeforeturningthesystemovertotheProduct
OwnerfromthosethattheProductOwnerdoesaspartofdecidingwhethertoacceptthesystem.
ReadinessassessmentisprimarilyabouttheprofessionalismoftheProductDevelopmentTeamwhile
acceptancetestingisaboutvalidatingthatthesystemasdeliveredwillsuitthepurposesoftheusers
andotherstakeholdersandconfirmingcompliancetocontractualagreements.
Theremaybeotherreasonstodividetheactivitiesintovariouscategories.Wementionthemonly
brieflybecausetheyarebeyondthescopeofthisguide:
DeadlinePressureSomemanagersmightbelievethathavinganearlier,separatemilestonebefore
handingoversoftwaretotheproductownerisaneffectivewaytomotivatedevelopers.
AccountingTheinitialconstructionofthesoftwaremaybetreateddifferentlythanfixingofbugs
fromanaccountingperspective(especiallyworkthatmaybeconsideredasachangerequest
mayaffectprojectaccountingdifferentlythanworkthatmaybeconsideredasabugfix).

EarlyValidationItisusefultohaverealuserstryusingearlyversionsoftheproducttovalidate
thattheproduct,oncefinished,willfillthenicheforwhichitwastargeted.Thisclearlyisntfull
onacceptanceoftheproductsoitcouldbeconsideredreadinessassessment.Priortodoingthis
typeoftestingwithusers,theProductDevelopmentTeamwouldwanttodotheirowndue
diligencetoensurethesoftwarewasworkingwellenoughtoallowtheuserstotryit.Therefore,
wewouldconsiderthisaformofAlpha/Betareleaseorpossiblyincrementalacceptancetesting
orevenconditionalacceptance.
EndUserTrainingExposingthesoftwaretousersbeforeacceptancecanbeaneffectivewayto
startthetrainingprocess.Ifthequalityofthesystemishighenough,itcanalsobeusefulasa
formofviralmarketing.BothoftheseusesfallintothecategoryofAlpha/Betareleasesrather
thanreadinessassessment.
ReadinessAssessmentorAcceptanceTesting?
Givenaparticulartestingactivity,shoulditbeconsideredpartofreadinessassessmentoracceptance
testing?Thereareseveralfactorsthatplayintothisdecision:
1. Whosdoingtheassessing:AcceptanceisperformedbytheProductOwnerortheirproxy
(someoneactingdirectlyontheProductOwnersbehalf)whilereadinessassessmentcanbe
donebyanyoneinvolvedintheproject.WheretheProductDevelopmentTeamandProduct
Owneraredistinctorganizationalentities,therolesshouldbefairlyclear.Thingsgetmurkier
whenthereisnorealuserorendcustomerinvolved,commoninproductcompanies,orwhen
thereisaseparateorganizationchargedwithtesting.Thesescenarioswillbecoveredinmore
detailinChapter2TheDecisionMakingModel.
2. Formalityofthetesting:AcceptanceisamoreformalactivitybecausetheProductOwneris
involved.Thisimpliesmoreformalrecordkeepingofanyconcernsthatwereidentified.
Readinessassessmentcanbemuchlessformal;itneedonlybeasformalasdictatedbythe
projectcontext.ProductDevelopmentTeamsthatneedtobeabletopassformalauditswill
keepmuchmoreformalrecords.Agileprojectstendtobeveryinformal;duringreadiness
assessmenttheyjustfixthebugsimmediatelyratherthandeferthemforfixinglater.

Thelinescangetsomewhatblurredwhentherearemorethantwodifferentstagesoftesting;these
scenariosaredescribedinAcceptanceinComplexOrganizationsandAcceptingComplexProductsin
Chapter2ElaboratingontheAcceptanceProcess.Intheend,itislessimportanttodecidewhichlabel
toapplytoaparticulartestingactivitythanitistoensurethattherighttestingactivitiesgetdoneand
thateachpartyknowsforwhichactivitiestheyareresponsible.Itcanbeusefultolistallthepotential
testingactivitiesandforeachonedecidewhetheritismandatory,optional,ornotrequired,andto
assignresponsibilityforittoaspecificpersonororganization.Asamplespreadsheetisprovided<online
athttp://testingguidance.codeplex.com>.

1.3.2 TheAccept
T tanceProccessforMu
ultiRelea
aseProduccts
Thusfar,w
wevefocusedontheacceeptanceproceessasitappliiestoasingleereleasecand
didate.Howd
does
theaccep
ptanceprocesssgetappliedwhenaprod
ductwillhaveemultiplereleeases?
mostpart,longglivedmultireleasesystemscanbethoughtofassimplyasequeenceofindiviidual
Forthem
products,whereeachproductisbeeingindividuallyassessedfforreadinessandacceptan
nce.Eachreleease
goesthroughtheentirredecisionmakingprocesss.Figure4TTheAcceptancceProcesswiithMultiple
Releasesiillustratesanexampleofthisprocess.
Figure4
TheAccep ptanceProcessswithMultip
pleReleases
mumMarketaableFunctionality
Thesetofffunctionalityyrequiredforreachreleasee,whichweccalltheMinim
1
orMMF ,isuniquebassedonthego oalsofthereleasedeterminedbythep productowneer.Thesetof
qualitycriiteria,whichw
wecalltheM
MinimumQualityRequirem mentorMQR,,issomewhattmoreconsisstent
fromreleaasetoreleaseebutitmayeevolveastheproductmaturesandprod ductcontextevolves.Thee
1
Theminimumleveloffunctionalityisalsosomeetimescalled dtheMinimum mMarketableeFeatures.TThis
termwascoindedbyM MarkDenneaandJaneClelaandHuanginthecontexto oftheIncrem
mentalFundin ng
Method,aanROIorienttedapproachtoscopingso oftwarereleaases[DenneHuangOnSoftw wareByNumbers],
toreferto
ounitsoffunctionalitythaatcanbedeployedtogetheerandwillstaartgeneratingvtangiblevvalue
afterdeplloyment.Weadoptavariaationofthisterm,Minimu umMarketablleFunctionaliity(MMF)incase
thedeliveeredunitisussable,butdoeesnotcoveraawholefeatuurebyourdeffinition.MMFFisalsoknow
wnin
theindusttryasMMP(MinimumMaarketablePro oduct)andMC CR(Minimum mCredibleRellease).
didate1 42

specificcriteriawouldbeselectedfromthesetofcriteriaineffectatthetimeoftheproject(whichmay
varyfromthosethatwereineffectforearlierreleases.)AnexampleofthisisthattheSarbanesOxley
Act(SOX)wasenactedin2002,soallsubsequentreleasesrequiredcompliancewiththisactasa
readinessand/oracceptancecriteria.Subsequentreleasesmayalsohavebackwardcompatibility
requirementsthatdidnotexistforearlierreleases.Ideally,theMMFandMQRusedbytheProduct
DevelopmentTeaminmakingtheReadinessDecisionshouldbethesameastheMMFandMQRusedby
theProductOwnertomaketheAcceptanceDecision.Inpractice,thisishardtoguaranteewithout
extensivecollaborationbetweentheProductDevelopmentTeamandtheProductOwnerTeam.
1.3.3 TheAcceptanceProcessforAlpha&BetaReleases
Alphaandbetareleasesarewaystouseendusersasfieldtestersbeforeaproductionqualitygeneral
releasetogathermoredataabouttheproductasitmightbeused"intherealworld."Theendusers
maybeinternaltotheorganization,orexternalbutfriendlier(meaningmorebugtolerant)orhavinga
stakeintheoutcome(wantingtoseetheproductearlierfortheirownbenefit.)Insomeorganizations
internalalphatestingiscalleddogfoodingsonamedbecausewhensomeoneproducesdogfood,they
shouldbepreparedtofeedittotheirowndog.
ThefinaloutcomeofalphaandbetatestingisaslikelytoresultinchangestotheMMFandMQRofthe
product(ineffect,newfunctionalityorqualityrequirements)asitistobeacollectionofbugreports
thatdescribefailuretomeettheexistingMMForMQR.
Eachalphareleaseandbetareleasecanbeconsideredaseparatereleasewithitsownreleasedecision
andacceptancedecision.Thatis,thedevelopmentorganizationneedstodeemittobereadyforan
alpha(orbeta)releaseandtheproductownerneedstoacceptitasbeingreadyforalphareleaseasin:
(Iacceptthisalphareleaseashavingsufficientfunctionalityandqualitytowarrantreleasingtousersto
collectfeedbackThisisillustratedinFigure5TheAcceptanceProcesswithAlphaandBeta
Releases.

Figure5
TheAccep ptanceProcessswithAlphaandBetaRelleases
Notethatteveryrelease,whetherallpha,betaorfinalhasareeadinessdecissionandanacceptance
MQR)foranalphareleasearetypicallyllowerthanth
decision.TThefunctionaality(MMF)aandquality(M hat
neededfoorabetarelease,whichislowerthann neededforaggeneralreleasse.Forexamp ple,theMMFFfor
theAlphareleasemaybeacoresub bsetoffunctiionality;notaallfeaturesneeedbepresent.TheMQRmay
be"nosevverity1bugs"and"respon ndsinunder2 2seconds95%ofthetimee(versusthe1second
responsetimerequired dinproductioon)."Typically,theMQRccriteriafortheeBetareleaseewillbebaseedon
theAlphacriteriawithadditionalcrriteriabasedoonimprovem mentspreviouslyplannedaaswellasfeed dback
fromtheAAlphatestinggasshowninFigure6Alp phaorBetaFFeedback.
didate1 44

Figure6
Alpha(orBeta)Feedback
Similarly,theMMFandMQRforthegeneralreleasewouldbeaffectedbyfeedbackfromusersofthe
Betarelease.
1.3.4 Soaking,FieldTrialsandPilots
Newproductsareoftentestedwithpilotgroupsbeforebeingrolledouttolargergroupsofusers.A
pilotistypicallythefirstproductionqualityreleasewithfunctionallypickedtosatisfyaparticular
targetaudience.Therefore,theMMFbarislowerbuttheMQRneedstobesufficienttosatisfythe
users.UnlikeanAlphaorBetarelease,thisisrealproductionsoftware.Apilotusergroupwilloften
receiveseveraldotreleasesbasedonissuestheyfindandreport.Theinitialpilotmaybefollowedbya
largerpilotgroup,whomayrequireadditionalfeatures,ortheproductmaygostraighttogeneral
availability.Eachofthepilotreleaseswouldbesubjecttotheacceptanceprocesswithbothreadiness
andacceptancetestingprecedingdeployment.
Asimilarstrategyisknownasfieldtrialorsoaking(thinkofthewashingmetaphor:justlikeclothes
youletthesoftwaresoakforawhile,tomakesureitcomesoutclean).Thesoftwareisdeployedto
friendlycustomers/usersforanextendedperiod(morethanjustatestcycle)toseehowitbehavesina
realcustomer/userenvironment.Aswithapilot,readinessassessmentandacceptancetestingwouldbe

donetoensurethattheMMFandMQRaresatisfied.Theoutcomeinallcasesistogatheruserfeedback
thatmaycausetheproductownertoadjusttheMMFandMQRexpectationsofsubsequentreleases,
nottorevisittheacceptancedecisionofthepilotorfieldtrialrelease.
1.3.5 SoftwareMaintenance
Anytimesoftwareneedstobemaintained(suchaswhensmallchangesaremadetothesoftwareand
thosechangesaredeployed),youare,ineffect,creatingaminorinterim,ordotreleaseofthesoftware
thatneedstogothroughtheentiredecisionmakingcycleyetagain.Itiscommontolookforwaysto
reducethecostofgatheringthedatatosupporttheacceptancedecision.Somewaysofdoingthis
increasetheriskofpossiblymissingnewlycreatedbugs(alsoknownas"regressionbugs")byreducing
theamountoftesting(forexample,riskbasedtestplanning)whileotherssimplyreducetheeffortto
getsimilartestcoverage(forexample,automatedregressiontesting).
Anotheruniqueaspectofsoftwaremaintenancerelatestothewarrantyperiodonasoftwarerelease.
Anychangesthatneedtobemadetothesoftwareshouldbemadeinthesourcecodemanagement
(SCM)system.Whenbuildingmultiplereleases,theremaybeongoingdevelopmentforthenextrelease
thatshouldnot,underanycircumstances,beinsertedintotheproductionsystemalongwiththe
warrantybugfixes.Thisrequiresmanagingseparatecodestreamsorbranchesduringthewarranty
periodandensuringthatallwarrantyfixesarealsoappliedtothenewdevelopmentcodestream.SCM
isalsoneededduringacceptancetestingifthedevelopmentforthenextreleaseismovingforwardin
parallel.Forpracticalstrategiesforusingsourcecodemanagementsystems,see[BerczukOnSCM]and
[MSOnSCM].
1.3.6 ConditionalAcceptance/Readiness
Frequently,theacceptancedecisionmakeracceptsaproductwithconditions.Acceptingaproductwith
conditionsisashorthandwayofsaying,
"Theproductisnotacceptableyet,butitisclosetomeetingourcriteriaforfunctionality(MMF)and
quality(MQR).Ifyouaddressthefollowingconcerns(andwefindnothingnewinthesubsequentround
ofacceptancetesting),weintendtoaccepttheproductinthenextpassthroughthedecisionmaking
process."
Conditionalacceptancebringstheprocessbacktotheconstruction/developmentphaseofthe
acceptanceprocess,butthistimewithamuchbetterideaofexactlywhatmustbedonetomakeit
throughboththereadinessdecisionandtheacceptancedecisiononthenextround.
1.3.7 TheAcceptanceProcessforHighlyIncrementalDevelopment
Theacceptanceprocessasdescribedthusfarlooksverysequentialinnaturebutitcanalsobeusedina
highlyincrementalfashiononagileprojectsbyapplyingworkflowscheduleindependence(seeSidebar
RecastingaSequentialProcessesthroughWorkflowScheduleIndependenceandParallelization).Each
chunkoffunctionality(oftencalledafeatureorauserstory)canbepassedthroughtheacceptance
processindependentlyasshowninFigure7TheAcceptanceProcessAppliedtoIncremental
Development.

Figure7
TheAcceptanceProcessAppliedtoIncrementalDevelopment
TheindividualMMFforeachfeatureoruserstorymaybe:
independentoftheMMFofotherfeaturesoruserstories;or
incrementalwhenitmaybuildontheMMFofpriorfeatures.
TheMQRmaybedifferentfordifferentincrements,althoughitsmoreconsistentacrossincrements.For
example,theMQRforanalphareleaseistypicallylowerthantheMQRforaregular,generalrelease.
WhenMQRrelatestononfunctionalrequirements,itappliesacrossfunctionalfeatures.
Atsomepoint,aproductlevelreadinessandacceptancedecisionsaremadetoensurethateverything
worksproperlytogether.RefertoAcceptingtheOutputofFeatureTeamsinChapter2Elaboratingon
theAcceptanceProcessforamorecompletediscussion.
Inincrementaldevelopment,thedevelopersmaypreexecutetheacceptancetestsWhenallthetests
passforthatfeatureoruserstory,theymayturnoverthefunctionalitytotheProductOwner(orother
customerproxysuchasabusinessanalystoracceptancetester)forimmediate"incrementalacceptance
testing."Therefore,acceptancetestingatthefeaturelevelmaystartimmediatelyaftertheproduct
developmentteamdecidesthatallormostofthefunctionalityforthatfeatureisbuiltandworking
correctly,allowingincrementstooverlap.Thisisaninstanceofincreasingefficiencybyintroducing

parallelism
mintothewoorkflowwhen nresourcesm maybeseparaate(seeSideb barRecastingaSequential
ProcessessthroughWo orkflowSched duleIndepend denceandParallelization)..Theremayaalsobearoundof
acceptanccetestingperrformedatth heendoftheiteration,asiillustratedinFigure4ofth
heIntroductio
onby
themediu umlengthverticalbarsrep presentingiteerationwidetesting.Theaactivitiescondductedforeaach
featurewwithintheiteraationareillusstratedinFigu
ure8.

Figure8
AgileDeveelopmentwitthIncrementa
alAcceptanceeTesting
Developmmentisdoneo onefeatureaatatimebyeitherasingledeveloperorrasmallteam mofdevelopeers,
documenttationwriterss,testersand duserexperieencepeople,ddependingon nthesizeoftthefeature.((See
thesidebaarWholeTeaamformoreinformation.)InFigure8 AgileDevelopmentwithIncremental
AcceptancceTesting,Deev1andDev2couldeach hbeasingled
developeroraasmallteam..Whenthe
developmmentworkisccomplete,theedeveloper,p possiblyassisttedbyatesteerorbusinesssanalystcond ducts
readinessassessmentonthefeaturrebyrunningalltheknownunit,compo onentandbu usinessaccepttance
ware.Iftheyfiindanyproblems,theyfixthembeforeeproceeding.Thesetestsare
testsagainstthesoftw
oftenautoomatedandaactasaformofmistakeproofingorbu ugrepellent.TTheyareconssistentwith
ShingosaadageInspecctiontofinddefectsiswaste;inspectio
ontoprefentdefectsisesssential
[ShingoDillonOnToyotaaProductionSSystem].
Whentheeyaresatisfie
edthatthesoftwareisworrkingproperlyy,theyasktheProductOw wner(ora
membero oftheProducctOwnerTeam)toruntheeiracceptanceetestsforthaatonefeaturee.IftheProduct
Ownerfin
ndsanyproblems,theysho owtheprobleemstotheteeamwhothen nfixestheprooblems
immediattelyandrepeaatsthereadinnessassessmeentontherevisedcodebeeforegivingitttotheProdu uct
Ownerforfurtheracceeptancetestinng.WhentheeProductOwnerissatisfieed,theyaccep ptthefeatureeand
thedevelo
oper(orteamm)startsworkkingontheirnextfeature.Thispracticee,calledIncreemental
AcceptancceTesting,haastwokeybeenefits.First,aanyconcernffoundbytheProductOwn nerduring
acceptanccetestingcan
nbediscusseddwiththedeeveloperswhiletheystillreemembertheedetailsofho ow
theyimplementedthefunctionalityy.Second,theedefectsord deficienciescaanbeaddresssedimmediattely
beforetheedevelopermmovesontotthenextfeatu ureinsteadofbeingstockpiledfora"bugfixingphase."
1.3.8 TheRoleof
T fBugFixin
ngintheA
Acceptance
eProcess
Findingb
bugsduringaccceptancetesstingisnormaal.Howwereeacttothebu
ugstellsusallotaboutour
overallap
pproachtoproductdeveloopment.Weccanusebugsttomotivateim mprovementstoour
didate1 48

developmentprocessorwecanjustfixthebugswithoutunderstandingtherootcause.Thelatter
approachoftenresultsinalargenumberofbugsbeingfoundandhavingtobefixed.Wecaneitherfix
thesebugsrightawayorwecanstockpilethemforabugfixingphase.Suchastockpileofbugsis
consideredaformofwasteinleanmethodologiesbecausewearebuildingupaninventoryofpartially
finishedsoftware.AccordingtoTomandMaryPoppendieck,thoughtleadersoftheLeanSoftware
Developmentapproach,itisirresponsibletoorganizetheworkinsuchawaythatTriageisnecessary.
Theyarguethatitisprimarilythedecisiontotestlateratherthantomistakeproofeverystepthat
escalatesthecostofrepairtothepointthatonewoulddeliberatelychoosetoleaveknowndefectsin
theproduct.Thus,theneedforbugtriagecanbeconsideredasaprocesssmellThereare,however,
circumstanceswhenyourdevelopmentisorganizedinawaywhenabugbacklogand,therefore,triage
activitiesarenecessary.Thistendstohappenwithlargeand/ordistributedteamsorwhenproduct
ownerisnotreadilyavailable.Inthiscase,itisimperativetokeepinmindthefollowingpoints:
1) Triageshouldonlylookatnewbugsreexamingoldbugsrepeatedlyiswaste.Thepurposeof
triageshouldbetodecidewhethera)weneedtofixthebugforthesoftwaretobeconsidered
done,theissueisnotreallyabugandthereforerequiresnoaction,orb)thebugrelatesto
futurefunctionalityandcanbeaddressedwhenthatfunctionalityisbuilt.
2) Toavoidwaste,weneedtokeepthedepthofthebugbacklogreasonablysmall.Reasonably
smallisafunctionofourdevelopmentandbugfixingcapacity.Ifonalargeprojectour
developmentteamcanfix200bugsperiterationonaverage,thenthebacklogof200bugsisnot
alot.However,ifourdevelopmentcapacityissuchthatwecanonlyfix10bugsperiteration,
clearlyabacklogofof100shouldbeconsideredasaredflag.
3) Thekeygoalistomakethebugfixlatency(measuredasaMeanTimeToFix)toaminimumby
performingregulartriagesandschedulingbugfixestopreventthebugbacklogfromexploding.
Averagetimetofixbugsshouldbenomorethanafewiterations.Longlivedbugsisasmell
indicatinginventory(aformofwaste.)Thiscanbeavoidedbyensuringwegettodonedone
oneachfeaturebeforemovingontonextfeature.
4) Batchingrelatedbugsforfixingorincludingtheminaplannedfutureuserstory/featurecanbe
consideredasapossiblestrategy(forexample,weknowthatthebugisrelatedtoafeaturethat
wewillbeworkingoninthenextiterationandwhichwillrequireaseriousupdateor
deprecationofsomerelatedfunctionality;inthatcaseitmaymakesensetowaittillthatworkis
completed),weneedtomakesurewearecognizantoftheimpactofsuchbatchingonour
latency.
Foradetaileddiscussionofbugbackloganalysis,bugtriageandtheuseofbugmanagementsystems,
refertoChapter19ManagingtheAcceptanceProcess.
1.3.9 UsingAcceptanceTeststoDriveDevelopment
TheacceptanceprocessworksbestwhentheProductOwnersuppliestheacceptanceteststothe
developmentteamearlyenoughtohelpthemunderstandwhatdonelookslike.Ideally,thisisbefore
developmentevenstartsbutitmustdefinitelybebeforedevelopmentisfinishedattheverylatest.The
bestresultsareachievedwhentheseacceptancetestsarenothandedoff,butratherdeveloped

together,forthiswaytheacceptancetestsarenotonlyusedforfindingbugsintheproduct,butrather
clarifyrequirementsandimproveconversationsbetweentheProductOwnerandtheproduct
developmentteam[MelnikEtAlOnExecutableAcceptanceTests].Thefocusisoncollaborationnot
handoff.ThispracticeisknownasAcceptanceTestDrivenDevelopment(ATDD)orStorytestDriven
Development(STDD).Whenusedinconjustionwithincrementaldevelopment,theteamwilltypically
automatemuchofthetestingtokeepthecostofrepeatedlydoingreadinessassessmentlow.The
sidebarWhatittakestodoIncrementalAcceptancedescribesothersuccessfactorsfordoinghighly
incrementalacceptancetesting.
Summary
Thischapterintroducestheconceptoftheacceptanceprocessasawaytothinkabouttheactivities
relatedtomakingthedecisiontoacceptsoftware.Theacceptancedecisionistheculminationofa
processthatinvolvesseveralorganizationseachwithspecificresponsibilities.Eachreleasecandidateis
passedthroughthedecisionpointsoftheacceptanceprocessonitswaytomakingthefinalacceptance
decision.TherearethreemajordecisionpointsbetweensoftwareconstructionbytheProduct
DevelopmentTeamworkingwiththeirProductOwnerandsoftwareusagebytheenduser.First,the
ProductDevelopmentTeammustdecidewhetherthesoftwareintensiveproductinitscurrentform,
knownasthereleasecandidate,isreadyforacceptancetesting.Ifitis,theProductOwnermustdecide
whetherthereleasecandidatemeetstheiracceptancecriteriabeforetheproductcanbemadeavailable
totheendusers.Ultimately,eachendusermustdecideforthemselveswhethertheywanttousethe
software.Thisusagedecisionhasverylittleinfluenceonthecurrentacceptancedecisionalthoughit
mayinfluencetheacceptancecriteriaforfuturereleasesoftheproduct.
Whileeachuserpotentiallymakestheirownusagedecision,thereadinessandacceptancedecisionsare
normallyeachasingledecision.Eachdecisionpointor"gate"shouldhavewelldefinedcriteriatoguide
thedecisionmaking.ThesecriteriashouldbeknowninadvancebyboththeProductDevelopment
TeamandtheProductOwner.
Theacceptancedescribedinthischapterappliestobothsequentialandagileprojectsbutinslightly
differentways.Thephasednatureofsequentialprojectsmeansthatalltestingisdonewithinaseparate
testingphaseandtheacceptanceprocessdescribeswhatgoesonwithinthatphase.Agileprojects
traversetheentiredevelopmentlifecycleforeachfeatureoruserstory.Therefore,theacceptance
processisexecutedatseverallevelsofgranularitywiththefinestgrainexecutionbeingattheindividual
featurelevelandthelargestbeingatthewholeproduct(orfeatureintegration)level.
WhatsNext
InChapter2ElaboratingonTHEAcceptanceProcesswegointomoredetailabouthowtheacceptance
decisionisrelatedtotheusagedecisionandhowtheprocessisimplementedincomplexorganizations
andwhenbuildingcomplexproducts.

References
[CooperOnDoingItRight]RobertG.Cooper,DoingitRight:WinningwithNewProducts,Ivey
BusinessJournal,July/August2000.AlsoavailableasaProductDevelopmentInstituteReference
Paper#10asathttp://www.stagegate.com.
[CooperOnStageGateProcess]Robert.G.Cooper,TheStageGateIdeatoLaunchProcessUpdate,
WhatsNewandNexGenSystems,J.ProductInnovationManagement,Volume25,Number3,
May2008.AmodifiedversionalsoavailableasasaProductDevelopmentInstituteReference
Paper#30Perspective:TheStageGateIdeatoLaunchProcessUpdate,WhatsNewand
NexGenSystemsathttp://www.stagegate.com.
[FewsterGrahamOnTestAutomation]MarkFewsterandDorothyGraham,SoftwareTest
Automation,1999,ACMPress
[ShingoDillonOnToyotaProductionSystem]Shingo,ShigeoandDillon,Andrew,AStudyofthe
ToyotaProductionSystem:FromanIndustrialEngineeringViewpoint:ProductivityPress,1989.
[HighsmithOnAgileProjectManagement]Highsmith,JimAgileProjectManagement:Creating
InnovativeProductsAWP2004ISBN13:9780321219770
[BerczukAppletonOnSCM]Berczuk,SteveandBradAppleton,SoftwareConfigurationManagement
Patterns:EffectiveTeamwork,PracticalIntegration,AddisonWesley(2003)ISBN:0201741171
[MSOnSCM]Patterns&PracticesTeamDevelopmentwithMicroosftVisualStudioTeam
FoundationServer,Ch.3,4,6:MicrosoftPress,2008.
[OMGOnSwimlanes]ActivityPartitionNotations.InUMLSuperstructureSpecification,pp.342345
http://www.omg.org/spec/UML/2.2/Superstructure/PDF/,Oct18,2009
[PoppendiecksOnLean]Poppendieck,Mary&TomLeanSoftwareDevelopmentAddisonWesley
(2003)ISBN:0321150783
[DenneHuangOnIFM]MarkDenneandJaneClelandHuang,"TheIncrementalFundingMethod:
DataDrivenSoftwareDevelopment,"IEEESoftware,vol.21,no.3,pp.3947,May/June2004
[LadasOnScrumban]CoreyLadas,Scrumban:EssaysonOSystemsforLeanDevelopment,Modus
CooperandiPress,2008
[HiranabeOnKanban]KenjiHiranabe,KanbanAppliedtoSoftwareDevelopment:fromAgileto
Lean,publishedonlineat<http://www.infoq.com/articles/hiranabeleanagilekanban>,
January14,2008
[MSOnTFSBranchingGuide]TFSBranchingGuide2.0,http://tfsbranchingguideii.codeplex.com/,Aug
15,2009
[PrimaveraOnScrum]PrimaveraScrumCasestudy.
http://controlchaos.com/download/Primavera%20White%20Paper.pdf,Oct19,2009

[KnibergOnScrumFromTrenches]HenrikKniberg,ScrumandXPfromtheTrenches,
http://www.infoq.com/minibooks/scrumxpfromthetrenches,Oct19,2009
[DenneHuangOnSoftwareByNumbers]MarkDenneandJaneClelandHuang,SoftwarebyNumbers:
LowRisk,HighReturnDevelopment,PrenticeHall,2003
[MelnikEtAlOnExecutableAcceptanceTests]GrigoriMelnik,FrankMaurer,MichaelChiasson,
"ExecutableAcceptanceTestsforCommunicatingBusinessRequirements:Customer
Perspective",Proc.Agile2006,IEEEComputerSocietyPress:3546,2006


Chapter2
2 Elabor
E ratinggonth
he
AccceptaanceProceess
Chapter1 1introducedttheacceptancceprocessandhowitfitsiintotheovera
allproductdeevelopment
lifecycle.TThedescriptio
onwaswasd
deliberatelysimple.Thischapterintrodu ucessomewa aystovarythee
acceptancceprocesstodealwithmo orecomplexssituationsinclludingcompleexorganizatio onsandprodu ucts.
2.1 TheRoleoftheUssageDecission
Thusfarw
wehavefocussedalmosten ntirelyontheereadinessan
ndacceptanceedecisions.W
Whataboutth he
usagedeccision?Itturn
nsoutthattheeusagedecissiondoesntd
directlyimpaccttheacceptaancedecision
n
becauseitthappensafttertheproductisalreadyaaccepted.
2.1.1 Separating
gtheAccep
ptanceDeccisionfrom
mtheUsag
geDecisio
on
Whenasooftwareproductisbeingsselectedorbu uctOwnerbyanotherparty,the
uiltspecificallyforaProdu
ProductO
Ownermayde ecidetoacceptthesoftwaareassuitableebeforeofferingittotheuserswhowiill
thenmaketheusaged
decisionindivviduallyasillustratedinFiggure9.

Figure9
AcceptancceandUsageeDecisionsan
ndCriteria
Almostallproductshavemanyuserrsandeachisslikelytohavvetheirownu usagecriteriawhichwhileit
mayhaveeinfluencedtheacceptanccecriteriauseedduringthereadinessandacceptanceedecisions,m mayin
factbequ
uitedifferentfromit.Inso
omecases,theeuseofthepproductisnottoptional,suchwhensoftware
isusedtocarryoutaspecificjobsu orairlinereseervationssystem.Inthesecases
uchasacallceenteragento
theusageedecisionism
madeonbehaalfoftheuserrsbytheirmaanagement;thisdecisionn neednotfollo
ow
didate1 53

immediatelyaftertheacceptancedecisionastheremaybeotherprerequisitesforusagesuchasuser
training.
Whileausercould,intheory,choosenottousethesoftware,theconsequenceoflosingtheirjobwould
makethisdecisionhighlyunlikely.Theusersfeedback,however,mightinfluencesubsequentreleasesof
thesoftwareaswellseeinFeedbackfromtheUsageDecisions.
2.1.2 DeterminingtheAcceptanceCriteria
Theacceptancedecisionismadebasedonsomecriteriaincludingbothfunctionalityandqualityrelated
elements.Theminimumleveloffunctionality(representedbyMMF)andtheminimumlevelofquality
(MQR)shouldbedeterminedwellinadvanceoftheacceptancedecisiontogivetheProduct
DevelopmentTeamofthesoftwareareasonablechanceatbeingabletosatisfythem.Therefore,the
decisionsabouttheMMFandMQRcomebeforetheAcceptanceDecisionasshowninFigure9
AcceptanceandUsageDecisionsandCriteria.
2.1.3 FeedbackfromtheUsageDecisions
Notethattheusagedecisionsmadebyindividualusersaftertheacceptancedecisiondonotdirectly
affecttheacceptancedecisionbutitmay,throughsalesdata,postsalessupportdataorfocusgroups
affectthedefinitionofMMForMQRforsubsequentreleasesoftheproduct.ThisisillustratedinFigure
10UsageDecisionFeedback.

Figure10
UsageDecisionFeedback

WhiletheMQRtendstobefairlyconstantfromonereleasetothenext,theProductOwnermayrevise
thequalitycriteriaforasubsequentreleasebasedonthefeedbackreceivedfromusers.Forexample,
usertestingofanalphareleasemightrevealthattheresponsetimesspecifiedintheMQRarenotfast
enoughinpracticeandmightcausetheproductownertomakethemmorestringentforthebetaor
finalrelease.
TheMMFforeachreleasetendstobeuniquebecauseitisthelistoffunctionalitythatneedstobe
includedtowarranthavingtherelease.FeedbackfromuserscaninfluencetheMMFbasedonrequests
formissingfunctionality,suggestionsforhowexistingfunctionalityshouldbechanged,andthroughbug
reports.
FeedbackfromusersofAlphaandBetareleasesoftenresultsinchangestotheMMForMQRofthefinal
release.
2.2 AcceptanceinComplexOrganizations
Thesimpleacceptanceprocessdescribedpreviouslywillsufficeinmanyorganizations.Butthereare
someorganizationswhosestructureisverycomplexasaretheproductstheydevelop.These
organizationsrequireadecisionmakingprocessthatmirrorsthecomplexityoftheorganizationor
product.Thedecisionsmaybemadeinseriesaseachdecisionmakerexaminestheproductbutthis
leadstolongerelapsedtimesbeforethevalueoftheproductcanberealized.Thecommonsolutionisto
haveeitherthereadinessoracceptancetestingactivitiesanddecisionbemadeinparallelbyseveral
organizationseachwithvetopowerovertheacceptabilityoftheproduct.
Notetoreaders:Ifthesescenariosdontapplytoyourorganization,feelfreetoskipaheadto2.3
AcceptingComplexProducts.2.3
2.2.1 ParallelAcceptanceDecisions
Manyorganizationshaveotherdepartmentswhoareresponsibleforoperatingandsupportingthe
systemonceitisdeployed.Thesedepartmentsneedtohaveasayintheacceptancedecision.Thisis
reflectedinFigure11ParallelAcceptanceDecisions;itshowsseveralacceptancedecisionsbeingmade
inparallel,eachbasedonadifferentsetofacceptancecriteria.

Figure11
ParallelA
AcceptanceDeecisions
Whentheereareseveraalorganizatioonsinvolvedinntheacceptaancedecisioneachwithth heirownareaof
specialty,eachmayinssistinconducctingitsownaactivitiesindeependentlyofeachother.Eachmakesttheir
owndecissionregardinggthereleasecandidatean ndanyoneoffthemcanefffectivelyvetootheothers.
Examplesofsuchstake
eholders,and
dtheirareao
ofinterest,co
ouldinclude:
B
BusinessUnitFunctionaliityisacceptab
ble.
O
OperationsDe
epartmentTTheproductccanbemanaggedandsupportedinprod
duction.
C
CorporateBra
andingProductcomplieswithbrandin
ngrequirements.
C
CorporateSec
curityProdu
ucthascomplliedwithsecu
urityrequirem
ments.
U
UsabilityGrou
upProductccomplieswith
husabilityreq
quirements.
ThisapprooachintroduccesriskiftheMMFandMQRusedbyeeachstakeholderarenotin ncludedinthee
MMFand dMQRusedb bytheProductDevelopmentTeam.Ideaally,eachstakeholderwou uldparticipatteat
leastparttimeinthePProductDevelopmentTeamtoensuretthattheteam mtrulyundersstandsthe
acceptanccecriteriaand ddesignstheebestpossibleesolutiontothebusinessneed.Thispaarticipationcoould
befortheeentiredurattionoftheproojectorforoneortwoiterationsfocusedonthatstakeholdersneeds.
Figure12AProductD
DevelopmenttLifecyclewitthParallelElaaborationand
dAcceptancebyProduct
OwneranndOperationssillustratessthescenario
owheretheP
ProductOwneeristhepartyywhohas
didate1 56

conceived
dthesoftwareasasolutio
ontoabusineessproblemandtheoperaationsdepartm
mentistasked
withsupp
portingthesyystemonceitisputintoprroduction.

Figure12
AProducttDevelopmen
ntLifecyclewiithParallelEllaborationan
ndAcceptanceebyProductO
Ownerand
Operationns
2.2.2 ParallelRe
P eadinessD
Decisions
Insomecasesseveraloorganizationssdotheirown nreadinessassessmentoffthereleaseccandidateand d
makeindeependentreaadinessdecisions.Theprod ductwillnotbehandedovvertothepro oductownerffor
theaccepptancedecisio
onifeitherorrganizationdeeemstheproductnotreaadyuntilanyyadditionalw work
theyidenttifyhasbeencompletedandtheydeem medthesoftw wareready..ThisisillustrratedinFigurre13
ParallelReadinessDeecisions.
didate1 57

Figure13
ParallelR
ReadinessDeccisions
Examplesofpotentialstakeholderswithparallelreadinessdeecisionsinclud
de:
P
ProductDevel
lopmentTeam
mCodeispaassingallunitttestsandfun
nctionaltestss
S
SecurityTeam
mDesignhasspassedsecu
urityreview
D
DataArchitect
tureTeamD
Dataarchitecturehasbeen
napproved
LegalDepartm
mentAllembeddedtechn
nologieshaveebeensuitablylicensed.
Again,eacchofthesesttakeholdersshouldbecon
nsideredmem
mbersofthew
wholeteam,eevenifonlypart
time,toeensurethatth
heMMF/MQR Rareaddressedbywhatth
heProductDevelopmentTTeambuilds.
Whetheradecisionisddeemedtobeepartofread dinessoracceeptanceissom mewhatarbittrary.Thecho oiceis
usuallybaasedonwhennthedecisionnmakerwanttstogetinvolvedandwhicchsideofalaarger
organizationalboundarythestakeh holdersits.Fo
orexample,in
nsomeorganizationsthessecurity
departmeentmaybeco onsideredabusinessfuncttionwhomusstacceptthereleasecandiidatewhilein n
mayhaveapresenceinorrneartheITd
othersitm departmento orproductdevvelopmentgrroupandtherreby
choosetooparticipatew
whenthereadinessdecisio onismade.O
Ofcourse,itissusuallyprefeerabletoinvo
olve
theseparttiesmuchearlierintheprrojectinamo
orecollaborattivefashionsoothatreadinessassessmeentis
conducted dincrementaallyasthepro
oductisdeveloped.
2.3 Accepting
gComple
exProduccts
Organizattionsthatbuildlarge,complexproductsstypicallyuseeamorecom
mplexversionoftheaccepttance
processth
hatinvolvesm
multistageaccceptance.Laargecomplexproductstyp picallyrequireelargenumbeersof
didate1 58

people,to
oomanytow workinasingleteam.Whilethelargeco
omplexprodu uctsaretypicallycomposeedof
components,theteam msthatbuildtthemmaybeorganizedinseveralwayss.Eachcompo onentmaybee
builtandownedbyaccomponentteeamorteamssmaybeorgaanizedaround dthefunctionnalityprovideedto
theusers..Thelatterap
pproachiscalledfeaturecrewsandisswidelyused
dbytheDeveloperDivision nof
Microsoftt[MillerCarterOnLargeScalleAgile].Themannerinw whichthesepeeopleareorgganizedimpaccts
theaccepptanceprocesss.Thekindso ofproblemsffoundduringtheacceptannceprocesscaanbeagood
indication
noftheappro opriatenessofthewaywehaveorganizzedourteams.
2.3.1 Acceptingt
A theOutputofCompo
onentTea
ams
Acommonwayofsubdividingworkkinlargeorgaanizationsisttodecomposeetheproducttinto
componentsthatareb builtandveriffiedinparalleelbyseparateecomponentteamsthateeachspecializeein
theirresp
ptivetechnolo
ogiesandtech hnicaloruserrdomains.Whenthisorgaanizationalsolutionisused
d,
eachcomponentshouldhaveitsow wnreadinessassessmentp priortocondu
uctingreadinessassessmeenton
theintegrratedproductt.ThisisillusttratedinFigure14ComponentTeamsswithSerialDDecisions.

Figure14
ComponeentTeamswitthSerialRead
dinessDecisio
ons
Eachcomponentmusttgothroughaacomponent readinessdeecisionbeforeebeingintegrratedintothee
largerpro
oductforsubssequentprod ductlevelread dinessdecisio
on.Theintegratedproductisthenturn
ned
overtoth
heProductOw wnerfortheaacceptancedecisions.Thereadinessasssessmentacttivitiesmaygo oby
variousnaamesincludinngcomponenttestingorrsystemtesttingfortheccomponentsandsystem
testingo
orintegrationntestingforrtheentireprroduct.EachccomponenthhasitsownM
MMFandMQR R;
hopefully,,thesearedirectlytracabllefromtheMMMF/MQRforthereadinessandaccepttancedecisions
madeabo outtheentireeproduct.
didate1 59

Thisreveaalsseveraloftheweaknesssesofthisap pproachtocomplexproducts.First,beccausethe
componentscannotbe edirectlyundderstoodandthereforeaccceptedbytheeProductOw wner,some
intermediiary,usuallyaanarchitecthhastoberespponsiblefortrranslatingProoductOwnerrequirementsinto
componentrequireme ents.Thiseffeectivelytransffersproducto ownershipoff thecomponeentfromthe
ProductOOwnertotheintermediaryytherebyredu ucingthedeggreetowhichthecomponentteamis
accountab bletothereaalProductOw opossibleforthecomponeentrequirementstodriftffrom
wner.Itisalso
whatisacctuallyrequireed.Second,theproductreeadinesscann noteasilybeassesseduntilallofthe
componentshavepasssedtheirresp pectivereadinnessdecisionssandareinteegratedtoforrmtheproduct.
Thisdelayystheproducctreadinessassessmentun ntilafterthebigbangintegrationoftthecomponents,
discoverthatsomeoftheccomponentreequirementswere
whichisratherlateinaaprojecttod
misunderstood.Thisproblemcanb besomewhatovercomethroughtheuseofcontinuo ouscode
on(CI)practiccescombinedwithincremeentalintegrattion[BasLarm
integratio manOnScalinggAgile]and
incrementalacceptanccetestingbuttinpracticeittismuchhard dertodoincrrementaldeliveryusing
componentteams(com mparedtofeaatureteams)andfeaturetteamswilltyp picallybeabletodeliver
productqqualitysoftwaareforindividdualfeaturesearlier.Italssoperpetuateesthetechniccalcomponen nt
etechnicalcomponentdevvelopersfrom
ghettobyyisolatingthe manyundersttandingoftheebusiness.
2.3.2 Acceptingt
A theOutputofFeaturreTeams
Anotherw wayoforganizingthedeveelopmentoflargecomplexxproductsisttousefeatureteams/crew ws
[BasLarmaanOnScalingA Agile,Eckstein
nOnLargeScalleAgileandM MillerCarterOn Agile].Eachfeature
nLargeScaleA
teamisreesponsibleforrbuildingand dverifyingaffeaturebased
dontheProductOwnersaacceptance
criteria.B
Buildingeachfeaturemayinvolvemodifyingseverallcomponentssbuttheteam mincludes
expertiseoneachcom mponentandeensuresthattthecomponeentsarepropeerlyintegrateedaspartofttheir
featurereeadinessassessment.Therrefore,theprroductreadinessassessmeentandtheprroductaccepttance
testingarebothfocuse edonfeatureeintegrationrratherthanooncomponenttintegration..Thisscenario
ois
illustrateddinFigure15FeatureTeeamswithSerrialAcceptancceDecisions.
didate1 60

Figure15
FeatureTeamswithSerialAcceptanceDecisions
EachfeaturesreadinessdecisionismadebasedontheMMFandMQRspecifictothatfeatureas
determinedbytheProductOwner(ideallywithinputfromtheProductDevelopmentTeam);no
intermediaryisrequiredtotranslatetheProductOwnersrequirementsintocomponentrequirements.
Thefeaturereadinessdecisionisaprerequisiteforthecorrespondingfeaturesacceptancedecision
whichshoulduseroughlythesameacceptancecriteria.Meanwhile,theproductisintegratedanda
productlevelreadinessdecisionismadebasedontheproductsMMFandMQR.Theproductlevel
MMFshouldsimplybeanaggregateofthefeaturelevelMMFsThereforetheinteractionsbetween
featuresneedtobeincludedaspartoftheMMFforeachaffectedfeatureorasseparatefeature
interactionfeatures.Whentheproductlevelreadinessandallfeaturelevelacceptancedecisionsare
positive,theproductlevelacceptancedecisioncanbemade.
Thisstyleoforganizationandacceptanceisusuallypreferabletocomponentteamsbecauseitensures
eachteamisworkingwitharealendProductOwnerandisresponsiblefordeliveringawholeproduct,
notjustaninternalcomponent.SeethesidebarFeatureCrewsonMicrosoftOfficeforanexampleof
howthisworksinpracticeinChapter9ProductManagersPerspective.
IftheProductOwnerisahighlyconstrainedresource,havingalargenumberoffeatureteams
contendingfortheirparticipationmayreducetheeffectivenessofthisapproach.Iftheinteractions
betweenfeaturesarealargerriskfactorthantheriskoftechnicalintegrationwiththetechnical
components,itmaybemoreappropriatetoreducethenumberoffeatureteamsbydelegatingmoreof
theworktocomponentteamssothatfeatureinteractionscanbeaddressedwithsinglefeatureteams
ratherthanbetweenthem.
2.3.3 AcceptingCustomizableProducts
Manyproductsarebuiltbyoneorganizationforsaletoanotherorganization,theircustomer.Whenthe
productneedstobecustomizedforeachcustomertherearetwodistinctacceptancephasesforthetwo
differentproductsinquestion.Thefirstproductisthegenericproductbuiltbythesellingorganization
totherequirementsspecifiedbytheproductmanager.Thesecondproductistheonebuilttothe
specificationofthepurchasingcustomer.Theconstructionofthelatterproductmayincludeeither
configurationorcustomcodinginadditiontothegenericcustomizableproduct.
Thefirstacceptancedecisioniswhentheproductmanageracceptsthecustomizableproductassaleable
tothecustomers.Thisacceptanceprocesscouldbeanyofthevariantsdescribedpreviouslyincluding
AcceptingComplexProducts.
Thesecondacceptancedecisioniswhentheacquirerofthecustomizableproductacceptstheproduct
ascustomizedtotheirownsituation.Thereisasingleacceptancedecisionforeachversionofthe
customizablegenericproductandoneacceptancedecisionperversionofthecustomizedproductas
showninFigure16Acceptingcustomizableproducts.

Figure16
Acceptingcustomizableproducts
Sidebar:AcceptingConfigurableERPProducts.
Manysoftwareproductsaresoldasgenericimplementationsofabusinessprocessthatcanbe
configuredtosuitabusinessspecificneeds.TwogoodexamplesofthisareERPsystemssuchasSAPs
ERPproductandMicrosoftsDynamics.Bothprovidegenericimplementationsofcorebusiness
processessuchastimereportingandpayroll,supplyorderingandaccountspayable,productorder
processingandaccountsreceivable.Theoutoftheboximplementationsoftheseprocessescanbe
modifiedthoughconfigurationdataorextendedbypluggingincustomwrittenprocedures.Fora
particularcustomersinstallationoftheERPsystem,itgoesthroughtwocompletelydistinct
acceptanceprocesses.First,theproductownerattheconfigurableproductvendordefinesthegoals
ofthenextreleasebysynthesizingnewfeatureideasfromtheproductdefinitionteam,bugreports
frompreviousreleasesandnewfeaturerequestsfromnewandexistingcustomers.Thisgetsturned
intothefunctional(MMF)andnonfunctional(MQR)requirementsfortherelease.TheERPproductis
built,goesthroughreadinessassessmentbytheproductdevelopmentteam,acceptancetestingbythe
productmanagementteamandprofessionaltestersbeforebeingdeclaredreadytoshipto
customers.

Meanwhile,thecustomersaredefiningtheirowngoalsandobjectivesfortheinstallationbasedonthe
expectedfeaturesetofthenewreleaseandtheirspecificbusinesscontext.Eachcustomershould
haveitsownproductownerforthecustomizedproduct.Theymaydobusinessprocessmodellingand
gapanalysisbetweentheirbusinessprocessandtheoutoftheboxproducttodefinetheirown
MMFandMQR.Whentheproductships,theirownproductdvelopmentteaminstallsitintotheirown
developmentsystemanddoesthecustomizationand/orconfiguration.Ifthisistheirfirstinstallation
oftheproductthentheyadjustthevariousconfigurationparametersandimplementnewplugin
procedures.IfthisisanupgradefromapreviousversionoftheERPsystemthenmuchofthework
consistsofreapplyingtheconfigurationfromtheirprevioussystemanddoingregressiontestingofthe
functionality.Inbothcases,thereleasecandidate(s)shouldtraversethefullacceptanceprocesswith
thecustomersproductdevelopmentteammakingtheReadinessDecisionandthecustomersproduct
ownermakingtheacceptancedecisionwhichisbasedontheMMFandMQRdefinedbythe
customersproductownerfortheirspecificinstallationoftheERPsystem,nottheMMFandMQR
definedbythevendorsproductownerforthegenericproduct.Inaninterestingparadox,ERPsystems
wouldseemtobeimminentlysuitabletoagilecustomizationandincrementalacceptancetesting
becausetheyworkrightoutoftheboxandcustomizationevolvestheproductfromagenericonetoa
customizedone.ButmanyERPsystemslackthetoolstoautomatethebuildandregressiontesting.
SeethesidebarWhatittakestodoIncrementalAcceptanceandthepapers[MeszarosOnAgileERP]
and[AstonOnAgileERP]formoreinformationonwhatneedstobeavailableandmightbemissingin
yourERPsystem.
2.4 Exitvs.EntryCriteria
Manyofthedecisionsintheacceptanceprocessinvolvehandoffsbetweenparties.Eachhandoff
involvestwosetsofcriteria.Thepartymakingthehandoffmayhavesomeexitcriteriathatmustbe
satisfiedbeforetheyarepreparedtomakethehandoff.Thepartyreceivingthehandoffmayhavesome
criteriatobesatisfiedbeforetheyarepreparedtoreceivethehandoff.Thesecriteriacanbeassessedat
separateexitandentrydecisionsasillustratedinFigure17SeparateExitvs.EntryDecisions&Criteria.

Figure17
SeparateExitandEntryyDecisionsan
ndCriteria
Thiseffecctivelyresultsinasituation
nwherethereeisnonomaanslandbeetweentheBu
uildandAccept
phasesassshowninFiggure18:

Figure18
AcceptancceProcesswiithSeparateEExit/EntryCritteriaforAcceeptance
WhentheeProductDevvelopmentTeeamandprod ductownerpaartieshaveagoodworkinggrelationship pthey
shouldbeeabletoagreeuponasingglesetofcriteeriathatsatissfiesboththeirneedsandhaveasinglee
decisionp
pointbasedoontheagreeduponsetofccriteriaasillu ustratedinFiggure19SinggleDecisionP
Point
withCombinedExitan ndEntryCriteria.
didate1 64

Figure19
SingleDeccisionwithCo
ombinedExitandEntryCriiteria
Thetwoppartieswouldagreeonwh hichpartydoeestheassessmmentforeach
hcriterionandwhomakessthe
decisionb
basedontheassessment.Typicallyitisthepartydoingthehando offthatwilld
dotheassessm
ment
andthereeceivingpartyymaybeinteerestedinseeeingtheresultts.
Themainentrycriterio onfortheaccceptancephaseisthattheProductDevelopmentTeamhasdeem med
thesoftwarereadyforracceptancettesting.SeconndarycriteriaamayincludeewhethertheeProductOwneris
sufficientllypreparedtooconducttheeacceptancetesting.Thedecisiontoentertheacceeptancephasee
shouldtakkeallthecriteriaintoacco
ount.
ple,aProductDevelopmentTeamsReeadinessDecissionshouldb
Forexamp bemadebasedona
combinationof:
E
Entrycriteriaf
foracceptanccetestingpro
ovidedbytheProductOwn
nerincluding::
AllfeaturesinMMFFfunctionalw
withnoknownseverity1&
&2bugsopen
n.
Stresstestedat110
0%ofratedcaapacityfor48
8hourswithllessthan1failedtransactiion
000.
per10
Regresssiontestofp
previousreleaasefeatureseexecutedwith
h100%passrate.
E
ExitcriteriaforconstructionprovidedbyytheProducttDevelopmen
ntTeamsuch
has:
odecoverageebyunittestss.
90%co
Dataaarchitecturereeviewedandapproved.
Staticanalysishasidentifiednodeficiencies.
Securittyreviewcom
mpletedwithnohighpriorrityissuesopeen.
didate1 65

InpracticeitishighlypreferableforthereadinesscriteriatoincludealltheProductOwnersacceptance
criteria.Thisavoidsfindingbugsduringtheacceptancetestingphase.
Sidebar:RepresentingtheAcceptanceProcesswithStagesandGates
Manycompaniesemployaprocessinvolvingstagesandgatestomanagetheirproductdevelopment
pipeline.Theseprocessesweredescribedinsection1.X(ProcesseswithStagesandGates).Thegates
serveasdecisionpointsbetweenstages.Themaingoalistomanagetheallocationofresources
(fundingbeingthemostcommon)amongandwithinprojects.Thegatesattheexitofstagestypically
involvego/nogodecisionswiththegodecisionsleadingtoprogressivelylargeredresource
commitmentsandinthesubsequentstages.
AtypicalsequentialStageGateTMprocessstartswithanewideaevaluatedataninitialgatetodecide
whethertheideaisworthpursuing.AsampleprocessisillustratedinFigureS0.
Attheideastageusuallyafixedamountofresourcesareallocatedtotheproducttodeveloptheidea
furtherandbuildabusinesscaseinthenextstage.Thenextgatedetermineswhethertheidea
warrantsadvancingtofullscaledevelopment.Ifso,resourcesareassignedfordevelopingtheproduct;
ifnot,theprojectiskilled.Thenextgatetypicallyinvolvesdecidingwhetherdevelopmentofthe
productiscomplete.Ifso,theproductentersthetestingandvalidationstage;ifnot,itmaystayin
developmentuntilitrunsoutofresources,inwhichcasetheprojectmayrevertbacktotheprevious
stagetodeterminewhetheradditionalresourceswouldbewarranted.Theexitgateofthetestingand
validationdetermineswhethertheproductisreadyforlaunch,thatistobesoldorused.

FigureS0.AsampleStageGateProcessTM[source:http://www.stagegate.comisfairlytypical].
Manysoftwaredevelopmentprocessescanbecastintermsofstagesandgates.TheRationalUnified
Process[KruchtenOnRUP]describes4phasesInception,Elaboration,Construction,Transition.Each
phasehasanassociatedmilestone.Theemphasiswithinaphaseisplacedonaspecificsetofactivities,
buttheprocessdoesnotprecludeperformingotheractivitiesoutsidethatcore.Thusthephasescan
beviewedasstagesinwhichmanykindsofactivitiestakeplaceandthesameactivitiesmaybe
repeatedoriteratedacrossmanydifferentstages.Themilestonesattheendofeachphasecouldbe
treatedasgates.
Agilesoftwaredevelopmentprocessesmayalsobecastintermsofstagesandgates,butwitha
differentspintocapturetheiriterativeandincrementalnature.Whenaniterativeandincremental
processisunwound,itcanstillbeviewedasastreamofgatesandstages,withfundingincrementally
allocatedatreleaseends.Thiscanespeciallybeveryeffectiveifhighvaluefeaturesaredeployedin

earlyreleasessothattheprojectstartsgeneratingvalueassoonaspossibleandcanbecomepartially
selfsustaining[DenneHuangOnSoftwareByNumbers].Inthisvariation,ratherthanstageshaving
distinctpurposesandgateshavingdifferenttypesofdecisions,thestages,activitieswithinstages,and
gatesarereplicated.Eachstagegatepairrepresentsaniterationforimplementingthenextsetof
featurestobedeveloped,togetherwithacontinuationorfundingdecision.Thegatesareusedto
approveadditionalfundsforthenextiterationbasedonwhatwaslearnedinpreviousiterations
[HighsmithOnAgileProjectManagement,ErdogmusOnEconomicsOfAgile].(Thisisaninstanceofthe
applicationoftheprinciplesdiscussedintheChapter1SidebarRecastingaSequentialProcesses
throughWorkflowScheduleIndependenceandParallelization.)
WereorganizepartofStage3,Gate4,Stage4,andGate5,intoacustomprocessforsoftware
products:theAcceptanceProcess.WhileitcouldbearguedthatGate4,GoToTesting,assumesthe
ProductDevelopmentTeamhasinfacttestedthesoftwareextensively,inmostorganizationsthe
majorityoftestingoccursinStage4,TestingandValidation.Weproposethatmuchofthetesting
activitybelowthelevelofbusiness,user,orcustomerrequirementsoftheproductbepushedback
intoStage3,Development,andbecomepartofthereadinessassessmentactivitywithinthatstage.
ThiseffectivelymakesthereadinessdecisionpartofGate4,GotoTesting,andleavesStage4focussed
oncustomeroruseracceptancetesting.ThenGate5,GotoLaunch,wouldrepresenttheAcceptance
decision.ThevariationsonwhomakeswhichdecisionandwhenaredescribedinChapter3The
DecisionMakingModel.
HowistheAcceptanceProcessforreleasecandidatesofaproductexpressedintermsofstagesand
gatesandhowdoestheresultrelatetothetypicalStageGateprocessTM?Werelaxthenormal
requirementofhavingagatebetweeneachpairofsubsequentstagesandseparatethepreparation
foradecisionassociatedwithagateintoadistinctstagethatimmediatelypreceedsthegate.Wealso
allowstagestobenestedtohaveinnerstagesandgates.FigureS1TheAcceptanceProcessas
NestedStagesandGatesillustratesthisthroughanexample.ThestagesBuild,Accept,andUsein
FigureS0correspondtothestagesDevelopment,ValidationandTesting,andLaunchinFigure0.it.

FigureS1
TheAcceptanceProcessasNestedStagesandGates
InsidetheBuildstageonthetop,Build1isthefirstreleasecandidatethatdevelopmentaskstheir
readinessassessorstoevaluate.Ithasseveralseverity1bugsthatmustbefixedandismissingsome
keyfeatures.Developmentfixesthesebugsandaddsthemissingfunctionality.Afterseveralmore
tries,BuildNfinallymakesitallthewaythroughReadinessAssessmentandisreleasedtotheproduct
ownerforacceptancetesting
ApositivereadinessdecisioninsidetheBuildstageautomaticallysatisfiesthecriteriaforGate4
thereforetheprojectentersintotheAcceptstage.Theproductownerconductsacceptancetesting
onlytodiscoversomekeyissues.TheseareprovidedtotheProductDevelopmentTeamwho
addressestheminBuildN+1afterseveralintermediatebuilds(notshown)thatfailtheirreadiness
assessment.NotethatatthispointtheprojectisstillconsideredtobeintheAcceptstagedespitethe
factthattheballisinTheProductDevelopmentscourttofixthebugs.Theproductownerfindsno
significantissueswiththisbuildandacceptsthesoftwareasfinished,satisfyingGate5scriteria.
TheprojectnowmovestotheUsestage.Afterafewweeksofusage,auserexperiencesaseverecrash
thatistracedbacktoamissedscenario.TheproductowneraskstheProductDevelopmentTeamfora
fix.DevelopmentcomesupwithafixinBuildN+2,whichpassesreadinessassessmentandacceptance
testing(mostlyregressiontestingplusanewtestcasecoveringthisbug)andBuildEisputinto
production.TheprojectstaysintheUsestage(oftenreferredtoasWarranty)untilthecriteriato
exittheUsestagearemet;thismaybedecidedbythepassageofanagreeduponperiodoftime
duringwhichtheproductiscoveredunderTheProductDevleopmentTeamswarranty.

AttheendoftheBuildstageandduringtheAcceptstage,eachreleasecandidateisassessedfor
whetheritsatisfiesfunctionalandnonfunctionalcriteria.Howevertypically,thetransitionsfromthe
BuildstagetotheAcceptandUsestagesinvolvealsobusinessandpoliticalfactorsthatmayhavelittle
todowithwhetherthesoftwaremeetsthefunctionalandnonfunctionalrequirements(SeeChapter
XSystemRequirementsModel).Examplesofsuchfactorsinclude:
Aneedtofreeuptheproductdevelopmentteamforotherwork
Changingwhopaysformodifications:beforeacceptanceitisusuallytheproductdevelopment
teamwhileafteracceptanceitistypicallytheproductowner.
Changingthefundingsourcefromcapitaltoexpense.
Managementpressuretoshortcutagateduetoamisalignmentofprojectgoalswithincentives.
Whilethesefactorsmayplayacriticalroleingatedecisions,theyareoutsidethescopeofthisguide.

Sidebar:QualityGatesandtheAcceptanceProcess.(MicrosoftQualityGatesexamplesasMMF/MQR
criteriaforreadinessandacceptancedecisions.)
<TBA>
2.5 Summary
Thischapterintroducestheconceptoftheacceptanceprocessasawaytothinkabouttheactivities
relatedtomakingthedecisiontoacceptsoftware.Theacceptancedecisionistheculminationofa
processthatinvolvesseveralorganizationseachwithspecificresponsibilities.Eachreleasecandidateis
passedthroughthedecisionpointsoftheacceptanceprocessonitswaytothemakingthefinal
acceptancedecision.Therearethreemajordecisionpointsbetweensoftwareconstructionbythe
ProductDevelopmentTeamworkingwiththeirProductOwnerandsoftwareusagebytheenduser.
First,theProductDevelopmentTeammustdecidewhetherthesoftwareintensiveproductinitscurrent
form,knownasthereleasecandidate,isreadyforacceptancetesting.Ifitis,theProductOwnermust
decidewhetherthereleasecandidatemeetstheiracceptancecriteriabeforetheproductcanbemade
availabletotheendusers.Ultimately,eachendusermustdecideforthemselveswhethertheywantto
usethesoftware.Thisusagedecisionhasverylittleinfluenceonthecurrentacceptancedecision
althoughitmayinfluencetheacceptancecriteriaforfuturereleasesoftheproduct.
Whileeachuserpotentiallymakestheirownusagedecision,thereadinessandacceptancedecisionsare
normallyeachasingledecision.Eachdecisionpointor"gate"shouldhavewelldefinedcriteriatoguide
thedecisionmaking.ThesecriteriashouldbeknowninadvancebyboththeProductDevelopment
Teamandtheproductowner.Complexorganizationsmayhaveseveralreadinessoracceptance
decisionsbeingmadeinparallel,eachbasedontheirowncriteria(forexample,operationalacceptance);
thereleasecandidatesprogressionthroughtheprocessmaybevetoedbyfailingtosatisfythecriteria

ofanyoftheparalleldecisions.ComplexproductsmayrequiremultipleProductDevelopmentTeam
teamseachwiththeirownreadiness,andpotentiallyacceptancedecisionspriortoanoverallreadiness
andacceptancedecisionfortheintegratedproduct;thereleasecandidatecanbesentbackforfurther
workfromanyofthedecisionpoints.
Theacceptancedescribedinthischapterappliestobothsequentialandagileprojectsbutinslightly
differentways.Thephasednatureofsequentialprojectsmeansthatalltestingisdonewithinaseparate
testingphaseandtheacceptanceprocessdescribeswhatgoesonwithinthatphase.Agileprojects
traversetheentiredevelopmentlifecycleforeachfeatureoruserstory.Therefore,theacceptance
processisexecutedatseverallevelsofgranularitywiththefinestgrainexecutionbeingattheindividual
featurelevelandthelargestbeingatthewholeproduct(orfeatureintegration)level.
2.6 WhatsNext?
InChapter3DecisionMakingModelweexaminetherolesandresponsibilitiesofthevariousparties
involvedinmakingthedecisionswhichmakeuptheacceptanceprocess.Wealsodescribetheroles
playedbythepartieswhoprovidethedataonwhichthedecisionsarebased.
2.7 References
[BasLarmanOnScalingAgile]Vodde,Bas.andLarman,CraigScalingLean&AgileDevelopment:Thinking
andOrganizationalToolsforLargeScaleScrum,AddisonWesleyProfessional,AgileSoftware
DevelopmentSeriesseries.,2008
[HighsmithOnAgileProjectManagement]Highsmith,JimAgileProjectManagement:CreatingInnovative
ProductsAWP2004ISBN13:9780321219770
[MeszarosOnAgileERP]Meszaros,Gerard,etalAgileERPYouDontKnowWhatYouveGotTillits
Gone,http://agileerppaper.gerardmeszaros.com/
[AstonOnAgileERP]Aston,Janice,etalAgileERPOvercomingtheChallenges,
http://asug2008agileerpchallenges.gerardm.com/
[EcksteinonLargeScaleAgile]JuttaEckstein,AgileSoftwareDevelopmentintheLarge:DivingIntothe
Deep,DorsetHouse,2004
[ErdogmusOnEconomicsOfAgile]HakanErdogmus,TheEconomicImpactofLearningandFlexibilityon
ProcessDecisions,IEEESoftware,Vol22,No6(Nov.2005),pp.7683
[DenneHuangOnSoftwareByNumbers]MarkDenneandJaneClelandHuang,SoftwarebyNumbers:
LowRisk,HighReturnDevelopment,PrenticeHall,2003
[KruchtenOnRUP]PhilippeKruchten,TheRationalUnifiedProcess:AnIntroduction,AddisonWesley,
2003
[MillerKarterOnLargeScaleAgile]AdeMillerandEricCarter,AgileandtheInconceivablyLarge,Proc.
Agile2007,IEEEPress,2007.

Chapter3
3 D
Decisio
onMakinggMod
del
Chapter1
1introducedttheacceptancceprocessanndthethreekkeydecisionsthatneedtobemadeas
oracceptabilitybywhoeveermakestheacceptanced
softwareisassessedfo decision:theReadiness
Decision,theAcceptannceDecisionaandtheUsageeDecision.
Figure1illlustratesthisssimplifiedveersionofthedecisionmakkingmodel.
Figure1
Simplified
ddecisionma
akingmodel
onelaboratessonhowthefirsttwodecisionsaremaadeandwhomakestheminavarietyof
Thissectio
businessmmodels.Thed decisionsarenotmadeinavacuum;th heyrequireinformationthatmustbem
made
availabletthroughactivvities.Figure2 orasingledecision:
2illustratesthisprocessfo
didate1 71

Figure2
Decisionmmakingmodeelsampleactiivities
Thediamo ondontherigghtsideofFiggure2repressentsthedecisiontobemadebasedon nthetestresuults
(thedecissioncanbeeithertheread dinessdecisioonortheacceeptancedecision).Thetesttresultsareb
based
onthetesstingandasse
essmentactivvities,whichaassessesthessystemunderrtestagainstttheexpectattions.
Theexpecctationsofthesystemund dertestwereedefinedbaseedontheuseersrequirements.Allofth hese
activitiesareexecutedwithintheco ontextofateestplan.
ManyoftthepracticesinVolumeIId describehowwtodotheassessactivity,andotherpracticesinVollume
IIdescribeewaystodeffinetheexpecctationsbasedontheneed ds.Thatison neofthereasonsthisguideehas
anumberrofrequireme entsrelatedp
practicesitisnotaboutttesting,itisaaboutacceptaance,and
acceptancceisbasedon nexpectationnsandrequireements.
3.1 TheSixA
AbstractR
Roles
Thejobtittlesofthede
ecisionmakerrsvarygreatlyyfrombusineessmodeltob
businessmod delandacrosss
businessd
domainsandorganizations,sothisguid deusesabstraactrolenameestodescribeetheroleswithin
thedecisionmakingm model.Thisguidealsoprovidesalistofccommonaliasses.Howeverr,beawareth hat
manyofthenamesare ehighlyoverloadedandth hatyourcusttomer(topickjustoneexxample)maybe
anentirelydifferentro onementioneedasanaliashere.Toseehowtheabsttractrolenam
olethantheo mes
maptojobtitleswithinnorganizationsinspecificbusinessmodels,seethesidebarDeccisionMakingg
ModelSteereotypes.
3.1.1 Readiness
R DecisionM
Maker
Thereadinessdecisionnmakermakeesthefinalreeadinessdecissionbasedon ninputfromoothers.When
na
singleperrsonperformssthisrole,thejobtitlemigghtbesometthinglikeChieefEngineer,P
ProjectManagger,
didate1 72

DevelopmentManager,orVPofEngineering.Thisrolecouldalsobeplayedbyacommittee,asis
commoninenterprisedeploymentwithmanystakeholders,ormadebyseveralpeopleorcommitteesin
parallelasdescribedinComplexOrganizationsinthepreviouschapter.
3.1.2 ProductDevelopmentTeam
Theproductdevelopmentteambuildsthesoftware.Generally,thisteammayincludeuserexperience
researchersanddesigners,graphicartists,requirements/businessanalysts,softwarearchitects,
softwaredevelopers,middlewareandintegrationspecialists,systemadministrators,DBAs,
release/deploymentspecialists,testersandtechnicalwriters.Inotherwords,thisteamincludesanyone
whoisinvolvedinanywayintheactualconstruction,customization,orintegrationofthesoftware.
3.1.3 ReadinessAssessors
Thereadinessassessors,astheirnamesuggests,assessthereadinessofthesoftwareforacceptance
testing.Theyprovideinformationthatisusedtomakethereadinessdecision.Thejobtitlesinvolved
dependsverymuchonthenatureoftheprojectandtheorganization,butittypicallyincludesrolessuch
asdevelopers,testers,anddocumentationwriters.Ineffect,areadinessassessorcanbeanyonewho
mightbeaskedtoprovideanopiniononwhetherthesoftwareisready.Insomecases,thisopinionis
basedonformaltestingactivities,butitmightalsobebasedontechnicalreviewsorevenqualitative
inputs.
3.1.4 AcceptanceDecisionMaker
Theacceptancedecisionmakeristhepersonorcommitteewhodecideswhethertoacceptthe
software.Inaproductcompany,ajobtitleforthisrolemightbeProductManager,butinaninformation
technology(IT)environment,thisroleistypicallyfilledbyacustomer,productowner,businesslead,or
businesssponsor.
3.1.5 AcceptanceTesters
Acceptancetestersprovidedataonacceptabilityoftheproduct.Theyperformactivitiestoassessto
whatdegreetheproductmeetstheexpectationsofthecustomerorenduser.Acceptancetestersmay
includetwoteamsonefocusingonfunctionalacceptanceofthesystem,whileanotherfocusingon
operationalacceptance.Theyprovideinformationtotheacceptancedecisionmaker.Theymaybe
dedicatedtestingstaff,endusersaskedtodotesting,oranywhereinbetweeninskillset.
3.1.6 Users
Usersmakeindividualusagedecisions.Eachuserdecideswhethertousetheproductasitiswhenitis
shippedordeployed.Theirfeedbackmightbeusedtoadjusttherequirementsforthenextreleaseorto
dousabilitytestingofthebetaversionsofthecurrentrelease.Peoplewhoareusersmayalsobe
involvedasacceptancetestersorbetatesters.Similarvitalfeedbackmightcomefromoperationsstaff,
supportstaff,trainers,fieldengineers,whoareexposedtoissuesofconsumabilityandusability.

3.2 MakingtheThreeDecisions
Thissectiondescribeshowtheprecedingsixabstractrolesareinvolvedinmakingthethreedecisions.
3.2.1 MakingtheReadinessDecision
Thereadinessdecisionismadebythereadinessdecisionmaker(s).Thereadinessdecisionisanexitgate
withadecisionaboutwhethertolettheproductbeseenbeyondtheboundariesofthesupplier
organization.Thedecisionisbasedonreadinessassessment(whichisbasedonthefeaturesincluded
andthequalityofthosefeatures)donebythereadinessassessors.Thedecisioncanbemadebyasingle
person(suchasaChiefEngineer)orbyacommittee(suchasengineers,architects,orotherproject
stakeholders).WhenitisnotasingledecisionasdescribedinChapter1TheAcceptanceProcess,each
decisionmaybemadebyasinglepersonoracommittee.Fromthepointofeachdecisionmakerthe
softwaresystemiseitherreadyoritisnotready.Ifitisnotready,theremaybealistofconcernsthat
needtobeaddressedbeforeitwillbeconsideredready.Formoreinformation,seeHowWillWe
ManageConcernssectioninChapter16PlanningforAcceptance.
Theremayhavebeenanumberofearlierdecisionmakingcheckpointsaspartofthedevelopment
process(suchas"requirementscomplete,""designcomplete,"or"codecomplete").Thesearebeyond
thescopeofthisguidebecausetheyareneitherdirectlypartofthereadinessdecisionnorarethey
easilytested.
3.2.2 MakingtheAcceptanceDecision
Theacceptancedecisionismadebytheperson(orpersons)playingtheAcceptanceDecisionMakerrole.
ThedecisionissummarizedbythequestionShouldweacceptthesoftwareandputitintouse
deliveringvaluetoourorganization?Theremaybeadditionalcontractualconsequencesformakingthe
acceptancedecision,suchasacommitmenttopaythesupplier,thestartofapredefinedwarranty
period,andsoon.Whileintheorytheseshouldnotbetheprimaryconsiderationswhenmakingthe
decision,inpracticetheyoftenare.Thedecisionshouldbewhetherthesoftwareiscompleteor
doneenoughtobedeployedorshipped.Formoreinformationaboutthedefinitionof"done,"see
Chapter7DonenessModel.Formoreinformationaboutthecompletedefinitionofthesystem
attributesthatmaybeconsideredwhenmakingtheacceptancedecision,seeChapter5System
Model.
Thedefinitionof"done"isinfluencedbyseveralfactors,includingthefollowing:
MinimumMarketableFunctionality(MMF)fortheproduct.Whatfeaturesorfunctionsmustthe
productsupporttobeworthreleasing?ThisisbasedonwhatevercriteriatheProductOwner
decidesareimportant,derivedfromproductplans,marketsurveys,competitiveanalysis,or
economicanalysis.WhiletheProductOwnerisaccountableforthedecisionthisdoesnot
removetheneedfortheProductDevelopmentTeamtounderstandtheproblemdomainin
generalandtheneedsofpotentialusers.Whereverpossible,theyshouldassisttheProduct
Ownerindefiningtheproduct.ForadefinitionoftheresponsibilitiesoftheProductOwner,see
Chapter1.

Minimumqualityrequirement(MQR)fortheproduct.Whatisthelevelofqualitythatmustbe
achievedbeforetheproductcanbereleased?Qualityhasseveraldimensions.Thepresenceor
absenceofbugs/defectsinfunctionalityisjustonedimensionofquality.Nonfunctional
requirements,alsoknownasqualityattributesisanotherdimension.TheMQRencompasses
thelatterwhileMMFencompassestheformer.
Harddeadlines.Bywhatdatemustaparticularversionoftheproductbereleasedtohaveany
value.Thesecanincludetradeshowdates,regulatorydeadlines,orcontractualobligations.
Eachdeadlinecouldbesatisfiedbyadifferentversion(orbuild")oftheproduct.Formore
information,see"ProjectContextModel."

Theacceptancedecisionismadebasedondataacquiredfromanumberofsourcesandactivities.
Acceptancetestinggeneratesmuchofthedataneededtomaketheacceptancedecision.Thisdata
includesthefollowing:
Pass/failresultsofallteststhatwereperformedaspartoftheacceptancetesting.Thiscouldverify
bothfunctionalrequirementsandparafunctionalrequirements.
FeaturecompletenessAsimpliedbythepass/failresultsoffunctionaltests

Theacceptancedecisionmayalsousedatagatheredduringreadinessassessment.Themostcommon
exampleofthisisdatarelatedtosystemperformance(capacity,availability,etc.)whichmanycustomer
organizationswouldnotbecapableofgatheringthemselvesbutwhichtheyarecapableof
understandingonceithasbeengathered.
Theacceptancedecisionisallaboutmaximizingvaluederivedfromtheproductbythecustomer
(representedbyProductOwner)andminimizingrisk.Timehasadirectvalueinthattimespent
collectingmoredatathroughtestinghasadirectcost(thecostofresourcesconsumedingatheringthe
data)andanindirectcost(thedeferralofbenefitthatcanonlyberealizedoncethesystemisaccepted).
Riskhascostthatcouldbecalculatedasthesumofthecostofallpossiblenegativeeventsmultipliedby
theprobabilityoftheiroccurrence.Thoughthiskindofliteralcalculationisnotfrequentlydoneour
perceptionsofriskareinherentlybasedonanintuitiveinterpretationofthecircumstancesalongthese
lines.Theintentisfortheacceptancedecisionmakertounderstandthetradeoffsanddecideon
whetherweneedmoredata,orwehavedoneenoughandwecanmakethedecision.Indoingthisitis
importanttobeawareofthetwoextremenegativepossibleoutcomes,oneintime,theotherinrisk.
Examplesofcostsofunmitigatedriskmightincludethefollowing:
costofpatchingsoftwareinthefield;
costofmanualworkaroundsforbugs;
costofmaintainingspecializedresourcesforsoftwaremaintenance;
losingcustomersthatneedspecificfeaturesthataremissing;
opportunitycostofunrealizedordelayedbusinessbenefitsifproductfailsusage
assessmentorendupnotbeingusedafterdeployment.

Thecostoftimecanbenonlinearinthatifadeadlineorneedismissed,allbenefitsmaybelostand
punitiveactiontaken.Theriskthecalculationisdifferentfor'blackswan'[TalebOnImpactOfImprobable]
eventsofextremelylowprobabilitybutextremelyhighimpactsuchthatagainallvaluemaybelost.
Thesecannotbeignoredintheacceptanceofnewsoftware.Inconcept,whenthecostofriskexceeds
thecostofdelay,moretestingshouldbeperformed.Whenthecostofmoretestingexceedstherisk
relatedcostthatwouldbereduced(byreducingprobabilityofoneormoreeventsoccurringorby
reducingtheexpectedcostgiventheeventdoesoccur),youcandecidetoaccepttheproductwithout
furthertesting.
3.2.3 MakingtheUsageDecision
Eachpotentialuserofthesystemhastomakeapersonaldecisionaboutwhethertousethesoftware.
Thisdecisionisdifferentfromtheacceptancedecisioninthatitismademanytimesbydifferentpeople
ororganizations.Infact,theremaybeseveraltiersofthesedecisionsascompaniesdecidewhetherto
adoptaproduct(oranewversionthereof)anddepartmentsorindividualsdecidewhethertocomply
withtheorganizationaldecision.Theimportantconsiderationfromtheperspectiveofthisguideisthat
thesedecisionshappenaftertheacceptancedecisionanddonotdirectlyinfluencetheacceptance
decision.Theymayindirectlyinfluenceitinoneofthefollowingtwoways:
Proactively.Usagedecisionsmayindirectlyinfluencetheacceptancedecisionforanupcoming
releasebyfutureuserscommunicatingtheindividualacceptancecriteriatotheproductowner
inresponsetomarketresearchorsurveys.Thistypeofcriteriamayalsobesubmittedtothe
productownerthroughunsolicitedinputs,suchasfeaturerequestsorbugreports.Also,user
feedback(includinglackofusage)fromalphaandbetareleasesmayinfluencetheacceptance
criteriaforthegeneralrelease.Seesection1.2.3TheAcceptanceProcessforAlpha&Beta
ReleasesinChapter1IntroducingtheAcceptanceProcessforamoredetaileddiscussion.
Retroactively.Usagedecisionsmayindirectlyinfluencetheacceptancedecisionbyproviding
feedbackonthereleasedproductindicatingalackofsatisfactionineitherfunctionalityor
quality.Thismayinfluencetheacceptancedecisioncriteriaofafuturerelease,butitrarely
causestheacceptancedecisionalreadymadetoberevisited.Thenotableexceptionwouldbe
thediscoveryof"severity1"bugsincriticalfunctionalitythatmightresultinarecallofthe
releasesoftware.

Theusagedecisionofteninvolvesdecidingwhetherornotthebusinessoruserisreadytostartusing
thesoftware.Thisissometimescalledbusinessreadinessanditfallsintheareaofchange
management.Oninternalinformationtechnologyprojects,theProductOwneristypicalresponsiblefor
businessreadiness.Thismightincludetrainingofusers,transforminglegacybusinessdataintoformats
compatiblewiththenewsystemorpreparationofnewdatareadytobeloadedwhenthenewsystem
becomesavailable.Whileitisanimportanttopic,itis,strictlyspeaking,notpartofacceptingsoftware
andisbeyondthescopeofthisguide.
Sidebar:ThePerilsofIgnoringtheUsageDecision

Acompanyoneoftheauthorsworkedforimplementedapopulartoolformanagingleads.The
ProductOwnerTeamconsultedwiththekeystakeholderswhowouldbeconsumingthereportson
potentialfuturebusinessinthesalespipelineandconfiguredthesystemwiththefieldneededto
generatingthereports.Mostofthefieldsweremarkedasmandatory.
Whenthesystemwasreadyforacceptancetesting,theProductOwnersteamtestedthesystemand
foundittobeworkingtotheirsatisfaction.Theyconductedextensiveusertraingingofthesalesforce
toensurethateveryoneknewhowtousetheirwonderfulnewtool.Theyacceptedanddeployedthe
system.Whenthesystemwentlive,theuseruptakewaspoor.Severalyearsafterdeployingthe
system,mostuserswerestillchoosingtonotusethetoolbecauseitwasprovidingthemwithvery
littlevaluewhileimposingconsiderableoverheadontheirdaytodaywork.Mostofthemdeveloped
theirquotesinspreadsheetsorinacustombuiltpricingapplicationthatdirectlyaddressedtheir
needs.Theleadmanagementtooldidntprovidetheequivalentfunctionalitythereforeitadded
additionalstepsreenteringdatabutindifferentformats.Tomakemattersworse,itforcedtheusers
topickthroughalonglistofpossiblecustomerseachtimeauserneededtocreateaquote.Asaresult,
thesystemlanguishedandmuchofthepotentialvaluewasalostopportunitycost.
TheProductOwnerofasubsequentprojecttoreplacethecustompricingtooltookthelessonsto
heartandincludedfunctionalitytoaddresstheshortcomingsbyfeedingthepricequotesintothelead
managementsystemandsynchronizethecustomersautomatically.Thissatisfiedtheneedsofthe
reportgenerationstakeholdersbyprovidingthemwithallthedataneededtocalculatethepresent
valueofleadsinthesalespipelinewhilenotburdeningtheuserswithduplicatedataentry.Asaresult,
usageoftheleadmanagementtoolsoared.Ahappythoughbelatedending.
3.3 Rolesvs.Organizations
Therolesdescribedinthisdecisionmakingmodelmaybeplayedbypeopleinseveraldifferent
organizations.Theprimaryvalueofdiscussingorganizationhereisinmakingiteasiertomap
terminologyfromvariousorganizationmodelstobetterunderstandwhoplayswhichdecisionmaking
role.Iftheorganizationalmodeldoesnothelpinthisendeavour,itcanbeignored.
Whenthesoftwareisbeingbuiltbyadifferentorganizationthantheonewhocommissionedits
construction,theorganizationthatcommissionedthesoftwareisoftenreferredtoasthecustomer,and
theorganizationthatisbuildingthesoftwareisthesupplier.Thisistruewhethertheorganizationsin
questionareseparate,unrelatedcompaniesorsimplydepartmentswithinasinglecompany.For
example,theITdepartmentistypicallyasupplierofsystemstothecorebusinessdepartments(suchas
TransportationorManufacturing)andsupportingdepartments(suchasHumanResourcesorFinance.)
Whenacceptancetestingisoutsourcedtoathirdpartytestorganization,itisoftenreferredtoasthe
(thirdparty)testlab.Thetestlabisasupplierofservicesasdistinguishedfromthesupplierofthe
software.
Anorganizationthatbuysanddeploysshrinkwrappedsoftwarecanalsobereferredtoasacustomer,
andtheorganizationtheybuyitfrommaybereferredtoasthevendororsupplier.Thefactthatthe

vendorcontractstheworktoanoutsourcer(anothervendorofwhichtheyarethecustomer)illustrates
theproblemwithusingtheterm"customer"todescribetheacceptancedecisionmaker(theproduct
owner)asadvocatedinextremeprogrammingwhichcustomerisitreferringto?
FigureXillustratesthisproblem.ThePurchaser(intheroleofcustomer)buysshrinkwrapfromS/W
Vendor(inroleofsupplier).S/WVendor(intheroleofcustomer)outsourcesdevelopmenttoS/W
Developer(thesupplier).S/WDeveloper(ascustomer)outsourcesreadinessassessmenttoTestLab(as
supplier).Sowehavethreeseparatecustomersupplierrelationshipsinthisscenariomakingithardto
tellwhichpartywearetalkingaboutwhenwerefertothecustomer.

FigureX
MultipleCustomersandSuppliers
3.3.1 WhoPlaysWhichRoles?
Thusfarthediscussionshavecenteredontheabstractrolesinvolvedinthedecisionmakingprocess.But
whoactuallyplaystheroles.
WhoPlaystheReadinessDecisionMakingRole?
TheReadinessDecisionMakerroleistypicallyperformedbysomeoneintheproductdevelopment
organization.Theyarethepersonwhoultimatelydecideswhetherthesoftwareisreadytobeshownto
theacceptancetesters.TypicaljobtitlesincludeDevelopmentLead,ProjectManager,Directorof
Development,ChiefEngineer,etc..Onagileprojectswherethisdecisionismadeseparatelyforeach

feature,thefeaturelevelReadinessDecisionMakerisoftenthedeveloperwhosignsupforthe
responsibilitytoimplementthestory/feature.Intheterminologyusedinthisguide,theReadiness
DecisionMakerisamemberoftheProductDevelopmentTeam.
WhoPlaystheAcceptanceDecisionMakingRole?
TheAcceptanceDecisionMakerroleistypicallyperformedbysomeoneinthebusinesssideofthe
organizationwhendevelopmentisdoneinternallyorbysomeoneinthecustomerorganizationwhen
developmentisoutsourced.Theyrethepersonwhoultimatelydecideswhetherthesoftwareisreadyto
beshowntotheacceptancetesters.TypicaljobtitlesforinternallydevelopedsoftwareincludeBusiness
Lead,ProductManager,orsomekindofbusinesstitle.Onagileprojectswherethisdecisionismade
separatelyforeachfeature,thefeaturelevelAcceptanceDecisionMakerisoftenthespecificbusiness
subjectmatterexpertwhoprovidedthedetaileddescriptionofthefunctionalitytobebuilt.Inthe
terminologyusedinthisguide,theProductOwneristheAcceptanceDecisionMaker.
WhoDoesWhatTesting?
ReadinessassessmentistheresponsibilityoftheProductDevelopmentTeamButdoesthismeanthat
developersaretheonlyonesdoingreadinessassessment?Bythedefinitionusedinthisguide,the
ProductDevelopmentTeamoftenincludespeoplewhowoulddescribethemselvesastesters.
Developerswoulddoataminimum,unittesting.Eithertestersand/ordeveloperswoulddowhole
systemtesting(functionalandnonfunctional).Otherpartieswhoarealsoconsideredpartofthe
ProductDevelopmentTeammaydootherformsofreadinessassessment.Forexample,dataarchitects
mightdodataschemadesignreviews.Securityarchitectsmightdosecuritycodereviews.
Ideally,acceptancetestingshouldbedonebyrealusersofthesoftwaresincetheyaretheoneswhoare
bestsuitedtodecidingwhetheritisacceptable.Butwhatiftherealusersareanonymous?Whocan
andshoulddoacceptancetestingontheirbehalf?Thisisacasewheretestersoftenactastheproxyfor
theenduser.InthissituationatleastsomeofthetesterswouldbepartoftheProductOwnerTeam
ratherthantheProductDevelopmentTeam.
InFigureX2WhoDoesWhatTestingbyTypeofOrganization,eachcelldescribeswhodoeswhich
testinginacommonprojectcontext.Thetoplefttriangleinthecellindicateswhoisdoingreadiness
assessmentandthebottomrighttriangleindicateswhoisinvolvedinacceptancetesting.

FigureX2
WhoDoessWhatTestin
ngbyTypeof
f Organization
n
Whensofftwareisbeingbuiltbyanorganizationforitsownuse,theusersshouldbeavvailablefor
participattingintheaccceptancetestting.Ifthereisnotestingffunction,thebusinesstypicallyexpects
developmmenttotestth hesoftwarethoroughlybeeforehandinggitover.Iftheereisatestin
ngfunction,th
hey
nd
wouldtyppicallyparticippateina2 pphaseofreaddinessassessm ment,oftencalledsystem
mtestingand d
integrationtestingbeforehandingthesoftwarreovertotheeusersforuseracceptancetesting(U UAT)
orbusineessacceptanccetesting(B BAT).
Whensofftwareisbeingbuiltforsalletoanonymouscustomers,realuserstypicallydon nttakepartinn
acceptanccetesting(exxceptperhapssasusersofaalphaorbetareleases.)Insstead,thetesstingorganizaation
doesacceeptancetestinngontheirbeehalfactingassaproxy(orsurrogate)usser.Ifthereissnospecializeed
testingfunction,theaccceptancetesstingcouldbeedonebytheeproductspeecifiers(membersofthe
ProductOOwnerTeam)ortheaccepttancetestinggcouldbeouttsourcedtoa3rdpartytesttlab.
Intheterm
minologyuse
edinthisguid
de,testersdoingReadinesssAssessmenttarememberrsoftheProd
duct
Developm mentTeamwh hiletestersdo
oingacceptanncetestingarrememberso
oftheProducctOwnerTeamm.
3.4 Summary
y
TheReadiinessDecision nismadebysomeoneorssomecommittteefromtheesupplierplayyingtheroleof
ReadinesssDecisionMaakerbasedon ninformation ngatheredbyyReadinessAsssessors.ReaadinessAssesssors
typicallyincludedeveloopers,architeectsandsometimestesterrs.Thegoalo
oftheReadinessDecision
Makeristtoensurethaattheproductthasenoughfunctionalityyandisofgooodenoughqu ualitytoexpo
oseto
theProdu uctOwner.ThheRDMistyp picallyasenio
orpersonintheproductdevelopmenttteams
organization.
didate1 80

TheAcceptanceDecisionismadebytheProductOwner(whichmaybeasinglepersonoracommittee
ofseveralkeystakeholders)playingtheroleofAcceptanceDecisionMakerbasedoninformation
gatheredbypeopleplayingtheroleofAcceptanceTesteraswellasanyReadinessAssessment
informationsharedbythesupplier.TheADMisusuallyakeypersoninthecustomerorganization.The
acceptancetestersmaybeendusers,representativesoftheenduserssuchasthespecifiersofthe
requirements,ortesters)whenendusersarenotavailabletodoacceptancetesting.Whenthe
customerorganizationshascommissionedtheproductionofaproductandtheyhavenointernaltest
capability,theymaychoosetooutsourcesomeoftheacceptancetestingtoathirdpartytestlab.
3.5 References
[TalebOnImpactOfImprobable] Taleb, Nassim Nicholas.TheBlackSwan:TheImpactoftheHighly
Improbable.RandomHouse.2007

Chapter4 ProjectContextModel
TheProjectContextModelisawaytobetterunderstandthegoalsoftheprojectandtheconstraints
underwhichitmustoperate.Itisnotaformalmodel;instead,itisasetofinformationtobecollected
andfactoredintootheractivities.Itincludesinformationsuchasthefollowing:
UsageContext.Whatistheoverallcontextinwhichthesoftwarewillexist?
Businessgoals.Anexamplequestiontogatherinformationfortheseis,"Whatisthebusiness
valuetobeprovidedbythesystemandhowisittobeachieved(strategy)?"
Scope.Anexamplequestiontogatherinformationforthisis"Whatfunctionalityisinscopeand
whatisoutofscopeforthisproject?"
Stakeholdersandusers.Anexamplequestiontogatherinformationfortheseis"Whoarethe
intendedusersandwhoaretheotherprojectandsystemstakeholders?"
Budget.Anexamplequestiontogatherinformationforthisis"Howmuchmoneyisavailableto
achievethebusinessgoal?"
Harddeadlines.Anexamplequestiontogatherinformationfortheseis"Whatdeadlinesmust
bemetfortheprojecttobeconsideredasuccess?"Examplesofharddeadlinesinclude
tradeshows,contractualdeadlines,andregulatorydeadlines.
Constraints.Anexamplequestiontogatherinformationfortheseis"Whatresources(suchas
people,space,andequipment)areavailabletotheproject?"Afollowupquestionmight
include"Whichresourcesarenegotiableandwhicharehardconstraints?"

Thesectionsofthischapterprovidemoreinformationabouteachofthesefactors.Othercontextual
factorsnotdiscussedinthischapteralsohaveaninfluenceonsoftwareacceptance.Thesefactors
include:
Criticalityofthesystem,whichaffectsdefecttolerancelevels,howcomprehensiveacceptance
testingshouldbe,andwhatconstitutesgoodenoughquality(Cockburn[Cockburn],for
example,definesfourzonesofcriticalityofthesystembasedonthefailureimpact:Lossof
Life(clearlythehighest),LossofEssentialMoney,LossofDiscretionaryMoney,orLossof
Comfort.Boehm&TurnerscriticalityscaleissimilarbutdistinguishesLossofManyLives
fromLossofSingleLife[BoehmTurner]);
ProcessesandToolsavailableandusedintheproject,suchascontinuousintegration,
automatedbuilds,andsourcecodemanagementtools,whichaffectthetiming,style,
amount,andfrequencyofacceptancetesting;and
Trust,whichaffectshowmuchandwhatkindofretestingtoperformbasedontestingdataand
resultsfrompreviousstagesorfromotherteams(e.g.,howmuchdoestheproductowner

performingacceptancetestingtrusttheproductdevelopmentteamperformingreadiness
assessment).
4.1 UsageContext
Softwarecanbebuiltformanydifferentkindsofusageandtheorganizationthatbuildsorcommissions
itaffectshowthesoftwareisbuiltandaccepted.Figure1TheProductContextillustratesonewayof
classifyingthesoftwareintensivesystemswebuild.

Figure1
TheProductContext(courtesyofJeffPatton;usedwithpermission)
Someorganizationsbuildsoftwarefortheirowninternalusagetoreducecostswhileothersbuild
productstoselltootherstogeneraterevenue.Theleftsideofthequadrantdiagramdescribeswhatis
commonlycalledenterprisesoftware;softwarethatisbuiltbytheenterpriseforitsownuse.Theright
sideofthediagramrepresentssoftwarethatisbuiltforsaletootherpartieseitherassoftwareinits
ownright(whichmaybeinstalledonsiteorsoftwareasaservice)orassoftwareembeddedina
hardwareproduct.Someoftheproductsbuiltontherightsideofthediagramshowupastools,
components,frameworksorproductsusedintheconstructionoftheenterpriseproductsontheleft
sideofthediagram.Independentlyofthis,theusersofthesoftwaremaybecaptiveuserswhoare

compelled dtousethep
productonceitisavailableeortheymayychoosetoussetheproducctoracompeeting
product.TTheprocessoofacceptingssoftwarecanvvarybasedon nthiscontextt.
4.1.1 Enterprise
E Systems
Enterpriseesystemsare ecommissionedbyanenteerpriseforitssowninternaaluse.Thesysstemsautomaate
allorparttofvariousbu
usinessproceesses.Theuseecasesofeacchsystemimp plementoneormorestep psofa
businessp process,eitheerinwholeorrinpartasillustratedbytheleftsideofFigure2.Th
heymaybebu uilt
fromscratchbywritinggcode,comp posedofpurchasedcompo onentsorcreaatedbyconfigguringapackkaged
productp purchasedfromavendor(ontherightssideofFigure2.)Theworkkmaybedoneebyapartoffthe
enterprisee,oftencalleddtheInformaationTechnollogy(IT)orInformationSyystems(IS)deepartment,orrthe
developm mentmaybeo outsourcedtooathirdpartyy.Manyofth hesesystemstendtobeintegratedwith h
othersysttemswithintheenterprisee(whichoften nincludelegaacysystems)andalargepartofthe
challengeofbuildingandacceptinggthesesystem msisensuringgthattheinteegrationworkkscorrectlyas
dbytherightsideofFigure2.
illustrated

Figure2
BusinessPProcessvs.SyystemsIntegra
ationView
Theintegrationrequireementsmayttakemanyformsincludinggdatareplicaation/synchroonizationand
transactio
onalinteractio
onstonamejjustafew,bu
utallshouldb
bedrivenfrom
mthebusinesssrequiremen
nts.
Acceptancceofenterprisesystemsinnvolvesdeterrminingwhettherornotthesystemwillmeetits
businesseenablemento orcostreductiontargets.
Figure2aalsopointsou
utthecauseo
ofmanypotentialissues:aacceptanceissdonefromabusinesspro
ocess
perspectivvewhileman nyProductDeevelomentTeamsarefocusedexclusiveelyonasubseet(oftenjustone)
ofthesystemsthatimplementtheprocess.Thisscanleadtom mismatchesb betweenbusinessexpectattions
didate1 84

andwhattheITdepartmentdelivers.ThisisespeciallytruewhenITisorganizedaroundthesystems(as
componentteams)ratherthanbusinessprocesses(asfeatureteams)(seediscussiononcomponent
teamsvsfeatureteamsintheAcceptingComplexProductsectionofChapter2)orwhendevelopmentof
thesystemisoutsourced.Thelatterisparticularlyfraughtwithperilastheoutsourcermaynot
understandanythingaboutthebusiness.
Becausethesoftwareiscommissionedbytheenterpriseitself,thereisthepotentialforalargedegree
ofbusinessinvolvementinthedevelopmentprocess.Thiscanresultinsoftwarethatisaverygoodfit
forthebusiness.Unfortunately,manybusinessesdelegatethehardworkofdefiningtheirrequirements
totheproductdevelopmentteam,whetherinternaloroutsourced,andeffectivelyabdicatecontrolover
whatisbuilt.Thisoftenresultsinbarelyusablesystemsthathavelong,painfulacceptancephases.A
goodwaytoavoidthisistohavestrongbusinessownershipandstewardshipofthedevelopmentof
thesesystemsintheformofaninformedproductownerwhomakesthecriticaldecisionsaboutwhat
thesystemsshouldsupportandwhatitshouldnot.Thispositiongoesbyvariousformaltitlessuchas
businesssponsor,businessleadorproductowner.Ifacriticaldecisionrequirestechnicalknowledgeand
theproductownerisnottechnicallysavvy,theproductownerneedsassistancefromtheproduct
developmentteamoroutsideadvicetomakethedecision.
ThesoftwareinthebottomleftquadrantofFigure1istypicallyembeddedinabusinessworkflow
implementingoneormanystepsofthebusinessprocess.Theusersdonothaveachoiceofwhetheror
nottousethesoftwarebecauseusingitispartoftheirjob.Usersarehighlyaccessiblethereforecanbe
usedforusabilitytestingandacceptancetestingbutbecauseusageisnonoptionalthereisan
unfortunatetendencyoftheproductownertoignoretheusabilityattributesofthesystem.Theusers
areofteninvolvedintheacceptancetestingprocessinanactivityoftencalleduseracceptancetesting
(UAT).
ThesoftwareinthetopleftquadrantofFigure1isdifferentinthatusersarenotcompelledtousethe
producteventhoughitwasbuiltforinternaluse.Theproductownertriestorepresenttheneedsofthe
realuserwhomayormaynotbeaccessible.Forexample,thesoftwaremaybeusedprimarilybyin
houseuserswhoareaccessible,oritmaybedesignedfortheuseofpeopleoutsidethecompany
includingcustomersorbusinesspartners.Thesystemmaybestandalonebutismorelikelytobe
integratedbehindthesceneswithothersoftwarewithintheenterprise.
4.1.2 SoftwareIntensiveProducts
Manyproductsinthemodernworld,fromtoystoautomobiles,haveasoftwarecomponentandthe
amountofsuchembeddedcomponentsissteadilyincreasing,especiallywithregardstotheportionof
thevalueaddedtothefinalproduct.Thissoftwareiswhatprovidesthecomplexbehaviorsthat
characterizeproductsinthe21stcentury.Theseproductshavebeenalargepartofthegrowthof
moderneconomies.Acceptanceoftheseproductsinvolvesdecidingwhethertheproductasbuiltwill
meetitsrevenuetargets,contributeordetractfromthebrandimage,andpossiblyotherfactors.The
productrequirementsaretypicallydrivenbyaproductmanager.
ThesoftwareinthebottomrightquadrantofFigure1isbuiltbyonecompanyforsaletoanother
company.Thissoftwaremaybepurchasedandusedasis.Oritmaybedesignedtobeconfiguredor

customizedbyorforthecustomertofitwithintheirenterpriseenvironment(ontheleftsideofthe
diagram.)Thenotionofacceptancethenhasaninterestingtwistinthatthegenericproductmustfirst
beacceptedbytheproductmanagerbeforebeingmadeavailabletothecustomerswhomaythenhave
itconfiguredorcustomizedbeforeacceptingitfordeploymentwithintheirorganizations.Thisis
describedinmoredetailinsectionAcceptingCustomizableProductsinChapter1TheAcceptance
Process.
Thesoftwareinthetoprightquadrantmaybepurelysoftwareforusebyconsumers(socalledshrink
wrappedsoftware)oritmaybeembeddedinahardwareproduct.Ineithercasetheenduseristoo
numerous(andisoftennotavailable)toinvolveintheacceptanceprocess.Notethattheremaybe
additionalspecializedtechniquesusedforbuildingandassessingembeddedproductsandthese
specializedtechniquesareoutofscopeforthisguidebutmanyofthetechniquesdescribedheredo
applytoembeddedproducts.
4.2 BusinessGoals
Businessgoalsaffecthowthesoftwarewillbeaccepted.Toomanyprojectsarerunwithamajorityof
thestaffnotunderstanding(orevencaringabout)thebusinessgoalsoftheproject.Sometimesthisis
theresultofdeliberatedecisionsbymanagementtowithholdinformation,andsometimesitisjustan
oversight.Eitherway,weshouldexpecttogetsuboptimalresultsifeachteammemberisfocusedon
optimizinghisorherownjobassignmentinsteadofensuringthatbusinessgoalsareachieved.
Asabareminimum,weshouldensurethateveryoneontheprojecthasaclearunderstandingofwhat
theprojectisexpectedtodeliverandhowthatwillprovidevaluetothebusiness.Exampleofdifferent
waystoaddvaluetothebusinessincludereducedcosts,increasedsales,moresatisfiedusers,and
improvedmarketperception/branding.
4.3 Scope
Scopedetermineswhatneedstobeincludedbeforetheproductwillbeaccepted.Thescopeofthe
projectshouldrealizethebusinessgoals.Atthebroadestlevels,thescopecanbedefinedintermson
thetypesofusersweexpecttosupportandthetypesoffunctionalitythattheproductwillsupplyto
them.Itisasimportanttoexplicitlyindicatewhatisoutsidethescopeasitiswhatisinside;otherwise
weriskwastingtimeandenergydiscussing,andpossiblybuildingandtestingfunctionality,thatis
supposedtobeexcluded.Whentherequirementschangeorevolveafterdevelopmentstartsandthe
scopeneedstoberevisited,thedecisiontochangealreadyscopedfunctionalityorexpandittoinclude
anypreviouslyexcludedfunctionalityshouldbeintentional,rational,andexplicit.Animportantquestion
theProductDevelopmentTeamshouldasktheProductOwnerinthatcaseiswhatexisting
stories/featuresintheproductbacklogshouldbereplacedwithanewstory.

4.4 StakeholdersandUsers
Stakeholdersandusersaffectwhoplayswhatroleintheacceptanceprocess.Acceptingsoftware
requirestheinvolvementofvariousstakeholders.Somearedirectlyinvolvedintheacceptancetesting
anddecisionmakingprocess,whileothersneedtohavetheirinterestsprotectedeventhoughtheyare
notinvolved.
4.4.1 Users
Themostobvioususagegoalholdersarethepeople(orsystems)whoaretousethecorefunctionality
ofthesystem.Typically,thesepeoplearereferredtoastheusers.TheyaretheoneswhouseaWeb
sitefordoingtheirbanking,entertainment,oronlineshopping;useanapplicationforexecutingoneor
morestepsofabusinessprocess;oroperateacombinedsoftware/hardwareproductsuchasamedical
imagingsystem.Butthesearenottheonlypeopleinteractingwiththesystem.Otherusersincludethe
peoplewhoadministerthesystembypopulatingthecatalogoftheonlinestoreorsetupthecontentin
anonlineentertainmentsystem;thepeoplewhorundiagnosticsonandmaintainthesystem.Thereare
alsopeople(orsystems)whoinstallthesystem,startit,monitoritsstatus,andshutitdownwhenthe
serversneedmaintenance.Theyallhaverequirementswithrespecttohowtheyusethesystem.For
moreinformationaboutusersandtheirgoals,seetheUserModelingactivityinVolume2ofthisseries.
4.4.2 SystemStakeholders
Therearealsostakeholders[ACUC]whowillnotdirectlyinteractwiththesystemwhenitisinoperation
butwhoexpectthesystemtolookoutfortheirinterestsasitisused(orabused)byothers.For
example,asystemthatcontainspersonalinformationaboutsomeonehasthatpersonasastakeholder
eveniftheythemselvesdonotdirectlyinteractwiththesystem.Thisisanexampleofanonfunctional
requirement.
4.4.3 ProjectStakeholders
ProductsandITsystemsareusuallydevelopedwithinaproject.Theremaybemanydefinedroleswithin
theproject,someofwhichmaybeplayedbyusersorsystemstakeholdersandsomebyuniqueparties.
Theseprojectstakeholdersmaybeinvolvedintheacceptancetestingoracceptancedecisionmaking
processwithoutbeingadirectsystemstakeholderoruser.Forexample,theproductmanageror
productownermayneverusethesystem,norbeasystemstakeholder,buttheydefinitelyhaveastake
intheacceptancedecisionprocess.ThebusinesssponsorofanITprojectmayneverusetheproductin
thefieldorevenviewareportitgenerates,butthepersoninthisrolehasaclearstakeintheprojects
outcomeintheformoftheexpectedbusinessbenefitsinreturnfortheinvestmentoftimeandmoney.
4.4.4 CommunicationBetweenStakeholders
Thelikelihoodofacceptanceofasoftwareintensivesystemisdirectlyproportionaltotheeffectiveness
ofthecommunicationbetweenthosewhocommissionedthesoftware(theproductownerorthe
productownerteam)andtheteambuildingthesoftware(theproductdevelopmentteam).The

requirementsandthetestsarepartofthiscommunication.Thecommunicationismademuchmore
effectiveifthereisacommonlanguagethateveryoneagreestouseconsistently.
Thecommonlanguagethateverybodyintheproductdevelopmentteamandtheproductownerteam
sharesisoftenreferredtoasaUbiquitousLanguage[EvansOnDomainDrivenDesign].Themembersof
theproductownerteamarethedomainexpertsandarticulatetherequirementsinthelanguageofthe
domain.Howeverthislanguageshouldbeaccessibletoandunderstoodbytheproductdevelopment
team.Conversely,thetechnicaljargonthatthedevelopmentteamusestoexpressgeneraldevelopment
conceptsrelatingtothecustomerfacingaspectsoftheproject(inthedesign,architecture,testing,and
acceptanceoftheproduct)shouldnotneedtobeaccessibletoandunderstoodbytheproductowner
team.TheUbiquitousLanguageofthewholeteamshouldbedevelopedthroughcloseinteractionacross
andwithintheproductowneranddevelopmentteamsearlyintheprojecttobeeffectiveinreducing
thecommunicationoverheadandavoidingcostly,latemisunderstandings.
4.5 Budget
Mostprojectshaveafixedbudget.Thistendstoconstrainthenumberofpeoplewhowillworkonthe
projectbothinnumberandinrateofpay.Oncetheteamsizeandcompositionisdefined,theteamwill
exhibitawelldefinedburnrateandanyextensionoftheprojecttimelinesduetolatedeliveryofthe
softwarewilltranslateintobudgetoverrunsunlessasignificantbufferisheldbackwhenplanningthe
project.
Anincrementalfundingapproach[DenneClelandHuang]isaviablealternativetoafixedbudget,and
providesflexibilityincostmanagement,scheduling,resourceallocation,teamcomposition,andscoping.
Incrementalfundingreliesonprioritizingandschedulinggroupsofusable,highvaluefeaturesearlyin
theprocessandcommittingresourcesforsuchchunksoffunctionalityinapiecemealmanner.
Incrementalfundingisparticularlysuitableforprojectsthatexhibithighcost,schedule,businessand
technical,uncertainty.
4.6 HardDeadlines
Allprojectsfacedeadlinesofonesortoranother.Someprojectsroutinelymissmanyofthesedeadlines.
Whenthebusinessconsequenceofmissingadeadlineissignificant,weshouldtaketheappropriate
measurestoensurethatthedeadlinesarenotmissed,suchasaplanningcontingencybufferorcutting
scope.Therefore,itisessentialtounderstandwhichdeadlinesarearbitrary(suchasin"Itwouldbe
reallygreatifwecouldhavethatfunctionalitybySeptember")orcritical(suchasin"Weneedto
demonstratethisfunctionalityatthetradeshowinSeptemberandfailuretodosocouldsignificantly
affectourfourthquartersalesandourshareprice.")Intheformercase,missingthedeadlinemayresult
inslightorevennoinconvenience.Inthelattercase,thevalueofthedeliverycoulddiminishtozero
makingtheproductandtheprojectthatdeliversitcompletelyirrelevant.

4.6.1 RecoveryfromDevelopmentScheduleSlippage
Amajorissuewithharddeliverydeadlinesrelatestotheperceivedcompressibilityoftheacceptance
testingphase(andtestingingeneral.)Thisisparticularlyanissuewhenthedeliverydateisfixedand
immovable.Whenthedevelopmentphaseisrunninglate,asitoftendoesonwaterfallproject,thereis
greatpressureontheentireteamtorecoverthescheduleslippagesomehow.Therearetwotechniques
commonlyemployed,eachwiththeirowndifficulties.
CompressingtheTestCycle
Readinessassessmentandacceptancetestingwilltypicallystartlatebutisoftenexpectedtomakeup
thescheduleslippagebyreducingthelengthofthetestcycle.Thisresultsinlesstestingthanplanned
andthereforelikelymorebugsslipthroughtoproduction.Inotherwords,theacceptancedecisionis
madebasedonincompletedataaboutthequalityoftheproduct.
OverlappingTestCyclewithDevelopment
Theotherschedulerecoverytechniquethatisoftenusedwithharddeadlinesistoaskthetestersto
starttestingtheproductbeforeitiscompletelyfinished.Whilethispractice,IncrementalAcceptance
Testing,isconsiderednormalonanagileproject,aprojectplannedasastrictlysequentialproject
cannotadoptthispracticeeasilymidproject.Forexample,manyfeaturesmaybe80%doneratherthan
80%ofthefeaturesbeingcompletelydoneandtestable.Thereforeitishardforthetesterstoknow
whattotestbecausethereisntaclearlistofwhatshouldbeworkingnow.Anotherissueisthat
IncrementalAcceptanceTestingusuallyincludesextensiveautomatedregressiontestingsothatthe
testerscanfocusonthenewlycompletedfunctionalityratherthanhavingtomanuallyretesteverything.
Theautomatedregressiontestingisalsoakeyformofbugrepellentusedbythedeveloperstoavoid
introducingregressionbugsbydetectingthesebugsaspartofreadinessassessmentsotheycanbefixed
beforethenewreleasecandidateispassedontothetesters.
4.7 Constraints
Mostprojectsoperateundersomekindofconstraints.Constraintscanincludepeopleandskillsor
facilitiesandequipment.Sometimestheseconstraintscanbeloosened,whiletheyarestrictlylimitedat
othertimes.Acompanymaynotbeabletohireadditionalstafforrecruitpeoplewithspecificskills.For
example,theskilllevelofthestaffperformingtestingmayhaveasignificantinfluenceonthetesting
approachesused,theamountoftestingdone,andthetimeittakestotest.Inthesecasesweneedto
createthebestpossibleplanthatallowsustomeetourgoalswithouthiringadditionalstaff.Thismay
resultinverydifferentplansthanifwewereabletohireadditionalstaff.
4.8 Resources
[ACUC]AlistairCockburnWritingEffectiveUsesCasesAddisonWesley

[BoehmTurner]BarryBoehmandRichardTurner.BalancingAgilityandDisciplineAddisonWesley
[Cockburn]AlistariCockburnMethodologyperproject,Figure9,
http://alistair.cockburn.us/Methodology+per+project,Oct19,2009
[DenneClelandHuang][DenneClelandHuang0]MarkDenneandJaneClelandHuang,"TheIncremental
FundingMethod:DataDrivenSoftwareDevelopment,"IEEESoftware,vol.21,no.3,pp.3947,
May/June2004
[EvansOnDomainDrivenDesign]EricEvans,DomainDrivenDesign,AddisonWesley,2003.

Chapter5
5 System
mReq
quirem
mentssMod
del
Theproceessofacceptinngasoftwareeintensivesyystemisinherrentlydepend dentonthesyystemtestedaand
thetestca
asesused.Yo
ourtestcasesaredesigned dtotestforsp
pecificrequireements.Thereeforebefore
designinggyourtestcassesitisimperrativeyoukno
owtherequirrementsthattheyareinten ndedtotest.
5.1 Requirem
mentsand
dAccepta
ance
Theaccepptanceofaso
oftwareintennsivesystemisclearlyrelattedtowhetheeritmeetsth
herequirements.
Assessingwhetheritmmeetstherequirementsinvvolvestheprocessoftestiingthesoftwareusingtestt
casesthattareinsomewayrelatedtotherequirements.Therrefore,requirrementsareaanimportant
componentoftheacce eptanceprocessevenifth
heyarenotdirectlyatestin
ngrelatedarttifact.
Thetermrequiremen ntsissomewwhatcontentio ous.Somepeeoplebelievethatyoumerrelyneedtotalkto
potentialusersandaskkthemfortheirrequiremeents.Frequen ntly,thisisrefferredtoasrrequirements
elicitation
norrequirem
mentsgatherinng.Thisisillu
ustratedinFiggure1.

Figure1
Gathering gRequiremen
ntsdirectlyfro
omusers
1weseetherequirementsfromtheussersbeingtranslateddirecctlyintoprodu
InFigure1 uctbythepro
oduct
developmmentteam.Ineffect,thereequirementsbecometheccoarsegraineedworkitemssfortheprodduct
developmmentteam.Initerativeand dincrementalprocesses,th quirementata
heproductisbuiltonereq
time,indicatedherebyythelineswithintheprod
duct.Therem
maybeanopp userstoprovide
portunityforu
feedbackononeversioonoftheproductbeforeaadditionalreq
quirementsarreimplementted.
Somepeo oplebelievethatrequirem mentscannotbegatheredllikestrawberriesormushrrooms;insteaad,
theybelieeverequireme entsmustbebasedontheedefinitionofaproductth hatisdesigneedtomeetthe
potentialusers'needs..Thisguidesu ummarizesthhisprocessassproductdesignwhichacttsasaplaceholder
forawideerangeofacttivitiesthatm
mayinvolvesp
pecializedpro
oductdesignsskills.Thisprocessisillusttrated
didate1 91

inFigure2.Foraserioustreatmentofthetopicsofproductdesignandexperiencedesignsee[HSPK]as
wellaseffectivelowfidelityproductexperiencedesigntechniquesby[BuxtonOnDesignForUsability].
Figure2
GettingRequirementsviaProductDesign
InFigure2weseetheuserrequirementsbeingaugmentedwithobservationsfromresearchers.These
observationsandrequirementsareinputintotheproductdesignprocessexecutedbytheproduct
designteamwhichmaybethedevelopmentteamoraseparategroupofspecialists.Theresulting
productdesign(oftenexternalizedintheformofprototypes)isthentestedwithpotentialuserstofind
outhowwellitsatisfiestheusersneeds.Thetestresultsarethenincorporatedintotheproductdesign.
Severalrefinementcyclesmaytakeplace.Theactualconstructionoftheproductisbasedontheresults
ofthisfeedback.Initerativeandincrementalprocesses,theproductisbuiltonerequirementatatime,
indicatedherebythelineswithintheproduct.Theremaybeanopportunityforuserstoprovide
feedbackononeversionoftheproductbeforeadditionalrequirementsareimplemented.
Theproductdesignapproachinvolvesexplicitactivitiestovalidatethattherequirementstrulysatisfy
theneedsoftheusers.Thisvalidationcanbedoneaspartofanacceptancetestingphase,butthatis
ratherlatetodiscoverthatwhatyoubuiltisgoingtorequiresignificantchangesbeforeitwillmeetthe
users'needs.Therefore,weadvocateacceptancetestingoftheproposedproductdesign(notthe
softwaredesign)beforethesoftwareisbuilt.Techniquessuchasexperiencesketching
[BuxtonOnDesignForUsability],paperprototypingandWizardofOztesting

[GouldLew wisOnDesignFForUsability]canbeusedttoverifythatyouare"buildingtherigh htsystem"very

earlyinth hilethereissttilltimetoadjjusttheproductdesign.Fo
heprojectwh orinformatio
onaboutthesse
techniquees,seetheUssabilityTestin
ngthumbnailinVolumeIV.
5.2 TypesofRequirem
ments
Therequirements,how weverderivedd,aretypicalllydividedinto
otwobroadccategories,fu
unctional
requiremeentsandnonfunctionalreequirements(alsoknownaasparafuncttionalrequireements,extra
functionalrequirements,systematttributesorqu nalrequiremeentsdescribethe
ualityattributtes.)Function
functionalitytobeprovidedtouserrsoradministtratorsoftheesoftwareinttensivesystemm.Nonfunctional
requiremeentstranscen ndthefunctio
onality.Thereelationshipbeetweenthetwwokindsofreequirementsis
illustrated
dinFigure3.

Figure3
Functionaalvs.nonfuncctionalrequirrements
mentsandforverifyingthat
Differenttechniquesareusedfordescribingthevariouskindssofrequirem
thoserequirementsarremet.
5.2.1 Functional
F Requirem
ments
Functionaalrequiremenntsdescribeh
howvarioustyypesofuserssexpectthesystemtohelp
pthemdotheeir
jobs.Thefunctionalrequirements,wwhetherexplloredandgatthereddirectllyorderivedfromaproduuct
design,caanbeorganize
edandcommmunicatedannumberofdiffferentways,includingtheefollowing:
didate1 93

Businessprocesses
Usecases
Userstories
Businessrules
Subsystems
Featurelists
Scenarios
Protocolspecifications
Functionalspecifications
Statemodels

Figure4illustratesaveryhighlevelsystemrequirementmodelthatincludesbusinessworkflowtying
togetherseveralusecases.TherequirementsincludetheneedtointeractwithSystem4aspartof
someoftheusecases.ItalsorequiresSystems1,2and3toexchangeinformationcreatedormodified
duringtheworkflow.

Figure4
RequirementsinvolvingUsers,UseCasesandUserStories
Thisguidehighlightsasmallsetofpopularrequirementspractices(seeVolumeII).Theyareusedto
illustratehowtherequirementspracticesandartifactsarerelatedtothetestsyoumayuseduring

acceptanceofasoftwareintensivesystem.TheUserModelingactivityisusedtocapturesalient
informationabouttheusersofthesoftware.Itcanhelptheproductdevelopmentteamunderstandthe
requirementsbetterbyunderstandinghowtheusersthinkandwhatmotivatestheirbehavior.UseCase
Modelingisusedtocapturewhattheuserswantthesystemtohelpthemachieve.Eachusecase
describesonekindofinteraction(e.g.transaction)betweenauserandthesystem.Ausecasetypically
requiresmanytestcasesbutoftencannotbetestedwithouttestingotherusecases.UserStoriesare
usedtobreakthefunctionalityofthesystemintoverysmallbutstillusefulandtestablechunksthatcan
bebuiltinjustafewdaysorweeks.Theyaretypicallymuchsmallerthanusecasesbutmayspan2or
moreusecases.Theymightdescribeabusinessruleorevenjustonescenarioofanintersystem
communication.Userstoriesmaptooneormoretestcases.
Theserequirementsactivitiesandartifactshelpusunderstandthetypesoftestswemayneedto
executetogatherthedataonwhichtheacceptancedecisionismade.
5.2.2 NonfunctionaRequirements
Nonfunctionalrequirementsarerequirementsthatdescribegeneralqualitiesorbehaviorsofthe
systemthatspanspecificusagescenarios.Frequently,theserequirementsaretracedbacktoprotection
ofvariousstakeholderswhomayormaynotbeusersofthesysteminquestion.
Unlikefunctionalrequirements,whichvarygreatlyfromsystemtosystem,thereisafairlystandardlist
oftypesofnonfunctionalrequirements.Manyoftheserequirementsendwiththesuffix"ility"because
theydescribecharacteristicsthewholesystemneedstosupporteitherwhileitisinuseoraspartofits
overallsystemlifecycle.
Requirementsthatthesystemneedstosatisfywhileitisoperatinginclude:
AvailabilityWhendoesthesystemneedtobeavailableforuse?
DataIntegrityThedataneedstobestoredandprocessedinawaythatitcanbereliably
retrievedwithoutchangingitsoriginalvalue.
SafetyThesystemshouldnotcausephysicaloremotionalharmtoauserorstakeholder.
RecoverabilityWhenitfails,thesystemshouldrestoreitspreviousstatewithoutundue
hardshiptotheusersorsupportersofplatform.
AccessibilityPeoplewithdiverselimitationsshouldbeabletousethesystem.Itmayneedto
conformtostandardslikethosestatedbytheADAorotherregulatoryorstandardsbodies.
SupportabilityItshouldbeeconomicaltoprovidesupporttousersoftheproduct.
ReliabilityItshouldworkwellandresistfailureinallsituations.
RobustnessItshouldtakeabuseandstillfunction(faulttolerance).

PerformanceWhatspeedorthroughputbenchmarksdoesitneedtomeet?Performance
includesresponsetime:thesystemshouldrespondtouseractionsandrequestsinatimely
manner.
UsabilityItshouldbeeasytousebyitsintendedusers.
InstallabilityTheproductshouldbeeasytoinstallontoatargetplatform.Thisissometimes
partofusability.
SecurityTheproductshouldprotectagainstunauthorizeduseorintrusion.
ScalabilityTheproductshouldbecapableofbeingscaledupordowntoaccommodatemore
usersortransactions.
CompatibilityWithwhatexternalcomponentsandconfigurationsdoesitneedtowork?
Requirementsrelatedtothetotalcostofownershiporproductlifecycleinclude:
TestabilityInwhatwaysshouldtheproductbetestable?
MaintainabilityHoweasyshoulditbetoevolve,fix,orenhancetheproduct?
PortabilityHoweasyshoulditbetoportorreusethetechnologyelsewhere?
LocalizabilityHoweconomicalwillitbetopublishtheproductinanotherlanguage?
ReusabilityHoweasilycansourcecode(e.g.classes,functions,etc.)beusedinother
circumstances?
ExtensibilityHoweconomicalshoulditbetoenhancetheproductwithotherfeaturesand
addons?
ConfigurabilityHoweasyshoulditbetopreparetheproducttobeusedonavarietyof
differentplatformsorforslightlydifferentusesandoperations?
Theprecedinglistoftypesofnonfunctionalrequirementsisnotintendedtobeexhaustive.Italsoisnot
intendedtobeuniversal;someoftheserequirementsmaybeirrelevantforsomesoftwareintensive
systems.Partoftheartofrequirementsgatheringorengineeringisdecidingwhichonesareimportant
andwhichonesarenot.Thosethatareimportantneedtobemadeexplicitandincorporatedintothe
testplans;thosethatarenotimportantcanprobablybeignored(buttheriskofignoringthemshouldbe
assessedbeforedoingso.)
Mostofthenonfunctionalrequirementscutacrosstheusecasesofthesystem.Thatis,theyapplyto
many,ifnotall,ofthediscretechunksoffunctionalitydescribedinthefunctionalrequirements.Note
thatsomeformsofnonfunctionalrequirementscanbedescribedatleastpartiallyinfunctionalterms;
securityisagoodexample.YoucansaythatUserRoleXshouldbepreventedfromchangingthevalueof
fieldFonscreenS.
Thekeytotestingconformancewithnonfunctionalrequirementsistheclassificationofeachofthese
requirementsindicatingtowhatdegreetheprojectstakeholderscareabouttherequirement.For
example,foranapplicationonebuildsforonespersonaluse,onemaynotcareaboutscalability

becausetherewillbeonlyoneuser,butonalargeecommerceapplication,scalabilityisveryimportant.
Itisworthreviewingthislistofnonfunctionalrequirementsandconsciouslydecidinghowimportant
eachoneistothesuccessofyourproductorproject.Thefollowingtableliststhegoals,importance,and
rationaleofavarietyofnonfunctionalattributesforaspecifichypotheticalsystem.
Attribute Goal Importance Rationale
Web site performance under Less than 500 milliseconds High Major source of
load response time revenue
Web server capacity under At least 300 transactions per Medium Large number of users
load second.
Graceful degradation under
load
Reliability/availability 7x24x52 Critical Users require instant

satisfaction when
worried about their
money
Usability Easily discoverable High Most users will use

infrequently
AppendixXinVolumeIVTestingActivities&Responsibilitiesisanexampleofachecklistthatcanbe
usedtorecordthesedecisions.
5.3 Summary
Inordertobeabletodoagoodjobofreadinessassessmentandacceptancetesting,thetargets
(requirements)mustbeclear.Readinessassessmentandacceptancetestingaredrivenbythequestion
"howwillweknowwearedone?"Assuch,theyshouldideallystartearlyintheproject.
Thereadinessassessmentandacceptancetestingactivitiesgatherdataabouthowwellthesystem
satisfiestherequirements.Therequirementsincludebothfunctionalandnonfunctionalrequirements.
Thereareanumberofalternativewaystodecomposetherequirementsintomanageablechunks.Some
formsofrequirementsmapmoredirectlytotestcasesthanothers.
5.4 WhatsNext?
Thingscananddogowrongwhenbuildingsoftwareintensivesystems.Commonlyoccurringproblems
includemisunderstoodrequirementsandfailuretomeetnonfunctionalrequirements.Thenextchapter
describestechniquesforidentifyingwhatmightgowrongandplanningreadinessassessmentand
acceptancetestingactivitiesthatwoulddiscoverthemasearlyaspossible.

5.5 References
[HSPK]HendrikSchifferstein,PaulHekkert(Eds.)ProductExperience.Elsevier,2008.
[BuxtonOnDesignForUsability]BillBuxton.SketchingUserExperiences:GettingtheDesignRightandthe
RightDesign.MogranKauffman,2007.
[GouldLewisonDesignForUsability]Gould,J.D.andC.Lewis,DesigningforUsabilityKeyPrinciplesand
WhatDesignersThink,inProc.ofACMCHI83ConferenceonHumanFactorsinComputingSystems.p.
5053,1983.

Chapter6 RiskModel
Everyprojectinherentriskswhetherornotwerecognizethem.Determiningwhatthoserisksarecanbe
achallenge.Projectrisksofproducingsoftwareintensivesystemsrelatetowhethertheproduct
developmentteamunderstandswhattheProductOwnerwantsbuiltandhowtobuilditandinabilityto
constructanddelploytheproductasintended,inatimelymanner,andcosteffectively.Opeationalrisks
relatetotheproductnotfunctioningasexceptedorfailingduringoperation.Ariskmodelcanhelpus
understandalltherisksandcomeupwithriskmitigationplanstoreducetheirlikelihoodorimpact.
6.1 WhatCouldPossiblyGoWrong?RiskAssessment
Onewaytodefineriskisbyaskingwhatkeepsusawakeatnight?Morespecifically,whatmighthappen
andwhatwouldbetheconsequencesifitdidhappen?
Wecanmakethediscussionofriskmoremeaningfulbytranslatingnebulousconcernsintoconcrete
eventsthatcouldhappenandtalkingaboutthelikelihoodthatitmighthappenandtheconsequencesif
itdoeshappen.
Forexample,supposeweorderedsomecriticalhardwarewhichwerequiretoconductcertaintypesof
acceptancetestingwithoutwhichwearenotpreparedtomaketheacceptancedecision.Whatcould
possiblygowrong?Thefollowingaresomeexamples:
Thehardwarecouldbedestroyedintransit.
Thewronghardwarecouldbeshippedeitherthroughanorderingerrororafulfillmenterror.
Thehardwarecouldbedefective.

Foreachoftheprecedingeventswecanestimatethelikelihoodthatitwilloccurandassesstheimpact
onourprojectifitdidoccur.Performingthesetwocalculationsseparatelyhelpsustobetterunderstand
theriskasshownintheriskmatrixinFigure1.Astheprobabilityofaneventincreases,therisk
increases.Astheconsequencesofaneventincreases,theriskincreases.


LowMediumHigh

Probability
Low Medium High

Consequence
Figure1
RiskMatrix
InFigure1RiskMatrix,thetwoareasoneithersideofthediagonalandthediagonalitselfrepresent
threedegreesofrisk.Thebottomleft(green)riskregimerepresentslowrisk,thetopright(red)risk
regimerepresentshighrisk,andthetoplefttobottomrightdiagonal(yellow)riskregimerepresents
moderaterisk.Ingeneral,risksthatfallinthesameriskregimeareequallyimportanttomitigate.
6.2 UnquantifiableRisks
Itisimportanttorecognizethatitisnottheknownandquantifaiablerisksthatcausethebiggest
problems,butthecompletelyunexpectedthingsthatdoso.Inhistreatmentoftheissue,PeteMcBreen
emphasizesthatflexibilityisthekeytosurvivingtheunexpected[McBreenOnRiskManagement].
6.3 WhoShouldIdentifytheRisks
Everyoneontheprojecthasadifferentandvaluableperspectiveonthepossiblerisksinvolvedin
buildinganddeliveringaproductwhetheritisforsaletocustomersorforusebyinternalusers.Risk
identificationshouldnotbeamonopolyoftheprojectmanagersbutshouldincludeproductdesigners,
productdevelopers,testersandallstakeholdersincludingtheProductOwner.Ultimately,itisthe
ProductOwnerwhoneedstounderstandtherisksanddecidewhichonestheycanacceptasisand
whichonestheywanttospendmoneytoreducethereforetheProductOwnermustbeintimately
familiarwithatleasttherisksinthered(topright)andyellow(diagonal)regionsoftheriskmatrix.
Ultimately,asTomGilbstatedinhisAskingPrinciple:Ifyoudontaskforriskinformation,youare
askingfortrouble[GilbOnSEManagement].

6.4 ShouldWeDoSomethingAboutIt?RiskManagement
Afterweunderstandtherisksofourproject,whatcanwedoaboutthem?Therearethreepossible
coursesofaction:
Wecanaccepttheprobabilityandconsequencethataparticulareventmighthappen.
Wecanperformactivitiestoreducethelikelihoodofithappening.
Wecanperformactivitiestoreducetheconsequenceifitdoeshappen.

Thecoursewechoosedependsonanumberoffactors,includingthefollowing:
Thefactorswehavecontrolover,suchasthefollowing:
Iftherearenocoursesofactionthatcouldreducethelikelihoodofsomething
happeningwemaybeforcedtofocusontryingtoreducetheconsequence.For
example,theonlywaytoavoidanextremeweathereventmightbetomovetoa
differentarea,whichmaysimplyexchangeonesetofextremeweathereventsfora
differentset.Butwecanpreparefortheoccuranceoftheextremeweatherbychoosing
appropriatefacilitiesorotherpreparations.
Ifthereisnowaytoreducetheconsequenceofanevent,weneedtofocusonreducing
thelikelihoodofitoccurring.Forexample,mostpeoplewouldagreethatitisbetter
(andeconomicallymorefeasible)totrytoreducethelikelihoodofaheartattackby
exercisingandeatingwellthantotrytoimprovetheprobabilityofsurvivingitbyhiring
aheartspecialisttobeatyoursideatalltimes.
Therelativecostoftheoptionsavailabletouse.Ifitismuchcheapertoreducethelikelihood
thantheconsequence,weshouldfirstfocusonloweringthelikelihoodandviceversa.Notethat
thecostistypicallynonlinearandgetsmoreexpensivetheclosertozerowetrytodrivethe
likelihoodorconsequence.Thereforeitmaybemoreappropriatetotrytopartiallyreduceboth
likelihoodandconsequenceratherthanreducejustonetoalargerdegree.
Thecostofriskreductionrelativetothecostwewouldincuriftheriskoccurred.Forexample,if
aparkingticketcoststwiceasmuchaspayingfortheparkingandthereisonlya20percent
chanceofgettingcaughtandticketed,wemaychoosetotakethechancebynotpayingfor
parking.

6.5 HowCanTestingHelp?RiskMitigationStrategies
Ifwedecidetomitigatearisk,howwegoaboutitdependsonthenatureoftherisk.Risksthatrelateto
thepossibilityofdeliveringadefectiveproductareamenabletoriskmitigationthroughsomeformof
testing.Risksthatrelatetodiscoveringsomethingtoolateforatimelyfixcanbemitigatedbyactivities
thatmovethediscoveryearlierintheproject.

6.5.1 DoingSomethingEarlier
Manyrisksonprojectsarerelatedtotime.Willsomethinghappenintime?Ifithappenstoolate,willwe
havetimetoreactwithoutaffectingtheprojecttimeline?
Agoodexampleofthisisthelatediscoveryofmissedormisunderstoodrequirements.Whenthis
discoveryoccursduringtheacceptancetestingphaseofaprojectshortlybeforetheproductisexpected
tobeturnedovertousers,theimpact(ofthediscovery)maybeasignificantdelayinachievingthe
businessbenefitsexpectedfromthesystem.Inthiscase,wecanreducetheimpactofthediscoveryby
doingtheacceptancetestingactivitiesearlierintheproject.
Theincrementalacceptancetestingpracticeusedonmanyagileprojectsisonewaytomovediscovery
ofproblemssuchasmisunderstoodrequirementsearlierintheprojectsothereisplentyoftimeto
addressthem.Sequentialprojectscanalsoreapthebenefitsofincrementalacceptancetestingby
movingtoanincrementaldeliverymodelwheretheproductisbuiltinfunctionalmodulesthatcanbe
acceptancetestedastheybecomeavailable.
6.5.2 DoingSomethingDifferent
Anextremeformof"toolate"discoveryiswhenwedonotdiscoveritatallandaproblemoccurswhen
theproductisbeingused.Iftheproblemissevereenoughtohaveseriousrepercussions,the
consequencescanbedisastrous.Thehighprofilelossesortheftofcustomers'privateinformationisjust
oneexampleofsomethingdiscovered"toolate."Thesetypesofriskmayrequireadditionalactivitiesto
reducethelikelihoodoftheiroccurrence.Thesolutionoftenliesindoingadditionalkindsoftestingto
improvethelikelihoodthatacertainclassofdefect,ifitexists,isfoundintime.Manytestauthoring
practicesarefocusedonwaystodefineadditionalteststhatimprovethetestcoverage(fromarisk
coverageinsteadofacodecoverageperspective).
6.6 Summary
Ariskmanagementmodelandawaytotrackrisksandriskmitigationisimportantonalltypesof
projects.Makingrisksexplicitallowsustodevisewaystochoosetherightriskstotakeandtomitigate
theriskseitherbyreducingtheprobabilityortheconsequence.Forrisksrelatedtomisunderstandingof
therequirements,earliertestingthroughincrementalacceptancecanreducetheconsequencebygiving
usmoretimetoreworkanyissues.Prioritizingacceptancetestingactivitiesshouldbepartoftherisk
modeltoensurethathighlycriticalfeaturesarethoroughlyaddressedduringsoftwareacceptance.The
ProductOwnerneedstobeinvolvedinidentifyingandunderstandingrisksandindeterminingwhich
risksneedtobeaddressed.

6.7 WhatsNext?
Asignificantriskonmostprojectsislatediscoverythattheproductismuchfartherfromdonethan
thought.Thenextchapterintroducesamodelforthinkingaboutthedegreetowhichtheproductis
done.
6.8 Reference
[McBreenOnRiskManagement]PeteMcBreen,TheMythofRiskManagement,BetterSoftware
Magazine:3034,June,2008.
[GilbOnSEManagement]TomGilb,PrinciplesofSoftwareEngineeringManagement,AddisonWesley,
1988.

Chapter7 DonenessModel
Determiningwhethertheproductisdoneistheultimategoalofsoftwareacceptance.Knowinghow
closetodoneweare,isakeyfunctionofprojectmanagementdiscipline.Thissectionintroducesamodel
fordetermininghowcompleteaproductisrelativetoitsgoalsfrombothfunctionalandnonfunctional
perspectives.
Akeypartofmakingtheacceptancedecisionrevolvesarounddecidingwhethertheproductisdone.
Comingtoconsensusondonenessrequiresagreementonwhatisbeingassessedaquestionof
targetgranularityandwhatitisbeingassessedagainstthedonenesscriteria.Initsmostbasicform,
theacceptanceprocessdefineshowwedecidewhetheraparticularreleaseoffunctionalityisdone.In
itsmoreadvancedformswecandeterminedonenessatthelevelofindividualfeaturesorworkitems.
Eachofthesedifferenttargetshasasetofdonenesscriteriathatmustbeagreedupon.Thefollowing
aresomeexamplequestionstoconsider:
Isasoftwareintensivesystem(forexample,asoftwareproduct)readyforanalphatestwitha
friendlyusercommunity?
Isanindividualfeatureoruserstoryreadyforacceptancetestingbyabusinesstester?
Isasoftwareintensivesystemreadyforthedesignclosemilestone?

Thedefinitionof"done"isdifferentforeachoftheseexamples.
7.1 ReleaseCriteriaDonenessofEntireProduct
Whendecidingtheacceptanceofanentiresoftwareintensivesystemforreleasetousers,donenessisa
binarystate.EithertheproductisacceptedbytheProductOwnerandripeforreleasetoitsintended
audienceorfortheusagedecision,oritsnot.Ifitsnot,thenwecantalkabouthowfartheproductis
fromthestateofbeingdonefortheunderlyingdecision.Thesameprinciplesapplytothereadiness
decisionthatprecedestheacceptancedecision:theoneaboutwhethertoreleasetheproducttothe
ProductOwnerfortheacceptancedecision.Thustherearetwomaincriteriafordeterminingwhethera
productisdone:
Isthereenoughhighvalue,ProductOwnerdefinedfeaturesincludedtomaketherelease
worthwhile?
Isthequalityofthefeatureimplementationshighenoughforthefeaturestobeusableintheir
expectedusagecontext?

Thefirstcriterion,knowninthisguideasMinimumMarketableFnctionality(MMF),istypically
determinedwhileplanningtherelease.Thecriterionmayberevisitedastheprojectisbeingexecuted

andmoreislearnedaboutthesystemcontext(suchasbusinessrequirements)andthetechnical
capabilitiesoftheProductDevelopmentTeam.
Whenreviewingacceptancetestresultsforeachfeature,itisfairlysimpletodeterminethepercentage
offeaturesthatisdone.ThisisthenumberoffeaturesthattheProductOwnerhasdecidedpasstheir
criticalacceptancetestsdividedbythetotalnumberoffeaturesfortherelease.Itisalsopossibleto
expressthisratioinstorypointstoaccountforfeaturesofdifferentsizes.
Thesecondcriterion,knowninthisguidanceasMinimumQualityRequirement(MQR),iswhatwe
constantlytestagainstwhilebuildingandassessingthesoftware.Tobeabletosaywhetherafeature
hasmettheMQR,weneedtohavetheacceptancetestsdefinedforthatfeature;inthisguidance,thisis
theperfeaturedefinitionof"Whatdonelookslike."Figure1illustratesgraphicallythedegreeof
donenessusingthesetwocriteria.Eachchartrepresentingthedonenessononeoftwoseparatedates
inthesameproject.

Figure1
DonenessrelativetoMMFandMQRattwopointsintime
InFigure1,thechartontheleftshowsthecompletenessofeachfeatureatpointXintime;thecharton
therightshowsthecompletenesstwoweekslater.Eachcolumnrepresentsafeaturewiththewidthof
eachcolumnindicatingtheestimatedefforttobuildthefeature.ThelinelabeledRAT(Readyfor
Acceptance)marksthelevelatwhichthefeatureisdeemedreadyforacceptancetestingbyvirtueof
havingconductedthereadinessassessment.Itistheperfeatureequivalentofthereadinessdecision
thatismadeatthesystemlevel.ThespacebetweentheRATlineandthelinelabeledMQRiswhen
acceptancetestingisdone.ThelinelabeledMMFisthedemarcationbetweenthefeaturesthatmustbe
present(leftoftheline)andthosethatareoptional(rightoftheline;omittedinthesediagrams)forthis
release.
WecanseeprogressovertimebycomparingtheleftandrightchartsinFigure1.Features1through5
(numberingfromtheleftstartingfrom1)werealreadycompletedatthepointintimerepresentedby
theleftchart.Features5through7werealreadystartedattimexandcompleted(deemeddone)in
theperiodendingattimex+2weeks.Features8through10werealreadyinprogressattimexand
werenotcompletedattimex+2weeks.Features11and12werestartedafterthedatedepictedbythe
leftchartandnotfinishedatthetimedepictedbytherightchart.

Theproductisdeemedacceptablewhenallfeaturespassalltheiracceptancetests.Thisistheupper
rightcornerofthechartwherethelineslabeledMQRandMMFintersect.Whentherectangletothe
lowerleftofthispointisentirelycoloredin,theproductisaccepted.Tosimplifythediscussion,thenon
functionalityrequirementsaredeliberatelyignored,buteachsetofnonfunctionaltestscouldbe
treatedasanother"featurebar"fromtheperspectiveofmeasuringdoneness.
7.1.1 GoodEnoughSoftware
ItisveryeasytosayItallhastobethere!whenaskedwhatfunctionalityismustbepresenttosatisfy
theMMF.Likewise,itiseasytoreplyTherecanbenobugs!whenaskedwhatMQRneedstobe.But
theseanswersaretakingtheeasywayout.Theproductownerhastomakeaconsciousdecisionabout
thevalueofqualityandfeatures.Thedogmaticanswer(allthefeaturesandabsolutelynobugs)may
delaydeliveryoftheproductbymonthsorevenyears.Meanwhileacompetitorsproductmaybe
earningmoneyandmarketsharebecausethatcompetitorhadamorerealisticgoalforMMFandMQR.
ThechoiceofMMFandMQRshouldbeaconsciousbusinessdecisionmadeinfullknowledgeofthecost
ofincreasedqualityandfunctionalityintermsofactualcosts(additionaldevelopment,testing)and
opportunitycosts(delayedincomeandotherproducts).
ForadditionaltreatmentofthetopicofGoodEnoughSoftware,see[BachOnGoodEnoughSoftware]and
[YourdonOnGoodEnoughSoftware]
7.2 DefiningWhatDoneMeans
Foreachchunkoffunctionalitywehavedecidedtodelivera"feature"thatneedstobedefinedforthe
minimumqualityrequirement(MQR)intheformofasetofacceptanceteststhatmustpassbeforethe
ProductOwnerwillacceptthefeature.Thesetofacceptancetestsforareleaseismerelytheaggregate
oftheacceptancetestsforallthefeatures("functionaltests")plustheacceptancetestsforeachofthe
nonfunctionalrequirements(the"nonfunctionaltests")thataredeemedmandatory.
7.2.1 Determining"Readiness"
"Readiness"iswhentheProductDevelopmentTeambelievestheproductis"doneenough"toaskthe
ProductOwnertoconsideracceptingtheproduct.ThisimpliesthattheProductDevelopmentTeamhas
areasonablyaccurateunderstandingofhowtheProductOwnerwillconducttheacceptancetesting.(In
somecases,theProductDevelopmentTeam's"readinesstests"maybemuchmorestringentthanthe
acceptanceteststheProductOwnerwillrun.)Thisunderstandingisknownasthe"acceptancecriteria"
andisusuallycapturedintheformofacceptancetests.Ideally,theacceptancetestsarejointly
developedandagreeduponbetweentheProductDevelopmentTeamandtheProductOwnerbefore
thesoftwareisbuilttoavoidplaying"battleship"or"blindman'sbluff"andtheconsequentrework
whentheProductDevelopmentTeamguesseswrong.Itisalsoidealforreadinesstestsandtestresults
tobesuppliedtotheProductOwnerbytheProductDevelopmentTeam.Thiscanassistinauditingthe
readinesstesting,streamliningacceptancetesting,andperformingamoreinformedstyleoftesting,
suchasexploratoryacceptancetesting.Thiskindoftestingcanbethoughtofablackhattesting
referringtoEdwarddeBonosoneofsixviewpointsforthinkingaboutaproblem

[DeBonoOnSixThinkingHats].Ablackhatapproachlooksataproblemfromacautiousandprotective
perspective,concentratingonwhatcangowrong.
7.2.2 DonenessofIndividualFeatures
Foranindividualfeaturewecandescribethedegreetowhichitisdoneinseveralways.Oneishowfar
alongitisinthesoftwaredevelopmentlifecycle.Forexample,Thedesignisdoneandcodingis80%
finished.Weexpectthefeaturetobereadyfortestingintwoweeksaspartofthetestingphase.This
formofprogressreportingisoftenusedinsequentialprojectsbecausethedesignandcodingcantake
manyweeksorevenmonths.Unfortunately,itisnotaveryusefulbecauseofitssubjectivity,low
transparency,andcoarsegranularity.Inprojectsthatadoptatoocoarselevelofschedulingand
clumpingwork,codingmayremain80%finishedformanyweeksormonthsevenifprogressisbeing
made.Itcanthereforebeverydifficulttotellwhenprogressisnotbeingmade.
Oncethecodinganddebuggingisfinished,itismoreusefultodefinedonenessintermsofwhichtest
casesarepassingandwhicharenot.Thisalternativewayofmeasuringprogressisobjective,
transparent,andfinegrained.Inthesequentialapproach,thepercentageoftestcasespassingstaysat
zeroformostofthelifecycleandthenrapidlyrisestosomewherenear100%.Initerativeand
incrementalprojects,featuresaresometimesimplementedonetestcaseatatimetherebyallowingthe
percentageoftestcasespassingtobeusedasanobjectiveprogressmetricatthefeaturelevelinstead
ofattheproductlevel.
Thenumberoftestcasesforaparticularfeaturemaychangeovertimeasthefeatureisunderstood
better.Thismayalsobeasignofworkcreep(moreefforttoimplementduetomorecomplexity)or
scopecreep(additionalfunctionalitythatwasntoriginallyintended).Theadditionaltestsmayresult
fromincreasedclaritycausedbyattemptingtowritetestsoritmaycomeasaresultoffeedbackfrom
otherformsoftestingincludingusabilitytesting,exploratorytestingoracceptancetestingofanAlpha
orBetarelease.Regardlessofthesourceoftheadditionaltests,thepercentdonemaytakeastep
backwardsifthenumberoftestsincreasesmorethanthenumberofadditionaltestspassing.
7.2.3 DonenessofNonfunctionalAttributes
Nonfunctionalattributesofthesystemareabitdifferentfromindividualfeaturesinthattheytendto
applyacrossallfeatures;thisiswhysomepeopleprefertocallthemparafunctional.Nonfunctional
attributes,whenquantifiedandobjectivelymeasurable,maybebinaryinthattheproducteither
satisfiestherequirementornot.Anexampleisscalability:eithertheproductcansupportamillion
transactionspersecondornot.Wemight,however,beabletobreakdownthefunctionalityofthe
systemintofunctionalareasanddeterminetheusabilityofeachareaindependently.Forexample,we
mayrequireahighdegreeofusabilityforcustomerselfservicefunctionswhileadministrativefunctions
canhaveamuchlowereaseofuse.Othernonfunctionalrequirementsarequantifiable.Forexample,
howmanyuserscanwesupportbeforeresponsetimeexceedsourresponsetimethreshold?Itcouldbe
1user,10users,or1000users.
Wecanchoosewhetherwewanttovisualizeprogressonnonfunctionalcompletenessaseither
additionalfeatures(support10users,support100users,support1000users)asillustratedinFigureZor

asasinglefeature(supportmanysimultaneoususers)withaminimumqualityrequirement(e.g.1000
users)asillustratedinFigureY.Ourchoicemaybeinfluencedbyhowweorganizetheworktodevelop
thefunctionality.Agileprojectsoftenoptforthefeatureperthresholdapproachbecausesupportfor
additionaluserscanberolledoutoveraseriesofiterations.Sequentialprojectsmayprefertotreat
themasdifferentlevelsofqualityforasinglefeatureasillustratedinFigureYorasadifferentlevelof
qualityacrossallfeaturesasillustratedinFigureX.(SeethediscussionofvaryingMMFforAlpha/Beta
releasesinsectionTheAcceptanceProcessforAlpha&BetaReleasesinChapter1foracounter
argument.)

FigureZ
IncrementalNonfunctionalFeatures
InFigureZtherequirementforsupportingmultipleusersisdividedintotwofeatures.Thefeatureto
supportatleast10usersisrepresentedbycolumn6(withhighlightedborder)andisreadyfor
acceptancetestingwhilethefeaturetosupport100users,representedbycolumn14(withhighlighted
border),hasnotyetbeenstarted.

FigureY
SingleNonfunctionalityFeature
InFigureYtherequirementforsupportingmultipleusersistreatedasasinglefeaturerepresentedby
column10.Itcurrentlysupportsatleast10usersbutisnotreadyforacceptancetestingatthefull
requirementof100users.

FigureX
BigBangNonfunctionality
InFigureXverificationoftherequirementforsupportingmultipleusersistreatedasanadditionallevel
ofqualitylabeledNFacrossallfeatures.Ithasnotyetbeenstarted.
7.3 Communicating"PercentDoneness"
Itcanbesaidyouareeither"done"or"notdone(yet)."Butinpractice,itisimportanttobeableto
clearlycommunicate"howclosetodone"youare.Morespecifically,itisimportanttobeableto
communicate"whatremainstodobeforewecansayweare'done'".Thisistheamountofworkleftfor
eachfeaturethathasnotyetpassedallitsacceptancetestssummedoverallthefeaturesthatarepart
oftheMMF.WhenlookingateitherdiagraminFigure1,wecanask"Whatpercentageoftherectangle
below/leftofMQR/MMFiscoloredin?"Thisgivesusasenseofhowmuchworkisremaining.Alotof
whiteimpliesalotofwork;verylittlewhitemeanswearenearlydone.Wherethewhiteislocatedtells
uswhatkindofworkislefttodo.Whiteacrossthetopmeanswehavelotsofpartiallyfinishedfeatures;
whiteprimarilyontherightimpliesthatentirefeatureshaventbeenbuiltyetbutmostfeaturesare
eithernotstartedorfullyfinished.Thesepatternsofprogressiondependprimarilyontheproject
managementmethodologyweareusing.ThediagraminFigure2showssnapshotsofcompletenessfor
threedifferentprojectstyles:

Figure2
Completenesscharts
InFigure2,thefirstrowofchartsrepresentsaclassicsequentialorphasedordocumentdrivenstyleof
projectmanagementwithwhiteacrossthetopindicatingthatallfeaturesareatsimilarstagesof
completion.ThebottomrowrepresentsaclassicExtremeProgrammingprojectwithwhiteprimarilyon
therightindicatingthatthereareveryfewfeaturespartiallydonesincemostareeitherdoneandnot
started.Themiddlerowrepresentsaprojectusinganincrementalstyleofdevelopmentwithlonger
featurecyclesthantheExtremeProgrammingproject.Noticethedifferenceinhowthecoloringinof
thechartadvancestowardtheupperrightcornerwheretheprojectwouldbeconsideredfullydone
basedon100%ofMMFand100%ofMQR.Thenextfewsectionsdelvemoredeeplyintohowwe
calculateandcommunicatedonenessonthesedifferentstylesofprojects.
7.3.1 CommunicatingPercentDoneonAgileProjects
Anagileprojectcansimplydividethenumberoffeaturesthatareacceptedbytheproductowneras
donebythetotalnumberoffeaturesscheduledfortherelease.Thisprovidesthepercentageof
featuresthataredone.Wecanexpressthismeasureinmonetarytermsbyweightingeachfeatures
percentcompletenessbythefeaturesestimatedcost.Theestimatedcostisrepresentedbythewidth
ofeachfeaturecolumnintheprogresschartsofFigure2.Howeverthemonetaryequivalentofafeature
calculatedthiswayisnotthesameasthefeaturesbusinessvalue.Itsessentiallyanexpressionofthe
featurescompletenessincostterms.Themonetaryequivalentiswhatisreferredtoasearnedvaluein
EarnedValueManagement(EVM).SeetheSidebartitledEarnedValueManagementandAgileProjects.
Sidebar:EarnedValueManagementandAgileProjects
EarnedValueManagement(EVM)isabudgetandscheduletrackingtechniquedevelopedbythe
ProjectManagementInstitute.Earnedvaluetrackstherelativeprogressofaworkunitasa
percentageoftheallocatedbudgetthathasbeenspentforthatworkunit.Ifaworkunitis80%
completeandwasallocateda$100budget,thenitscurrentearnedvalueis$80.Thetotalvalue
earnedbytheprojectisthesumoftheearnedvaluesofalltheworkunitsscheduledandbudgeted

fortheeproject.Whentheprojecctiscompletee,itstotalearrnedvalueeq
qualstheprojjectstotalbu
udget.
Thereffore,unliketh
henamesugggests,earnedvalueisessentiallyaneffo orttrackingm
metric.
InEVM
M,actualcostsarealsotracked.Ifatanypointintheeproject,theactualaccum
mulatedcost
exceeddstheearned dvalueoftheprojectatthhattime,thep
projectiscon
nsideredoverbudget,withhthe
shortfaallrepresentingthebudgeetoverrun(inEVMterms,costvariancee).Iftheaccumulatedearn ned
valueaatthatpointintimeisbelo
owtheestimatedtotalexpendituresacccordingtothheplanned
budget,thentheprrojectisconsiideredoversschedulewith hthedifferencerepresentiingthemoneetary
expresssionoftheprojectssched duleshortfall(inEVMterm
ms,schedulevariance)attthatpoint.
Schedu ulevarianceccanalsobecaalculatedincaalendartime.
FigureWillustratessschedulevarriance(inbotthmonetaryaandtemporallterms)andccostvariancefora
projectwithanestiimatedtotalb budgetof$5millionandaanestimatedscheduleof1 1year.InJulyy
2010,aatanearnedvalueof$2.5 5million,theprojecthasatotalearnedvaluethatisonly50%ofits
planneedtotalcost,butitshouldhaveaccumu ulated75%offitaccordingtotheprojecctplan.Therefore
itsoveerschedulebby$1.75millio
onor50%inmonetaryterrms.Sincetheeprojectshouldhave
accum mulatedthatm muchearnedvaluebackin nApril2010,incalendartim
meterms,itiis3months
behind d.Theprojecctisalso120%
%overbudgeetonJuly2010becauseith hasaccumulaatedanactualcost
of$5.55millionatth
hatdatecomp paredtothe$ $2.5millionitthasearned.
FigureW
Calcula
ationofsched
duleandcosttvarianceinEEVM.
AnappproachtoporrtingEVMtoaanagileconteextisdiscusseedbyGriffithhsandCabri
[CabriGGriffithsOnAggileEVM].GrifffithsandCabbriadvocateaapplyingEVM Monanincrem mentalbasis
(iterationbyiteratio
onorreleaseebyrelease).Inanagileprroject,agrand
dplanfortheeprojectisno
ot
meaningfulforproggresstrackinggpurposessin ncethegrand dplanwouldsubjecttoconstantchangge
duringgthecourseooftheproject.Howevermminiplansinteermsofworkkscheduledin nthecurrent
scope(featuresorsstories)andeestimatedcossts(resourcessallocatedtothescope)areusually
didate1 111

availableandreliableenoughtoserveasareferencepointonaperiterationorperreleasebasis.
Budgetandscheduleoverrunscanthenbecalculatedwithineachincrement,whetherforasingle
iterationorrelease,inthesamewayasinEVM.Effortoraproxyforeffortsuchasstory/feature
pointscanbesubstitutedforcost.
Inagileprojects,customerorbusinessvalueismoreimportantthanearnedvalue.Thereforeearned
valueinitselfmaynotbeverymeaningful.Thecustomervalueorbusinessvalueequivalentof
earnedvaluecanalsobeeasilytrackedprovidedthateachfeaturehasavalueorutilityassignedby
theproductowner.Thebusinessvalueorcustomervaluecanbeexpressedinabsolute,monetary
terms(incurrency)orinrelative,utilityterms(inartificialutils).Thebusinessvalueisanestimate
basedontheproductownersassessment.Thereforeitissubjective,butitcanstillbeinformedand
backedupbyseriouseconomicanalysis.Actualbusinessvalueisstilloftenhardtomeasure,
especiallyatthefeaturelevel.Unlikeinplainearnedvaluecalculation,onlydeliveredfeaturesshould
earntheirrespectivebusinessvalue:eachfeatureearnsitsfullestimatedbusinessvalueonce
delivered.However,ifafeatureisonlyusefulinthefieldwhendeliveredtogetherwithanother
feature,itshouldntearnbusinessvalueuntilallthefeaturesonwhichitdependsarealsodelivered.
Percentweightingstoadjustforafeaturesstatusofcompletenessdontmakemuchsensefor
trackingbusinessvalue.Consequently,individualfeaturesdontearnpartialbusinessvalue.Partial
orpercentagebusinessvalueearnedisonlymeaningfulattheprojectlevel.
ThediagraminFigure3illustratessnapshotsofhow"done"eachfeatureisatvariouspointsintime.
Eachminichartrepresentsapointintime.Theheightofthecoloredinportionofeachfeaturebar
representswhatdegreethatfeatureisdone.Asimplewaytocalculatethisisdividingthenumberof
acceptancetestspassingbythetotalnumberofacceptancetestsforthatfeature.Notethatthenumber
oftestscouldincreaseovertimeastheproductownersunderstandingofthefeatureimprove.
Figure3
Donenesschartsforanagileproject
Notehowagileprojectsfocusonbothmakingthefeaturessmallenoughthattheycanbedoneina
cycleandnottakingontoomanyfeaturesatthesametimetherebyminimizingthelengthoftimethata
particularfeatureisindevelopment.(Thegoalistocompleteeachfeatureinthesameiterationitwas
startedin,oratworstcase,theverynextiteration.)ThisallowstheProductOwnertodoincremental
acceptancetestingaseachfeatureisdelivered.Anybugsfoundcanbescheduledforfixingatthe
appropriatetime(whichmayberightawayorinsubsequentiterations.)
Figure4illustratesa"burndownchart"basedonplottingthenumberoffeatureslefttobe"done"
againsttime.

F
e
a
t
u
r
e
s
l
e
f
t
Time

Figure4
Burndownchartforanagileproject
Thesimplicityoftheburndowncharthelpsquicklygrasptheprogressoftherelease.However,itmay
masksomeotherdynamicsthattakesplaceontheproject.Forexample,when
Forsuchsituations,MichaelCohnintroducedanalternativeburndownchart(Figure4B),inwhichthe
progress,asusually,isshownabovethebaseline,andchangesinscopearereflectedbelowthebaseline.
[CohnOnBurndownChartAlternative]
<TBA>

Figure4B
Alternativeburndownchart
Insteadofhaving100percentofthefeatures50percentdoneatthehalfwaypointoftheproject,agile
projectsstrivetohave50percentofthefeatures100percentdone.ThisgivestheProductOwner
optionsifspecificationanddevelopmentofthefunctionalitytakeslongerthanexpected,whichisnot
uncommon.Theycandecidewhethertoadjust(reduce)theproductscopetodeliverontimeorto
adjust(delay)thedeliverydatetoincludeallfunctionality.Italsomeansthattheworkofreadiness
assessmentandacceptancetestingarespreadoutmoreorlessevenlyacrosstheproject.Thisis
discussedinChapter17PlanningAcceptance.
Figure5illustrateschartsforalessagileproject.

Figure5
Donenesschartsforalessagileproject
Onthisexampleproject,mostfeaturesaretakingseveraliterationstocompleteandacceptancetesting
onlystartsafterallfeaturesaredeemedready.Becausetheyarefoundverylateintheproject,thereis
lesstimetofixdeficienciesfoundduringacceptancetesting(suchasmissedrequirements)sotheyneed
tobefixedmuchmorequicklyortheytendtostayunfixedforthisrelease,withworkarounds,
restrictions,andspecialcases..
7.3.2 CommunicatingPercentDoneonSequentialProjects
Sequential(a.k.a.waterfall)projectshavemoreofachallengebecausethephases/milestones
synchronizedevelopmentinsuchawayastoensurethatallfunctionalityisavailablefortestingat
roughlythesametime.Thispreventsusing"percentoffunctionalityaccepted"asameaningful
predictivemeasureofprogress.Instead,Sequentialprojectsusuallyasksomeonetodeclarewhat
percentageeachfeatureisdone.Forexample,thedevelopermaysaytheyare80percentdonecoding
anddebugging(thoughthisnumberisoftenstuckat80formanyweeksinarow!).Consideringthe
subjectivenatureofestimationtechniques,sequentialprojectsoftenchoosetousetechniquessuchas
"earnedvalue"todeterminea"degreeofdoneness"metric.Unfortunately,thesetechniquesareprone
toerrorandfudging,andtheyarebothdifficultandtimeconsumingtoproduceandmaintain.
Figure6containsatypicalsequenceofdonenesschartsforthisapproach.
Figure6
Donenesschartsforasequentialproject
Inthissequentialversionofthecharts,wecanseehowphased/sequentialdevelopmentencouragesus
toworkinparallelonmanyfeaturesbecauseeachfeatureissynchronizedbygatingmechanismssuchas
themilestonesRequirementsFrozen,DesignComplete,andCodingComplete.Thismeansthatallthe
featuresareavailableforacceptancetestingatroughlythesametimeandmustbefinishedacceptance
testinginaveryshortperiodoftimebecauseitisonthecriticalpathoftheproject.Thishas
implicationsforthestaffinglevelsrequiredforthereadinessassessmentandacceptancetestingroles.
Whendevelopmentislate,theperiodforreadinessassessmentandacceptancetestingisfurther
shortenedandtheresourcesfurtherstressed.Italsohasimplicationsontheimpactoffindingbugs
duringthetestingbecausethefixesareonthecriticalpathtodelivery.
Figure7illustratesa"burndownchart"basedonplottingthenumberoffeatureslefttobe"done"
againsttime.

F
e
a
t
u
r
e
s

l
e
f
t
Time

Figure7
Burndownchartforasequentialproject
Sidebar:WhatittakestodoIncrementalAcceptance
InprojectsthatroutinelyapplyIncrementalAcceptanceTesting;theProductOwner(ortheonsite
customerproxy)isshowntheworkingfeatureassoonasitisdeemedreadybythedevelopers,often
justafewdaysintoadevelopmentiteration.Projectsthatdonotemploythispractice,inparticular
thoseemployingasequentialprocess,encountermanyobstacleswhentheytrytooverlaptestingwith
development.Theproblemismoreseverewhentheoverlapisintroducedtorecovertheschedule
whendevelopmentisrunninglate.Whythedifference?
Thereareanumberoffactorsatplayhere,eachofwhichmaycontributetothesituation.The
problemsincludedifficultycommunicatingwhatistestableandwhatisnot;thisresultsinfrustration
amongsttestersthatthetestingresultsinmanybugstowhichthedeveloperrespondsthatfeature
isntdoneyet;youshouldnotbetestingityet.
Crossfunctionalteamsadvocatedbyagileprocessesaddressthecommunicationproblem.The
productowner,requirementsanalysts,usabilityexperts,developers,andtestersalltouchbasedaily.
Theyparticipateinthesameiterationplanningmeetingsandendofiterationreviews.Thismeansthat
thereismuchbetterawarenessamongsttesters,whethertheybereadinessassessorsoracceptance
testers,ofwhatcanbetestedandwhatshouldbeleftforlater.
Adisciplinedsourcecodemanagementapproachalsohelps.Applyingtestdrivendevelopment
practicesandusingacontinuousintegrationserverthatrebuildstheentirecodebaseafterevery
checkinarepracticesthatsupportdisciplinedsourcecodemanagement.Inanenvironmentusing
thesepractices,thebuildprocessinvolvesrunningalltheautomatedunittests.Anytestfailuresare
treatedasastopthelineeventwherethetopprioritybecomesgettingthebuildworkingagain.This
ensuresastablecodebaseatalltimes;theteamsimplywonttoleratemembersdumpinghalffinished
codeintothebuild.

Anotherformofdisciplineisthedecompositionoflargefeaturesintosmallerfeaturesandeven
smallertaskswhichcanbecompletedinunderaday.Thisallowsworking,testedcodetobechecked
inseveraltimesadayavoidingtheoverheadofbranchingandmerging.
Controllingthenumberoffeaturesinprogressisyetanothermeasurethatsupportsincremental
acceptancetesting.Thispractice,borrowedfromtheworldofleanmanufacturingandleanproduct
design,minimizestheamountofmultitaskingpeopleneedtodotherebyimprovingfocus,flowand
productivity.Averyusefulsideeffectisthatthelengthoftimefromdecidingonwhatisrequiredto
whenthefunctionalityisreadytobeacceptancetestediskepttoaminimum;oftenjustafewdays.
Thisreducesthelikelihoodoftherequirementschangingwhiledevelopmentisinprogressorthe
ProductOwnerforgettingwhatitwastheymeantbythatrequirement.
Highlyincrementalacceptancetestingimpliesthattheproductwillbetestedmanytimes.Thecostof
testingwouldrisewitheveryiterationasmorefunctionalityisadded.Agileteamsavoidthisbyhaving
extensiveautomatedregressiontestsuites(atleastunittestsbutusuallyfunctionaltestsaswell)that
theyrunbeforedeclaringthatthecodeisreadyfortesting.Theseregressiontestsactasalarge
changedetectorwhichinformstheteamwheneverthebehaviouroftheproductchanges
unexpectedly.Forexample,itmaybeanunexpectedsideeffectofafeaturebeingintroduced.
Acceptancetestersaretypicallyinvolvedinpreparationoftheautomatedfunctionaltestssothey
knowwhatisalreadytestedandfocusonareasnotcoveredbytheautomationsuchasuserinterface
behaviour.
Ateamthatisencounteringdifficultyindoingincrementalacceptancetestingmaybeexhibiting
symptomsofmissingoneormoreofthesepractices.Thisisespeciallyprevalentonteamsnewtoagile
andteamsthathavetodealwithlargelegacycodebases.
7.4 Summary
Itmayseemlikeasimpleconcepttodetermineifafeatureorproductisdone,butinpracticeitcan
provetobesomewhatslippery.Wemustknowhowtodetermineifwearedone,whatcriteriatouseto
establishwhetherwearedoneandhowtotestthosecriteria.Itinvolvesdeterminingiftheproper
featuresorfunctionalityispresent,ifitworksasexpectedorrequiredanddoesitmeettheexpectations
oftheProductOwner.Havingaprocessinplacewillincreaseourchanceofsuccessfullydeliveringa
productwearehappywith.
7.5 WhatsNext?
Thisfinishesourintroductiontothemodelswecanuseastoolsforthinkingaboutacceptance.Thenext
fewchaptersdescribeperspectivesonacceptancefromanumberofdifferentplayers.

7.6 References
[BachOnGoodEnoughSoftware]Bach,J.TheChallengeofGoodEnoughSoftware.
http://www.satisfice.com/articles/gooden2.pdf
[YourdonOnGoodEnoughSoftware]Yourdon,E."WhenGoodEnoughSoftwareIsBest,"IEEESoftware,
vol.12,no.3,pp.7981,May1995.
[CabriGriffithsOnAgileEVM]Cabri,A.andGriffiths,M.Earnedvalueandagilereporting,Proc.ofthe
Agile2006Conference,2328July2006,pp.1722.
[DeBonoOnSixThinkingHats]EdwarddeBono,SixThinkingHats,BackBayBooks,1999.
[CohnOnBurndownChartAlternative]MikeCohn,AnAlternativeReleaseBurndownChart,
http://www.mountaingoatsoftware.com/altreleaseburndown,Oct19,2009

PartIIPerspectivesonAcceptance
Theprocessleadinguptotheacceptancedecisioninvolvesanumberofdifferentplayersandeachhas
theirownperspectiveontheprocess.Thechaptersinthispartdescribeacceptancefromanumberof
theseperspectivesincludingseveralpartiesthatmayplaytheroleofacceptancedecisionmaker
(BusinessLead,ProductManager,EnterpriseArchitect,LegalDepartmentandTestManager)andother
partiesthatmayplaytheroleofreadinessdecisionmaker(UserExperienceSpecialist,Development
Manager,SolutionArchitectandTestManager.)Eachoftheperspectivesthatfollowfocusonaspectsof
acceptancethatareparticularlysalienttotheplayer.Itisworthreadingalltheperspectivesevenifone
seemstofitthereaderlikeaglove.
[Notetoeditors:ThetextinPart2deliberatelyspeaksdirectlytothepersonineachoftheroles.Thatis,
itaddressesthepersonasthoughwewereinconversationwiththem.Itusestheyoupronounon
purposeasopposedtothePart1chapterswhichtypicallyusedwemustorwedo.]

Chapter8 TheBusinessLeads
Perspective
Anorganizationmaycommissiontheconstructionorcustomizationofsoftwareforitsownuse.The
constructionwouldbedonebyanotherorganizationwhichmaybeaninternaldepartment(suchasthe
InformationTechnologydepartment)oranothercompany(anoutsourceddevelopmentcompany.)In
suchanorganization,theBusinessLeadisthepersonwhobringstotheprojectteamthevisionofwhat
thesoftwareisintendedtodo.TheBusinessLeadmayalsobeknownastheProjectSponsor,Business
Sponsor,BusinessOwner,orBusinessPartner,andactsinthecapacityoftheProductOwner.
SeetheBusinessPersonintheMarketingDivisionpersonainAppendixAOrganizationStereotypesand
ReaderPersonas.
Thedecisiontocommissioncustom(orbespokeasitiscalledinsomelocales)softwareisabusiness
decisionmadebasedontheexpectedbusinessbenefitsoftheinvestmentthereturnoninvestment
(ROI).Thebenefitsmaybeincreasedincomethroughadditionalsalesorservices,costreductionthrough
fewerstaff,moreefficientworkprocessesorreducederrorsessentially,thesebenefitsareintendedto
contributetotheeconomicbottomlineoftheorganization.Sonowthatyouvemadethedecisionto
commissionconstructionofthesoftware,nowwhat?Beforeyoucanrealizethebenefitsyouwillbe
askedtomakethedecisiontoacceptthesoftwareasbuiltorprovidealistofdeficienciesthatmustbe
addressedbeforeyouwillacceptit.Acceptanceisasignificantstageofanycontractualprocess(See
LegalPerspectivesonAcceptance).Howyouapproachthisacceptancedecisioncanhaveabiginfluence
onhowpainfulorpainlesstheprocesswillbe.Itwillalsoimpactyourpostacceptanceactivities
(includingactualusage,operationalsupport,maintenanceetc.)
ThewrongapproachcanbesummarizedwiththefamousquoteIcannotdescribegoodartbutIll
recognizeitwhenIseeit.Takingthisapproachtotheacceptancedecisionissuretoresultinlate
delivery,costoverruns,deliveryofsoftwarethatdoesnotbringvaluetoyourorganizationanda
strainedrelationshipwithyourProductDevelopmentTeambecausethetheywillhaveapoorchanceof
meetingyourneedsiftheydontknowwhattheyare.Whentheydeliverthesoftwareyouwanted,
youllbesuretofindallsortsofissueswithhowitworks.YoullreportthesebugstotheProduct
DevelopmentTeamandexpectmostofthemtobefixedbeforeyouacceptthesoftwareandputitinto
production.Thatwilltakeadditionaltimeandresourcestherebyincreasingcostsanddeferringthe
benefits.Intheworstcase,thevendormaybeunabletofixallthedefectstoyoursatisfactionandyoull
eitherhavetoacceptsoftwareyouarenothappywithorwriteoffthesystementirely.
ToquoteLudwigWittgenstein,oneoftheleadingphilosophersofthe200thcentury,thereisagulf
betweenanorderanditsexecution.Ithastobefilledbytheactofunderstanding
[WittgensteinOnPhilosophicalInvestigations].Tobesuccessful,donotunderestimatethisactof
understanding.SuccessfulprojectstypicallyhaveaclearagreementbetweentheProductOwnerand

theProductDevelopmentTeamonwhatconstitutesasuccessfuloutcomefortheproject.Thisproject
charterprovidesahighlevelviewofthebusinessleadsexpectationsoftheprojectandhoweachparty
willbehavethroughouttheproject.Italsolaysoutanykeyconstraintsregardingschedule,costandany
otherrelevantfactors.Dontfocusthecharter(orcontract,StatementofWork,etc.)onwhatremedies
youwillhaveiftheProductDevelopmentTeamfailstodeliver;bythentheprojecthasalreadyfailed!
Instead,focusonthebehaviorsthatwillensureprojectsuccess.Allpartiesshouldtreattheprojectasa
miniaturejointventureandstriveforawinwinoutcome.Ifanyofthepartiesistryingtowinatthe
othersexpense,allpartieswilllose.Dysfunctionalrelationshipsleadtodysfunctionalproducts.Seethe
sidebarTargetcostcontractingfordescriptionofanalternativetotraditionalfixedpriceortime&
materialscontracts.RefertoSectionResponsbilitiesoftheProductOwnerinChapter1forasummaryof
theresponsibilities.
Sidebar:TargetCostContracting:fromadversarialcontractstowardspartneringcontracts
LeanSoftwareDevelopmentvisionariesTomandMaryPoppendieckadvocatethepartneringor
relationalapproachtocontracting,inwhichpartiesagreetoworkinacollaborativewaytocarryout
theproject[PoppendiecksOnLean,Ch.9].TargetCostContractsareatypeofincentivescontractsthat
canbeusedtosupportsharingofbothfinancialrisksandrewards.Thishelpstoalignthebestinterest
ofeachpartywiththebestinterestsofthejointventure.ThePoppendiecksencourageforging
synergisticrelationshipacrosscompanyboundaries.Theyenvisionthatthosewhodowillseeatleast
the25percentto30percentcostreductionofthecostofdoingbusinesspredictedbyPeterDruckerin
[DruckerOnManagementChallenges].
TargetCostContractsareanapproachtoidentifyingandconstraininguncertainty.Itsdoneinthe
followingway.Atargetcostfortheprojectisagreedatthebeginningandbothpartiesareproactively
workingonachievingthetargetcost.Uponprojectscompletion,theactualcostismeasuredagainst
theoriginallysettargetcost.Iftheactualcostisgreatthanthetargetcost,thesupplierandthe
customereachbearanagreedelementofthatcostoverrun.Iftheactualcostislowerthanthetarget
cost,thesuppliersharesthesavingswiththecustomer.Theshareofthecostoverrun/underrunmay
beaconstantproportionormayvarydependingonthesizeofthedelta.
Note:TheanalysisofPerry&Barnesrevealsastrongcaseforsettingthesuppliersshareofcost
overrunorunderrunatavaluethatisnotlessthan50%[PerryEtAlOnTargetCostContracts].
- ThekeycharacteristicsoftheTargetCostContractsareasfollows:Targetcostincludesallchanges
- Targetisthejointresponsibilityofbothparties
- Targetcostisclearlycommunicatedtoworkers
- Negotiationsoccuriftargetcostisexceededwithneitherpartybenefiting.
- Thisenticesworkersatallleveltoworkcollaboratively,compromise,andmeetthetarget.
ForadditionalcasestudyofusingTargetCostContractsinsoftwaredevelopment,seethisexperience
report[EckfeldtEtAlOnTargetCostContracts].

8.1 TestPlanning
Acceptingsoftwaresoundslikeadiscreteeventbutthereismuchtobedoneleadinguptoit.Asa
ProductOwneryouneedtobepreparedtoparticipateinthedevelopmentofyoursoftware.Anyone
whohashadahousebuiltspecificallyforthemknowsthattherearealotofdecisionstobemadeand
inspectionstobedonethroughouttheentireconstructionprocess.Softwareshouldnotbeany
different.Expecttogetasmuchoutofyoursoftwareasyouarepreparedtoputin.Theabsentee
ProductOwnergetstheresultstheydeserve;anunsuitablesystemdesignedforsomeoneelse.
8.1.1 CommunicatingExpectations
Toavoidthisfateyoucertainlymusthaveaplanformakingtheacceptancedecision.Butyoumustalso
planforregularcommunicationwiththeProductDevelopmentTeamaboutthecriteriaforthat
decision.Oneofthemajoracceptancecriteriawillbesuccessfulimplementationoftherequirements
youhavespecified.Theserequirementsmaytakemanydifferentformsincluding:
Functionalspecificationswritteninplainlanguage(e.g.Thesystemshall)
Usecasemodelsanddocuments
Protocolspecificationsforinterfacestoothersystems
Userinterfacemockups,prototypes,cognitivewalkthroughs,orexistingsystemstoemulate.
Descriptionsofbusinessrulesandalgorithms
Theserequirementsdocumentsformanimportantbasisbothforthevendorbuildingthesystemand
youracceptancetestingplans.Butrecognizethatthesedocumentsandmodelswillneveranswerallthe
questionsthevendormayhave.Norwilltheyexhaustivelydescribeallyourexpectations.Infact,you
maynotevenbeawareofsomeofyourexpectationsuntiltheyfailtobesatisfiedbythesoftware.No
amountofupfrontrequirementsplanningwillflushoutallyourexpectationsthereforeitiscrucialto
interactwiththesoftwareearlyandoftentogiveyourselftimetodiscovertheseunrealized
expectationsearlyenoughtoaffordtheProductDevelopmentTeamtimetoimplementthem.
ExpecttheProductDevelopmentTeamtodiscovermanyambiguitiesandunforeseencircumstancesin
whateverrequirementsandexpectationsyouhavedescribedtothem.Plantobeavailabletothe
ProductDevelopmentTeamonaregularbasis,themorefrequentthebetter.Ifyoucannotparticipate
personally,assignsomeonetoactasyourProductOwnerproxyandensurethattheyhavefull
understandingofwhatyouwanttoachieveandaclearmandateonwhattheycandecidethemselves
andwhatneedsyourapproval.Onagileprojects,expecttoprovideatleastonepersonfulltimeforthe
entiredurationoftheprojecttoclarifyrequirementsanddocontinuousincrementalacceptancetesting.
8.1.2 PlanningWhentoTest
AkeypartoftheagreementwiththeProductDevelopmentTeamshouldbethefrequencywithwhich
youaresuppliedworkingsoftwareonwhichyoucanperformacceptancetesting.Inthepuresequential
orclassicwaterfallmodel,youwilllikelyneedtowaituntilneartheendoftheprojectbeforeyou
receiveanysoftware.Thismaybeasinglefinalreleasethatthevendorbelievesiscomplete,high

qualitycode,ortheremaybeaseriesofreleaseswithnamessuchasAlpha,BetaorReleaseCandidate
thatareprovidedearlyenoughtoallowrevisionofhowthesoftwareworksbasedontheresultsof
acceptancetestingofeachrelease.Inmostcases,thepreliminaryreleasesareeitheroflowerqualityor
containlessfunctionalitythanthefinalrelease.Makeyourincrementalacceptancetestingplans
accordingly;dontexpecteverythingtobeavailableintheearlyreleases.Whatistobeavailableineach
releaseshouldbethesubjectofdiscussionswiththeProductDevelopmentTeam.
Intheagilemodel,youshouldexpecttoreceiveworking,potentiallydeployablesoftwareonaregular
schedulethroughouttheentireprojecttimeline.Youllbeaskedtodecidewhatfunctionalitythe
ProductDevelopmentTeamshouldbuildineachiteration(akasprint)andyoullneedtobepreparedto
providethedetailedacceptancecriteriaforeachfeatureoruserstorytotheProductDevelopment
Teamearlyintheiteration.ThisprocessisknownasAcceptanceTestDrivenDevelopment.Whenthe
softwarefeatureisfinished,youllbeaskedtotryoutthenewlybuiltfunctionalityanddecidewhetherit
workstoyoursatisfactionorrequireschangesbeforeyouwillacceptit.Inthisincrementalacceptance
testingmodelyouwillneedtotestadditionalfunctionalityeachiteration.Youmayalsoneedto
dedicateoneormoreiterationsattheendoftheprojecttothefinalendtoendacceptancetestingand
releaseactivities.
RegardlessoftheprocessmodelyoushouldexpecttheProductDevelopmentTeamtodoadequate
testingaspartoftheirreadinessassessment.Theexpectationshouldbeclearlyunderstoodbyall
partiesandyoushouldplantoprovidetheacceptancecriteriaanddetailedacceptanceteststothe
ProductDevelopmentTeamsothattheycanrunthetestsduringtheirreadinessassessmentactivities.
Youwillalsoneedtohaveplansinplaceforhowyouwillassessthesoftwarewhenitbecomesavailable.
Ataminimum,youwillneedtoknowhowmanypeopleandhowmuchtimewillberequiredtodothe
assessment.Thedetailsofwhatteststheywouldrunmayalsobeincludedorthismaybedetermined
nearertotheactualtesting.
8.1.3 RegressionTesting
Assumingyouwillreceiveanewversionofthesoftwaremorethanonce,andthisisaverysafe
assumptionregardlessofwhetheryouuseasequentialoragileprocessmodel,youllneedtoretestnot
onlythenewlyintroducedorchangedfunctionalitybutalsoallthepreviouslybuiltfunctionality.This
canturnouttobealotofeffort.Onestrategyistochoosetheteststoberunbasedonwhatwas
changedbutthisincreasestheriskofmissingnewlyintroducedbugs.Anotherstrategyistohaveasuite
ofautomatedtests(see
AutomatedTestinginCh.16:PlanningforAcceptance)thatcanberunquicklyaftereverychangetothe
software.Thiscangreatlyreducetheeffortinvolvedinretestingorregressiontesting.Thereforemake
suretoaskyourProductDevelopmentTeamtoincludeautomatedtestsaspartoftheircostingto
ensureyoudonthaveaneverincreasingworkloadasmoreandmorefunctionalityisdelivered.The
automatedtestsarereallyfortheProductDevelopmentTeamsbenefitasithelpsthemensurethat
theyarenotintroducingdefectsintotheexistingsoftware.Also,youmaywanttorequestthe
regressiontestssuitestobeshippedaspartoftheprojectdeliverables.Testsareassetsandtherefore
youshouldplanonkeepingthemcurrentforaslongasyouplantousetheproduct.Havingthe

acceptancetestsinplacemayreducethecostofyourfuturemaintenance/extensionprojects(forwhich
youmaychoosetouseadifferentsupplier).Buttheacceptancetestsmaythemselvesrequiretest
maintenanceorrefactoringasthebehaviorofthesystemundertestevolves.
8.1.4 TestResourcingandOutsourcing
ThetestingyouplantodomaybeinfluencedbyyourknowledgeofwhattestingtheProduct
DevelopmentTeamwillhavedonepriortogivingyouaccesstothesoftwareandbytheskillsand
experienceofthepeopleyouhaveavailabletotest.Whenyoudonthavethenecessaryexpertiseand
youcannotorchoosenottobringitinhouseyoucanelecttouseathirdpartyorganizationtodoyour
acceptancetestingonyourbehalf.Beaware,however,thatveryfewtestoutsourcershaveperfected
mindreading!Youwillneedtoprovidethemwithdetaileddirectionregardingthenatureandobjectives
ofthetestingyouwantthemtodoandtheexpectedbehavior(i.e.requirements)ofthesystemtobe
tested.Youllalsowanttobeveryclearabouttheacceptancecriteriaandwhethertheyaremakingthe
acceptancedecisiononyourbehalforprovidingyouwithdatatosupportyouracceptancedecision.Test
outsourcingiscommonforspecializedtypeoftesting(e.g.securitytesting).Testoutsourcingmayalso
bemandatedbyacertificationbody/regulatoryauthoritythatrequiresanindependent
verification/audittobeperformed.Forexample,alltaxpreparationsoftwarepackagesmayneedtobe
testedforcompliancetothetaxauthoritiesstandardsfortaxreturninformation.
8.1.5 TestEstimation
Estimatingtheamountoftimeandefforttoallocatetoacceptancetestingisdifficultwithoutagood
understandingofthenatureofthefunctionalityinvolvedandthethoroughnessofthereadiness
assessmentthatwillbedonebytheProductDevelopmentTeam.
Acommonstrategyistotimeboxthefinalacceptancetestphaseandguessthenumberofcyclesof
acceptancetestingplusbugfixingthatwillberequired.Thisrequirespastexperiencetocomeupwith
reasonableguesses.Ifindoubt,guesshighandallowforseveralcyclesoftestingtogivethevendortime
tofixbugs.Warning:Thereisalwaysalotofpressuretoguesslowtoallowearlierdeliveryofthe
softwareandthelowercostthatthisimplies.Butguessingtoolowwilltypicallyresultinhavingto
choosebetweenacceptingthesoftwareonschedulewithmanyknowndeficienciesordelayingthe
releasetoallowadditionaltest&fixcyclestobeperformeduntilallimportantdeficienciesarefixedand
retested.ThetestingphaseisoftensqueezedbylatedeliveryofthesoftwarebytheProduct
DevelopmentTeamandthetestersareoftenaskedtomakeupthedelaybyreducingtheelapsedtime
fortesting.
Thealternativestrategyinvolvesincrementalacceptancetesting;acceptancetestingthatisdone
frequentlythroughouttheproject.Thisimprovesthepredictabilityofthedeliverydateintwoways.
First,itfindsbugsearlierallowingthemtobefixedlongbeforethefinalacceptancetestcycle.Second,it
providesyoutheopportunitytolearnhowmuchtestingisreallyrequiredandhowmuchtimeandeffort
itwilltaketodoit.Thislearninghappensearlyenoughintheprojectthatyoucanadjustyourplansfor
lateriterationsandthefinaltestcyclesothattheprojectcanbedeliveredontime.Adjustmentscould

includeincreasingordecreasingtheplannednumberoftestcycles,addstafforskills,increasingor
decreasingtheamountoftestautomation,contractingathirdpartyforspecializedtestingandsoon.
8.2 WhatDoYouNeedtoTest?
EnsurethatyouhaveaclearunderstandingofwhatwillbetestedbytheProductDevelopmentTeam
andwhatyouareexpectedtotest.Typically,theProductDevelopmentTeamisexpectedtodo
thoroughtestingofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,usability
etc.)beforehandingoffsystemforacceptancetesting.Ofcourse,thisisbasedontheProduct
DevelopmentTeamsunderstandingofyourexpectationswhichmayvaryfromyoursifthe
communicationislessthanperfectwhichisalmostalwaysthecase.Yourresponsibilityduring
acceptancetestingisensuringthesystemissuitableforyourownpurpose.
Thetestcasesyourunduringacceptancetestingshouldcovereverythingyoucareabout.Thiscan
includebusinessfunctionality,operationsfunctionalityandnonfunctionalcharacteristicsofthesystem.
Eachoftheseareasmayrequiredifferentexpertisetotestandpossiblydifferentstakeholdersignoffs.
Decidingwhowillsignoffbasedonwhattesting,donebywhom,isanimportantpartofacceptancetest
planning.Thebusinessleadmayneedtonegotiatewithotherpartiestodeterminetheacceptance
decisionprocessandthetestingthatprovidestheinformation.
8.2.1 FunctionalTesting
Functionaltestingshouldincludebusinessworkflowteststhatverifythatallreasonablebusiness
scenariosorworkflowsfunctionproperly.Moredetailedusecasetestscanbeutilizedtoverifythat
eachbusinesstransactionorusecaseisimplementedcorrectly.Thehappypathormainsuccess
scenarioofeachusecaseisnotwherebugswilltypicallybefound;theytendtolurkinthedarkcorners
ofthelesscommonlyexercisedscenarios.Businessruletestscanfocusonverifyingthatthebusiness
rulesandalgorithmsareimplementedcorrectly.Applicationswithrichuserinterfacestypicallyrequire
extensivetestingoftheuserinterfaceitself.Thistestingcaneitherbebundledwiththeusecasetestor
youcanhaveseparatetestsessionsdedicatedtoexploratorytestingthebehavioroftheuserinterface.
Thedegreetowhichyouneedtotestshouldbebasedonthelevelofconfidenceyouhaveinyour
ProductDevelopmentTeamandthecomplexityofthedeliverable.Thiscanbebasedonpastexperience
onpreviousprojectsoronwhattheyhavedemonstratedonthisproject.AProductDevelopmentTeam
thathasprovidedfullvisibilityoftheirinternalprocessesandespeciallytheirreadinessassessment
activitiesmayinspiremuchmoreconfidenceinthequalityoftheproduct.Thiscouldchangethefocusof
youracceptancetestingactivitiesfromlookingforobviousmisimplementationsofthebusinesslogicor
rulestohighervalueaddingactivitieslikeusabilitytestingandbusinessworkflowtestingbothofwhich
improvetheproductdesignandthereforethefitnessforpurposeratherthanjustverifyingthequalityof
theconstruction.Inparticular,thepresenceofanextensivesuiteofbusinessreadableautomated
workflow,usecaseandbusinessruleacceptancetestscouldchangetheemphasisfromconducting
extensivemanualtestingofthisfunctionalitytospotchecking.SeeAutomatedExecutionofFunctional
TestsinVolumeIIIforhowtoautomatethesetests.

8.2.2 NonFunctionalTesting
Nonfunctionaltesting(akaextrafunctionaltesting)isoftendonebytheProductDevelopmentTeam
duringthereadinessassessmentorbyaseparatetestorganization.Itshouldincludeverifyinganynon
functionalcharacteristicsofthesystemthataredeemedcriticalforacceptanceaspertheSystem
RequirementsModeldescribedinChaper5,suchas:
Performance(capacityandresponsetime)andstress(behaviorwhenoverloaded)tests,
Securitytesting(enforcementofprivacypolicies,resistancetoattackbyhackers,etc.)
Scalability(theabilitytohandleadditionalusersortransactionloadsinthefuture)
Usability(theefficiency,effectivenessandsatisfactionwithwhichuserscanemploythe
softwaresysteminordertoachievetheirparticulargoals)
andothers.
EnsurethatyouhaveacommonunderstandingwiththeProductDevelopmentTeamastowhatProduct
DevelopmentTeamfunctionaltestingtheywilldoandhowmuchinformationyouexpecttoreceiveas
inputintotheacceptancedecision.
Thepeoplechargedwithoperatingandsustainingthesystem(hereincalledtheoperationsdepartment)
willhaveveryspecificrequirementsthatneedtobeaddressed.Youllneedtobeawareoftheseasyou
workwiththeProductDevelopmentTeamexploringanddefiningtherequirementsforthesystem.Ina
largerorganizationyoumayhaveacounterpartintheoperationsdepartmentwhocanprovidethe
requirementsandactastheacceptancedecisionmakerfromtheoperationsperspective.See
OperationsManagersPerspectiveinPartII.
Ataminimumyoullneedtobeawareofthetimeandresourcescomplyingwiththeserequirementswill
entail.Insomecasesyoullneedtogiveupnewfunctionalityinordertoensurethattheoperational
requirementsaremet.Someexamplesofoperationalrequirementsmightinclude:
Thesystemneedstointegratewiththespecificsystemsmonitoringsoftwareinuseatyour
company.
Dataneedstobemigratedfromapreviousversionofthesystemorfromasystemthisone
replaces.
Thesystemneedstohaveupgrade,startupandshutdownscriptsthatcanbeusedduring
automatedservermaintenancewindows.
Thesystemneedstobebuiltusinganapprovedtechnologystack.Theoperations
departmentmaynothavetheskillstosupportsometechnologies.E.g.A.NETshopmaynot
havetheskillstosupportanapplicationbuiltinJavaorPHP.Sometechnologiesmaybe
incompatiblewithtechnologiesusedatyourcompany.
Theoperationsgroupmayhavespecificwindowsduringwhichtheycannotinstallortest
newversionsofsoftware.Thismayaffectthereleasescheduleyouhavedevisedanditmay

makeontimedeliveryofsoftwarebytheProductDevelopmentTeamevenmorecriticalas
smallscheduleslipscouldresultinlonginservicedelays.
Therecouldberequirementsforperformingsuchserviceoperationswithoutdegrading
thecurrentsystem
Reliabilityrequirementsdictatethatthesystemmeetstheagreeduponservicelevel
agreements(SLAs)suchasuninterrupteduptime,transactionlatency,.
Thesystemmustprovideselfmonitoring,errordetection,andsystemloggingcapabilities.
Thesystemmustprovideforrobustrecoverability(includingtheabilitytoverifymultiple
failoversandrecoveries),disasterrecovery,databackup&recovery,andallotheraspectsof
systemrecovery.
Afullrangeofdocumentationmustbeprovidedincludingsupportdocumentation,bothpre
andpostrollout,trainingmanuals,andotherreferencedocumentation.
Thesystemmustbemigratedtothenewversionoftheplatform.
Multipleversionsofthesystemmustrunsidebyside.

8.2.3 OtherPartiesInvolvedinAcceptanceTesting
Ifthereareotherpartiesinvolvedintheacceptancedecisionensurethatyouhaveacommon
understandingaboutwhattheywilltestandhowthatfeedsintotheacceptancedecision.Common
partiesincludetheoperationsdepartmentwhowouldhaveasayinwhetherthesystemcanbeinstalled
andmanaged,usabilitypractitionerswhomightneedtoensurethatthesystemcomplieswithany
usability(includingaccessibility)standards,andcorporatesecuritydepartmentswhoneedtoensurethe
systemissecurebeforeitcanbeaccepted.
8.3 WhereWillYouTest?
Akeypartoftestplanningisdecidingwhatenvironmentswillbeusedforwhatkindsofactivitiesand
howthesoftwareismovedbetweenenvironments.
8.3.1 TestEnvironments
Expecttohavemanyactivitieshappeningconcurrentlyasyoudoacceptancetesting.TheProduct
DevelopmentTeamwilllikelybefixingthebugsthatyoureportwhileyoucontinuetolookforfurther
issues.Thisrequiresaseparateenvironment.WhentheProductDevelopmentTeamhasfixedenough
bugstowarrantdeliveringanewversionofthesoftwaretheywillneedtodoreadinessassessment.This
mayrequireyetanotherenvironmentifbugfixingistocontinueinparallel.Ifatestorganizationneeds
totestthesoftwarebeforepassingitontoyouforacceptancetestingtheywillneedtohavetheirown
testenvironmentaswell.

8.3.2 SoftwareDeliveryProcess
Giventhatdifferentgroupswillbeworkinginseparateenvironments,howdoesanewversionofthe
softwaregetinstalledintheenvironment?Forexample,youvefinishedacceptancetestingonrelease
candidate1(RC1)ofthesoftware,reportedanumberofbugs,andtheProductDevelopmentTeamhas
fixedenoughbugstoproposedeliveringanewversion,RC2,toyourtestenvironment.Whatisthe
processforreplacingtheRC1softwareintheacceptancetestingenvironmentwithRC2?Doesthe
ProductDevelopmentTeampushthenewversionwithorwithoutanyadvancenotice?Doesthe
ProductDevelopmentTeamnotifyyouoftheavailabilityofRC2andwaitforyoutoaskforittobe
installedintheacceptancetestingenvironmentwhenyouarereadytoreceiveit?Andhowlongwillthe
acceptancetestingenvironmentbeunavailablewhiletheupgradeisbeinginstalled?
8.4 TestExecution
Whenthesoftwareisdeliveredforacceptancetesting,acceptancetestexecutionmustbegininearnest.
Itisimportanttoknowwhereyoustandatalltimeswithrespecttotheprogressoftestingandyour
currentunderstandingofthesuitabilityofthesoftware.
8.4.1 ProgressMonitoring
Asacceptancetestingisinvariablytimeboxedbyeithercontractsorcustomerexpectations,itis
importanttoensurethatthetestingactivitiesareprogressinginatimefashion.Thenatureofthetest
plandeterminesthenatureofprogressmonitoring.Agileprojectshaveacceptancetestingspreadout
overthedurationoftheprojectandtheresultsoftestingarehighlyvisibleintheformoffeedbackto
theProductDevelopmentteam.Thereisplentyoftimefortheacceptancetesterstolearnfromtheir
mistakesandmisassumptionsintimeforthenextroundofacceptancetesting.Traditionalsequential
projects,ontheotherhand,havealltestinghighlyconcentratedattheendoftheproject.Itiscritical
thatthesetestactivitiesbeverycloselymonitoredtoensurethatanydeviationsfromthetestplanare
quicklyidentifiedandrectified.Thereisntmuchtimeforlearningbetterapproachestotestingso
everyonemustbehighlytrainedbeforetheacceptancetestperiod.Thisapproachtotestingisfraught
withmanydangersandshouldbeavoidedatallcosts.
8.4.2 BugTracking&RecordKeeping
Thenatureoftheproductbeingbuiltandthecontextinwhichitwillbeusedhasalargebearingonthe
natureanddegreeofrecordkeepingforbothtestexecutionandanybugsfound.Itmaybesufficientto
simplyruntestsandtrackanybugsfoundinaninformalmannerasdescribedinthesidebarIncremental
AcceptanceTestingandBugTrackingonAgileProjects.Inotherenvironments,recordsneedtobekept
aboutwhattestswererun,againstwhichversionofthesoftware,when,bywhomandtheactualresults
thatwereobserved.
Ataminimum,anydefectsthatwerefoundduringacceptancetestingneedtoberecordedandtriaged.
Eachbugshouldincludewhichversionofthesoftwareitwasfoundin,whetheritwillprevent
acceptanceofthesoftware(severitylevel)andwhofoundit.Bugsthatmustbefixedbeforethe

softwarewillbeacceptedmustbeclearlycommunicatedtotheProductDevelopmentTeam.Lesssevere
bugsshouldalsoberecordedbutshouldalsobeclearlylabeledasnotgatingacceptance.Keepinmind
thatinmanycontextsanythinglabelnongatingwilllikelynevergetfixedsotherealdecisioniscanyou
livewiththeproductexhibitingthisbug,onnot?
8.4.3 RegressionTesting
Eachtimethesoftwareischanged,whethertoaddnewfunctionalityortofixbugsfoundduring
acceptancetesting,newbugsmayhavebeenintroduced.Therefore,anytestsperformedbeforethe
softwarewaschangedmayneedtobeexecutedagainonthenewversionofthesoftware.Thiscanbe
verytimeconsumingiftheregressiontestsarenotautomated.
SomeProductDevelopmentTeamsautomatetheirunittestsaswellasthefunctionalteststhatyouwill
usetomaketheacceptancedecision.OtherProductDevelopmentTeamsdontknowhowtodothisor
dontfeeljustifiedinincludingthecostofautomatedtestsintheircostestimates.Theymayneedtobe
pushedintodoingthisbyyou,theProductOwner,insistingthatautomatedregressiontestsasjustas
importantasusermanualsandbuiltinhelp.Ifyouplantousethisproductforanylengthoftime,
expecttheproducttoneedtoevolve.Theevolutionwillbemucheasierifthetestsareautomatedand
willsaveyoulargeamountsofeffortandwillhelpavoidhavingthequalitydegradeastheproduct
evolves.
Animportantconsiderationinregressiontestingisthechoiceofdatatobeusedinthetests.Regression
testscanbeexecutedwiththesamesnapshotofproductiondataforgenuinerepeatability.Anapproach
thatmaybemorerepresentativeofareal,evolvingproductionenvironmentistorefreshtheregression
testingdataovertime.Thismoredynamicapproachtoregressiontestingsacrificesrepeatabilityin
returnforthepotentialtofindnewproblems.
8.4.4 MaintainingMultipleVersionsoftheSoftware
Therewillbetimesthatyoumaychoosetomaintainseveralversionsofthesoftwareatthesametime.
Acommonsituationisafterrelease,duringthewarrantyperiod,ifyouarealsodevelopingasubsequent
releaseatthesametime.Anothersituationiswhenyouchoosetosupportmultipleversionofsoftware
inproductionandneedtoapplyanybugfixestoallsupportedversions.Beforewarnedthatthereisan
extracosttomaintainingversionsinparallelasanybugfixesdoneinthesupport/warrantystreamwill
needtobepropagatedintothedevelopmentstreamatsomepointandretestedthere.
Sidebar:IncrementalAcceptanceandBugTrackingonAgileProjectsCaseStudy
ArecentITprojectIworkedonwasprettytypicalofhowbugsaredealtwithonagileprojects.The
developerswouldconsultwiththebusinesslead(actingastheProductOwner)astheydevelopedthe
functionalitytomakesuretheyunderstoodwhatshe(thebusinesslead)wantedbuilt.Theywould
jointlydefinetheaccetancetestsforthefunctionalitybeforeorwhilethefunctionalitywasdeveloped.
Whenthedeveloperhadfinishedwritingthecodeandhadfullyunittestedit,theywouldrunallthe
previouslydefinedacceptancetests.Assumingthetestsallpassed,theywoulddeemtheuserstory
readyforacceptancetestingbythebusinesslead.Theywouldwalkovertheher(the15personteam

allsatinthesameteamroom)andaskforafewminutesofhertime.Theywoulddemonstratethe
functionalitytoherbyexecutingthemainacceptancetestsinfrontofher.Shewouldtryafewmore
thingsthatoccurredtoher.Ifeverythingworkedasexpectedthusfar,shewouldsaysomethinglike
Thislooksgoodthusfar.Letmedosomemoredetailedtestingandgetbacktoyouwiththeresults
laterintheday.Thedeveloperswouldthengobacktotheirdeskstocatchontheiremailorpickand
startresearchingthenextuserstorywhiletheywaitedforthefinalacceptancedecision.
Ifthebusinessleadfoundsomethingshedidntlikeduringthispreliminarytesting,shewoulddiscuss
theissuewiththedevelopersandtheywouldjointlycomeupwithoptionsanddecidedhowto
proceed.Ifthismeantchangingthesoftware,thedeveloperswouldgobacktotheirdeskstomakethe
changesandthiswholeprocesswouldberepeatwhentheyweredone.
Ifthebusinessleadfoundanybugswhiledoingthemoredetailedacceptancetestingshewouldgo
overtothedevelopersdesktodiscussthem.Theywouldthenagreediscusswhattodo.Thebusiness
leadhadseveralchoices.Somebugswereseriousenoughthatshewantedthemfixedbeforeshe
wouldaccepttheuserstory;thedeveloperswouldaddresstheseproblemsrightawaydropping
anythingelsetheywereworkingontodoso.Thekeyincentivewasthatanyuserstoriesthatwere
almostfinishedbutwhichhadntbeenacceptedbytheProductOwnerwouldntcounttowardsthe
teamsvelocityfortheiteration.(Theteamwasveryproudofitsvelocityandwasalwayslookingfor
waystoimproveit.)
Anotheroptionshehadwastoaddressthebuginanalreadyplanneduserstorythatwouldbe
implementedinafutureiteration.Shewouldchoosethisoptionifthefunctionalitywasotherwise
workingandtheparticulartestcasewasborderlineastowhichstorysscopeittrulybelongedto.
Athirdoptionwastoturnthebugintoafullblownuserstory.Shechosethisoptionifitdidntfitinto
anyoftheotherstoriesalreadyintheproductbacklog.
Thefinaloptionwastotrackthebugonaseparatebuglist.Eachbugwaswrittenona5x7cardand
addedtothebugmanagementsystemwhichwassimplyadesignatedareaofthewhiteboard.
Becausemostbugswerefixedwithinafewhoursofbeingfound,thislistofbugsonthewhiteboard
rarelyexceededadozenorso.Duringourbiweeklyiterationplanningmeetingsthebusinesslead
wouldpickthestoriesfromtheproductbacklogshewantedbuiltandthebugsfromthebuglistshe
wantedaddressed.Sometimesdeveloperswouldsuggestthataparticularbugbeincludedintheplans
becausedevelopmentofauserstorywouldaffectthesamecode.
Thisapproachtobugmanagementworkedverywellforourteambecausewewerecolocated.We
didntneedadatabaseorfancyreportsbecauseweneverhadmorebugsontheboardthantheteam
couldkeeptrackofatanyonetime.Anditensuredthatthequalityoftheproductalwaysmetour
MQRbecausewedidnthaveaplacetohidethebugs.
Thiswhiteboardbasedapproachwontworkonlarge,distributedprojectsbutthephilosophyoffixing
bugsrightawayratherthanlettingthembuilduptounmanageablenumberswill.
GerardMeszaros

8.5 Summary
TheBusinessLeadisthepersonnominatedbythebusinesstoplaytheroleofProductOwnerforthe
durationofaprojecttodevelopasoftwareintensivesolutiontoabusinessproblem.Theyare
responsibleformakingthedecisionabouthowtobestutilizetheallocatedmoneyorresourcesin
buildingorextendingthesystemfunctionality.
AsummaryoftheresponsibilitiesoftheProductOwnercanbefoundinChapter1TheBasicAcceptance
Process.

Chapter9 ProductManagers
Perspective
AproductmanagerinaproductcompanyplaysaverysimilarroletothatoftheBusinessLeadonanIT
projectwithafewexceptions.Whilethebusinessleadistypicallyengagedonlyforthedurationofthe
projecttodeliverthenewfunctionality,aproductmanageristypicallyresponsiblefortheprofitabilityof
theproductoveritsentirelifetime.Theyaremoreinclinedtothinkintermsofmultiplereleasesofthe
software/product.Theirstrategymayalsobefocusedonaproductline,notindividualproducts,inwhich
casetheintegrationandinteroperabilityamongtheseproductswouldbeofprimaryconcern.Companies
thatbuildproductsarealsomorelikelytohavededicatedtestingresourcesinaseparatetest
department.ManyofthefactorsdiscussedinTheBusinessLeadsPerspectivealsoapplytotheproduct
managersincetheProductManageeralsoplaystheroleofaProductOwner,butinaproductcompany.
Thissectionfocusesonthedifferences.
SeetheProductManagerpersonainAppendixXOrganizationStereotypesandReaderPersonas.
AsaProductManager,youshouldfeeltotalownershipofthesuccessoftheproductinthemarketplace.
Yousurveythemarketforwhatitneedsanddefinetheproducttobebuilt.Youdecidehowmuch
revenuetheproductislikelytogenerateandhowmuchmoneyitisworthinvestingtobuildthe
product.Itisyourjobtoensurethatthesupplierteamunderstandswhatyouwantbuiltandtherelative
prioritiesofthepiecesoffunctionality.
9.1 DefiningDone
Akeypartoftherelationshipbetweentheproductmanagerandtheproductdevelopmentteamis
havingacommonlyunderstooddefinitionofwhatconstitutesdone.Thiscomesintwodimensions:
functionalityandquality.
9.1.1 DefiningMMF
MMFstandsforMinimumMarketableFunctionality(alsoknownasMinimumMarketableProduct
(MMP)orMinimumMarketableFeatures(MMF));itisthesmallestamountoffunctionalitythatyou
coulddelivertoyourcustomerswithoutlosingcredibility.AnythingoverandabovetheMMFaddsvalue
butshouldnotdelaythereleaseifitcannotbecompletedbythedeliverydeadline.Ittakesself
disciplineandagoodunderstandingofthemarkettodefinetheMMF;itissomucheasiertojustthrow
ineverythinganycustomerhaseveryaskedforandcallitareleasebutthisisalmostsuretoresultin
spendingtoomuchandtakingtoolongtodelivertheproduct.
Youmayneedhelptotranslatethehighlevelrequirementsintoadetailedproductdefinition.Business
analysisexpertisemaybeneededtounderstandthebusinessprocessesusedbythepotentialusers.

Productdesignexpertise(e.g.interactiondesignersandgraphicdesigners)maybeneededtohelp
designthefunctionality.Productdevelopmentexpertisemaybeneededtounderstandwhatis
technicallypossibleandconsistentwithhowtheproductalreadyworks.Youmaywanttoinclude
peoplewiththeseskillsaspartoftheproductmanagementteamoryoucanenlistthehelpofother
organizations.Whicheverwayyouchoosetodealwiththeissueyoullneedtoensurethateveryone
understandsyourvisionfortheproductandhowitsatisfiesspecificmarketneeds.
9.1.2 DefiningMQR
Youdontlikereceivingunsatisfactorysoftwareandthesupplierteamhatesbeingtoldthattheir
carefullycraftedsoftwareisnotverygood.Establishingtheminimumqualityrequirement(MQR)isan
importantstepinbuildingatrusting,productiverelationship.AjointlycraftedDoneDoneChecklist
postedinthesupplierorganizationsofficeisagreatwayforthesupplierteamtokeepthefocuson
qualityfrontandcenter.AreleaselevelchecklistasshowninFigure1isagoodwaytosettheoverall
expectationsofwhatthesuppliershouldbeprovidingbeforeacceptancetestingcanstart.
SampleDoneDoneChecklistforRelease1.0
- AllMMFfeaturesareincludedintheRCbuild.
- Asecurityreviewhasbeenconductedandasignoffobtained.
- Testteamisconfidentthatnoneoftheincludedfeatureshasasignificantriskofcausingproblems
intheproductionenvironment,i.e.MQRmet,including:
- configtesting,sidebyside
- performance
- localization
- globalization
- userexperience.
- Businesscomplianceachieved.
- Thereareclear,concisedeploymentandrollbackinstructionsfortheoperationsteam.
- Therearecleartroubleshootingscriptsandknowledgebasearticlesforusebythehelpdesk
representatives.
- Allincludedfeatureshavebeendemoedtoandacceptedbythecustomer.
Figure1
SampleDoneDoneChecklistforaRelease
ThequalitycriteriadescribedinareleaseleveldonedonechecklistasshowninFigure1mayrequire
severalreadinessassessmentcyclesbeforethecriteriaaresatisfied.Toreducethenumberofreadiness
assessmenttest&fixcyclesweneedtoinfluencethelevelofqualitydeliveredbyindividualdevelopers

tothereadinessassessors.Thiscanbeaffectedbyafeatureleveldonedonechecklistasshownin
Figure2.
Theacceptancecriteriaarespecifiedandagreedupon
Theteamhasatest/setoftests(preferablyautomated)thatprovetheacceptancecriteriaaremet
Thecodetomaketheacceptancetestspassiswritten
Theunittestsandcodearecheckedin
TheCIservercansuccessfullybuildthecodebase
TheacceptancetestspassonthebitstheCIservercreates
Nootheracceptancetestsorunittestsarebroken
Userdocumentationisupdated
Userdocumentationisreviewed
Thefeatureisdemoedtothecustomerproxy
Thecustomerproxysignsoffonthestory
Figure2
SampleDoneDoneChecklistforindividualfeatures
Donedonechecklistssuchasthesearecommonlyusedonagileprojects.
9.1.3 ManagingYourOwnExpectations
Themosteffectivewaytoensurethatpoorqualitysoftwaregetsdeliveredistoaskthesuppliertoover
commit.Softwarerequirementsarenotoriouslyhardtopindownandjustashardtoestimate.Forcinga
suppliertodelivertoanunrealisticscheduleissuretobackfireasyouwilllikelygetalatedeliveryof
poorqualitysoftwarethatyouthenneedtoinvestextratimeintobringingthequalityuptobarely
sufficientlevel.Thisisnotarecipeforsuccess!
Itisfarbettertodefinethelevelofqualityrequiredandmanagethescopeoffunctionalitytoensureit
canallbefinishedinthetimeandmoneyallotted.Timeboxedincrementaldevelopmentisaproven
techniqueforachievingontimedeliveryofqualitysoftware.Itistoyouradvantagetoworkcloselywith
thesupplierorganizationtoselectthefunctionalitytobeimplementedduringeachiteration.Youllhave
muchbettervisibilityofprogresstowardsthereleaseandmuchearlierwarningifthefullslateof
functionalitycannotbecompletedbywhateverdeliverydateyouhavechosen.Thisgivesyoutimeto
prunethefunctionalitytofitintothetimeavailableratherthantryingtocramintoomuchfunctionality
andtherebysacrificingquality.

9.2 WhoMakestheAcceptanceDecision?
Asproductowner,youneedtoberesponsibleformakingthebusinessdecisionaboutwhethertoship
theproductasisortodelaydeliverytoallowmorefunctionalitytobeaddedortoimprovethequality
byfixingknownbugs.Youwilllikelyrelyondatafromotherpartiestomakethisdecisionbutdelegating
thisdecisiontooneofthosepartiesisatyourperilbecausetheyareunlikelytounderstandthebusiness
tradeoffsnearlyaswellasyoudo.
Itisreasonableforyoutoexpectthatthepartiesprovidinginformationaboutthequalityoftherelease
candidatedosointermsthatyouunderstand.Theimpactofeverymissingfeatureorknownbugshould
bedescribedinbusinessterms,nottechnicaljargon.Ideally,thisshouldbeeasilytranslatedinto
monetaryimpact(lostordelayedrevenue)andprobability(likelihoodofoccurrence).
9.3 OperationalRequirements
Mostsoftwareproductshaveoperationalrequirements.Thesearetherequirementsofwhoeverwill
havetoprovisiontheruntimeenvironment,installthesoftware,monitoritsperformance,install
patchesandupgrades,startupandshutdownthesoftwareduringservermaintenancewindows,andso
on.Eventhesimplestshrinkwrapproducthasoperationalrequirements(suchasautomaticupdating)
andcomplexserverproductsoftenhaveveryextensiveoperationalrequirements.Ensurethatyouhave
theappropriatepeopleengagedduringtheproductdefinitionandproductacceptanceprocessto
ensurethattheserequirementsarenotmissed.Makesureyouhavetherightpeopleinvolvedinthe
decisionmakingprocesstoensurethattherequirementsaresatisfied.Operationalrequirementsmay
requirespecializedtestingandtesttools.
SomeexamplesofoperationalrequirementsforSoftwareasaServicearesimilartothoseofanITshop:
Thesystemneedstointegratewiththespecificsystemsmonitoringsoftwareinuseatyour
company.
Thesystemneedstohaveupgrade,startupandshutdownscriptsthatcanbeusedduring
automatedservermaintenancewindows.
Thesystemneedstobebuiltusinganapprovedtechnologystack.Theoperations
departmentmaynothavetheskillstosupportsometechnologies.E.g.a.NETshopmaynot
havetheskillstosupportanapplicationbuiltinJavaorPHP.Sometechnologiesmaybe
incompatiblewithtechnologiesusedatyourcompany.
Theoperationsgroupmayhavespecificwindowsduringwhichtheycannotinstallortest
newversionsofsoftware.Thismayaffectthereleasescheduleyouhavedevisedanditmay
makeontimedeliveryofsoftwarebythesupplierevenmorecriticalassmallscheduleslips
couldresultinlonginservicedelays.

Theoperationalrequirementsofasoftwareproductinclude:

Thesoftwarewillneedtointegratewithavarietyofsystemsmonitoringsoftware
frameworks.
Thesoftwaremayneedtoworkinavarietyofhardwareplussoftwareconfigurationsfor
bothserverandclient(possiblybrowser)components.Thismayrequireextensive
compatibilitytesting.Itmayalsoconstrainthefeaturestobedeliveredduetocrossplatform
issuesthatrestrictthefunctionalitytoalowestcommondenominator.

9.4 DealingwithLargeProducts
Whenaproductistoolarge,complexorincludestoomanydifferenttechnologies,itcanbedifficultto
commissionasingleteamtodevelopitforyou.Anotherchallengeistheintegrationofseveralexisting
productseachofwhichhasitsownsupplierorganization.Ifyouhavethisissueyouhaveseveralwaysto
dealwithit.Themoreobvioussolutionistobreakdowntheproductrequirementsintoseparate
requirementsforeachsubcomponentandcommissioncomponentteamstodothework.Themain
issuewiththisfromaproductmanagementperspectiveisthatitmakesproductarchitectureandthe
subsequentsubcomponentintegrationyourproblem.SeethesidebarComponentTeamsforamore
detaileddiscussion.
Theotheralternativeistocommissionfeatureteamsthatworkacrosscomponentsanddividethe
productrequirementsintosubsetofproductrequirementsforeachfeatureteam.Thishasthe
advantageofdealingwithanyintegrationissueswithinthefeatureteam.Thesupplierorganizationmay
needtoprovideamechanismtoensureconsistencyofapproachacrossthefeatureteamsuchasa
virtualarchitectureteamorstandardsteam.SeethesidebarFeatureTeamsonMicrosoftOfficefora
moredetaileddiscussion.Regardlessofwhichapproachyouchoose,youllwanttohavesome
mechanismforcoordinatingtheactionsoftheteams.Twocommonapproachesaretheuseofproject
reviewsandtheScrumofScrumsapproach.
Sidebar:ComponentTeams
Alargeclienthadorganizedtheirteamsaroundthecomponentsoftheirarchitecturewhichconsisted
ofmanycomponentsorganizedinseverallayers.ThebottomlayeriscalledThePlatform.Builtatop
ThePlatformaremanyreusablecomponentsandserviceswecansimplyrefertoasA,BandC.These
componentsareusedbythecustomerfacingcomponentsX,YandZ.Whenmarketingreceivesanew
featurerequesttheproductmanagerdecomposestherequestedfunctionalityintorequirementsfor
eachofthecustomerfacingcomponentsX,YandZ.Thearchitectsofthecorrespondingteamsare
eachaskedtodoafeasibilitystudyofthefunctionalityandtoprovideanestimateoftheeffort.This
mayrequirerequestingnewcapabilitiesfromoneormoreofcomponentsA,BorCwhichmayinturn
requirenewcapabilitiesfromThePlatform.Oncethefeasibilitystudywascompletedandanestimate
wasavailablefromeachoftheaffectedteams,marketingwoulddecidewhethertobuildthefeature
andifso,whichreleasetoincludeitin.Intheappropriaterelease,eachoftheteamsinvolvedinthe

designwoulddotheworkrequiredintheirindividualcomponents.Whenallthecomponentswere
finished,bigbangintegrationwouldstart.Thiswouldoftendiscoverseriousmismatchesbetween
expectationsandwhatwasbuilt.
Otherissuesassociatedwiththecomponentteamapproachincludebalancingworkloadamongthe
teamsanddifficultiesimplementingmultiplefeaturessimultaneously.LarmanandVoddestressthat
theseissuesleadtodelaysandincreasedoverheadduetofrequenthandoffs.Analternativeto
componentteamsisorganizingdevelopmentaroundfeatures,discussedinthesidebarFeatureTeams
atMicrosoft.See[VoddeLarmanOnScalingAgile]and[LeffingwellOnFeatureVsComponentTeams]fora
comparisonbetweencomponentandfeatureteams

Sidebar:FeatureTeamsatMicrosoft
Afeatureisanindependentlytestableunitoffunctionalitythatiseitherdirectlyvisibletoacustomer
(customerfacingfeature)orisapieceofinfrastructurethatwillbeconsumedbyanotherfeature.
Featureteamsaresmallinterdisciplinaryteams(512people)thatfocusondeliveringspecificproduct
features.AsamplefeatureteamcouldbetheoneinchargeofRibboninMSWord.Anotheroneisthe
oneinchargeofSpellchecker.AthirdoneisinchargeofAddressLabels.
MicrosoftimplementstheFeatureTeamsstrategyusingamethodologycalledFeatureCrews.It
extendsthegeneralstrategywithspecificguidelinesonhowtomanagethecodebaseissuchawayas
toavoidpartiallyfinishedfeaturesfromaffectingthestabilityofthemainbranch.Theideaisforall
disciplines(development,PM,UxD,testing)toworkcloselytogetheronprivatebuilds(intheirown,
isolatedprivatefeaturebranches)andonlyaddthefeaturetotheproduct(themainbranch)whenitis
FeatureComplete(similarlytoDoneDonedescribedearlier).ThefeatureinFeatureComplete
stateisexpectedtobefullyimplemented,sufficientlytestedandstableenoughforeithera)dog
fooding(usingonesownproductinhouse)andsharingwithcustomersinaCTP(Community
TechnologyPreview)forcustomerfacingfeatures,orb)codingagainstbyanotherfeaturecrewfor
featuresthatdealwithunderlyinginfrastructure.
Thetypicaldurationforfeaturedeliveryisbetweenthreetosixweekstheideaistoensureasteady
flowoffeaturesandregular,frequentfeedbackfromthecustomers.Importantly,eachfeatureteamis
freetodecidewhatdevelopmentapproach/processtouse,aslongastheiroutputmeetsallquality
gates(commonforallfeaturecrews)anddoesntdestabilizetheproduct.(Foracasestudyontheuse
oftheFeatureCrewmethodologyatMicrosoftVisualStudioToolsforOfficeproductunitanddetails
onintegrationoffeaturesintoparentandmainbranches,see[MillerCarterOnLargeScaleAgile])
9.5 BugTrackingandFixing
Usersandtesterswillreportbugsnomatterhowgoodajobyoudobuildingthesoftware.Someof
thesebugswillbelegitimatedefectsintroducedintothesoftwareaccidently.Otherswillberequests
fornewfunctionalityyouconsciouslychosetoleaveoutoftherelease.Exceptfromacontractual(who
pays)perspective,bugsneednotbetreatedanydifferentlyfromfeaturerequests.Eitherway,youneed

todecidewhattimeframeyouwantthefunctionalityofthesystemchangedandthereforeinwhich
streamofsoftwareyouwanttoincludeit.TheConcernResolutionModelprovidesacommonwayof
thinkingaboutbugs,featurerequestsandotherprojectissues.
9.6 MaintainingMultipleVersionsoftheSoftware
Therewillbetimesthatyoumaychoosetomaintainseveralversionsofthesoftwareatthesametime.
Acommonsituationisafterrelease,duringthewarrantyperiod,ifyouarealsodevelopingasubsequent
releaseatthesametime.Anothersituationiswhenyouchoosetosupportmultipleversionofsoftware
inproductionandneedtoapplyanybugfixestoallsupportedversions.Beforewarnedthatthereisan
extracosttomaintainingversionsinparallelasanybugfixesdoneinthesupport/warrantystreamwill
needtobepropagatedintothedevelopmentstreamatsomepointandretestedthere.
9.7 Summary
TheProductManageristypicallytheProductOwnerofasoftwareintensiveproductandisresponsible
formakingthedecisionabouthowmuchmoneyorresourcestoinvestinbuildingorextendingthe
productfunctionality.
AsummaryoftheresponsibilitiesoftheProductOwnercanbefoundinChapter1TheBasicAcceptance
Process.

Chapter10 TestManagersPerspective
SomeorganizationshaveaseparateTestManagerrolewhileothershavethetestersreportingdirectlyto
theDevelopmentManageroreventheBusinessLeadorProductManager.WhentheTestManagerrole
exists,thetestmanagerisusuallyresponsibleforplanningthebulkofthetestingactivitiesand
managing/coordinatingtheirexecution.Thetestmanagermayormaynotactasthegatekeeperthat
is,maketheacceptancedecisiononbehalfofotherstakeholders.Itisimportantforallpartiesto
understandtheroleoftheTestManagerinthisregard.
SeetheTestManagerpersonainthecompanystereotypecalledAProductCompanyinAppendixX
ReaderPersonasforanexample.
TheTestManagersrolecanbeadifficultonetodowellbecausesomanyprojectfactorsareoutside
yourcontrol.Testplanningisessentialforallbutthemosttrivialprojects;whatthetestplanspecifies
dependheavilyontheroleoftestingasitrelatestotheproductdevelopmentteamandtheproduct
manager.
10.1 TestManagersRoleinAcceptanceDecision
Theroleoftestingseemssimpleenough:usetheproductinvariouswaysandreportanybugsthatyou
find.Butthisisnottheonlyresponsibilityofthetestorganizationinmanyenterprises.Makesureyou
understandwhetheryouandthetestingteamaredoingreadinessassessment,acceptancetestingor
makingtheacceptancedecision.
10.1.1 TestingasAcceptanceDecisionMaker
Insomecircumstances.Thetestorganizationisexpectedtocollectdatatohelptheproductowner
decidewhetherornotthesoftwareisreadytobereleasedtousers.Makesureyouhaveaclear
understandingofwhetheryouarethegatekeeper(acceptancedecisionmaker)oraresupplying
informationtosomeoneelsewhowillbemakingthedecision(preferred.)Recognizethatthe
acceptancedecisionisabusinessdecision.Ifyouaregoingtomakeit,youhadbetterunderstandthe
businessfactorswellenoughtomakeagooddecisionthatyoucanexplaintoeveryone.Otherwise,
youlljustbethebadguyholdinguptheshow.
IfyouaretoactastheAcceptanceDecisionMaker,youmusthaveaccesstoallthebusinessrelevant
informationtomakeagooddecision.Thatis,youmustunderstandthebusinessconsequencesof:
acceptingthesoftwaregiventhenatureofthebugsthatareknowntoexistandthelevelof
confidencethatallseriousbugshavealreadybeenfound,and,
notacceptingthesoftwaretherebydelayingthedelivery.

Eachofthesechoicescanhaveseriousnegativebusinessconsequencesandtheacceptancedecision
cannotbemadewithoutafullunderstandingofthem.TheTestManagershouldnotaccept
responsibilityformakingtheacceptancedecisionlightly.Inmostcasesitisbetterforthisbusiness
decisiontobemadebyabusinessperson,typicallytheproductmanager,businessleadoroperations
managerbasedoninformationprovidedbythetestmanager.
10.1.2 TestingasAcceptanceTesters
Whenyouareprovidinginformationtoanotherpartytomaketheacceptancedecision,makesureyou
havecommonunderstandingofwhatinformationthatparty(orparties)willrequiretomakethe
decision.Ensurethetestersunderstandtheusersforwhomtheyareactingasaproxysotheycandefine
testthatreflectrealistic(notnecessarilyalwaystypical)usageofthesoftware.UserModels(eitheruser
rolesorpersonas)canbeaneffectivewaytogainunderstandingofrealusers.
10.1.3 TestingasReadinessAssessors
IfyouaredoingreadinessassessmenttohelptheDevManagerdecidewhetherthesoftwareisreadyfor
acceptancetestingbythebusinessusers,youllwanttobuildagoodrelationshipwiththedevteamso
thatyoucanhelpthemunderstandthequalityofwhattheyhaveproducedandhowtheycanimprove
it.Embeddingtesterswiththedevelopmentteam(oftenusedinconjunctionwithatechniquecalled
pairtesting)canbeagoodwaytohelpthemlearnhowtobuildqualityinbytestingcontinuouslyrather
thantossinguntestedsoftwareoverthewalltothetestteam.(Kohldescribesbenefitsofpairtestingin
thisexperiencereport[KohlOnPairTesting])
10.2 TestPlanning
Youwillprobablybeexpectedtocoordinatealltestingactivitiesandkeeptrackofthetest
environmentsused,bugslogged,andversioningrelatedtothefixesperformedonthesourcecode,all
thewayfromreadinessassessmentthroughtoacceptancetesting.Thisworktypicallygoesbythename
TestPlanning.Youllneedtocommunicatethisplantoallinterestedparties;especiallythosewhoare
expectedtoexecutepartsoftheplanandthosewhoareinterestedinknowingwhatkindsoftestingwill
bedonebywhom,whereandwhen.ThiscommunicationoftentakestheformofaTestPlandocument
butitcanalsobedoneorcomplementedorallythroughtargetedpresentations,discussionsor
workshops.Manyofthebesttestplanscomeaboutbyinvitingtheinterestedpartiestoparticipatein
thetestplanningprocesseitherinoneononesessionsorviaworkshops.
10.2.1 TestStrategy
Youwilllikelyendupbeingthedefactoowneroftheoverallteststrategy.Theonlyothercontender
wouldbethesupplierorganizationandiftheywanttobeinvolved,probablyonthetestautomation
side,byallmeansencouragetheirinvolvementbecauseitwillmaketestautomationmucheasier.
Theteststrategyincludesthemajordecisionssuchaswhethertestingwillbedoneinafinaltestphase
orincrementally,whethertestingwillbeprimarilyscriptbasedtestingorexploratorytestingora

combinationofthetwo.Exploratorytestingisaveryeffectivewayoffindingbugsfastbutitisnt
intendedtoberepeatable.Scriptbasedtestingisbetterforensuringaknowndegreeoftestcoverage
andshouldbeautomatedtoallowrapidregressiontestexecution.
Anotherkeydecisionisregardingtheroleanddegreeofuseofautomatedtoolsincludingautomated
executionoffunctionalandparafunctionaltests,theuseofmodelbasedtestcasegeneratorsandthe
useofautomationaspowertools(includingtheuseofautomatedcomparatorsorverifiers)tosupport
manualtestactivities.Someofthesedecisionsrequirelongleadtimestodecidewhethertochoose&
acquireorbuildtoolsortohiretheappropriateresourcesandthereforemustbemaderelativelyearly
intheproject.Thisdoesntimplythatyouwontrefinethemovertimeasmoredetailedinformation
comestolight.
Theteststrategywillbeheavilyinfluencedbythenatureofyourrelationshipwiththeorganization
developingthesoftware.Theidealcaseiswhentheyarepreparedtocollaboratewithyoubydoing
incrementaldeliveryoffunctionalitytosupportincrementalacceptancetesting.Collaborationontest
automationanddesignfortestabilityisalsoakeyfactorinreducingtestexecutiontimeandcost.Ifthis
levelofcollaborationisnotavailableyoumayneedtohiretestautomationengineersandtest
toolsmithswithstrongprogrammingskillstosupportyourtestautomationactivities.
10.2.2 TestAutomation
Anotherareathatgreatlybenefitsfromcollaborationwiththeproductdevelopmentteamistheareaof
testautomation.Akeysuccessfactorisdesignfortestabilityofthesystem.Onlythesupplier
organizationcandothissoyouneedtoworkwiththemtobuilditin.Agoodwaytoincentthemto
collaborateistoprovidethemwithaccesstothe1buttonautomatedexecutionoftheacceptancetests
youdefine.Thisprovidesthesupplierorganizationwithaninstantscorecardthattellsthemhowthey
aredoingatdeliveringqualitysoftware.Everyonehatessurprisesandnothingannoysadevelopment
teammorethanworkinghardtodeliverqualitysoftwareandbeingtoldafterthefactthatitwasnt
goodenough.
Whilethedevelopmentteamneedstomakethesystemamenabletotesting,thetestautomationtools
canbebuiltbyeitherthesupplierorganizationorthetestteam.Thelatterwillrequiretechnicaltesters,
sometimescalledtestautomationengineers.Eitherway,thetestsshouldbedevelopedprimarilybythe
testersorbusinesspeople/analyststoensuretheyreflectrequirementsratherthandesigndecisions.
Testautomationisnotappropriateinallsituations,whereformsofmanualtestingaremoreeffective
oreconomical.SeeSection17.2inChapter17(PlanningforAcceptnace)inPartIIIforadiscussionof
whereautomationisappropriate.Itsthejobofthetestmanagertodeterminethetestautomation
strategy.
10.2.3 AgreeingonExpectations/Requirements
Onepurposeoftestingistoverifythesystemmeetstherequirements.Unfortunately,therequirements
areoftenvagueorambiguousandthismakesverifyingthemadifficultproposition.Onepersonsbug
canbeanotherpersonsfeature.Testersthinkdifferentlyfromanalystsorbusinesspeopleandthisis
bothablessingandacurse.Theyllcomeupwithallmanneroftestscenariosthattherequirement

specifiersneverimagined.Thisresultsinadefactodivergencebetweentherequirementsandthe
specificationsthatneedtobemanaged.Theproductowner(ProductManager,BusinessLead,etc.)
needstoagreethattheseadditionaltestscenariosareinfactrequirements.
Therequirementsneedtoaddresstheneedsofallstakeholders,notjusttheusers.Operational
requirementsneedtobetested.Ifthetestteamdoesnthavetheskillstoexecutetheparafunctional
teststhenmakesurethattheproductdevelopmentteamexecutesthemandprovidestestplansto
reviewaheadortimeandtestresultsaspartofthehandovertotesting.Theacceptancedecisionmaker
willtoknowthattheparafunctionaltestingwasdoneandmayevenwanttoseetheresults.
Thepreferredsolutionistotreattestscriptsasanextensionoftherequirementsbyusingthemto
illustratehowspecificrequirementsplayoutintheproduct.Thisimpliesthattheproductmanageror
businesslead(ortheirteammembers)needtoagreethatthetestsinterprettherequirementscorrectly.
Ideally,thetestcaseswouldbearticulatedbeforethesoftwareisbuiltsothatthedevelopmentteams
canusethetestcasestounderstandhowthesystemshouldbehave.ThisisknownasAcceptanceTest
DrivenDevelopment.
10.2.4 AgreeingonDone
Arewedoneyet?Thatisaquestionthatneedstobeaskedandansweredcontinuously.Anddone
doesntalwaysmeanthesamething.Onepersonsdefinitionofdonemaybeanotherpersons
definitionofnotevenclosetodone!Itiscriticaltogetagreementonhowdonesoftwareneedstobe
beforeitgoesthrougheachofthequalitygatesofthegatingmodel.Whenissoftwareconsidered
readyforacceptancetesting?Whenisitconsideredacceptable?Theseneedtobeagreedupon
betweenthevariouspartiesinvolved.Thesupplierandtestingneedtoagreeontheminimumquality
requirementofsoftwarebeforeitisreadyfortesting.Thisbecomestheexitcriteriaforthereadiness
acceptancephasethatthesupplierteammustensureismet.AprominentlypostedDoneDone
Checklistisagoodwayforthesupplierteamtokeepthisqualitycriteriafrontandcenter.
10.2.5 EstimatingTestEffort
Estimatingtheamountoftimeitwilltaketogetthroughonecompletecycleoftestingisachallengebut
onethatcanbeovercomewithexperience.Therealchallenge,however,isguessinghowmanytest&fix
cycleswillberequiredbeforethesoftwareisofgoodenoughqualitytoreleasetoacceptancetestingor
tothemarket.Theneedtoguesscanbeavoidedbydeliveringsoftwareincrementallyandtestingeach
incrementassoonasitisavailable.Thishelpsinseveralways:
Itreducestheamountofsoftwarethathasntgonethroughthetest&fixcycleatleastoncewhen
thefinalroundoftestingoccurs.Hereuntestedsoftwarecanbethoughtofasinventoryand
incrementaltestinganddeliveryasinventoryreduction,whereinventoryisconsidereda
formofwaste(see[PoppendiecksOnLean])
Itprovidesdataonhowmanytest&fixcyclesarerequiredforsoftwaredeliveredbythissupplier
organizationearlyenoughtoallowthetestplanstobeadjustedtoensureontimedelivery
ofsoftware.

Itprovidesfeedbacktothesupplierteamonthequalityofthesoftwaretheyaredeliveringearly
enoughtoallowthesupplierteamtimetolearnhowtodeliverbetterqualitysoftwarethat
requiresfewertest&fixcycles.TheDoneDoneChecklistmayneedtobeupdatedbasedon
whatwaslearned.

10.3 ConcernResolution
Testingwillinvariablyresultinanumberofconcernsbeingraised.Someoftheseconcernswillbebased
onfailedtestcasesandothersmaybebasedongeneralimpressionsabouttheproduct.Allconcerns,
however,arebasedonexpectationsanditisimportanttoensurethattheexpectationsarecorrect.Test
casesthatdontmaptoactualrequirementscanresultinbugsthatwillbeclosedasByDesign.Some
failedtestcasesmaypointoutinconsistenciesbetweenhowthesystemoperatesandhowthetester
expectsthesystemtooperate.Thesemaypointtolegitimateusabilityissuesifthetestersmental
modelofthesystemmatchesthatofrealusers.
Partoftheroleofmanytestorganizationsistoactasthesecondopinion,thehouseofsobersecond
thought,aboutthesoftwarebeingbuilt.Itisimportanttobalancethisformofrequirementselaboration
withtheneedtogettheproductoutinatimely,andevenmoreimportantly,predictablefashion.
Discoveryofsuchrequirementsissuesduringafinaltestingphasemakestheproductmanagersjob
harderbecausetheinformationcomestoolatetoaddresswithoutcompromisingtheschedule.
Therefore,thetestmanagershouldstrivetofindsuchissuesasearlyaspossiblesothattheproduct
managercanmakedecisionswithouttheirbacktothewall.
TheConcernResolutionModeldescribesagenericwaytothinkaboutallmannerofconcernsincluding
bugs,requirementsissues,projectissuesandothermismatchesbetweenexpectationsandactualor
perceivedbehaviors.

Chapter11 DevelopmentManagers
Perspective
Thedevelopmentmanagermaygobymanyofficialtitlesbutforourpurposes,thisisthepersoninthe
supplierorganizationwhoisaccountablefordeliveringsoftwaretotheproductowner(PO)whetherthe
POistheinternalbusinesssponsorofanITprojectortheproductmanagerinaproductcompany.Where
aseparatetestorganization(oftencalledQualityAssurance,IndependentVerificationorsomevariation
onthese)existsforthepurposeofdoingacceptancetesting,thedevelopmentmanagerisresponsiblefor
makingthereadinessdecision,thatis,thedecisiontodeliverthesoftwaretowhoeverisresponsiblefor
makingtheacceptancedecision.Forlargerproductsthedevelopmentmanagermaybeassistedbya
solutionarchitect;forsmallerproductsthedevelopmentmanagershouldunderstandthesolution
architectsresponsibilitiesandeithercarrythemoutthemselfordelegatetosomeinthedevelopment
team.
SeetheDevelopmentManagerpersonainthetwocompanystereotypesinAppendixXReader
Personasforanexample.
AstheDevManageryourkeyresponsibilityistomanagethedevelopmentofthesoftwareinsuchaway
thattheproductowner(productmanager,businesslead,etc.)willacceptthesoftware.Youshoulddo
thisinawaythatmaximizesthevalue(orutility)thatthesoftwarewillprovidetotheproductowner(by
meansofvalueprovidedtotheirusersorcustomer)whileminimizingthecost.Thebestwayto
minimizecostistobuildtherightsoftwareandtobuildittherightwaythefirsttimeanddeliveritto
theacceptancetestingorganizationand/orproductownerassoonaspossible.Thisavoidsexpensive
andtimeconsumingtest&fixcyclesinwhichsomeoneelseteststhesoftware,findsbugswhichthey
thenaskyoutofix.Thisdelay(aformofwaste)contributessignificantlytochurnandcost.Whetherthis
istheproductownerorsomeonelookingoutfortheirinterests,thefewerbugstheyfind,theless
reworkyouneedtodo,thelowertheoverallcostandthemorepredictabletheschedulebecomes.
11.1 TheRoleofReadinessAssessment
Mostdevelopmentteamswanttodogoodwork.Theydontwanttodeliversubstandardsoftwareto
theircustomer.Mostcustomerswanttoreceivegoodqualitysoftware.
Deliveringgoodqualitysoftwareiswhatthecustomerexpectsfromthesupplier.Thisisareasonable
expectation,onethataprofessionalsoftwaredevelopmentorganizationshouldbepreparedtomeet.
Deliveringgoodqualityworkingsoftwareisnteasyortrivial;ifitwere,theproductownerprobably
wouldntneedadevelopmentmanager!

So,whatdoesittaketodelivergoodqualitysoftware,firsttime,everytime?Whatdoesittaketobea
professionalsoftwaredevelopers?Thecompletedetailsofasoundsoftwareengineeringprocessare
beyondthescopeofthisbook.But,therelationshipbetweenthesoftwaredevelopmentorganization
andthepartiesinvolvedinacceptancethesoftwarearenot.Partofthedevelopmentprocessneedsto
beanhonestselfassessmentofthesoftwaretheteamhasproduced.Iftheteamsaysitisready,giveit
tothetestteam.Iftheteamsaysitisntready,askthemwhatstillneedstobedonetomakeitready.A
gooddevelopmentteamwillalwaysbeabletosaysomethingisreadyandshouldbeabletoclearly
articulatetothetestteamwhatisreadyandworthtesting.
RobertC.Martinarguesthatitisirresponsibleforadevelopertoshipasinglelineofcodenotcovered
byatestandtestsmustbekepttothesamelevelofcodequalityastheproductioncode
[MartinOnProfessionalismAndTDD].Althoughthisargumentappliestodifferentdegreesdependingon
whatisbeingbuilt,thedevelopmentmanagerisresponsibleformakingsurethatthecodeisadequately
securedbysolidsuiteoftestsbeforetheproductownerevaluatesit.
11.2 EffectiveReadinessAssessment
11.2.1 WhosJobisQuality?
Howwouldyoufeelaboutbeingthefirstpersontotryabrandnewproductthatnoone,noteventhe
builderhadtriedbefore.Howmanypeopleyouknowwouldbecomfortableinthatsituation?Ifwe
applythesamelogictothesoftwarewebuild,howmanypeopledoyouknowthatwouldwanttobe
thefirstonetotryapieceofsoftwarethatnooneelsehastriedbefore?Yetthisisexactlywhatmany
developmentteamsdowhentheythrowuntestedorpoorlytestedsoftwareoverthewalltothetest
organizationorthecustomer.Manypeoplewouldarguethatthisisjustplainunprofessional.
Allsoftwareshouldbetestedbythedevelopmentorganizationbeforebeinghandedtoanyoneelsefor
testing.
11.2.2 BuildingQualityInStartwiththeEndinMind
Buildingsoftwareshouldnotfeellikeaguessinggamewheredevelopersguesswhatsrequiredand
testersorusersshoutoutwrong!Thedevelopmentorganizationneedstohaveaclearunderstanding
ofwhatsuccesslookslikebeforeitstartsbuildingthesoftware.Partoftheproblemisthatdevelopment
teamsoftendonthaveagoodsenseofhowthesoftwarewillactuallybeused.Developersdonotthink
likeauserbecausemostdevelopersthriveoncomplexitywhilemostusersabhorit.Thismakesit
challengingfordeveloperstodoagoodjobofreadinessassessmentwithoutoutsidehelp.Acceptance
testsprovidedbytheproductownerortestingdonebytesterscanbridgethisgap.Theformeris
proactive,positiveinputthathelpsdevelopmentteamsunderstandwhatdonelookslikebeforethey
buildthesoftware.Thelatterisnegative,reactivefeedbackthattellsthedevelopmentteamyou
haventdoneagoodjob.Whatkindofguidancewouldyouprefer?
Aprojectcharterisonewaytostarttheprocessofdevelopingthiscommonunderstanding.
Requirementsdocuments,usermodels,usecasesanduserstoriesareallwaysthatwetrytodevelopa

commonunderstandingbetweenthesupplierandthecustomer.Butthesestaticdocumentswrittenin
naturallanguagetypicallycontainmanyambiguousstatementsandinmostcasesdonotprovide
enoughtosufficientlydescribewhatsuccesslookslike.Thedesignsandworkestimatesprovidedbythe
developmentteamthereforelikelycontainassumptions,manyofwhichareeitherwrongorwillresultin
aproductthatismorecomplexandexpensivetobuild.Itiscriticaltoclarifythesepotential
misunderstandingsasquicklyaspossible.
Asthedevelopmentmanageryoushouldworkwiththeproductownerandacceptancetestteamto
ensurethateveryoneontheteamhasaccesstothedefinitionofsuccessintheformofacceptancetests
beforethesoftwareisbuilt.Thisshouldincludeanythingthatisusedasanacceptancecriteriaincluding
bothfunctionalteststhatverifythebehaviorofthesystemfeaturebyfeatureandparafunctional
qualitycriteriasuchassecurity,availability,usability,operationalcriteriaetc.Thisprocessisknownas
AcceptanceTestDrivenDevelopment(akaExampleDrivenDevelopmentorStorytestDriven
Development).Thismayrequirethatproductownerortestersprepareacceptancetestsearlierinthe
projectthantheymighthavetraditionallydone.Itmayalsorequirethemtobemoreinvolvedforthe
durationoftheprojectratherthanjustatthebeginningandtheendsothattheycananswerany
questionsthatcomeupduringtheinterpretationoftherequirementsandalsodoincremental
acceptance.
11.2.3 DefectPreventionbeforeDefectDetection
Traditionalapproachestotestingfocusondefectdetection.Thatis,theemphasisisonfindingbugs
ratherthanpreventingtheiroccurrenceinthefirstplace.Howcantheemphasisbechangedto
prevention?Itisntenoughtodefinethetestsaheadoftime;wemustalsorunthemfrequently.By
doingso,wealwaysknowthescore.Thetestresultstellushowfarfromdoneweare.Theyprovidea
veryclearindicationoftheprogresstowardsdone,onethatdoesntrequirealotofextraworkto
calculateandonethatishardtofudge.Thetestresults,availablethroughouttheproject,providethe
stakeholderswithvisibilityintotheprojectinamuchmoretransparentfashionthantraditionalmetrics
measuringprogressagainstaphasedactivityprojectplan.
Topreventdefectswedefinethetestsbeforebuildingthesoftware,automatethem,andrunthem
frequentlywhilebuildingthesoftware.Weinstitutesimpleteamnormstoavoidregressing:teststhat
havepassedbeforemustcontinuetopassasnewfunctionalityisadded.Noexceptions!Eitherthetest
hastobechanged(changestofunctionalacceptancetestsmayrequirethecustomersagreement)or
thecodehastobechangedtoreturnthetesttopassingstatus.
Whatroledothevariouskindsoftestsplayinthisprocess?Functionaltestsdefinedbytheacceptance
testersdefinewhatdonelookslikeandthesoftwarecannotbedeliveredtothemwhenanyacceptance
testsarefailing.Automatedunitandcomponenttestsdefinethedesignintentofthesoftware;writing
thesefirstandwritingjustenoughcodetomakethempassensureswedontbuildunnecessary
softwareandthatthesoftwarewedowritesatisfiesthedesignintent.Italsoensuresthatthesoftware
youbuildisdesignedfortestability,acriticalsuccessfactorfortestautomation.Anotherbenefitisthat
theyactasalargechangedetectorthatwillinformtheteamofanyunexpectedchangesinbehaviorof
thesoftware.Thishelpstheteamcatchregressionbugsbeforetheycansneakthroughtotheusers.

11.2.4 ReduceUntestedSoftware
InleanthinkingsuchasexemplifiedbytheleanmanufacturingparadigmusedbyToyota
[ShingoOnToyotaProductionSystem],unfinishedwork(inventory)isconsideredaformofwaste.In
softwaredevelopment,softwarethathasbeenwrittenbuthasnotbeenacceptedisunfinishedwork.
Weshouldstrivetofinishthisworkassoonaspossiblebydoingacceptancetestingassoonaspossible
afterthesoftwareiswrittenandreadinessassessmentiscompleted.Thisincrementalacceptance
testingrequirescollaborationbetweendevelopment,testingandtheproductownerasallpartiesmust
bepreparedtoworkfeaturebyfeatureratherthanwaitingforthewholesystemtobeavailablebefore
anyacceptancetestingisstarted.
Theminimumqualityrequirement(MQR)shouldbeagreeduponaheadoftimewiththetesting
organizationortheproductowner.Ideally,anyandallteststhatwillberunaspartoftheacceptance
testingshouldberunbythesupplierorganizationbeforemakingthesoftwareavailableforacceptance
testing.Knowndefectsshouldbeidentifiedaheadoftimetoavoidwastingpeoplestimetestingbroken
functionality.
11.2.5 WhatKindsofTestsAreRequired?
Functionaltestsshouldberunassoonasthecorrespondingfunctionalityisbuilt.Thisshouldinclude
businessworkflowteststhatverifyendtoendbusinessprocesses,usecaseteststhatverifythevarious
scenariosofasingleusecaseorbusinesstransaction,andbusinessruleteststhatensurethatbusiness
rulesareimplementedcorrectly.
Operationalrequirementsalsoneedtobeverifiedbyensuringthatacceptancetestsprovidedbythe
operationsstakeholdersarerunandresultsareanalyzedregularly.
Parafunctionaltestingshouldbedoneonaregularbasisassoonasenoughsoftwareisbuilttoallow
themtoberun.Theearliertheyarerun,themoretimethesupplierhastocorrectanydeficienciesthat
arediscovered.Thisisespeciallyimportantwithparafunctionaltestsbecausechangingthepara
functionalattributesofthesystemmayrequireachangeinarchitecture,apropositionthatmayget
moreexpensivethelaterthechangeismade.
Manyofthesetestscanbedonemuchearlierintheprojectiftheearlyfocusoftheprojectistobuilda
walkingskeletonoftheapplication.Thewalkingskeletonimplementsthefullarchitectureinavery
minimalistway.Forexample,allthelogicmightbehardcodedtherebysupportingonlyasinglehighly
simplifiedbusinesstransactionorworkflow.Butallthemajorarchitecturalcomponentswouldbe
presenttoensuretheruntimecharacteristicsweretrulyrepresentativeofthefinishedproductevenif
thefunctionalbehaviorwasnotimplemented.
11.2.6 SharpeningtheSaw
Anybugsthatarefoundduringacceptancetestingshouldbeasurpriseandshouldpromptthequestion
howdidthatslipthroughourreadinessassessment?Clearly,thesoftwarewasnttrulyready.Ifthe
answerisbecausetheyuseditadifferentwayfromwhatweexpectedthenthesupplierorganization
hastodoabetterjobunderstandingtheusersofthesoftware.Therefore,everybugfoundbecomesa

learningopportunityfortheteambypromptingthemtolookforwaystoimprovehowtheybuild
software.ForexampleatMircrosoftspatterns&practicesgroup,duringtheWebServiceSoftware
Factory:ModelingEditionproject,theteamdidjustthatonseveraloccasions,modifyingtheirCIsystem
topreventissuesfromreoccurring.
Asmanagerofthedevelopmentorganizationyouneedtocreatetheconditionsinwhichthe
developmentteamcanproducetherightsoftwarebuilttherightway.Thismayrequireovercoming
organizationalandculturalhurdles.Youneedtoworkwithyourcounterpartsinthetestorganizationto
breakdownthetraditionaladversarialrelationship(ifoneexists)andworktogethertocreateamore
collaborativerelationship.AsAdeMiller,patterns&practicesdevelopmentlead,eloquentlyputit:I
thinkoneofthekeythingsaDevManagercandohereistrytosendtheclearmessagethattestingand
thetestorganizationareimportant.Infartoomanycasestestersareseenassomesortoflesser
functionbecausetheylackthetechnicalskillsofdevelopers.Thetestorganizationhastheskillsto
helpyourteambuildbetterqualitysoftware,notjusttotellyouwhenyouhavent.

Chapter12 UserExperienceSpecialists
Perspective
Someorganizations,especiallythosethatbuildconsumerproducts,haveaseparaterolethatis
responsibleforensuringagoodandconsistentuserexperience.Whatistheroleofpeoplewiththiskind
ofskillsetinthemakingoftheacceptancedecision?
SeetheUserExperienceSpecialistpersonainthecompanystereotypecalledAProductCompanyin
AppendixXReaderPersonasforanexample.
Asapersonwiththeresponsibilityofuserexperienceyoumayberesponsiblefordesigningtheuser
experienceassociatedwiththeproduct.Thismayinvolvevariouskindsofactivitiestobetterunderstand
andcharacterizethepotentialusersincludingethnographicresearch,usermodeling,taskmodeling,etc.
Itlikelyalsoincludesverificationactivitiessuchasusabilitytestingatvariouspointsthroughoutthe
designanddevelopmentprocess.
12.1 TheRoleofUserExperienceintheAcceptanceDecision
Itisimportanttohaveaclearunderstandingwiththeproductowner(productmanagerorbusiness
lead)astowhatyourroleshallbeinmakingtheacceptancedecision.Itcouldbeanyof:
1. Beinganinputintotheproductdefinitionprocessbutnodirectinvolvementinthe
acceptancedecision.Thatis,helpingtheproductownerdefinetheproductbutleavingitto
otherpartiestodefinetheacceptancecriteria.
2. Beinganinputintothesoftwaredevelopmentprocesswithnoinvolvementinthe
acceptancedecision.Thisissimilartothefirstpointbutworkingmorecloselywiththe
softwaredevelopmentteamratherthantheproductownership/definitionteam.
3. Asinpoint1butalsoprovidingusabilitytestingdatafortheacceptancedecisionmadeby
theproductowner.
4. Asinpoint1and3butalsohavingasayintheacceptancedecisioneitheraspartofthe
productownersteamorasapartofaseparate,parallelacceptancedecision.

Forproductswhereusabilityisakeydifferentiator,thepreferredrolewouldbeeither3or4.For
productswithcaptiveuserssuchassystemsdevelopedforinternalusersitismorelikelytobe1or2.
Inanyofthesemodelsitisimportanttoworkwiththebusinessleadorproductmanagertodefinethe
usabilitycomponentoftheacceptancecriteriaasearlyaspossibletogivethedevelopmentteamas
muchguidanceaspossible.Earlyintheprojectthismayconsistofdescriptionsofthekindsoftesting

proceduresthatwillbeusedtoverifyusability;laterintheproject,itmaybespecificfunctionalitytobe
testedsuchasthenoviceuserstop5transactions,specificusabilityguidelinestobecompliedwith,and
soon.
Whereusabilityisimportantandhowtoachieveitisunclear,itiscrucialtoaddresstheusabilityrisksas
earlyaspossible.Someoftheserisksmaybeaddressedthroughtheuseoflowcostusabilitytesting
techniquessuchassketchingandWizardofOztesting,aformoflowfidelityusabilitytestingconducted
withpaperproductsandmockups.Visualcommunicationisverypowerfultopresentandvalidate
ideas.Abilitytogatherfeedbackfromtheuserseffectivelyisasimportantasproductionofthe
prototypes.Figure1presentsasamplelowfielectronicprototypebuiltwithMicrosoftSketchFlow
[SketchFlow]thatauserisabletopreviewinabrowserandproviderapidfeedback.

Figure1.
UseraddingfeedbackonaSketchFlowprototype
Othersituationmayrequiretheconstructionofelectronicprototypesforhigherfidelityusabilitytesting.
Constructionofspecialpurposehighfidelityprototypesmaybeavoidediftheactualsystemcanbebuilt

anddeliveredincrementallysequencedbyusabilityrisk.Thismayrequireworkingwiththeproduct
ownertohelpthemunderstandtherisksandconsequencesandtousethesetoinfluencetheorderin
whichfunctionalityisbuilt.Usabilityrelatedriskscanbemitigatedbydefininganinternalmilestoneat
whichsufficientfunctionalityisavailabletoallowusabilitytestingoftheactualproductwithrealor
proxyusers.Thisrequirescollaborationbetweentheproductownerandthesupplierorganizationtodo
incrementaldevelopmentanddeliveryofthesoftwarebasedontheusabilitytestschedule.Theareas
withhighestimpactofusabilityissuesshouldbebuiltandtestedfirsttoensureenoughtimeforany
usabilityrelatedissues,whichmaybepervasiveenoughtohaveawiderangingimpact,tobeaddressed
beforethecostofchangehasbecometoohigh.
Wheretheproductownerhasnotspecifiedincrementaldeliveryitmaybepossibletoworkdirectlywith
thedevelopmentmanagertoarrangeforopportunitiestodousabilitytestingusingearlyversionsofthe
actualsoftwaresystem.
Sidebar:MarryingUsabilityPracticeswithIncrementalDevelopmentMethods
AcommonlyheldopinionisthatInteractionDesignandUsabilityTestingneedtobefinishedbefore
softwaredevelopmentcanbestarted.Thisisindirectcontradictionwiththeagileapproachof
incrementaldevelopment.Modernproductdevelopmenttimescalesoftenprecludeaseparatedesign
phasesotheworkofusabilityspecialistsneedstobedonewithinthedevelopmentiterationsasthe
projectunfoldsratherthanbeforetheirstart.
AnumberofcasestudiesdemonstratethatInteractionDesignandagilemethodsdonotneedtobeat
oddswitheachother.In[AstonAndMeszarosOnAgileUsability]JaniceAstonandGerardMeszaros
comparetwophasesofanagilewherelightweightusabilitypracticeswereincorportatedintothe
secondrelease.TheyusedpaperprototypingandWizardofOztestingtovalidatetheuserinterface
designandsawsignificantimprovementsinusabilityanduseracceptance.Thebenefitswereso
significantthatinasubsequentprojectthebusinesspartnersinitiatedtheprototypingandusertesting
activitieswithoutpromptingfromtheITdepartment.
In[MillerOnCustomerInput]LynnMillerconcludes:TheUsabilityEngineeringteamatAliashasbeen
gatheringcustomerinputformanyyears,butneveraseffectivelyaswhenweworkwithanagile
developmentteam.ForSketchBookPro,wewereabletomaximizethequantityandimpactof
customerinputbyhavingtheinteractiondesignersworkinaparallelandhighlyconnectedtrack
alongsideofthedevelopers.Dailyinteractionbetweenthedevelopersandinteractiondesignerswas
essentialtothesuccessofthisprocess.
In[SyOnAgileUCD],DesireeSydescribeshowAliasSoftware(nowAutodesk)adaptedtheirUCD
processtosupportagiledevelopment.Sheconcludes:
*Agileusercentereddesignresultedinbetterdesignedsoftwarethanwaterfallusercentered
design.Agilecommunicationmodesnarrowedthegapbetweengatheringusabilitydataandactingon
it.
*BecauseAgiledevelopmentishighlyfeedbackdriven,productteamsmayrelyonuseropinionin
situationswhereuserobservationismoreappropriate.Usabilitypractitionerscanbethebestsuited

membersofanAgileteamtomitigatethisbiasbecauseoftheirskillsingatheringandanalyzinguser
experiencedata.
*ItispossibletousethefamiliararsenalofusabilityinvestigationmethodsonAgileprojects,
includingformativeusabilitytesting,userandtaskanalysis,interviews,andevenfieldbasedworklike
contextualinquiry.Thisisachievedbychangingthetimingandgranularityoftheinvestigations,and
howresultsarereported.
*usabilityanddesignactivitiescanbescopedasincrementalminidesigns.Differentvalidationand
elicitationactivitiescanbeblendedwithinsinglesessionsconductedatausabilitylaborinthefield.
DesignactivitiesoccuratleastoneAgilecycleorsprintaheadofthedevelopmentteaminan
InteractionDesignerTrackseparatefromtheDeveloperTrack.Developersreceivevalidateddesigns.
*Prototypedemonstrationsanddailyconversationhavelargelyreplaceddetaileddocumentssuch
asusabilitytestreportsandUIspecificationswhencommunicatingwiththeproductteam.Documents
arenowwrittenforinteractiondesigners,torecordahistoryofdesigndecisions.
In[McInerneyMaurerOnAgileAndUCD],PaulMcInerneyandFrankMaurerdescribethreecasestudies
ofhowuserexperiencespecialistsintegratedwithagileteamsandreportvarioustechniquesfrompair
observationstoinformalusabilitytestingintheteamroomwiththelatestbuildinvolvingboth
developersandcustomerproxies.
Manynotableadvocatesofusabilitypracticesdescribetechniquesforadaptingthemintoagile
methods.JeffPattonhasawebsite[PattonOnAgileProductDesign]andaYahoo!Groupdedicatedto
thetopicofagileusabilityandproductdesign.
In[MeszarosOnAgileProductDesign]GerardMeszaroswritesthatonmostprojectsthereisplentyof
timetounderstandtheusersandtheirgoalsandtodotheoverallproductdesignactivitiescalledfor
byUxDmethodsbeforeiterativedevelopmentstartsbecausethebusinesscaseneedstobemadeand
fundingneedstobesecuredbeforetheprogrammerscanstartdevelopoment.
LarryConstantinehaspointedoutthatevenwhenthisisntthecase,thereisusuallysomebackend
(ortechnicaldevelopment)thatcanbestartedimmediatelythatdoesnotdependontheinteraction
designortheuserinterface[ConstantineOnAgileUsability].)
In[NielsenOnAgileUsability],JakobNielsen(knownasthekingofusability)writes:Therearegood
reasonstobelievethatusabilityandAgiledevelopmentmethodscanworktogetherandimproveuser
experiencequality:
*Agileoffersmanyopportunitiesforovercomingproblemswithtraditionaldevelopmentmethods
thathavelongimpededusability.
*ApproachingAgilenarrowly,asaprogrammingmethodologyratherthanasystemdevelopment
methodology,threatenstodestroythelastdecade'sprogressinintegratingusabilityanddevelopment.
But,asoutlinedabove,therearewaysaroundeachofthesethreats.Solongasteamsrecognizethe
threatsasexplicitissues,theyneednotharmproductquality.

*Finally,weknowfromourresearchthatmanycompanieshavemadethingsworkswimmingly
oncetheyadaptedtheAgilemethodologytosuitqualityfocusedsystemdevelopment.
ForuserexperiencepractitionerswhosupportAgileteams,themainchangeisinmindset.Having
good,generaluserexperienceknowledgewillhelpyouunderstandhowtochangetraditionaldesign
andevaluationmethodstomeetyourAgileteam'sdifferentfocus.Ultimately,however,youmust
bothbelieveinyourselfandembraceAgiledevelopmentconceptsifyouwanttosucceed.
12.2 RecommendationsforSuccess
HerewesummarizealistofrecommendationsbyJeffPatton,anexpertinUserCenteredDesignand
agilemethods[PattonOnAgileProductDesign]andothers:
Helpproductmanagersunderstandandbalancebusinessandusergoals;Useendusertesting
asawaytogivevoicetoenduserneedstobalancethemwithbusinessgoals[SyOnAgileUCD].
Helpproductdevelopmentteamunderstandtheunderlyingproductexperienceanddefine
interactionandvisualdesignpatterns
Dojustenoughuserresearchanddesigntogetstartedbutplanforongoingusercontactfor
researchandvalidation.
Involvethedevelopmentteamintheinteractiondesign;becomeadesignfacilitatorratherthan
purelyabehindcloseddoorsdesigner.
Focusonbigpicture,helptheproductdevelopmentteamsetthecontext
Validateusabilitybeforedevelopmentthroughtheuseoflowfiprototypingandtesting
[AstonMeszarosOnAgileUsability]
Validateusabilityafterdevelopment(throughlightweightusabilitytestingondeveloped
features)[AstonMeszarosOnAgileUsability]
Userparalleltrackdevelopmenttoworkaheadandfollowbehind[MillerOnCustomerInput]and
allowdevelopmenttoproceedonbackendfunctionalityinparallelwithinteractiondesign
[ConstantineOnParallelUx]toavoidseparateinteractiondesignphase.
Increasethefrequencyandtimingofenduserinvolvement:buildareadysupplyofusersand
usersurrogatesinsideandoutsideofyourorganizationtoleverageforcontinuoususertesting.
Considerschedulingregularusertestsessionsanddeferdecidingwhattotestjustintime.


Chapter13 OperationsManagers
Perspective
Someorganizationsbuildorcommissionsoftwarethattheyinstallandruntoprovideaservicetotheir
customersorinternalusers.Intheseorganizationstheacceptabilityofthesoftwaremaybedetermined
bytwoentirelyseparatecommunities,thosethatbenefitfromthefunctionalitythatthesoftware
providesandthoseforwhomoperatingthesoftwareisacost.Intheseorganizationstheacceptance
decisionmustincludeoperationalcriteriaandtheroleoftheoperationsgroupinthemakingofthe
acceptancedecisionmustbeclear.Thiscategorycouldalsobeexpandedtoencompassothergroups
suchassecurity,dataarchitecture,ITarchitectureandinfrastructurethatprovidestewardshipof
corporateintereststocounterbalancepotentialprojectpressureinducedshortcutssuchassecurity,data
architecture,ITarchitectureandinfrastructure.(SeeChapter14EnterpriseArchitectPerspective.)
SeetheOperationsManagerpersonainthecompanystereotypecalledITBuildingSoftwareforthe
BusinessinAppendixXReaderPersonasforanexample.
Asanoperationsmanageryoulikelyhaveacompletelydifferentperspectiveonthedeploymentofa
neworupgradedsoftwaresystem.Otherpartiescareaboutwhetherthenewfunctionalityworks
properlyandwhetheritwillprovidesufficientbusinessbenefitinthenearfuture;youcareaboutwhatit
willdotoyoursupportcostsandwhetheritwillbesustainableinthelongerterm.Therefore,youridea
ofacceptabilitywillbequitedifferentanditshouldinfluencetheacceptancedecision.
13.1 RoleinAcceptanceDecision
Itiscrucialthatyourroleinthemakingoftheacceptancedecisioniscleartoeveryoneconcerned.Do
youprovideinformationtotheacceptancedecisionmakerorareyouoneofseveralacceptancedecision
makerswhooperatebyconsensus,votingorveto?Orperhapsyouparticipateinthereadinessdecision
beforethesoftwareismadeavailableforacceptancetesting?
Aswithotherformsofacceptancecriteriaitishighlydesirableforthesecriteriatobeclearlydefined
beforethesoftwareisbuiltsoastoavoidunnecessarytest&fixcyclesbeforethesoftwarecanbe
deployed.Itisalsodesirabletobeingabletoconductthetestingincrementallythroughouttheproject
ratherthanjustattheend.IncrementalAcceptanceTestingdecreasesprojectrisksbyuncoveringissues
earlyenoughtohavethemfixedbeforethefinalacceptancetestphasewhichisthenmuchmorelikely
topasswithoutuncoveringanyshowstopperproblems.

13.2 OperationalAcceptanceCriteria
Theactualcriteriathatanoperationsgroupwillusetodetermineacceptabilitywillbedependentonthe
natureoftheorganizationandtheservicesitprovidesaswellastheservicelevelagreementforthe
softwareapplicationinquestion.Thebaselevelrequirementstypicallyinclude:
Installeranduninstallertesting
Autoupdateorpatchinstallation
CompatibilitywithyourSystemsmonitoringframework
Performanceunderexpectedloads
Systemsmanagementscriptsforstartup,restartandshutdown.
Additionalacceptancecriteriamayinclude:
Frequentlyaskedquestionsandanswers
Troubleshootingscriptsforthehelpdesk
Failoverandrecoverytestingforhighavailabilityapplications(includingapplicationshutdown,
OSfailure,hardwarefailure,occasionalconnectivityetc.)
Testingofhowsystemhandlesbad(illformated)data
Penetrationtestingforpublicfacingapplications
Stresstestingforapplicationswithpotentiallylargenumbersofusers
Datamigrationfromapreviousversionofthesystemorfromasystemthisonereplaces
Transactionaldatapreservationandarchiving
ITArchitectureapprovalofthetechnologystack.
Simultaneousmaintenance(performingserviceoperationswithconstrainedornoperformance
degradationofthecurrentsystm)Reliability,availability,meetingcertainSLAs(uptimeetc.)
Selfmonitoring,errordetection,systemlogging
Recoverability(mustverifymultiplefailoverandrecovery),disasterrecovery,databackup&
recoveryetc
Supportdocumentation(preandpostrollout),training
Thesystemmustbemigratedtothenewversionoftheplatform
Multipleversionsofthesystemmustrunsidebyside.

13.3 OtherConsiderations
Asanoperationsteam:
Weshouldcontributetotherequirementsdiscussionsasearlyaspossibletoensurethat
realistictargetsarebeingproposedfortheproduct.Weshouldhelptheproductownerandthe
productdevelopmentteamunderstandoperationalcontexts.
Werecognizecriticalityofoursystemundertestandunderstandhowsignificantlyoperational
outagesaffectourbusinessandreputation.
Weunderstandusagepatternsandfluctuationsbasedonthetimeofday,season,special
events.Theprojectedusagewillmostlikelychangeovertime.
Managingtestenvironmentisimportant.Ourtesterswouldlikelyneedtheflexibilitytomake
tuningandconfigurationchanges.Forexample,manyoperabilityscenariosoftenrequiretesters
topurposefullyconfigurethesystemwrongtoobservetheoutcome.
Thechoiceswemakefortestenvironmentwillreflectacostbenefitanalysisbasedonyourrisk
tolerance.
Testlabproceduresarecrucial.Itisimperativetokeepalogofallchangesthataremadetothe
environmentinorderforustobeconfidentthatwhenacceptancetestingisdone,the
configurationandstateoftheproductinthetestenvironmentisalignedwiththeintended
productionconfiguration.
Wecommunicateregularlywiththedeploymentteamandtheproductdevelopmentteam.
OperationalacceptancepracticesarediscussedinVolumeIVofthisGuide.Foraserioustreatmentand
discussionofthepatternsforperformanceandoperability,see[PnPonPerfGuide]and
[FordOnOperabilityPatterns].


Chapter14 SolutionArchitects
Perspective
Thesolutionarchitectistypicallypartoftheproductdevelopmentteam.Theirroleintheacceptance
processusuallyinvolvesapprovingdesignandtechnologydecisions,designingdatamodelsanddomain
models,andensuringthattheproductasbuiltsatisfiestheneedsoftheendusers.Theroletranscends
differentpartsoftheproductdevelopmentteamsorganizationasthesolutionarchitectisresponsible
forthewholesolutionwhereasindividualcomponentteamsorfeatureteamsareeachonlyresponsible
forapart.
14.1 ResponsibilitiesOverview
Asasolutionarchitect,yourjobistoensuretheentiresolutionsatisfiestheproductownerandthe
potentialusersofthesystem.Youresponsibilitiesinclude:
conductingarchitecturalreviewsandaudits;evaluatingarchitecturalalternatives;
ensuringtheuseofagreeduponarchitectureanddesignpractices;
ensuringthatarchitecturaldocumentsareuseful,uptodate,andsufficientlycomplete;
makingdocumentationtradeoffdecisions;
planningarchitecturalmigration;
andcoordinatingandcommunicatingwitharchitectsandotherstakeholdersregarding
architecturalgovernance,compliance,andpolicymatters.
Yourroleintheacceptanceprocesswillprimarilyfocusonthereadinessassessmentactivitiesbutyou
shouldalsobeinvolvedinhelpingtheproductownerunderstandwhattheywanttheproductowner
teamtobuildandfacilitatethecommunicationofthatinformationtotheproductownerteam.Your
involvementinacceptancetestingmaybeminimalbutyoumaybecalledupontointeractwith
acceptancetesterswhichmayincludeenterprisearchitectsassessingcompliancetoarchitectural
guidelinesandstandards.
14.2 UnderstandingtheSolution
Youneedtoworkwiththeproductownertounderstandtheirvisionoftheproductandtohelpthem
understandwhatoptionsareavailabletothemwithrespecttobuildingthatproduct.Lesstechnical
productownersmayneedyourhelptodefinetheproductbasedonmoregeneralneeds.Youwilllikely
wanttoengageuserexperienceexpertsifgoodusabilityisarequirement.Youwillalsoneedtohelpthe
productownerunderstandtherelativecostsofvariousoptionspresentedtothemsotheycanmakethe

righttradeoffsbasedonreturnoninvestment.Thismayrequireunbundlingofsomefunctionalityinto
smaller,moreatomic,featuresthatcanbeprioritizedseparately.
Beawarethattheproductownermaynotbeawareofalltherequirementsfortheproduct;itmaybe
uptoyoutohelpthemengagetheoperationsmanager,securityarchitects,andotherstakeholdersso
thatalltherequirementsfortheproductbecomeknown.Discoveringsecurityrequirementsor
mandatoryservicelevelagreementslateinaproductdevelopmentcycleissuretoresultinalate
product.
Forlargerproductsyoumaybecalleduptohelpmaketheprojectmanagerorproductownerdecide
whethertodividetheproductownerteamintofeatureteamsorcomponentteamsandtosubdividethe
featurebacklogintofeatureteambacklogsordecomposethefeaturebacklogintocomponentbacklogs.
Whereamixoftechnologiesisinvolved(includingmixedhardware/softwareproducts),youmustbe
preparedtobeinvolvedinnegotiatingtheinterfacesbetweenthevarioustechnologies.
Regardlessofthesizeoftheproductownerteam,itisimportantforeveryonetounderstandtheoverall
solutionandhowtheirpartfitsintoit.Thiscanhelpavoidsituationswhereeachcomponentworks
accordingtospecificationbutwhenintegratedfailtoprovidetheseamlessexperiencetheproduct
ownerwouldexpect.
14.3 EnsuringReadiness
Assolutionarchitect,theonusisonyoutodoeverythingyoucantoensurethattheproductisready
beforethereadinessdecisionneedstobemade.Thingswillgoalotsmootherifeveryoneontheteam
understandswhatdonelookslike.Thisisespeciallytrueforcomponentteamswhoseworkneedstobe
integratedbeforeacceptancetestingcanbedone.Withcomponentteams,youmayneedtodoaform
ofacceptancetestingofthecomponentsfollowedbyintegrationtestingoftheintegratedcomponents
beforeyoucanconductreadinessassessment.Thiswillbeeasiertodoifyouhavebeendoingdesign
reviewsofeachofthecomponentsandhavedefinedclearinterfacesandacceptancecriteria.
Someoftheacceptancecriteriamaybederivedfromenterpriserequirements.Whereenterprise
architectsexist,itisimportanttoengagethemearlyintheproductdevelopmentlifecycletoensureyou
understandwhatstandardstheproductneedstocomplyto,whatreusablecomponentsareavailable
and/orareexpectedtobeusedandwhatenterprisedataimpactstheproduct(andviceversa.)Thiswill
avoidnastysurprisesduringreadinessassessmentandacceptancetesting.
14.4 AssessingReadiness
Assolutionarchitect,youwilllikelybeaskedforyouropinionastowhethertheproductis
readyforacceptancetesting.Youllwanttomakethisrecommendationbasedonharddata
andwillthereforewanttoconductarchitecturereviewsandsecuritycodeinspectionsandto
runstaticcodeanalysisandcoveragetools.Youwillalsowanttounderstandhowwellthe
nonfunctionalcharacteristicsofthesystemcomplywiththerequirements;thatwillrequire
testingofthenonfunctionalcharacteristics.Youmaybeinvolvedintheactualtesting,either

inpersonorthroughplanninganddelegating,oryoumayreviewresultsprovidedby
someoneelsesuchasthetestorganization.Eitherway,youwillwanttoensurethatthe
systemmeetsanyoperationalrequirements,includingSLAs,inadditiontothefunctional
requirements.

Chapter15 EnterpriseArchitects
Perspective
Theenterprisearchitectisresponsibleforensuringconsistencybetweenproductsemployedinorbuiltby
anenterprise.Theirgoalistoreducetotallifecyclecosts(includingoperatingcosts,technologylicensing
costs,etc.)andtomaximizetheROIofanycommoninfrastructure.Theenterprisearchitectoftenhas
vetopowersoverindividualproductsthatfailtocomplywithenterprisearchitectureprinciples.This
sometimemakesthempartoftheacceptancedecisionwhentheproductisbuiltbyanexternalsupplier
andsometimespartofthereadinessdecisionwhentheproductisbeingbuiltinhousebyanIT
department.
Asanenterprisearchitect,yourjobistoensurethealltheproductsusedbytheenterpriseworkwell
togetherandprovideacceptableperformanceatanacceptablecost.Dependingonthenatureofyour
organizationyoumaybecalledupontoprovidedataforeitherthereadinessdecisionortheacceptance
decision.Insomeorganizationsyoumightevenbeaskedtobeinvolvedinmakingtheacceptance
decision.Theroleyouplaywilldependonwhereintheorganizationyoureportrelativetothesupplier
organization.Inoutsourceddevelopmentsituationsyouaremorelikelytobeinvolvedinacceptance
testingandtheacceptancedecision;forinhousesoftwaredevelopmentandproductcompaniesyou
maybemoreinvolvedinreadinessassessment.Whoyouinteractwithwillalsodependonthenatureof
theorganization.Ifthereisasolutionarchitectforeachproduct,muchofyourinteractionwiththe
supplierteamwillbewiththesolutionarchitect.Inothercasesyoumightbeinteractingprimarilywitha
developmentleadoraprojectmanageroreventheproductowner.
15.1 UnderstandingtheSolution
Whileitisprimarilythesupplierteamsresponsibilitytoworkwiththeproductownertounderstandthe
desiredproduct,youmayneedtobeinvolvedinthesediscussionifthesupplierteamdoesnthavea
solutionarchitect.Otherwise,youwouldlikelyworkwiththesolutionarchitecttoundersandhowthe
requirementsofthespecificproductrelatetotherestoftheenterprisearchitecture.Youwould
determine(alongwithSolutionArchitect)whatformsofintegrationarerequiredandhowthat
integrationshouldbeimplementedfromatechnicalpointofview.Youwouldalsocollaborateto
understandhowthenewproductimpactsanycommoninfrastructuresuchasnetwork,applicationand
databaseservers,etc..Iftheproducthasanyimpactonthelongtermvisionoftheenterprise
architecture,eitherbychangingthevisionorbyimplementinganykeycomponents,thatinformation
needstobecommunicatedtoallpartiesconcernedasthelattermaybecomepartofthereadiness
and/oracceptancecriteria.

15.2 EnsuringReadiness
Insomeorganizationsyouwillbeaskedtoprovidereadinessassessmentinformationtothereadiness
decisionmaker.Youwillneedtoengagethesolutionarchitectordevelopmentleadstoensurethey
understandwhatstandardsorguidelinesareapplicabletotheproductandwhatreusablecomponents
shouldbeused.Youmayalsobeinvolvedindefiningextensionstothecorporatedatamodelsto
supportthenewproduct.Itiscrucialthatyoucommunicateanyexpectationsofhowthesupplierteam
wouldengagetheenterprisearchitectsduringthereadinessassessmentand/oracceptancetesting
(whicheverisapplicable.)
15.3 AssessingReadinessorAcceptability
Thenatureofyourorganizationwillinfluencewhetheryouareformallyinvolvedinreadiness
assessmentoracceptancetestingbutthenatureoftheactivitiesyoudowillprobablybethesame
eitherway.Unfortunately,thelateryouareinvolvedthebiggeranimpactanoncompliancediscovery
willhaveontheoveralldeliveryschedule.Thereforeitispreferabletoengagethesupplierteamsearlier,
beforeevenreadinessassessment,toensurethatthereadinessassessmentoracceptance
testing/reviewsarejustaformalsignoffofinformationthatwaspreviouslydiscoveredratherthanan
unpleasantsurpriseforeveryoneinvolved.
Someoftheactivitiesyoumightwanttostartdoingevenwhiledevelopmentisinprogressinclude
designandarchitecturereviewsthatfocusonstandardscomplianceparafunctionalattributessuchas
security,capacity,scalability,andinfrastructureimpact.Insomecasesyoumayevenbecalleduponto
arrangefortestingofparafunctionalcharacteristicsofthesystemsuchaspenetrationtesting.
Ifyoufindyourselfonanacceptancedecisioncommittee,youwillneedtobepreparedtogatherand
reviewtheresultsoftheparafunctionalreviewsandtestingandcastyourvoteontheacceptabilityof
theproductfromatechnicalperspective.Lettheproductownervotebasedonthefunctionality
providedandtheoperationsmanagerontheirsatisfactionwiththeoperationalrequirements.

Chapter16 Legalperspective
Thedecisiontoacceptsoftwareisoftengovernedbytermsinalegallybindingcontract.Theacceptance
decisionmaythereforehavelegalramificationswhichmayoverrideconsiderationsrelatedtofunctional
andnonfunctionalacceptancecriteria.
Disclaimer:Thischapterisnotmeanttobeusedaslegaladvice.
Sinceacceptanceisamajorlegalmilestoneofcommercialsoftwaresystemdevelopment,itisimportant
tokeepinmindthefollowing:
16.1 AcceptanceMayHaveLegalRamifications
Aspartofthecontractualprocess,acceptancetypicallyresultsinrenderingapaymentaswellasit
affectstheapplicationofanywarrantyprovisionsandpotentiallyanyremediesavailabletothe
customer.Whendefiningthecontract,makesurethattheappropriatefactorsaretakeninto
consideration.Therelationshipbetweentheacceptanceoftheproductintoaproductionenvironment
andthecontractualacceptanceoftheproductascompleteshouldbeclearlyspelledout.
Acceptancecanbeapproachedfromtwopositions:explicitprovisionsspecifiedinthesoftware
developmentcontractandthetermsimpliedintothecontractbylaw(whichalsodependsonthe
geographicjurisdiction).Implicitprovisionsmayalsodifferbasedonwhetherthesoftwareisrendered
asaproductorasaservice.Toseewhatprovisions(explicitandimplicit)doapplytoyourcase,consult
yourlegalcounselor.
16.2 ContractShouldStipulateAcceptanceProcess
Whensoftwaredevelopmentisoutsourced,thecontractshouldclearlystipulatetheacceptanceprocess
andhowtheacceptancecriteriawillbedetermined.Itisimportanttoalsospecifyanyexpectations
abouthowmuchreadinessassessmentwillbeconductedbythesupplierandhowthecriteriafor
readinessassessmentwillbedetermined.Forexample,willthecustomerbeprovidingthesupplierwith
theacceptancetests,arepresentativesampleoftheacceptancetestsorleavingthesupplierto
determinewhatteststorun(theBattleshipapproach.)
Thecontractshouldnotjuststipulatewhenthesuppliermustdeliverthesystembutalsohowlongthe
customercantakeforacceptancetesting.Acceptancetestingisusuallytimeboxed.Thecustomeris
givensomuchtimetoacceptsoftwareandreportdeficiencies,afterwhichthesoftwareisdeemed
completeiftherearenoshowstopperbugs.Thecontractmuststipulatethecriteriausedto
determinewhetherabugisindeedashowstopperorgatingbug.

Thecontractshouldalsostipulatewherethedifferentstepsoftheacceptanceprocesswillbecarried
out.Incaseofcustombuildsoftwaresystem,preliminaryacceptancetestingmaybeperformedatthe
locationofsupplier,whilefinalacceptancetesting(bothfunctionalandoperational)isperformedwhen
thesoftwareisdeployedtothecustomer.Thecontractmaystipulatethat,aspartoftheacceptance
procedure,thesoftwaresystembesuccessfullyrunwithcurrentdatainaliveoperatingenvironmentfor
aperiodoftime,withminimumbugs,beforethesystemiscontractuallyaccepted.
Thecontractshouldnotbeonesided;itshouldencouragebothpartiestogettoamutuallyagreeable
acceptanceasquicklyaspossible.Contractsthatpenalizeonepartyoftenleadtodysfunctionalbehavior
thatresultsinloseloseoutcomes.

PartIIReferences
[AstonMeszarosOnAgileUsability]Aston,JaniceandGerardMeszarosAddingUsabilityTestingtoan
AgileProject,Proc.OfAgile2006.
[CecilOnAgileAndUCD]RichardF.Cecil,"ClashoftheTitans:AgileandUCD"
http://www.uxmatters.com/mt/archives/2006/12/clashofthetitansagileanducd.php,Sep20,2009
[ConstantineOnAgileUsability]Constantine,Larry,LucyLockwood,Processagilityandsoftwareusability:
Towardlightweightusagecentereddesignhttp://www.foruse.com/articles/agiledesign.pdf
[ConstantineOnParallelUx]Constantine,Larry,TutorialonAgileMethodsandUsabilityIntegrating
UsageCenteredDesign(slide19),presentedatSoftwareDevelopmentBestPracticesSeptember2005
[DruckerOnManagementChallenges]Drucker,P.ManagementChallengesforthe21stCentury,Harper
Business,2001,p.33.
[EckfeldtEtAlOnTargetCostContracts]Eckfeldt,B.;Madden,R.;Horowitz,J.SellingAgile:TargetCost
Contracts.InProc.Agile
2005.IEEEComputerSociety:160166,2005.
[FordOnOperabilityPatterns]ChrisFordetal.PatternsforPerformanceandOperability:Buildingand
TestingEnterpriseSoftware,Auerbach:2008.
[KohlOnPairTesting]Kohl.J.PairTesting.BetterSoftwareMagazine.
www.kohl.ca/articles/pairtesting.pdf
[LeffingwellOnFeatureVsComponentTeams]DeanLeffingwell,ScalingSoftwareAgility,TheBlog:Best
PracticesforLargeEnterprises,
http://scalingsoftwareagility.wordpress.com/2009/07/15/organizingagileatscalefeatureteams
versuscomponentteamspart1/
[MartinOnProfessionalismAndTDD]Martin,C.ProfessionalismandTestDrivenDevelopment.IEEE
Software24(3):32362007.
[McInerneyMaurerOnAgileAndUCD]PaulMcInerneyandFrankMaurer."UCDinAgileProjects:Dream
TeamorOddCouple?",ACMInteractions,12(6):1923,2005.
[MeszarosOnAgileProductDesign]Meszaros,GerardConcepttoBackWhatHappensBeforeIteration
Zero,http://concept2backlog.gerardm.com.
[MillerOnCustomerInput]LynnMiller,"CaseStudyofCustomerInputForaSuccessfulProduct,"Proc.
AgileDevelopmentConference(ADC'05):225234,2005.
[MillerCarterOnLargeScaleAgile]Miller,A.andCarter,E.AgileandtheInconceivablyLarge,inProc.
Agile2007,IEEEComputerSociety:304308,2007.

[NielsonOnAgileUsability]Nielson,Jakob,AgileDevelopmentProjectsandUsability,
http://www.useit.com/alertbox/agilemethods.html
[PattonOnAgileProductDesign]JeffPatton,"AgileProductDesign",http://www.agileproductdesign.com,
Oct19,2009
[PerryEtAlOnTargetCostContracts]Perry,J.,Barnes,M.Targetcostcontracts:ananalysisofthe
interplaybetweenfee,target,
shareandprice.InJ.Engineering,ConstructionandArchitecturalManagement,7(2):202208,2000.
[PnPOnPerfGuide]patterns&practices:PerformanceTestingGuidanceforWebApplications,Microsoft
Press:2007.Alsoonline:http://www.codeplex.com/PerfTestingGuide
[PoppendiecksOnLean]Poppendieck,M.,Poppendieck,T.ImplementingLeanSoftware
Development:FromConcepttoCash.AddisonWesley,2007
[ShingoOnToyotaProductionSystem]Shingo,S.AStudyoftheToyotaProductionSystem,Productivity
Press,1989.
[SyOnAgileUCD]Sy,DesireeAdaptingUsabilityInvestigationsforAgileUserCenteredDesignJournalof
UsabilityStudies,Volume2,Issue3,May2007,pp.112132
http://www.upassoc.org/upa_publications/jus/2007may/agileucd.html
[VoddeLarmanOnScalingAgile]Larman,CraigandBasVodde.ScalingLeanandAgileDevelopment.
AddisonWesley,2009.
[WittgensteinOnPhilosophicalInvestigations]Wittgenstein,L.Philosophicalinvestigations,3rdEd.,
Oxford:Blackwell,1967.

PartIIIAcceptingSoftware
PartIintroducedseveralmodelstohelpusthinkabouttheroleofacceptancetestingintheoverall
contextofthesoftwaredevelopmentlifecycle.PartIIdescribedtheacceptanceprocessfromthe
perspectivesofcommonlyoccurringrolesplayedbythepeopleinvolvedinmakingtheacceptanceand
readinessdecisionsontypicalprojects.PartIIIaddressestheprocessofacceptingsoftware,applying
theconceptsintroducedinPartsIandII.InPartIII,weoutlinethemainactivitiesthatleadtothe
acceptancedecisionanddescribestrategiesfortheireffectivemanagement.
Chapter17PlanningforAcceptanceintroducesthepracticesusedbyoneormoreoftheroles
describedinPartItoplantheactivitiesthatwillprovidethedecisionmakerswiththedatathatthey
needtomakeanacceptancedecision.
Chapter18AssessingSoftwareintroducestheacceptancetestingandverificationtechniquesusedin
softwareassessment,withparticularemphasisontestdefinition,executionandmaintenance.It
addressesbothfunctionalandnonfunctionalacceptancetesting.
Chapter19ManagingtheAcceptanceProcessintroducesthepracticesusedtomanageacceptance
testexecutionandacceptancedecisionmakingandaddressesthetradeoffsbetweendifferent
approachestotheseactivities.
Chapter20StreamliningtheAcceptanceProcessdescribestheorganizationoftheacceptanceprocess
tooptimizetheacceptancescheduleandtheresourcesusedintheacceptancedecisionbyfocusingon
threemainstrategies:incrementaldevelopment,wastereductionbyavoidingoverproductionand
concurrentexecutionoftheactivitiesunderlyingsoftwareacceptance.

Chapter17 PlanningforAcceptance
PartIintroducedseveralmodelstohelpusthinkabouttheroleofacceptancetestingintheoverall
contextofthesoftwaredevelopmentlifecycle.Italsointroducedtheabstractrolesofthepeopleinvolved
inmakingtheacceptanceandreadinessdecisions.Thischapterintroducespracticesusedbyoneormore
ofthoserolestoplantheactivitiesthatwillprovidethedataonwhichtheacceptancedecisionmaker(s)
canbasetheirdecision.
Acceptanceisanimportantpartofthelifecycleofaproduct;itisimportantenoughthatitshouldbe
theresultofacarefullythoughtthroughprocess.Thetestplanistheendresultofallthisthinking.Like
mostsuchdocuments,itcanserveanimportantroleincommunicatingtheplansfortesting,butthereal
valueliesinthethinkingthatwentintoproducingit.
Testplanningbuildsontheworkdoneduringprojectchartering,whichdefinestheinitialprojectscope.
Intestplanning,youwedefinethescopeofthetestingthatwillbedone,selecttheteststrategy,and
drilldowntodetailedtestingplansthatdefinewhowilldowhat,when,andwhere.
Mostprojectsprepareatestplanthatlaysoutthefollowing,amongotherthings:
Thescopeoftheacceptanceprocessandthebreakdownintoreadinessassessmentandacceptance
testingphases
Theoverallteststrategyincludingbothmanualandautomatedtesting
Theactivities,testingandotherwise,thatwillbeperformedineachphase(thereadinessphaseand
theacceptancephase)
Theskillsthatarerequiredtoperformtheactivities
Theotherresourcesthatwillbeutilizedtocarryouttheactivities(suchasfacilitiesandequipment)
Thetimeframeandsequence,ifrelevant,inwhicheachactivitywillbeperformed

17.1 DefiningTestObjectives
Weneedtohaveaclearunderstandingofthescopeoftheprojectandthesoftwarebeforewestart
thinkingabouthowwewillacceptthesoftware.Mostorganizationshavesomekindofproject
charteringactivitythatdefinestheproductvisionorscope.Itmayalsoincludeariskassessmentactivity.
Oneformofriskassessmentactivityinvolvesbrainstormingallthepotentiallynegativeeventsthat
couldcausegrieffortheproject.Someofthecommonlyoccurringrisksthatwemayaddressthrough
testingare:
TheProductDevelopmentTeammisinterpretedtheProductOwnersdescriptionresultingina
productthatbehavesdifferentlyfromexpected.

TheProductOwnersdescriptionoftheproductleftoutimportantbehaviorsresultinginmissing
functionality.
TheProductDevelopmentTeamimplementedthebehaviorspoorlyresultinginfrequentcrashesof
theproduct.
Theproductexhibitstherightbehaviorwhenusedbysmallnumbersofusersbutbehaveserratically
orrespondsslowlywhenthenumberofusersortransactionsincreasestoexpectedlevels.
Foreachpossibleeventweclassifyeachofthelikelihoodandtheimpactaslow,medium,orhigh.
Anythingrankedmedium/highorhigh/highneedstobeaddressed.Somerisksmaycauseustochange
thewayweplanourproject,butotherrisksmaycauseustotakeonspecifictestplanningactivities.
Together,thevision/scopeandriskassessmenthelpdrivetheteststrategydefinitionandtestplanning.
17.1.1 TraditionalApproachestoTesting
Traditionally,mostproductsaretestedfromtheusersprespective.Thatis,oncetheproductisfinished
theproductsbehaviorisverifiedviawhateverinterfacestheuserswilltypicallyuse.Often,verylittleif
anytestingisdoneofthepartsofthesystembeforeitisassembledintothefinalproduct.Thisresultsin
adistributionofteststhatlookslikeFigureAInvertedTestPyramid.

FigureA
InvertedTestPyramid

Thisfigureshowsahighlevelviewofhowvariouskindsoftests(andconsequently,testeffort)are
typicallydistributed(basedontheleveloftestgranularityfromunittocomponent/subsystemto
system/wholeproduct)whentestingisdoneprimarilybytestersusingthesameinterfacesasendusers.
Systemtestingisdoneatthelevelofthewholeproductandcanincludearangeofsystemsizes:
individualsystemsorintegratedworkflowsthroughmultiplesystemsthathavetointeract.Similarly,
componenttestsareusedfortestingpackages,individualexecutablesorsubsystems;whilethe
granularityofunittestingcanbeonlittlethingslikemethods/functionsorbiggerthingslikemodules
andclasses.SeethesidebardTestingTerminologyPitfallsforadiscussionaboutwhatwecallvarious
kindsoftests.
TheproblemwiththeapproachpresentedinFigureA(wheremosteffortisfocusedonsystemtesting
andverylittleonunittesting)isthatitishardtoverifythatthesoftwarebehavesinallpossible
circumstancesbecausemanyofthecircumstancesaretriggeredinsidethesoftware.Evenwhenthisis
notthecase,muchoflogicwithinthesystemiscumbersometoverifyviatheenduserinterface.This
makesverifyingthebehaviorandfindinganybugsveryexpensiveandtimeconsuming.Mostofthe
behaviorcanbeverifiedandmanyofthebugscanbefoundmuchmorecheaplythroughappropriate
unittesting,componenttestingandsubcutaneoussystemtesting.Thesesortsoftestscanandshouldbe
automatedaswillbediscussedinsection17.2.3AutomatedTestingbelow.
Sidebar:TestingTerminologyPitfalls
WhenpreparingFigureA(andFigure4laterinthischapter)westruggledwithwhattocallthetestsin
thetoplayeroftheupsidedownpyramid.Someofthealternativesweconsideredwere:Acceptance
Tests,WholeProductTests,CustomerTests,FunctionalTests,andIntegrationTests.Whilethese
namesareincommonusetodescribetheteststhatoccupythetoplayer,wewerenthappywithany
ofthesenamesbecausetheydidntclearlyconveythedifferencebetweenthetoplayerandthelayers
belowit.Theproblemwitheachofthesehistoricalnamesisthattheyoftenexisttohelpdifferentiate
onekindoftestsfromanotherkind.Forexample,FunctionalTestsasopposedtowhat?Theobvious
answerisNonfunctionalTests,butwefeelnonfunctionaltestsalsofitintothetoplayerofthe
(inverted)pyramid.IfnotAcceptanceTeststhenwhat?ThelogicalalternativeisnotRejectionTests
butRegressionTests!Acceptancetestshelpeusverifynewfunctionaltyworkswhileregressiontests
verifythatexisitingfunctionalitystillworks.Theacceptancetestsforthisreleasearelikelyaddedto
theregressiontestsuiteforthenextrelease.Sothetermsacceptanceandregressionrefertopointsof
differentiationonthetimehorizon.
Examiningthepotentialnameswiththiscriticallenshelpeduseliminatemanypotentialnamesleaving
usonlywiththosenamesthathelpedusfocusonwhetherweweretestingthefullyassembled
productoritsconstituentparts.ThisledustoWholeProductTest,SystemTestandIntegrationTest.
Unfortunately,thetermIntegrationTestcanbeusedatvariouslevelsdependingonwhetherweare
integratingunits,components,subsystemsorsystems(intoasystemofsystems.)Wholeproductis
probablytheclearestnamebutmanyteamsthinkofthemselvesasbuildingsolutiostobusiness
problemsratherthanproductsforsale.ThisledustofocusonSystemTestsandthecontrastwith
ComponentTestsandUnitTests.

Therearresimilarissu
uesintheterm
minologyaroundhowtesttsareprepareedandexecuted(Explorattory
vsScripttedtesting)andhowtestssareautomatted(Handscrriptedorproggrammedvs.record/playb back,
scriptedvs.datadrivvenvs.keywoorddriven,etcc.)Themoralofthissidebaristhatweneedtobecaareful
tomakeesureweund derstandwhatsomeonem meanswhenth heyuseoneo oftheseworddstodescribee
testing.Theymaywe ellmeansomeethingentirelydifferentfrromwhatwewouldhavemeanthadw we
usedtheesameword..
17.1.2 Flipp
pingthePy
yramidRig
ghtSideU
Up
Theupsid
dedowntestp
pyramidcanbeturnedrigghtsideupbyymovingmucchofthetestingactivityto
othe
unitandccomponentle
evel.Butwhaatkindsoftesstingcanbep
pushdown?
Theagilesoftwaredevvelopmentcommunityhassshownthatitispossiblettoproduceco onsistentlyhigh
qualityso
oftwarewitho outsignificanttlyincreasingtheeffortbyyintegratingttestingthrougghoutthe
developm mentlifecycle
e.Thishasled
dtoarethinkiingoftheroleeoftesting(ttheactivity)andoftestteaams.
BrianMarrick,hasdefin
nedamodel[MarickOnTeestingDimensions](seeFiggure1)thath
helpsus
understan
ndthepurpossebehindtheedifferentkin
ndsoftestsw
wecouldexecute.

Figure1
PurposeoofTests
oftestingwecandoalongtwokeydimensions:
ThediagraaminFigure1classifiesvaarioustypeso
Whettherthetestsarebusinessfacingortecchnologyfacin
ng
Whettherthetestsareintended development(byhelpingth
dtosupportd hemgetitrigght)ortocritiique
thheproductaftteritisbuilt
didate1 169

17.1.3 TestsThatSupportDevelopment
TestscansupportdevelopmentbyhelpingtheProductDevelopmentTeamunderstandwhatthe
productissupposedtodobeforetheybuildit.Thesearetheteststhatcanbepreparedinadvanceand
runaswebuildtheproduct.Aspartofthereadinessassessment,theProductDevelopmentTeamcan
runtheseteststoselfassesswhethertheproductimplementsthenecessaryfunctionality.
Thetestsinthiscolumnfallintotwocategories:thebusinessfacingteststhatdescribewhattheproduct
shoulddointermsunderstandablebythebusinessorProductOwnerandthetechnologyfacingtests
thatdescribehowthesoftwareshouldworkbeneaththecovers.
BusinessFacingTests
Thebusinessfacingteststhatdrivedevelopmentarethesystemaccepatancetests(alsoknownas
customertests).Thesetestselaborateontherequirementsandtheveryactofwritingthesetestscan
exposemissingorambiguousrequirements.Whenwe(ProductOwner,oftentogetherwithsomeone
fromtheProductDevelopmentTeam)preparethesetestsbeforedevelopmentstarts,wecanbesure
thattheProductDevelopmentTeamunderstandswhattheyneedtobuild.ThisisknownasAcceptance
TestDrivenDevelopment.
Ifwepreparethesystemacceptancetestsafterdevelopmentiscompleteorwepreparetheminparallel
withdevelopmentanddonotsharethem(thisisreferredtoas"independentverification"),thetestsdo
nothelpusbuildtherightproduct;instead,thetestsactasanalternativeinterpretationofthe
requirements.Iftheyfailwhenwefinallyrunthem,weneedtodeterminewhichinterpretationofthe
requirementsismoreaccurate:theoneimplementedbythedevelopmentteaminthecodebaseorthe
oneimplementedinthefunctionaltestsbythetestteam.Ifthelatter,timewillbeconsumedwhilethe
developmentteamreworksthecodetosatisfythisinterpretation,reworkthatcouldhavebeenavoided
ifwehadsharedthetests.Eitherway,wehavesetupanadversarialrelationshipbetweendevelopment
andtesting.Itishighlypreferabletopreparethetestsbeforethesoftwareisbuiltsothattestingcan
helpdevelopmentunderstandwhatneedstobebuiltratherthansimplycriticizewhattheyhavebuilt.
Thesetestsmayberunmanuallyortheymaybeautomated.ThelatterallowstheProductDevelopment
Teamtorunthemthroughoutthedevelopmentcycletoensurethatallspecifiedfunctionalityis
correctlyimplemented.TheProductOwnerwillwanttorunadditionalacceptanceteststomakethe
finalacceptancedecision,butsupplyingasetofteststotheProductDevelopmentTeamearlysothey
candrivedevelopmentgoesalongwaytowardbuildingthecorrectproduct.Thisismuchmorelikelyto
happenwhenthetestsareeasyandcheaptorunandthatrequiresautomatedexecution(formore
information,seetheAutomatedFunctionalTestExecutionthumbnail).Thesetestsmaybeimplemented
asprogrammatictests,buttheyaremoretypicallyimplementedusingKeyworddrivenTestAutomation.
TechnologyFacingTests
Therearemanytestsusedbyproductdevelopmentthatarenotbusinessfacing.Developersmay
prepareunitteststoverifythatthecodetheywrotehassuccessfullyachievedthedesignintent.Thisis
howtheydeterminethattheycorrectlybuiltthecode(asopposedtobuildingthecorrectproduct).
Testdrivendevelopment(TDD)iswhendevelopersimplementautomatedunittestsbeforetheybuild

thecodethetestsverify.Thisdevelopmentprocesshasbeenshowntosignificantlyimprovethequality
ofthesoftwareinseveralwaysincludingbettersoftwarestructure,reducedsoftwarecomplexity,and
fewerdefectsfoundduringacceptancetesting[JeffriesMelnikOnTDDEffectiveness].Thesetestsareever
morefrequentlyautomatedusingmembersofthexUnittestingframework.Formoreinformation,see
[MeszarosOnUnitTestPatterns]
17.1.4 TestsThatCritiquetheProduct
Assumingthattheproducthasimplementedthecorrectfunctionality,weneedtoknowwhetherthe
productmeetsthenonfunctionalrequirements.Thesetestssupporttheacceptancedecision.Wedo
thisbyassessingthenonfunctionalattributesoftheproductafterithasbeen(atleastpartially)built.
Thesetestscritiquetheproductinsteadofdrivingthedevelopmentprocess.Theytelluswhetheritis
goodenoughfromanonfunctionalperspective.Wecandividethesetestsintothetwocategories:
businessfacingandtechnologyfacing.
TechnologyFacingTests
Technologyfacingteststhatcritiquetheproductmeasurehowwelltheproductmeetstechnically
orientedqualityattributes(scalability,availability,selfconsistency,etc.)
Thesetestsprovidemetricswecanusewhendecidingwhethertheproductisreadytobeshipped.In
mostcases,thesetestswillberunaspartofreadinessassessmentbecauseoftheirtechnicalnature.
However,aProductOwnerchargedwithdecidingwhethertoacceptaproductmaybeinterestedin
seeingtheresultsandcomparingthemwiththeminimumrequirement.Theymayevenhireathird
partytestlabtoconductthetestingontheirbehalf.Formoreinformation,seetheTestOutsourcing
thumbnail.
BusinessFacingTests
Ifsystemacceptancetestsareusedtodrivedevelopmenttobuildtheproductpertherequirements,
howdowemakesurewearebuildingtherightproduct?Thebusinessfacingteststhatcritiquethe
productfulfillthisrole.Thesetestsassesstheproduct(eitherasbuiltorasproposed)forfitnessfor
purpose.Usabilitytestsareexamplesofteststhatcritiquetheproductfromabusinessperspective
Typically,thesetestscannotbeautomatedbecausetheyarehighlysubjectiveandsomeevenrequireus
toobservepeopletryingtousetheproducttoachievetheirgoals.
17.2 WhatTestingWillWeDo?AndWhy?
Nowthatwehavebeenintroducedtoawayofreasoningaboutthekindsoftests,wecandecidethe
typesoftestsweneedtorunandwhichteststoautomate.Thisisanoverallteststrategythathelpsus
determinehowtobestaddressourtestingneedsatthelowestcost.

17.2.1 TestStrategy
Definingtheteststrategymaybeconsideredtobepartofthetestplanningprocessoradistinctactivity.
Eitherway,thepurposeofdefiningateststrategyistomakesomehighleveldecisionsaboutwhat
kindsoftestingneedtobedoneandhowtheywillbeexecuted.Teststrategyistightlylinkedtothetest
objectives.Forexample,youwouldhaveadifferentteststrategyifyourobjectivesaretoverifyand
acceptfunctionalityasopposedtoverifycompliance.
Oneofthekeydecisionsiswhatkindsoftestsshouldbeautomatedandwhichapproachtotesting
shouldbeusedformanualtests.Thegoalofthesedecisionsistotrytominimizeprojectriskwhilealso
minimizingthetimeandeffortspenttestingthesoftware.
Previoussectionsintroducedtheconceptsoffunctionalandnonfunctionalrequirements.Aspartofthe
teststrategyweneedtodecidewheretofocus.Clearly,testingcannotprovethatsoftwareworks
correctly;itcanonlyprovethatitdoesnotworkcorrectly.Therefore,wecouldspendaninfiniteamount
oftimetestingandstillnotprovethesoftwareisperfect.Theteststrategyisaboutmaximizingthe
returnoninvestment(ROI)oftestingbyidentifyingthetestingactivitiesthatwillmitigatetherisksmost
effectively.Thisimpliesthatsomerequirementsmaybetestedlessthoroughly,bychoice.
Wealsoneedtodecidewhetherwewilldoalltheacceptancetestingattheendoftheproject(ina
separatetestingphase,orinatestlastorbigbangtestingstyle)orincrementallyasfunctionality
becomesavailable.Thelatterapproach,incrementalacceptancerequireschangesinhowtheprojectis
plannedandhowthesoftwareisdevelopedtoensureacontinuousstreamoffunctionalityisdelivered
startingfairlyearlyintheproject.Thisisthesytlethatwestronglyadvocatesincebigbangtesting
carriestoomanyrisksandisavoidableinmanycontexts.Thepaybackofincrementalacceptanceisearly
discoveryofmisunderstoodandmissedrequirements,therebyallowingtimeforremediationoffthe
criticalpathoftheproject.
Anotherstrategicdecisionmayrelatetotestoracles;oursourceoftruth.Howwillwedefinewhata
correctoutcomelookslike?Isthereacomparablesystemthatwecanuseasanoracle?(Formore
information,seeComparableSystemTestOracle.)Canwehandcraftexpectedresults?(Formore
information,seeHandCraftedTestOracle.)OrwillweneedtouseaHumanTestOracle?Ifso,whatcan
wedofromadesignfortestabilityperspectivetoreducethedependencyonhumantestoracles?
17.2.2 ManualTestingFreedom
Forfunctionaltesting,thekeystrategydecisionsrelatetohowwewillexecutethetests.Whenwe
manuallyexecutethetests,weneedtodecidehowmuchfreedomtograntthetesters.Figure2
containsthe"freedom"scaleusedbyJonandJamesBachtodescribethechoiceswehave.

Figure2
"Freedom"scalefortesters(JonBach,JamesBachusedwithpermission)
Atoneextremeofthetestfreedomscale,thereisfreestyleexploratorytesting,inwhichthetesterscan
testwhatevertheythinkisimportant.Attheotherendofthescaleisscriptedtesting,inwhichtesters
attempttofollowawelldefinedtestscript.Inbetween,thereischarteredexploratorytesting,which
haschartersofvaryingdegreesoffreedomincludingscenarios,userroles/personas,andcharters.
Scriptedtestinginvolveshavinganexpertpreparedetailedtestscriptstobeexecutedmuchlaterby
someoneelse(oracomputerwhenautomated).Thereisverylittleopportunityfortestdesignduring
testexecution.
Exploratorytestingisaneffectiveandefficientapproachtotestingthatleveragestheintelligenceofthe
testertomaximizethebugsfoundinafixedamountoftime.Unlikewithscriptedtesting,testersare
encouragedtoconceivenewthingstotrywhiletheyareexecutingtests.Somepeopledescribeitas
concurrenttestcasedesignandexecutionwithanemphasisonlearning.Basedonasurveyof
companiesusingexploratorytesting,ItkonenandRautiainen[ItkonenRautiainenOnExploratoryTesting,
[ItkonenEtAlOnDefectDetection]reportseveralreasonsforapplyingthisapproach:
whentestingaspectsthatrequiretheusersperspective;
whenwritingsufficientlycomprehensiveautomatedtestsareimpractical;
forleveragingthetestersexperienceandcreativitytodiscoverunusualbehavior;
whenfastfeedbackonproductsbehaviorisrequiredintheabsenceofscriptedtests;
whenrequirementsandfeatureschangerapidly;and
tolearnabouttheproductsbehaviorwhentherequirementsarevague,incomplete,orpossibly
misunderstood.
Formoreinformationonexploratorytesting,see[KanerOnExploratoryTesting]and
[BachOnExploratoryTesting].

17.2.3 AutomatedTesting
Automatedtestingcoversawiderangeoftopics:fromautomatedexecutiontoautomatedtestcase
generation.Somekindsofnonfunctionaltestsrequireautomatedexecutionbecauseofthenatureof
thetestingbeingperformed.Acommonlyoverlookedareaforautomationistheuseof"powertools"
whileperformingmanualtesting.Toolscanalsobeusedtogeneratetestdataortododatainspection.
Thevarioususesoftestautomationneedtobedeterminedonaprojectbyprojectbasis.Formore
informationaboutthisprocess,seethePlanningTestAutomationthumbnailofthisguideand
[FewsterGrahamOnSoftwareTestAutomation].
MaximizingAutomationROI
AneffectivetestautomationstrategystrivestomaximizetheROIoftheinvestmentinautomation.
Therefore,thetestsweautomateshouldcostless,atleastinthelongrun,thanwewouldhavespent
manuallyexecutingthecomparabletests.Sometestsaresoexpensivetoautomatethatwewillnever
recouptheinvestment.Thesetestsshouldberunmanually.
AutomatingtheRightTests
ToensurethatwegetthebestpossibleROIforourtestautomationinvestment,weneedtofocusour
energiesonthefollowing:
Teststhathavetobeautomatedbytheirverynature
Teststhatareinherentlyeasiertoexecuteusingacomputerthanahuman
Teststhatneedtoberunmanytimes
Tasks(nottests)thatcanmakemanual(orautomated)testingfasterandmoreeffective

17.2.4 AutomatedExecutionofFunctionalTests
Automatedfunctionaltestexecutionisapowerfulwaytogetrapidfeedbackonthequalityofthe
softwareweproduce.Whenusedcorrectly,itcanactuallypreventdefectsfrombeingbuiltintothe
product;whenusedincorrectly,itcanrapidlyturnintoablackholeintowhichtimeandeffortare
sucked.Whenautomatedregressiontestsarerunfrequently,suchasbeforeeverycodecheckin,they
canpreventnewdefectsfrombeinginsertedintotheproductduringenhancementormaintenance
activities.EnsuringtheProductDevelopmentTeamunderstandstheacceptancetestsaheadoftimecan
ensuretheProductDevelopmentTeambuildsthecorrectproductthefirsttimeinsteadofasaresultof
testandfixcycles.Forinformationabouthowthisworks,seetheAcceptanceTestDrivenDevelopment
thumbnailinVolumeII.
Acommonstrategyonprojectsthathaveanextensivesuiteofautomatedtestsistorunthesetestsfirst
asaformofregressiontestasthefirstactivityinatestcycle;thisisaformofextendedsmoketest.This
ensuresthatthesoftwarefunctionsproperly(totheextentoftheautomatedtestcoverage)beforea
humantesterspendsanytimedoingmanualtesting.

Thekeytoeffectiveautomatedfunctionaltestingistouseanappropriatetoolforeachtypeoftest
onesizedoesnotfitall.Thetwomostcommonapproachestoautomatedtestpreparationaretest
recording(seetheRecordedTestthumbnail)andtestscripting.Recordedtestsareeasytoproduce,but
theyareoftenhardtomaintain.Scriptedtestscaneitherbeprogrammatictestautomation,which
involvestechnicalpeoplewritingcodetotestthecode,orkeyworddriventestautomation,whichnon
technicalpeoplecanusetowritetestsusingamuchmoreconstrainedtestingvocabulary.Because
keyworddriventestsaretypicallywrittenintheubiquitouslanguagedefinedfortheproduct,theyare
alsomucheasiertounderstandthanmostprogrammatictests.Whateverapproachweuse,weshould
thinkbeyondtheinitialtestauthoringandalsoconsiderthelifecyclecostsofthetests.Recordedtest
toolsdohavesomevaluableuses.Theycanbeusedtoquicklyrecordthrowawaytestsuitestosupport
thedevelopmentteamwhiletheyrefactortestabilityintotheproductundertest.Theycanalsobeused
inarecordandrefactorstyleasawayofquicklybuildingupacollectionofkeywordsortestutility
methodstobeusedinkeyworddriventestsorprogrammatictests,respectively.
Keyworddriventestinginvolvesspecifyingtestscriptsinanonprogrammaticstyle.Thestepsofthetest
aredatainterpretedbyakeywordlanguageinterpreter.Anotherstyleofdatadriventestautomationis
thereuseofatestscriptwithmultipledatasets.Thisisparticularlyeffectivewhenwecangeneratetest
data,includinginputsandexpectedoutputs,usingacomparablesystemtestoracle.Thenwerunthe
datadriventestonetimeforeachsetofinputs/outputs.Commercialrecordedtesttoolstypically
providesupportforthisstyleoftestingandoftenincludeminimalsupportforrefactoringofthe
recordedtestscriptsintoparameterizedscriptsbyreplacingtheconstantvaluesfromtherecordedtest
withvariablesorplaceholderstobereplacedbyvaluesfromthedatafile.
TestAutomationPyramid
Thetestautomationpyramidisagoodwaytovisualizetheimpactofdifferentapproachestotest
automation.Whentestautomationisanafterthought,thebestwecanusuallydoistousegraphical
userinterface(GUI)basedtestautomationtoolstodrivetheproductundertest.Thisresultsina
distributionoftestsasshowninFigureAInvertedTestPyramid.
Frequently,thesetestsareverydifficulttoautomateandverysensitivetoanychangesinthe
application.BecausetheyrunthroughtheGUI,theyalsotendtotakealongtimetoexecute.Soiftest
automationisanafterthought,weendupwithalargenumberofslow,fragiletests.
Animportantprinciplewhenautomatingtestsistousethesimplestpossibleinterfacetoaccessthe
logicwewanttoverify.Projectsthatusetestdrivendevelopmenttechniquesattackthisproblemat
multiplelevels.Theydodetailedunittestingofindividualmethodsandclasses.Theydoautomated
testingoflargergrainedcomponentstoverifythattheindividualunitswereintegratedproperly.They
augmentthiswithsystemtests(whichincludeusecaseorworkflowtests)atthewholeproductlevel.At
eachhigherlevel,theytrytofocusontestingthosethingsthatcouldnotbetestedatthelowerlevels.
Thisleavesthemwithmuchfewersystemteststoautomate.ThisisillustratedinFigure4ProperTest
Pyramid

Figure4
ProperTestPyramid
Onewaytoreducetheeffortinvolvedistominimizetheoverlapbetweentheunittestsandcomponent
testswiththesystemtests.Aspecificexampleofthisistheuseofbusinessunitteststotestbusiness
logicwithouthavingtogothroughtheuserinterface.Anothertechniqueistheuseofsubcutaneous
(literally,undertheskinmeaningbehindtheGUI)workflowteststotestbusinessworkflowswithout
beingforcedtoaccessthefunctionalitythroughtheuserinterface.Bothoftheseapproachesrequire
theproducttobedesignedfortestability.
17.2.5 AutomatedTestingofNonfunctionalRequirements
Manytypesofnonfunctionaltestsrequiretheuseofautomatedtesttools.Manyofthesetoolsare
speciallycraftedforthespecificpurposeofassessingtheproductwithrespecttoaparticularkindof
nonfunctionalrequirement.Commonexamplesincludeperformancetestingtoolsthatgenerateloadto
seehowtheproductcopeswithhightransactionratesorusabilitytestingtoolsthatrecordtestsessions
andanalyzedataonusersperformance(e.g.successfultaskcompletion,timeontask,navigationpaths,
errorratesetc.)
17.2.6 AutomationasPowerToolsforManualTesters
Automatedtestsprovideahighdegreeofrepeatability.Thisworksveryeffectivelyasachangedetector,
butitprobablywillnotfindbugsthathavealwaysbeenthere.Forthatweneedhumantesterswhoare

continuallylookingforwaystobreakthesoftware.Forhumantesterstobeeffective,theymustbeable
tofocusonthecreativetaskofdreamingupandexecutingnewtestscenarios,notthemundanetasksof
settinguptestenvironments,comparingoutputfiles,orgeneratingorcleansinglargeamountsoftest
data.Alotofthesetaskscanbemadefairlypainlessthroughappropriateuseofautomation.
Wecanuseautomatedscriptsandpowertoolsforthefollowing:
Tosetuptestenvironments
Togeneratetestdata
Tocompareactualoutputfilesordatabaseswithtestoracles
Togenerateandanalyzetestreports
Toteardowntestenvironments

Ifweneedalargedataset,wecanwriteaprogramtogenerateonewithknowncharacteristics.Ifwe
needtotesthowparticulartransactionsbehavewhentheproductisstressed,wecanwriteaprogram
thatusesupallthememoryordiskspaceorCPUoncommand.Theseareallexamplesofpowertools
thatmakehumantestersmoreeffective.ForspecificexamplesseeaseriesofVisualStudioTeamTest
walkthroughs[SterlingOnTestTools].
17.2.7 AutomatedTestGeneration
Oneofthegrandobjectivesofsoftwaretestingisautomatedtestgeneration.Theindustryisstillsome
distancefrombeingabletopushonebuttonandhaveatoolgenerateandrunallthetestswewillever
need,buttherearesomeselectedsituationswhereautomatedtestgenerationispractical.Oneexample
iscombinatorialtestoptimization.Forexample,ifwehaveamodulewearetestingthattakesfive
differentparameters,eachofwhichcouldbeanyoneoffourvalues,eachofthevaluescausesthe
moduletobehavesomewhatdifferently,butindifferentway.Totestthiseffectively,wewouldhaveto
test1024(4*4*4*4*4)differentcombinations,whichisnotverypractical.Wecanuseatoolthat
analysesthefivedimensionsandgeneratesaminimalsetoffivevaluetuplesthatwillverifyeach
interactionofaparticularpairofvaluesatleastonce(seeallpairstestgenerationandstressfulinput
testgenerationin[BachOnTestTools]includesall).
AnotherformoftestgenerationisModelBasedTesting(MBT).InMBT,webuildamodelofthesalient
aspectsoftheproductundertestanduseittoderivetests.Thetestscanbederivedmanuallyorwecan
buildatestgenerationtoolthatanalysesthemodelandgeneratesthetestsneedtoachievefulltest
coverage.Thetoolcansimplygeneratetestscriptsthatwillbeexecutedbyothertoolsorhuman
testers,oritcanexecuteeachtestasitisgenerated.
17.2.8 ReadinessvsAcceptance
AsdescribedChapter1TheAcceptanceProcessandChapter2DecisionMakingModel,the
acceptanceofsoftwarecanbedivided,atleastlogically,intotwoseparatedecisions.Thereadiness

decisionismadebytheProductDevelopmentTeambeforegivingthesoftwaretotheProductOwner
whomakestheacceptancedecision.Akeydecisionisdeterminingwhichtestsarerunaspartof
readinessassessmentandwhicharerunaspartofacceptancetesting.Inmostcases,thereadiness
assessmentismuchmoreextensivethantheacceptancetesting.Whenfunctionaltestsareautomated,
itislikelytheyruninreadinessassessment,andthesoftwareisnotreleasedtoacceptancetestinguntil
allthetestspass.ThisresultsinabetterqualityproductbeingpresentedtotheProductOwnerfor
acceptancetesting.
17.2.9 ManagingTestDevelopment&Automation
Itisusefultohaveacommonunderstandingofhowthedevelopmentoftestsrelatestothe
developmentoftheproduct.Insequentialprocesses,thedevelopmentofthetestsstartswellafter
productdevelopmentanddoesntinfluenceit.Weprefertothinkoftestdevelopmentasoccurringin
parallelwithproductdevelopmentspecificallysothatitcaninfluencewhatisbuilt.Itisalsousefulto
thinkabouthowthetestsaredefinedandhowthatrelatestotestautomation.Ifautomationisfocused
onentiretestcases,theautomationcannotbeginuntilatleastsomeofthetestcasesarefullydefined.
Analternativeistothinkoftestautomationattheteststeplevel.Thenautomationdevelopmentcan
beginassoonassomeofthestepsrequiredbytestcasesaredefined.Thisalsoallowsthetest
automationdevelopmenttobecarriedoutbypeoplewithdifferentskills(testautomation)thanthose
doingthetestdesign(testing).Theautomationworkcanbecarriedoutinparallelwiththetest
definitionandtheproductdevelopment.Thismakesitpossibletohaveautomatedtestsavailableas
soonastheproductsoftwareisavailableratherthanatsomelatertime.
TheteststepsoractionsshouldbedefinedusingtheUbiquitousLanguageadoptedbyeveryoneinthe
project.ThetestscriptsdefinedusingthesetermscanbeexecutedbyhumantestersoriftheROIis
sufficient,automated.Ifallthestepsneededbyatestscripthaveautomatedimplementations,the
entiretestscriptcanbeexecutedautomatically.Ifonlysomeofthestepsareautomated,ahuman
testermaystillbeneededbutthetimerequiredtoexecutethetestmaybereducedsignificantly.
17.3 WhoWillAccepttheProduct?
Ultimately,theacceptancedecisionbelongstotheProductOwner.Insomecases,theProductOwner
maynotbeasingleperson.Inthesecases,wemayhaveaProductOwnerCommitteethatmakesthe
acceptancedecisionusingsomesortofdemocraticorconsensusbasedprocess.Or,wemayhaveaset
ofacceptancedecisionmakersthateachcanvetoacceptance(seeChapter1TheAcceptanceProcess
inPartI)Inothercases,theProductOwnermaybeunavailable.Inthesecases,wemayneedaProduct
Ownerproxytoactasthe"goaldonor"whobothprovidesrequirementsandmakestheacceptance
decision.TheproxymaybeeitheradelegateselectedbytheProductOwner,amediatorbetweena
groupofcustomers,orasurrogatewhoactsonbehalfofalargegroupofanonymouscustomers.The
latterroleisoftenreferredtoastheproductmanager.Formoreinformationaboutthisprocess,seethe
CustomerProxySelectionthumbnailinVolumeII.

Arelatedquestioniswhowilldotheacceptancetesting?Andbyextension,whowilldothereadiness
assessment.Thisverymuchdependsonthebusinessmodelandthecapabilitiesandskillsetofthe
partiesinvolved.Thesidebar"DecisionMakingModelStereotypes"enumeratesseveralcommon
scenarios.WheneithertheProductDevelopmentTeamortheProductOwnerfeelstheyneedassistance
conductingthereadinessassessmentoracceptancetesting,theymayresorttoaTestOutsourcing
model.Thethirdpartytestlabwoulddotheassessment,butthereadinessdecisionandacceptance
decisionstillbelongtotheProductDevelopmentTeamandProductOwnerrespectively.
17.4 WhenWillWeDotheTesting?
Thetestplanneedstoaddresswhenthetestingwillbedone.Sometestingactivitieswillbedonebythe
ProductDevelopmentTeamaspartofreadinessassessment,whileothersaretheresponsibilityofthe
ProductOwnerwhowillbedecidingwhethertoaccepttheproduct.Thetestplanneedstoincludethis
infurtherdetailtothepointwherewehaveanunderstandingofhowmuchtimeweneedforreadiness
assessmentandacceptancetestingandwhatwewillusethattimefor.Therearetwomainstrategiesfor
whentestingisdoneonaproject.Thesequentialapproachtoproductdevelopmentlifecycleplaces
mosttestingactivitiesintoaseparatetestphaseafteralldevelopmentiscompleted.Theagileapproach
istotesteachincrementoffunctionalityasitiscompleted.
17.4.1 TestPhase
Onewaytoplantestingistodefineatestingphaseoftheprojectafterallnewfunctionality
developmentiscomplete.ThisisillustratedinFigure5TestingPhaseinaSequentialProduct
DevelopmentLifecycle.

Figure5
TestingPhaseinaSequentialProductDevelopmentLifecycle
Thistestingphasetypicallyconsistsofseveraltimeboxedtestcycles,eachofwhichcontainsboth
readinessassessmentandacceptancetestingactivitiesasillustratedinFigure6.

Figure6
MultipleTTestCyclesinAcceptPhaseeofProject
Eachtestcyclewouldb befocusedon nonereleaseecandidate(R RC)versionoffthesoftwaree.Withinthetest
cycletheProductDeve elopmentTeaammakesareeadinessdeciisionbeforeinvolvingtheProductOwn ner
Teamintheacceptanccetesting.Inttheearlytesttcycles,thisrreadinessdeccisionismadeebyanswerin ngthe
question,"Isitgoodennoughtoboth herhavingtheProductOw wnerTeamteestit?"Itisvaaluabletoget
ProductO Ownerfeedbaackonthesofftwareevenifweknowth herearesomeedefects.Theetestcycleseeach
resultincconcernsthattneedtobein
nvestigated.AAnyconcernssthatneedso oftwarechanggesarethen
addressed dbytheProductDevelopm mentTeam,aandanewreleeasecandidateisbuilt.Thissetsthestaage
fortheneexttestcycle.Werepeatthhisprocessuntilthereleassecandidateisacceptedb bytheacceptaance
decisionm maker(s).
predefinedseetoftestingaactivities;thessemaybelaidout
Withineaachtestcycle,,itislikelyweewillhaveap
asaPertcchartoraGanttcharttoreeflecttimingandinterdep pendencies.W Wemaydecid detoreuseth
he
sameplan nforeachoftthetestcyclees,orwecoulddefineaun niqueplanforreachcycle.Inpractice,wwith
goodauto omatedregre essiontestinginplace,weshouldfindfeewerandfew werdefectseaachtestcyclee.
Becauseo ofthis,ariskbasedapproaachtoplannin ngthesubseq quentcyclesccouldresultinshortercycle
timesand dfastertimettomarket.Wemayalsodeeliberatelych hoosetodefeersomekindsoftesting
activitiesttolatertestccyclesortodo
otheminearrliertestcyclees.
Thetestresourcesonaasequentialp projectaretyypicallyinvolvvedmostlyatthebackend doftheprojeect.
Evenwheentheyareinvvolvedinflesshingouttherequirementsatthebegin nningofthepproject,their
involvemeentwhilethesoftwareisb beingbuiltten
ndstobemin nimal.Thestaaffingprofileofahypothettical
10weekpprojectisillusstratedinFigu
ure7SequeentialTestSpeecialistStaffin
ngProfile.
didate1 180

12
10
8 BugFix
Regression
6
TestExecution
4 TestDesign
Requirements
2
0
1 2 3 4 5 6 7 8 9 10

Figure7
SequentialTestSpecialistStaffingProfile
Thisprojectinvolvestenfeaturesandeachfeaturerequiresonepersonweekofrequirements,one
personweekoftestdesignandonepersonweekoftestexecutionfollowedbyapersonweekofbug
fixingbydevelopmentandonepersonweekofregressiontesting.Notehowthebulkoftheactivity
occursatthebackendoftheprojectandrequirestentestspecialistsforweeks8and10(week9
involvesprimarilydevelopmentresources.)Thisburstynatureoftestingisusuallyaccommodatedby
testerssplittingtheirtimeacrossseveralprojects.Thismakesithardforthemtokeepuptodateon
whatisbeingbuiltandhowtherequirementsareevolvingbecausemultiplexing(taskswitching)isa
formofwaste.(SeethesidebaronMultiplexingvs.UndividedAttention).Also,thisrequires
developmenttobefinishedinweek7toleavetimefortwo1weektestcycleswithaweekofbugfixing
inbetween.
Sidebar:Mutliplexingvs.UndividedAttention
Havingthebulkoftestingactivitydoneattheendoftheproductdevelopmentlifecyclegivesthe
managerstheillusionthattestersexperienceslackintheirworkloadsatdifferenttimesandtherefore
canbeassignedtomultipleprojectsthatwouldenablethemtousethetimemoreefficientlyand
effectively.Notonlydoesthisapproachplaceconstraintsonwhendevelopersreceivefeedbackon
theproduct,butalsoonhowmuchtimetheyhavetoactonthatfeedback.
Multiplexingrequirescontextswitiching.Thecostofcontextswitchingincludestheeffortfor
understandingeachprojectsstatusandreimmersingonselfintheotherprojectcontextaswellas
findingpropertestenvironments,resettingthem,relearningcertainrequirementsoftheproduct,re
establishingcontactwiththerightpeople,andoftenregainingfocusandrethinkingissuesthathave
alreadybeensettled.AllofthesearereferredtobyTomDeMarcoasimmersion[DeMarcoOnSlack].
Multiplexingmayalsorequirecatchingupontheworkdoneinonesabsence.

GeraldWeinbergestimatesthatthecontextswitchingcostofassigninganadditionalprojecttoone
personisamassive20%[WeinbergOnContextSwitching].Forthreeprojects,itsa40%dropin
productivity!
JoelSpolskystestimonyindicatesthatwithsoftwareengineersthecostofcontextswitchingeven
greater75%[SpolskyOnContextSwitching].Spolskyarguesthatsoftwaredevelopmentisthekindof
intellectualactivitywherewehavetokeepalotofthingsinourheadatonce.Themore[information]
you[can]keepinmind,themoreproductiveyouare.Suchinformationincludesvariablenames,APIs,
datastructures,requirements,helperfunctions,testcases,detailsofstacktraces,debugging
information,therepositorystructure,configurationparametersandwheretheyareset.
Researchersincognitivepsychologyandorganizationalbehaviourwhospecificallystudiedsoftware
engineersalsocametothesameconclusion:multiplexingisharmful.Itsoneofthecontributing
factorstotimefamine,whichisdestructivetoindividualslivesandnotinthebestinterestoftheir
employerseither[PerlowOnTimeFamine]and[OcasioOnWorkFragmentation].

Ooopsthepieceofcodeaboveclearlydoesntbelonghere.Itcrawledinfromadifferentproject.
Whennoticedit,Irealizedwhatatravestythiswas!Iwasworkingonademoofthefluent
configurationinterfaceforfortheEnterpriseLibrary5.0projectwhilealsoworkingontheAcceptance
TestEngineeringGuide.
Thiswassuchawonderfulillustrationofanotherpointhowthebugscaneasilycrawlinwhile
multiplexingandnotgivingthefullattentiontooneprojectthatIvedecidedtoleavethisinhereas
acaseinpoint!
Thebottomlineisthis:fulltimeinvolvementinasingleprojectimprovesindividualperformance.
17.4.2 IncrementalTesting
Thealternativetoleavingallthetestingtotheendofaprojectistodoincrementaltestingassoftwareis
developed.ThisrequiresthattheProductDevelopmentTeamorganizesitsworksuchthatthereisa
steadystreamoftestablesoftwareavailablestartingearlyintheproject.ThisisillustratedinFigure8
IncrementalProductDevelopmentLifecycle.

Figure8
TestinginanIncrementalProductDevelopmentLifecycle
Thisincrementalproductdevelopmentlifecycleallowsacceptancetestingtobespreadoutovermostof
theprojectdurationinsteadofbeingjammedintoamuchshortertestingphaseonthecriticalpathnear
theendoftheproject.(Thisisratherdifferentthanthesequentialapproachusedinmanyorganizations;
thesidebarWhatitTakestodoIncrementalAcceptancedescribessomeofthechallengesand
solutions.)Aseachfeatureoriterationisfinished,itisassessedforreadinessimmediatelybythe
ProductDevelopmentTeamandturnedovertotheProductOwnerforacceptancetesting.Ideally,any
bugsfoundarefixedimmediatelyasdescribedinthesidebarIncrementalAcceptanceandBugTracking
onAgileProjectsinChapter8ratherthanbeingstockpiledfortheendoftheproject.
Ifthesametenfeaturesdescribedinthesequentialexamplearebuiltbyanagileteamusing
IncrementalAcceptanceTesting,thetestingstaffprofilelooksmorelikeFigure9AgileTestSpecialist
StaffingProfile.
5
4
Regression
3
TestExecution
2
1 TestDesign
0 Requirements
1 2 3 4 5 6 7 8 9 10

Figure9
AgileTestSpecialistStaffingProfile.
Onthisprojectwestartoutbyspecifyingtwofeaturesinthefirstweek,designthetestsforthosetwo
featuresinthesecondweekaswellasspecifyingathirdfeature.Inthethirdweekwetestthefirst
featureanddotestdesignforthethirdfeature.Notehowthestaffingrampsuptofourpersonweeksof

workandstaystherefortherestoftheduration.Towardstheendasnewfeaturetestingrampsdown
theslackistakenupbyregressiontestingoftheearlyfeatures.Becausethetestersarepartofthe
developmentteamtheyareabletokeepabreastofhowtherequirementsandproductdesignare
evolving.Sincethesamepeoplewillbedesigningandexecutingtheteststheneedtowritedetailedtest
scriptsarereducedandmoreeffortcanbespentonexploratorytestingandtestautomation.Sincethe
testingisdonesoonafterthecodeisdevelopedthedeveloperscanfixthebugsimmediatelysothereis
noseparatetestingandbugfixphaseattheendthereforedevelopmentcancontinueclosertothe
deliverymilestone.
17.4.3 SessionBasedTestManagement
Analternativetousingaplandrivenapproachwithinthetestcycleistouseamoreiterativestyle
knownasSessionBasedTestManagement.Wecreateaprioritizedbacklogoftestingactivitiesthatwe
addressinaseriesoftestsessions.Asnewconcernsareidentifiedintestsessions,wemayadd
additionaltestactivitiestothetestbacklog.Thekeyistokeepthebacklogprioritizedbythevalueofthe
testing.Thisvalueistypicallybasedontheexpecteddegreeofriskreduction.Thedepthofthebacklog
givesusanideaofhowmuchtestingworkwehaveleft(forexample,bycreatingatestingburndown
chart)andifwearemakingheadwaybyaddressingconcernsorifyouarelosingground(ifthisisthe
case,thebacklogisincreasingindepth).SessionBasedTestManagementiscommonlyusedwith
exploratorytesting.
17.5 WhereWillWeDotheTesting?
Thetestplanneedstoidentifywherethetestingwillbeperformed.Whenallthetestingwillbe
performedinhouse,theprimaryconsiderationiswhichphysical(orvirtual)environmentswillbeused.
Thisisparticularlyimportantwhennewenvironmentsneedtobecreatedorsharedenvironmentsneed
tobebooked.Ifwelackphysicalresourcesortheskillstodothetesting,wemaychoosetodotest
outsourcingtoathirdpartytestlab.
Wealsoneedtodefinethecriteriaformovingthesoftwarebetweentheenvironments.Thetransition
fromthereadinessassessmentenvironmenttotheacceptanceenvironmentisgovernedbythe
readinessassessmentcriteria.Whendevelopershavetheirownindividualdevelopmentenvironments,
wealsoneedcriteriaforwhensoftwarecanbesubmittedintotheteam'sintegrationenvironment
wherereadinessassessmentwilloccur.ThesecriteriaareoftenreferredtoastheDoneDoneChecklist
becausethedefinitionof"done"ismorestringentthanwhatadevelopertypicallyreferstoasdone.
17.6 HowLongWilltheTestingTake?
Becausetestingisusuallyonthecriticalpathtodeliveryofsoftwareintensiveproducts,project
sponsorsandprojectmanagersusuallywanttoknowhowlongtestingwilltake.Themostcommon

answeris"Howmuchttimedoweh have?"Frequeently,thetimeavailableisnotlongeno oughtogatheer

enoughdatatomakeaahighconfideencereadinesssoracceptancedecision.Thisanswerisnotquiteaas
flippantasitsoundsbe ecauseoftheenatureoftesting.Wecannnotproveso oftwareworksscorrectlyw we
canonlyd disproveitbyyfindingbugs.Evenifweh
hadaninfiniteeamountofttimetotest,aafterapoint,
diminishin ngreturnswillkickinandwewillnotfindmanymorredefects.So o,wehaveto odeterminewwhat
isbarelyssufficienttoggetenoughcoonfidenceabo outthequalittylevel.Thisrrequiresatleastaminimal
levelofteestestimation oundofthetimeandefforrtweneedtoexpend.
ntoestablishthelowerbo
Sometesttsmayhaveddependenciesstherestrictwhentheycaanberun.Theesecouldbedependencieeson
othertesttenvironmen
ntssuchaswh
hendoinginteegrationtestingwithotheersystemsortheymaybe
dependen nciesondate.
needtoknowhowlongweellneedtowaitforanydeefectsweneeedaddressedtobefixedbythe
Wealson
ProductD
DevelopmentTeam.Willth herebedead dtimebetweentestcycleeswhilewew
waitforfixed
softwareasillustratedinFigureYAlternatingTTest&Fix?

FigureY
Alternatin
ngTest&Fix
Thedeadperiodsoccuuriftheaccep
ptancedecisioonisthefirstpointatwhicchtheProducctDevelopmeent
TeamisprovidedthelistofmustfixbugsafterrwhichtheProductDevelopmentTeam mstartsthe
processofcharacterizingandfixingthebugs.Thismaytaked daysorweeksstodevelopaanewreleasee
candidateewhichmustthenundergo oreadinessassessmentbeeforebeingpresentedtottheProduct
Ownerforthenextrouundofaccepttancetesting..
Thedurattionoftheacceptancephaase(consistingofseveraltestcycles)caanbereduced dsignificantlyyif
theProdu uctDevelopmmentTeamisnnotifiedofbu
ugsassoonasstheyarefouund.Thisallow
wsthemtobee
fixingdefeectscontinuo
ouslyanddeliveringanew
wreleasecanddidateonareegularschedu uleasshowniin
FigureXContinuousTest&Fix.
didate1 185

FigureX
Continuou usTest&Fix
ThisdepeendsonthePrroductOwner(orproxy)d
doingbugtriageonallnew
wlyfoundconcernssothattthe
ProductDDevelopmentTeamismadeawareofalllgatingbugs(bugsthatm mustbefixedb
beforeacceptting
thesoftware)assoonaspracticablee.
planningtoautomatetestts,weshouldhaveaneffortestimatefo
Ifwearep ortheautomation.Inmosst
cases,wewantseparateestimatesfortheconsttructionoftheeautomation ninfrastructureandthe
preparatioonofthetesttsbecauseoffthedifferenttskillsandkn
nowledgeneeededtodotheetwojobs.Fo
or
moreinfoormation,seetheTestAutomationPlan nningthumbn nail.
17.7 HowWilllWeDete
ermineOu
urTestEfffectivene
ess?
Alearninggorganization
nisonethatiisconstantlystrivingtoim
mprovehowittworks.Thisiinvolves
understanndinghowwe ellwearedoingtodayand dtryingnewaapproachestooseewhetheertheymakeus
moreeffeective.Measu uringeffectiveenessrequireesTestMetriccs.Thesemettricsmeasuretwokeyareaasof
performance:
Theymeasurehow wfaralongareweinexecu
utingourtesttplan.Thatis,howmuchw
workisleftbeefore
w
weknowenou ughtomaketthereadinessoracceptanccedecision?FFormoreinfo
ormation,seeethe
TeestStatusReportingthum
mbnail.
Theymeasuretheeffectivenesssofourtestin
ng.Formoreinformation,seetheAssessingTest
Efffectivenesstthumbnail.

17.8 HowWilllWeMana
ageConce
erns?
Thepurpo
oseoftestinggistoidentifyyanyconcern
nswiththeso
oftwareoftho
osewhomattter.Manyoftthese
concernswillrequirecchangestotheesoftwareeitherbecauseesomethingw wasincorrectllyimplementted(a
didate1 186

bug)orbecausetheProductOwnerrealizedthatwhattheyhadrequestedwillnotsatisfythebusiness
need(anenhancementorchangerequest).Thetestplanneedstolayouthowtheseconcernswillbe
managedandtracked;italsoneedstoincludetheprocessfordecidingwhatneedstobechangedand
whatisacceptableasis.
Asweconductthevariousreadinessassessmentandacceptanceactivities,wenoteanyconcernsthat
comeup.Investigatingtheseconcernsmorecloselyrevealsthat,eachconcernfallsintooneofthe
followingcategories:
Bugordefect
Requirementschange
Projectissue
Nonconcern

Figure3illustratesthelifecycleforeachofthemajortypesofconcerns.

Figure3
ConcernResolutionModel
Partoftheadvantageofcategorizingtheseconcernsistohelpidentifyhowtheywillbeaddressed.Each
categoryofconcernshasitsownresolutionprocessandeachmustbeaddresseddifferently.
17.8.1 BugsorDefects
Bugsaredefectsfoundinthesoftwarethatrequireasoftwarechange.Thebugsneedtobeunderstood
wellenoughtomakedecisionsaboutwhattodoaboutthem.Acommonprocessfordoingthisisknown
asBugTriage,whichdividesthebugsintothreecategorieswithrespecttothenextmilestoneorrelease:
MustFix,WouldLiketoFix(ifwehavetime),andWillNotFix.Ofcourse,thesoftwaremustberetested

afterthefixesaremade,whichiswhytherearetypicallymultipletestcycles.Fixesmayalsoneedtobe
propagatedintootherbranchesofthesoftware.Ifthebugisfoundtoexistinpreviousversionsthatare
currentlyinuse,apatchmayneedtobepreparedifthebugisseriousenough.Securityrelatedbugsare
anexampleofbugsthataretypicallypatchedbackintoallpreviousversionsstillinuse.Likewise,abug
fixedintheacceptancetestbranchmayneedtobepropagatedintothecurrentdevelopmentbranchif
newdevelopmenthasalreadystartedinaseparatedevelopmentbranch.
17.8.2 RequirementsChanges
TheProductOwnermayhaverealizedthateventhoughtheProductDevelopmentTeamdeliveredwhat
theProductOwneraskedfor,itwillnotprovidetheexpectedvalue.TheProductOwnershouldhavethe
rightandresponsibilityformakingthebusinessdecisionaboutwhethertodelaythereleasetomakethe
changeorcontinuewiththelessusefulfunctionality.Oncewevedecidedtoincludeachangeinthis
releasewecantreatitmoreorlessthesameasabugfromatrackingandretestperspective.
17.8.3 OtherIssues
Someconcernsthatareexposeddonotrequirechangestothesoftware.Theymaybeprojectissues
thatneedtobetrackedtoresolution,additionalthingsthatshouldbetested,andsoon.Thesetypically
donotgettrackedinthebugmanagementsystembecausemostprojectshaveothermeansfortracking
them.Otherconcernsmaybenotedbutdeemedtonotbeconcernsatall.
17.9 Summary
Thischapterintroducedtheactivitiesandpracticesinvolvedinplanningatestingeffort.Akeyactivityis
thedefinitionofateststrategybecausethisiswhatguidesusaswestrivetomaximizetheROIofour
efforts.Thereisaplaceforbothautomatedtestingandmanualtestingonmostprojectsbecausethe
twoapproachesarecomplementary.Automatedfunctionaltestsarehighlyeffectivechangedetectors
thatgoalongwaytowardpreventingnewbugsfrombeingintroducedduringsoftwaremaintenance
activities.Theyarealsorepeatable.Carehastobetakentousetheappropriatefunctionaltest
automationtoolstoavoidtheslow,fragiletestsquagmire.Exploratorytestingmanualtestingcanbe
effectiveandefficientwaysoffindingbugs,especiallyunusualones,thosethatarehardtoautomate,
andthosethatstemfromincomplete,misunderstood,orvaguerequirements.Theuseofpowertoolsby
humantesterscansignificantlyincreasetheeffectivenessofmanualtesting.
17.10 WhatsNext?
Inthischapterwehavedescribedthepracticesinvolvedinplanningfortheacceptanceoftheproduct.
Inthenextchapterwewillexaminethepracticesweusewhileexecutingtheacceptanceprocess.
17.11 References
[MarickOnTestingDimensions]Marick,Brianhttp://www.exampler.com/oldblog/2003/08/21/

[CrispinGregoryOnAgileTesting]Crispin,LisaandJanetGregory.AgileTesting,AddisonWesley,2008.
[MeszarosOnUnitTestPatterns]Meszaros,Gerard.xUnitTestPatterns:RefactoringTestCode.Addison
WesleyProfessional.2007.
[KanerOnExploratoryTesting]Kaner,Cem,JackFalk,andHungQ.Nguyen.TestingComputerSoftware,
2ndEdition.Wiley.1999.
[BachOnExploratoryTesting]Bach,James.WhatisExploratoryTesting?
http://www.satisfice.com/articles/etarticle.pdf
[FewsterGrahamOnTestAutomation]Fewster,M.,Graham,D.SoftwareTestAutomation.Addison
Wesley,1999.
[ItkonenRautiainenOnExploratoryTesting]JuhaItkonenandKristianRautiainen,ExploratoryTesting:A
MultipleCaseStudy,InternationalSymposiumonEmpiricalSoftwareEngineering,ISESE2005,1718
Nov.2005,pp.8494.
[ItkonenEtAlOnDefectDetection]Itkonen,J.;Mantyla,M.V.;Lassenius,C.DefectDetectionEfficiency:
TestCaseBasedvs.ExploratoryTesting,inProc.ofFirstInternationalSymposiumontheEmpirical
SoftwareEngineeringandMeasurement,ESEM2007.2021Sept.2007,pp6170.
[JeffriesMelnikOnTDDEffectiveness]Jeffries,R.andMelnik,G.,TDD:TheArtofFearlessProgramming,
GuestEditorsIntroduction,IEEESoftwareSpecialIssueonTestDrivenDevelopment,MayJune2007,
pp.2430.
[SterlingOnTestTools]CharlesSterling,VisualStudioTeamSystem2010TestFeatureswalkthrough
withscreenshots.http://blogs.msdn.com/charles_sterling/archive/2008/11/05/visualstudioteam
system2010testfeatureswalkthroughwithscreenshots.aspx,October20,2009
[BachOnTestTools]JamesBach.RepositoryofTestTools.http://www.satisfice.com/tools.shtml,October
20,2009
[WeinbergOnContextSwitching]GeraldWeinberg,QualitySofwtareManagement,Vol.1:Systems
Thinking,NewYork,NY:DorsetHousePublishing,1992.
[SpolskyOnContextSwitching]JoelSpolsky.HumanTaskSwitchesConsideredHarmful
http://www.joelonsoftware.com/articles/fog0000000022.html,2001,visitedOctober20,2009
[DeMarcoOnSlack]TomDeMarco,Slack,Broadway,2001.
[OcasioOnWorkFragmentation].WilliamOcasio.Towardsanattentionbasedviewofthefirm.
StrategicManagementJournal,18:187206,1997.
[PerlowOnTimeFamine]LesliePerlow.Thetimefamine:Towardasociologyofworktime,
AdministrativeScienceQuaterly,44(1):5781,1999.

Chapter18 AssessingSoftware
Inthepreviouschapterswehaveintroducedmanyoftheconceptsaroundhowweplantheassessment
oftheproductagainsttheMinimumMarketableFunctionality(MMF)andminimumqualityrequirement
set(MQR)towhichwehaveagreed.Inthischapterwewillintroducethevarioustechniquesthatweuse
aswedotheassessmentincludingtestconception,testdesignandtestexecution.
Assessmentisagenerictermwecanusetodescribetheactivities,testingorotherwise,thatweuseto
evaluatethesystemundertest.Someoftheseactivitiesarefocussedonpreparingandexecutingtests
whileothersmaybereviewactivities.Someoftheactivitiesaredoneaspartofreadinessassessmentby
theproductdevelopmentteamwhileothersmaybedonebyorfortheproductownerunderthebanner
ofacceptancetesting.Wherethesamepracticeisusedinbothformsoftesting,themechanicsofthe
practicestypicallydontchangeverymuchalthoughtheemphasisofhowmucheachpracticeisdone
andtheobjectivesofapplyingthatpracticemightvary.
Westartthediscussionwithanoverviewofthelifecycleofanindividualtests,somethingthatboth
functionalandnonfunctionaltestsdoshareandthenmoveintoadiscussionofthepracticesusedin
eachstateoftheindividualtestlifecyclewhereweintroducethetechniquesthatareuniquetospecific
ofkindsofrequirements.
18.1 TheLifecycleofanIndividualTest
Everysingletest,howeversimpleorcomplex,whethermanualorautomated,goesthroughanumberof
stagesduringitslifetime.ThislifecycleisillustratedinFigure1.

Figure1
IndividualTestLifecycle

Thestatesofthelifecycleare:
Conception Anacceptancetestisconceivedtoaddressaparticularriskand/orachievea
specificgoal.
Authoring/Design Thetestiswritteneitherindetailedstepformorsomekindofoutlineofwhat
needstobedone,bywhom,when,where,andhow.
Scheduling Theexecutionofthetestisplannedorscheduledforaspecifictimeframeand
resources(people,testenvironment(s),etc.)
Execution Thetestisexecutedagainstthesystemundertest.
ResultAssessment Theresultsofthetestareassessedagainsttheexpectations.(Thismayoccuras
partofexecutionorseparately.)
Reporting Theassessedtestresultsareaggregatedandreported.
Actioning Thetestresultsmayresultineitherfurthertestingbeingidentifiedand/orbug
reportsbeingcreatedandtriaged.
Maintenance Eachtestisanassetandmostmustbemaintainedsothatitcanprovidevaluein
thefuture.Sometests,however,e.g.whendoingexploratorytesting,,arenot
meanttobemaintainedorrepeated.
EndofLife Thetesthasoutliveditsusefulness.Itisabandonedandnolongerrunor
maintained.
Notethattestconceptionandtestauthoringareoftenlumpedtogetherunderthelabeloftestdesign.
Someformsoftesting,oftencalledstatictesting,involveinspectingartefactsthatdescribethesystem
undertestratherthanrunningtheactualcode.Theterminologyusedfortheseformsoftestingis
somewhatdifferent(forexample.theyareoftencalledreviewsratherthantests)butforthepurposeof
discussingtestlifecycle,weshallusethetestterminology.
Letsexamineeachofthesestatesinabitmoredetail.
18.1.1 TestConception
Atsomepoint,someonedecidesthatweneedtoverifyoneormoreaspectsofthesystemundertests
behavior;wecallthesethingstotesttestconditions.Atthistimethetestisjustafigmentof
someonesimagination.Itstartsitstransitionfromanimplicitrequirementtoonethatismuchmore
explicitwhenitgetswrittendownorcapturedinadocument.Itmightappearinalistoftestconditions
associatedwithafeature,requirementoruserstory.Typically,itwilljustbeaphraseornamewithno
associateddetail.Nowthatthetestexistsinconcept,wecanstartmovingitthroughitslifecycle.
18.1.2 TestAuthoring/TestDesign
Testauthoringortestdesignisthetransformationofthetestortestconditionfrombeingjustanamed
itemonalistintoconcreteactions.Itmayalsoinvolvemakingdecisionsaroundhowtoorganizetest

conditionsintotestcaseswhicharethesequencesofstepsweexecutetoverifythem.Notethattest
authoring/designmayhappenlongbeforetestexecutionorconcurrentlywithtestexecution.
18.1.3 TestScheduling
Onceatestcasehasbeenidentifiedandauthoredoratestcharterdefined,wemustdecidewhenitwill
beexecuted.Theschedulemayindicatetheonetimethetestisrunorthefrequencywithwhichitisrun
andthetriggeringmechanism.Itmayalsoidentifywhoorwhatisexecutingthetestandinwhichtest
environment(s).
18.1.4 TestExecution
Onceauthoredandscheduled,weneedtoactuallyexecutethetests.Fordynamicteststhisinvolves
runningthesystemundertest;statictestsinvolveinspectingvariousartefactsthatdescribethe
systemundertestbutdonotinvolvedexecutingthecode.Dependingonthekindoftestinquestion
thetestmaybeexecutedmanuallybyaperson,byanautomatedtestingtool,orbyapersonusingtools
thatimprovetesterproductivitythroughautomation.
18.1.5 ResultAssessment
Dependingonthetoolsinvolved,thepass/failstatusofthetestsmaybedeterminedasthetestsare
executedortheremaybeaseparatesteptodeterminethetestresultsafterthetestexecutionhasbeen
completed.Wedeterminewhetheratestpassedorfailedbyinspectingtheactualresultsobservedand
determiningwhethertheyareacceptable.
18.1.6 TestReporting
Onceasuiteoftestshasbeenexecutedandassessed,wecanreportonthetestresults.Agoodtest
reporthelpsalltheprojectstakeholdersunderstandwheretheprojectstandsrelativetotherelease
gate.Chapter1TheAcceptanceProcessprovidesdetailsonwhatinformationmightaffectthis
decision.Testreportingincludesbothteststatusreportingtoindicatehowmuchtesteffortremainsand
testeffectivenessreportingwhichdescribesourlevelofconfidenceinourtests.
18.1.7 TestActioning
Thepurposeofexecutingtestsistolearnaboutthequalityofourproductsothatwecanmake
intelligentdecisionsaboutwhetheritisreadyforuseorrequiresfurtherdevelopmentortesting.The
AcceptanceProcessdescribestheprocessfordecidingwhetherornottoacceptthesoftwarebutbefore
wecanmakethatdecisionwemayneedtofixsomeofthedefectswehavefound.TheBugTriaging
processisusedtomaketheIsitgoodenough?decisionbydeterminingwhichbugsneedtobefixed
beforewecanrelease.(SeetheDonenessModelformoredetails.)
18.1.8 TestMaintenance
Sometestsareonlyrunoncewhileothersmayneedtoberunmanytimesoverlongperiodsoftime.
Teststhatwillberunmorethanoncemayrequiremaintenancebetweenrunsasaresultofchangesto

thepartsofthesystemundertestthattheyinteractwith(forexample,thedatabasestate).These
kindsoftestsmaywarrantmoreofanupfrontinvestmenttoensurethattheyarerepeatableand
robust.Testsintendedformanualexecutionwillneedtobeupdatedwheneverthepartsofthesystem
beingtestedundergosignificantchangesinfunctionality.Whereashumantesterscanusuallywork
aroundminorchanges,fullyautomatedtestswilltypicallybeimpactedbyeventhesmallestchanges
(withteststhatinteractwiththesystemundertestthroughtheGUIbeingthemostfragile)and
thereforemayrequiresignificantlymorefrequentmaintenance.
18.1.9 EndofLife
Soonerorlateratestmaynolongerbeworthexecuting.Perhapsthefunctionalityitverifieshasbeen
removedfromthesystemundertestormaybewehavedeterminedthatthefunctionalityiscovered
sufficientlywellbyothertestssothatwenolongergetmuchadditionalvaluefromrunningthistest.At
thispointthetesthasreacheditsendoflifeandnolongerwarrantseitherexecutionormaintenance.
18.2 VariationsinTestLifecycleTraversal
Sometestsspendalotoftimeineachstateofthetestlifecyclewhileothersmaypassthroughthe
statesveryquickly.Forexample,inautomatedfunctionaltestingwemightspendsweekspreparinga
complextest,waitseveralweeksbeforewecanfirstexecuteit,andthenrunitseveraltimesadayfor
manyyears.Incontrast,duringasingleonehourexploratorymanualtestingsession,thetestermay
conceiveofseveraltestconditions,designatesttoexplorethem,learnsomethingaboutthesystem
undertest,conceiveseveralmoretestconditionsanddesignteststoexplorethemalso,allinthe
spaceofafewminutes.Theautomatedtestswillspendmostoftheirlifetimeinthemaintenancestate
whileexploratorytestsareveryethereal;thereisntaconcreterepresentationthatneedstobe
maintained.
18.2.1 HighlyCompressedTestLifecycleExploratoryTesting
ExploratorytestingissummarizedbyCemKanerasSimultaneoustestdesignandexecutionwithan
emphasisonlearning.Fromthisdescriptionitshouldbeclearthatthereisnoclearseparationofthe
variousstagesofthetestlifecyclewhendoingexploratorytesting.Thetesterlearnsaboutthesystem
undertestbyusingitandforminghypothesisabouthowitshouldbehave.Basedonthesehypotheses
thetesterconceivesofoneormoretestconditionstowhichtheymightsubjectthesystemundertest.
Theyrapidlydesign,intheirmindseye,atestcasetheycouldusetoachievethis.Theyexercisethetest
caseandobservetheresulttherebyformingmorehypotheseswhichinturnleadtomoretest
conditions.Thereisnotanattemptmadetohavethetestpersistbeyondthetestsessionunlessit
revealedabug.Thisremovestheneedtodocumentandmaintainthetests(thoughagoodexploratory
testerkeepsasetofnotes/journalofkeypointsanddiscoveriesduringhisorhertesting);theobvious
consequenceisthatexploratorytestingisnotintendedtobeveryrepeatable.
Thisprocessisverylightweightwithverylittleoverheadgettinginthewayofthetesterinteractingwith
thesoftware.Thismakesitpossibleforexploratorytesterstoformulatealotofhypotheses,testthem

andfindalotofbugsinaveryshortperiodoftime.Thetestertakesnotesastheygofocussedprimarily
onthefollowingoutputs:
1. Whatfunctionalitytheyhavetestedandtheirgeneralimpressionsofwhattheyhaveseen.
2. Whatbugstheyfoundandwhattheyhaddonetocausethem.
3. Whattestconditionstheyhadconceivedthattheywerenotabletogetto.Thesemaybe
usedasthecharterforasubsequentexploratorytestsession.
4. Howmuchtimewasspentactuallytestingversushowmuchwasspentgettingready.This
informationisusefulwhendecidingwhatkindofpowertoolswouldmaketheexploratory
testermoreefficientinthefuture.

Despiteitssomewhatchaoticappearanceexploratorytestingcanbequitedisciplinedandmethodical
eventhoughitisnotveryrepeatable.Exploratorytestingcanrangefromcompletelyunstructuredto
highlydisciplined.Themoredisciplinedformsofexploratorytestinguseasequenceoftimeboxedtest
sessionstostructurethetestingactivities.
Planningofexploratorytestingconsistsofdefininganinitiallistofhighleveltestconditions(asinkinds
ofthingsweshouldtest)foruseastestsessionchartersanddecidinghowmanytestsessionstobudget
forexecutingthecharters.Examplesofchartersmightincludethefollowing:
Pretendthatyouareahackerandtrybreakingintothesystem(apersonabasedcharter)
Tryoutvariationsoftheinvoicingworkflowfocussingonrejecteditems(ascenariobased
charter)
Tryusingtheuserinterfaceusingonlythekeyboard(adevicebasedorpersonabasedcharter).
Tryscenarioswhereseveraluserstryaccessingthesameaccountatthesametime.(ascenario
basedcharteroftencalledtugofwar.)
Thetestchartersareprioritizedbeforebeingscheduledviaassignmenttoatesterexecutingaspecific
testsession.Uponcompletionofthetestsession,thetestermayrecommendadditionalchartersbe
addedtothebacklogofchartersbasedonwhattheyhadlearnedaboutthesystemundertest,business
domain,usersneedsetc.Exploratorytestingisoftendoneinaniterativestylewithmanyofthetest
chartersforlateriterationsbeingdiscoveredduringexecutionofthetestsessionsinearlieriterations.
Thisallowsexploratorytestingtofocusontheareasofthesoftwarethathavebeenfoundtobethe
mostsuspiciousratherthanprovidingthesameamountofeffortforallareasregardlessofthequality
levelactuallyobserved.This,combinedwiththelowoverheadnatureofthesimultaneoustestdesign
andexecution,iswhatallowsexploratorytestingtobesuchaneffectivewayoffindingthebugswe
knowmustlurkinthesoftware.
18.2.2 ASpreadOutTestLifecycleScriptedTesting
Inscriptedtesting,theaveragetestlifecycleismuchlongerthatinexploratorytesting.Thetestsmaybe
conceivedaspartofthetestplanningexerciseorinmoredetailedtestdesignactivities.Theactualtests

arethendocumentedorprogrammed,andpotentiallyreviewed,oftenbeforethesoftwareisavailable
fortesting.Thetestsmayrequiremaintenanceevenbeforetheyarefirstexecutedagainstsystem
undertestifthedesignofthesoftwarehasevolvedsincethetestsweredesigned.Eventually,we
determinethescheduleforexecutingthetestsandthetestsareexecutedattheappropriatetime
(whichmaybeweeksorevenmonthslater.)Anybugswefindareloggedandthetestresultsare
reportedtothestakeholders.Thebugsareactioned,oftenweeksorevenmonthsaftertheywere
found.Ifthetestsaretoberepeatedatalatertime,usuallyagainstasubsequentversionofthesystem
undertest,thetestsmayrequiremaintenancetotrackchangesinthesystemtheytest.Eventually,
someonedecidesthatthisparticulartestisnolongeraddinganyvalueandthetestisabandoned.
Thiscyclecouldtakeanywhereforseveraldaysorweektomanyyears.ThetestteamforMicrosoft
Officepreparesextensiveautomatedtestscriptstoverifythebehavioroffeaturesinapplicationslike
MSWord.Thetestsforaspecificgenerationoftheproduct(e.g.Word2003)havealifetimeofover10
yearsbecauseofMicrosoftscommitmentof5yearsofmainstreamsupportandafurther5yearsof
limitedextendedsupportforeachbusinessanddevelopmentsoftwareproduct,followedbyaminimum
of1yearofselfhelponlinesupportviatheknowledgebase[Ref
http://support.microsoft.com/lifecycle/].Newbuildsaretypicallycreatedeveryweekthroughthis
productlifetimeandthetestsarerunagainsteachnewbuild.Inthisexamplethemaintenancephaseof
thetestdominatestheindividualtestlifecycle.Microsoftpatternsandpracticesteamusecontinuous
integrationthatpotentiallyproducesseveralbuildsadaywithcontinuousautomatedtestexecution.
18.2.3 IntermediateTestLifecyclesHybridTestApproaches
Thetwoprevioussectionsdescribedthetwoextremesoftestlifecycleduration.Inpractice,thetest
lifecyclescanfitanywherebetweenthesetwoextremes.Therecouldalsobeamixoftestlifecycle
durationseveninthesametestingsession.Forexample,amanualtestercouldbefollowingadetailed
testscriptthatwaswrittenmonthsago.Theynoticesomethingoddthatisntspecificallyrelatedtothe
scriptanddecidetogooffscripttoexploretheoddity.Inthisoffscriptexcursiontheyaredoing
simultaneoustestdesignandexecution(inotherwords,exploratorytesting).Atsomepointtheymay
returntotheoriginalscriptaftereitherconfirmingthatthesystemisworkingproperlyorloggingthe
bugsthattheyhavefound.
18.3 PracticesforAssessingSoftware
InChapter16PlanningforAcceptanceweintroducedmanypracticesinthecontextofplanningthe
readinessassessmentandacceptancetestingactivities.Nowitistimetolookatthepracticesthatwe
usewhiledesigning,executingandactioningtheindividualtests.Atthispointwefocusonthepractices
andnotonwhodoesthem;itreallydoesntmatterwhethertheyaredoneaspartofreadiness
assessmentoracceptancetestingasthepracticesthemselvesarenotchangedbywhenandwhodoes
them.VolumeIIcontainsacollectionofthumbnailsandjobaidesforeachpracticedescribedherewith
VolumeIIIpresentingsampleartefactsofapplyingthosepracticesinaproject.

18.3.1 TestConceptionPractices
Therearequiteafewpracticesforconceivingtestsortestconditions.Somearemorestructuredor
formalthanothers.Theyallsharethegoalofcreatinganextensivetodolistforoursubsequenttest
designandexecutionefforts.Moststartwitheitherrequirements,whetherfunctionalornonfunctional,
orrisks(concernsaboutsomethingthatmightgowrong.)Someofthemorecommontechniques
includethefollowing:
Riskbasedtestidentification
Threatmodeling
Heuristicsorchecklists
UsecasebasedtestingDefinetestsbasedonspecificusecasesofthesystemundertest(see
theFunctionalTestingthumbnail.)
BusinessruletestingDefinetestsforvariouscombinationsofvaluesusedasinputstobusiness
rulesorbusinessalgorithms.
InterfacebasedtestingDefinetestsbasedonthecharacteristicsoftheuserinterface(human
computerinterface)orcomputercomputerinterfaceprotocol.
Scenariobasedtestingusingrealworldusagescenariostoinspirethedesignoftestcases.
SoapoperatestingUsingexaggeratedrealworldusagescenariostoinspirethedesignoftest
cases.
ModelbasedtestgenerationBuildingoneormoremodelsofkeycharacteristicsofthe
systemundertestandgeneratingtestsfromthemodel(s).
GroupBrainstorming.
Paired/collaborativetestingWorkingtogethertodesignbettertestcases.
RiskBasedTestIdentification
Inriskbasedtestidentificationwedoriskmodelingtoidentifyareasrelatingtofunctionalornon
functionalaspectsthatweareconcernedmightnotbeimplementedcorrectlyormighthavebeen
adverselyaffectedbychangestothefunctionality.Weusethisinformationtoidentifytestconditions
wewanttoensureareverifiedbytests.Agoodexampleofakindoftestthatmightbeidentified
throughriskanalysisisthefaultinsertiontest.Forexample,theriskweidentifiedwasThenetwork
connectionfails.WeaskourselvesWhywouldthenetworkconnectionfail?andcomeupwith3
differentpossiblecauses:Unpluggedcable,networkcardfailure,networkcardoutofservicedueto
maintenanceactivityinprogress.Thesearethreetestconditionswewouldwanttoexerciseagainstour
systemundertest.Otherformsofriskmightrelatetopotentialmistakesduringsoftwaredevelopment.
E.g.Thisshippingchargealgorithmisverycomplex.Thismightcauseustodefinealargenumberof
testconditionstoverifyvariousaspectsofthealgorithmbasedonthekindsofmistakesadeveloper
wouldbelikelytomake.Forexampleapplyingthevarioussurchargesinthewrongorder.

ThreatModeling
Anotherformofriskmodelingisthreatmodelingforsecurity.Thepotentialthreatsidentifiedbythe
threatmodelcanleadustochoosefromasetofsecurityassurancepractices.Wemightdefinespecific
penetrationtestscenariostoensurethesoftwarerepelsattemptsatpenetration.Wemightconduct
securityreviewsofthecodebasetoensuresafecodingpracticeshavebeenfollowed.Wecandecideto
doFuzzTestingtoverifythatthesoftwarecannotbecompromisedbyinjectingspeciallychosendata
valuesviauserinputfields.
UseCaseBasedTestIdentification
Whenwehavebusinessrequirementsdefinedintheformofusecaseswecanidentifytestconditions
byenumeratingallthepossiblepathsthroughtheusecaseanddeterminingforeachpathwhatinput
value(s)wouldcausethatpathtobeexecuted.Eachpathconstitutesatleastonetestcondition
dependingonhowmanydistinctcombinationsofinputsshouldcausethatpathtobeexecuted.
InterfaceBasedTestIdentification
Anothersourceoftestconditionsisthedesignoftheinterfacethoughwhichtheusecaseisexercised
whetheritbeauserinterfaceusedbyahumanoranapplicationprogramminginterface(API)or
messagingprotocol(suchasawebservice)usedbyacomputer.Theinterfacemayhaveverydetailed
designintricaciesinadditiontotheelementsrequiredtoexercisetheusecase.Theseintricaciesarea
richsourceoftestconditions.Forexample,auserinterfacemayhavepulldownlistsforsomeinput
fields.Testconditionsforthesepulldownlistswouldincludecaseswheretherearenovalidentries
(emptylist),asinglevalidentry(listof1item)andmanyvalidentries(longlistsofitems.)Eachofthese
testconditionswarrantsverification.
BusinessRulesandAlgorithms
Businessrulesandbusinessalgorithmsareanotherrichsourcefortestconditions.Forarulethat
validatesausersinputsweshouldidentifyatestconditionforeachkindofinputthatshouldbe
rejected.Rulesthatdescribehowthesystemmakesdecisionsaboutwhattodoshouldresultinatleast
onetestconditionforeachpossibleoutcome.Rulesthatdescribehowcalculationsshouldbedone
shouldresultinatleastonetestconditionforeachformofcalculation.Forexample,whencalculatinga
graduatedshippingchargewiththreedifferentresultsbasedonthevalueoftheshipment,wewould
identifyatleastonetestconditionforeachofthegraduatedvalues.
ScenarioBasedTestIdentification
Scenariobasedtestingistheuseofreallifescenariostoderivetests.Therearevariouskindsof
scenarios.Thescenariobasedtestingthumbnaildescribesawiderangeofscenariotypesthis
introductiontouchesononlyafewofthem.Acommontypeofscenariobasedtestisthebusiness
workflowtest.Thesetestsexercisethesystemundertestbyidentifyingcommonandnotsocommon
endtoendsequencesofactionsbythevarioususersofthesystemasaparticularworkitemispassed
frompersontopersonasitprogressestowardssuccessfulcompletionorrejection.Wecanexaminethe

workflowdefinitionslookingforpointsintheworkflowwheredecisionsaremade,anddefinesufficient
workflowscenariostoensurethateachpathoutofeachdecisioniscovered.
Aparticularlyinterestingformofscenariotestisthesoapoperatestinwhichthetesterdreamsupa
particularlytorturousscenariothattakesthesystemundertestfromextremesituationtoextreme
situation.Thenamecomesfromitssimilaritytoasoapoperatelevisionprogramwhichcondensesmany
days,monthsandpotentiallyyearsofextraordinaryeventsinpeopleslivesintoshortmelodramatic
episodes.Thisformoftestidentificationisgoodforthinkingoutsidethebox.
Scenariosabouthowsoftwareisinstalledbyapurchasercouldleadtoidentificationofpotential
compatibilityissuesandtheneedforcompatibilitytesting.
ModelBasedTestGeneration
Inmodelbasedtestgenerationwebuildadomainorenvironmentalmodelofthesystemundertests
desiredbehaviour(expressedinmathematicaltermsorinsomeotherabstractnotation)anduseitto
generatealltherelevanttestcases.Forexample,whentestingafunctionthattakes4parameterseach
with3possiblevalues,wecouldgenerate81testconditions(3*3*3*3)byiteratingthrougheachvalue
foreachparameter.Forlargeandcomplexsystems,thenumberofsuchcaseswillbehuge.Various
guidingorselectiontechniquesareusedtoreducethetestcasespace.Tofullydefinetheexpected
result,wewouldneedtohaveanindependentwaytodeterminetheexpectedreturnvalue,perhapsa
ComparableSystemTestOracleoraHeuristicTestOracle.Generationofthetestsfromthemodelmay
befullyautomatedormanual.
IdentifyingNonfunctionalTests
Thetestcasesusedtoassesscompliancetothenonfunctionalrequirementscanbeidentifiedinmuch
thesamewaysasthoseforfunctionalityrequirementswiththemaindifferencebeinghowthe
requirementsarediscoveredandenumerated.
OtherTestIdentificationPractices
Allofthesepracticescanbeappliedbyasinglepersonworkingaloneattheirdesk.Butasingleperson
canbebiasedorblindtocertainkindsoftestconditions;therefore,itisbeneficialtoinvolveseveral
peopleinanytestidentificationactivities.Onewaytodothisistoreviewthetestconditionswithat
leastoneotherperson,aformoftestreview.Anotherpracticeispairedtestinginwhichtwopeople
worktogethertoidentifythetestconditions(ortowriteorexecutetests.)Wecanalsousegroup
techniqueslikelisting,brainstormingorcardstormingtoinvolvelargergroupsofpeople
[TabakaOnCollaboration].
Allthesepracticescanbenefitfromthejudicioususeofchecklistsandheuristics.Thesecanbeusedto
triggerthoughtprocessesthatcanidentifyadditionaltestsorentirecategoriesoftests.Theyarealso
usefulwhendoingexploratorytesting.

18.3.2 TestAuthoring/TestDesignPractices
Nowthatwehaveoflistofthingswewanttotest,ourtodolist,wecangetdowntodesigningthetest
cases.Akeytestdesigndecisioniswhetherwewillprepareaseparatetestcaseforeachtestcondition
oraddressmanytestconditionsinasingletestcase.Thereisnosinglebestwayasitdependsverymuch
onhowthetestswillbeexecuted.Whenhumantesterswillbeexecutingthetestcasesitmakesalotof
sensetoavoidexcessivetestenvironmentsetupoverheadbytestingmanytestconditionsinasingle
testcase.Thehumantesterhastheintelligencetoanalysetheimpactofafailedteststepanddecide
whethertocontinueexecutingthetestscript,abandontestexecutionortoworkaroundthefailedstep.
Automatedtestsarerarelythisintelligentthereforeitisadvisabletotestfewertestconditionspertest
case.Inthemostextremecase,typicalwhenautomatingunittests,eachtestcaseincludesasingletest
condition.
TestsasAssets
Testsareassets(notliabilities)thatneedtobeprotectedfromlossorcorruption.Theyshouldbe
managedwiththesamelevelofcareanddisciplinethatisusedformanagingtheproductcodebase.
Thismeanstheyshouldbestoredinaversioncontrolledrepositorysuchasasourcecodemanagement
(SCM)system.Thelineupofteststhatcorrespondtoaparticularlineupofproductcodeneedstobe
labelledortaggedinthesamewaytheproductcodeislabelledsothattheteststhecorrespondtoa
specificproductcodebuildcanberetrievedeasily.
TestsasDocumentation
Regardlessofwhetheratestcasewillbeexecutedbyapersonoracomputer,thetestcaseshouldbe
writteninsuchawaythatapersoncanunderstanditeasily.Thisbecomescriticalforautomatedtests
whenthetestsneedtobemaintainedeitherbecausetheyhavefailedorbecausewearechangingthe
expectedbehaviorofthesystemundertestandweneedtomodifyallthetestsforthechanged
functionality.
Manyofthepracticesusedforidentifyingtestconditionscarrythroughtotestcasedesignbutthe
emphasischangestoenumeratingthestepsofthetestcaseanddeterminingwhattheexpected
outcomeshouldbeforeachtest.Forusecasetestswemustenumeratetheuseractionsthatcausethe
particularpathtobeexercised.Wealsoneedtoincludestepstoverifyanyobservableoutcomesaswe
executethestepsandawayofassessingwhethertheoutcomematchesourexpectations.(Seethe
section17.3.5onResultAssessmentPractices.)
PickinganAppropriateLevelofDetailforTestScripts
Aswedefinethestepsofourtestsitisveryimportanttoensurethatthelevelofdetailisappropriate
forthekindoftestwearewriting.Forexample,inabusinessworkflowtest,eachstepofthetestcase
shouldcorrespondtoanentireusecase.Ifweweretousethesametestvocabularyandlevelofdetail
asinausecasetest,ourworkflowtestswouldbecomeexceedinglylongandreadersofthetestwould
haveahardtimeunderstandingthetest.Thisisaclassicexampleofnotbeingabletoseetheforestfor
thetrees.Soapoperatestsarewrittenmuchlikeaworkflowtestexceptthatthestepsand
circumstancearemoreextreme.Again,eachstepinthetestshouldcorrespondtoanentireusecase.

BusinessRuleTesting
Businessrulestestscanbedesignedmuchlikeusecasetestsbyinteractingwiththesystemundertest
viatheuserinterface.Iftherearealotoftestconditions,itmaymaketestingproceedmuchmore
quicklyifweuseadatadriventestapproachwhereinweenumerateeachtestconditionasarowina
tablewhereeachcolumnrepresentsoneoftheinputsoroutputs.Thenwecansimplywritea
parameterizedtestcasethatreadstherowsfromthetableoneatatimeandexercisesthesystem
undertestwiththosevalues.Anevenmoreeffectiveapproachrequiresmorecooperationwiththe
productdevelopmentteambecauseitinvolvesinterfacingthetestcasethatreadstherowsofthetable
directlytotheinternalcomponentthatimplementsthebusinessrule(wecallthemsubcutaneous
tests).Thisapproachresultsinautomatedteststhatexecutemuchfasterthanteststhatexercisethe
softwarethroughtheuserinterface.Thetestscanusuallyberunmuchearlierinthedesignphaseofthe
projectbecausetheydontevenrequiretheuserinterfacetoexist.Thesetestsaremuchmorerobust
becausechangestotheuserinterfacedontaffectthem.Thesetestsareparticularlywellsuitedtotest
drivendevelopment.
ModelBasedTesting
Amoresophisticatedwayofusingmodelsistogenerateexecutabletestcasesthatincludeinputvalues,
sequencingofcallsandoracleinformationtochecktheresults.Inordertodothat,themodelmust
describetheexpectedbehaviourofthesystemundertest.Modelbuildingiscomplexbutisthekey.
Oncethemodelisbuilt,atool(basedonsomemethodornotation)istypicallyusedtogenerateabstract
testcases,whichlatergettransformedintoexecutabletestscripts.Manysuchtoolsallowthetesterto
guidethetestgenerationprocesstocontrolthenumberoftestcasesproducedortofocusonspecific
areasofthemodel.
UsabilityTesting
Thedesignofusabilitytestingrequiresanunderstandingofthegoalsofuserswhowillbeusingthe
systemundertestaswellasthegoalsoftheusabilitytestingitselfandthepracticestobeused.The
goalsoftestingwillchangefromtestsessiontotestsessionandthepracticeswillevolveastheproject
progresses.Earlyroundsofusabilitytestingmaybefocusedongettingtheoveralldesignrightandwill
involvepaperprototypes,storyboards,andWizardofOztesting.Laterroundsoftestingaremore
likelytoinvolvetestingrealsoftwarewiththepurposebeingtofinetunethedetailsoftheuser
interfaceanduserinteraction.Allofthesetests,however,shouldbebasedontheusagemodelsdefined
aspartofUserModelingandProductDesignpractices.
OperationalTesting
Functionalrequirementstendtofocusontheneedsoftheendusersbutthereareotherstakeholders
whohaverequirementsforthesystem.Theneedsoftheoperationsdepartment,thepeoplewho

supportthesoftwareasitisbeingused,needtobeverifiedaspartoftheacceptanceprocess.The
specificneedsmayvaryfromcasetocasebutcommonformsofoperationalacceptancetestinginclude:
Testingofinstallers,uninstallersandsoftwareupdates.
Testingofbatchjobsforinitialdataloading
Testingofdatareformattingforupdatedsoftware.
Testingofdatarepairfunctionality.
Testingofstartupscriptsandshutdownscripts.
Testingofintegrationwithsystemmonitoringframeworks.
Testingofadministrativefunctionalitysuchasusermanagement.
Testingdocumentationcontent.
ReducingtheNumberofTestsNeeded
Ifwehavetoomanytestconditionstobeabletesteachone,wecanusevariousreductiontechniques
toreducethenumberoftestconditionswemustverify.Whenwehavealargesetofpossibleinputsto
verify,wecanreducethenumberoftestcasesweneedtoexecutebygroupingtheinputvaluesinto
equivalenceclassesbasedontheexpectedbehaviorofthesystemundertest.Thatis,anequivalence
classincludesalltheinputsforwhichthesystemundertestshouldexhibitthesameorequivalent
behavior(includingbothendstateandoutputs.)Wethenselectafewrepresentativeinputvaluesfrom
eachequivalenceclassforuseinourtests.Whenthevaluesarenumericandordered(e.g.,integersor
reals)wepickthevaluesrightattheboundariesbetweenthedifferentbehaviors,atechniqueknownas
boundaryvaluesanalysis(BVA).Whentheyarenominal,i.e.,theyrepresentclassificationofbehaviours
withnonaturalordering(e.g.,afinitesetofartbitrarystringsorenumerationtypeswithnomeaningful
orderinginthecontext),wecandesignasingledatadriventestforeachequivalenceclassandrunthe
testforeachinputvalueintheclass.
Ifwehaveseveralinputsthatcaneachvaryandwesuspectthatthebehaviorofthesystembasedon
theindividualinputsisnotindependent,weavoidtestingallcombinationsofinputvaluesbyusing
combinatorialtestoptimizationtoreducethenumberofdistinctcombinationswetest.Examples
include:
Manyindependentvariationsorexceptionpathsinausecase.
Manydifferentpathsthroughastatemodel.
Algorithmsthattakemanyindependentinputvaluesthateachaffecttheexpectedoutcome.
Manysystemconfigurationsthatshouldallbehavethesameway.
ThemostcommonvariationofcombinatorialtestoptimizationisknownasPairwiseorAllPairstesting;
itinvolvespickingthesmallestsetofvaluesthatensurethateveryinputvalueispairedwitheveryother
valueatleastonce.Thistechniqueisusedsofrequentlythatthereisa
website[PairWiseOnAllPairsTesting]dedicatedtolistingthemanyopensourceandcommercial

programsthatexisttohelpuspickthevaluestouse.Ithasbeenshownthatpairwisetestingprovides
bettercoverageandresultsinfewerteststhanrandomtesting.
18.3.3 TestExecutionPractices
Thedetailsofhowthetestsareexecutedvarygreatlyacrossthedifferentkindsoftestsbutseveral
thingsarecommon.Akeyoutcomeoftestexecutionisthedatathatwillbeusedasinputtothe
subsequentreadinessandacceptancedecisions.Thoughvariousprojectcontextswillrequiremuch
moreextensiverecordkeepingthanothers,itisreasonabletoexpectaminimumlevelofrecord
keeping.Thisrecordkeepingconsistsofthefollowingkeypiecesofinformation:
Whattestshavebeenrun,bywhom,whenandwhere?
Whatresultswereobserved?
Howdidtheycomparetowhatwasexpected?
Whatbugshavebeenfoundandlogged?
ThecomparisonwithexpectationsisdescribedinthesectionResultAssessmentPractices.
FunctionalTestExecutionPractices
Thenatureofatestdetermineshowweexecuteit.Dynamictestsinvolverunningthesystemundertest
whilestatictestsinvolveinspectingvariousartefactsthatdescribethesystem.Dynamicteststypically
fallintooneofthefollowingcategories:
AutomatedFunctionalTestExecutionThisinvolvesusingcomputerprogramstorunthetests
withoutanyhumanintervention.Thetestautomationtoolsetsupthetestenvironment,runs
thetests,assessestheresultsandreportsthem.Itmayevenincludeloggingofanybugs
detected.Thetestsmaybestartedbyahumanorbyanautomatedtestschedulerorstarted
automaticallywhencertainconditionsaresatisfiedsuchaschangestothecodebase.
ManualtestexecutionThisinvolvesapersonexecutingthetest.Thepersonmaydoallsteps
manuallyormayusesomeautomationaspowertoolstomakethetestinggofaster.Thehuman
mayadjustthenatureofthetestastheyexecuteitbasedonobservedbehaviorortheymay
executethestepsofatestcaseexactlyasdescribed.
ExploratorytestexecutionThisisaformofmanualtestexecutionthatgivesthetestermuch
morediscretionregardingexactlywhatstepstocarryoutwhiletesting.Eachtestsessionis
usuallyscopedusingatestcharter.
NonfunctionalTestExecutionPractices
Nonfunctionaltestsaresomewhatdifferentfromfunctionaltestsinseveralways.Asintimatedbytheir
name,parafuncationaltestsspanspecificfunctionsofthesystem.Staticnonfunctionaltestsmay
involverunningtoolsthatanalysethesoftwareinquestionortheymanyinvolvereviewerswholookat

thecodeorotherartefactstofindpotentialdesignorcodingdefects.Dynamicnonfunctionaltestsmay
involverunningtoolsthatinteractwiththesystemundertesttodetermineitsbehaviorinvarious
circumstances.
StaticNonfunctionalTestPractices
Staticanalysisisdonebyexaminingthecodeorhigherlevelmodelsofthesystemtounderstandcertain
characteristics.Specificformsofstaticanalysisincludethefollowing:
ADesign/ArchitectureReviewisusedtoexaminehigherlevelmodelsofthesystemtounderstand
certaincharacteristics.Themostcommoncharacteristicsofinterestincludecapacity/scalability,
responsetimeandcompliancewithstandards(internalandexternal.)
ASecurityReviewisaspecializedformofDesignorArchiecturereviewthatinvolvesexaminingthe
architectureandthecodelookingforwaysamalicioususerorprogrammightbeabletobreakintothe
system.
StaticCodeAnalysisinvolvesexaminingthesourcecodeeithermanuallyorusingtoolstoascertain
certaincharacteristicsincluding:
Reachabilityofcodesegments
Correctusageofkeylanguageconstructssuchastypesafety
DynamicNonfunctionalTestPractices
Performanceandstresstestsaregoodexamplesofnonfunctionalteststhatrequirespecializedtools.
Sometimeswehavecomplexorlongrunningtestproceduresthatexercisethesystemundertestjustto
seewhatwillhappen;thereneednotbeenumeratedexpectationsperse.
18.3.4 ResultAssessmentPractices
Thevalueinexecutingatestcaseistodeterminewhetherornotsomerequirementhasbeensatisfied.
Whileasingletestcannotprovetherequirementhasbeenmet,asinglefailingtestcancertainlyprove
thatithasnotbeenmetcompletely.Therefore,mosttestcasesincludesomeformofassessmentor
checkingofactualresultsagainstwhatweexpect.Thisassessmentcanhappeninrealtimeasthetestis
executedoritmaybedoneafterthefact.Thischoiceisapurelypragmaticdecisionbasedonthe
relativeeaseofoneapproachversustheother.Therearethefollowingthreebasicapproachesto
verifyingwhethertheactualresultisacceptable.
Comparetheactualresultwithapredeterminedexpectedresultusingacomparatorwhichmay
eitherbeapersonoracomputerizedalgorithm.
Examinetheactualresultforcertainpropertiesthatavalidresultmustexhibit.Thisisdone
usingaverifier.
Justrunthetestsandnotchecktheresultsatall.Thismaybeappropriatewhenthetestingis
beingconductedexpresslytogatherdata.Forexample,thepurposeofusabilitytestingisto
findoutwhatkindsofissuespotentialusershaveusingtheproduct.Wewouldntreportan

individualusabilitytestsessionashavingfailedorsucceeded.Rather,weaggregatethefindings
ofalltheusabilitytestsessionsforaspecificpieceoffunctionalitytodeterminewhetherthe
designofthesystemundertestneedstobechangedtoimproveusability.
UsingComparatorstoDetermineTestResults
Themostconvenientformofassessmentiswhenwecanpredictwhattheactualresultsshouldbeina
highlydeterministicfashion.Themechanismthatgeneratestheexpectedresultissometimescalleda
trueoracleandthereareseveralwaysthetestresultscanbegenerated.Teststhathaveatrueoracle
areveryusefulwhendoingAcceptanceTestDrivenDevelopmentbecausethereisacleardefinitionof
whatdonelookslike.
Whenthereisanexistingsystemwithsimilarfunctionalityandweexpectthenewsystemtoproduce
exactlythesameresults,itmaybeconvenienttousetheexistingsystemasaComparableSystemTest
Oracle.
Whenwehaveanewsystemforwhichnocomparablesystemexists,weoftenhavetodefinethe
expectedresultsmanually.ThisisknownasaHandcraftedTestOracle.Oncethesystemisupand
runningwemayalsohavetheoptionofcomparingsubsequentreleasesofthesoftwarewithprevious
versions,anapproachwecallPreviousResultTestOracle.Thisistheapproachthatmostrecordand
playbackorcapture/replaytesttoolsuse.InsomecasesitmaybeappropriatetoforgotheHand
CraftedTestOracleandwaitforthesystemtogenerateresultswhicharetheninspectedbyaperson,a
HumanTestOracle,beforebeingusedasaPreviousResultTestOracle.Thisapproachforgoesthe
benefitsofAcceptanceTestDrivenDevelopment.
UsingVerifierstoDetermineTestResults
Whenwecannotpredetermineexactlywhattheexpectedresultshouldbe,wecaninsteadexaminethe
actualresultforcertaincharacteristics.Ratherthanhaveanoracledescribewhattheresultshouldbe,
weasktheoracletomakeajudgementastowhethertheresultisreasonablegiventheinput(s).This
approachhastheadvantageofnotrequiringthatwepredeterminetheexpectedresultforeach
potentialinput.Themaindisadvantageisthatwemayacceptsomeresultsthatsatisfytheinvariantbut
whicharenotactuallycorrect.
Forexample,aHumanTestOraclecouldexamineageneratedgraphictodeterminewhethertheycan
recognizeitasanacceptablerenderingofanunderlyingmodel.Oracomputeralgorithmcouldverify
thatwhentheactualresultisfedintoanotheralgorithmtheoriginalinputvalueisrecovered.The
programthatimplementsthisalgorithmissometimescalledaHeuristicTestOracle.HeuristicTest
Oraclesmaybeabletoverifysomeresultsareexactlycorrectwhileforotherresultsitmayonlybeable
toverifytheyareapproximatelycorrect.
Forexample,wecouldwriteatestscriptthatstepsthroughallintegerstoverifythatthesquareroot
functionreturnstherightvalue.Ratherthaninspecttheactualvaluesreturnedbyeachfunctioncalland
comparethemtoahandcraftedtestoracleoracomparablesystemoracle,wecouldinsteadmultiply
theresultbyitselfandverifythatwegetbacktheoriginalnumberwithinaspecifiedtolerance,say,+/
.001.Inthisexample,weshouldalsotestagainstanotherinvarianttoensurethattheactualresultisnot

negativewhichwouldclearlybeatestfailure.Acomputerizedheuristicoracleissometimescalleda
verifier.
LoggingBugs
Oneofthekeyreasonsfortestingandreviewsistofinddifferencesbetweenexpectationsandreality.
Whenwedofindsuchadifferenceitisimportantthatitbeloggedsothatitcanbefurtherinvestigated,
prioritizedandtheappropriateactiondetermined.Theinvestigationcouldrevealittobeabug,a
misunderstandingbytestersabouthowthesystemwasintendedtobeused,missingdocumentation,or
anyofamyriadofotherreasons.Toavoidpresumingtheoutcome,weprefertocalltheseconcerns.To
allowtheinvestigationtobeconductedefficiently,itisimportanttologthekeycharacteristicsofeach
concern.Ataminimum,weneedtologthefollowing:
1. Theexactstepsrequiredtoreproducetheproblem.Thismayrequirererunningthetesta
numberoftimesuntilwecanconfidentlydescribeexactlywhatittakestocausetheproblem
tooccur.
2. Whatactuallyoccurred.
3. Whatweexpectedtohappen.Weshouldprovideasmuchdetailaswouldbenecessaryfor
thereadertounderstand.Weshouldnotjustrefertoarequirementbutratherdescribe
exactlywhatweexpected,whathappened,andhowwhatactuallyoccurredwasdifferent
fromwhatweexpected.

Everypotentialbugreportshouldbegivenacleartitlethatdescribesspecificallywhatwastested;this
avoidsconfusionbetweenbugswithverysimilartitlesyetcompletelydifferentexpectedandactual
behaviours.
Referto[Kaneretal,Chapter.Reporting and Analyzing Bugs.] on bug reporting guidance and
http://blogs.msdn.com/productfeedback/archive/2004/09/27/235003.aspxforanadditionalexample.
18.3.5 TestMaintenancePractices
Sometestsareonlyintendedtoberunoncewhilesomeareintendedtoberunmanytimesoveralong
periodoftime.Somekindsoftestsholdtheirvaluelongerthanothers;somekindsoftestsdeteriorate
veryquicklybecausetheyaresotightlycoupledtotheSUTthatevensmallchangestotheSUTmake
themobsolete.Teststhatareexpectedtobeusedmorethanoncemaywarrantanupfrontinvestment
toensurethattheyarerepeatableandrobust.
Usefultechniquesformakingtestsmorerobustincludethefollowing:
Buildmaintainabilityintothetests.Writethetestsatappropriatelevelofabstraction.Dont
couplethetesttoanypartofthesystemitisnttesting.Dontprovideanyunnecessaryor
irrelevantdetailinanyofthestepsofthetest.Strivetodescribetheteststepsinbusiness
ratherthantechnicalterms.

Designthesystemundertestfortestability.Designingfortestabilityiscommonpracticein
computerhardwarebutistoooftenneglectedinsoftwaredesign.Designthesystemtomakeit
easytoputitintoaspecificstate.Makeiteasyfortestprogramstointerfacewiththesystem
throughapplicationprogramminginterfaces(API)ratherthanforcingprogramstouse
interfacesintendedforhumans.Writingthetestsbeforethesystemisdesignedisagoodway
toinfluencethedesigntosupporttestability.
Refactortheteststoimprovemaintainability.Testsshouldbeassets,notliabilities.Teststhat
arehardtounderstandarehardtomaintainwhenthesystemundertestismodifiedtomeet
changingrequirements.Automatedtestsinparticularshouldberefactoredtoavoid
unnecessaryduplicationandirrelevantinformation.SeetheTestEvolution,Refactoringand
Maintenancethumbnail.
18.4 Summary
Thischapterintroducedthepracticesweusewhiledefining,executingandmaintainingindividualtests.
Whilesomeofthesepracticesarespecifictofunctionaltestingandothersarespecifictononfunctional
testing,theoveralllifecycleofatestisconsistentforbothcategoriesoftests.Thepracticesusedfor
readinessassessmentaremoreorlessthesameasthoseusedforacceptancetestingalthoughthe
specifictestsproducedforeachwilldependontheoveralltestplanasdescribedinChapter16
PlanningforAcceptance.
18.5 WhatsNext?
Thenextchapterdescribespracticesrelatedtomanagingtheacceptanceprocess,especiallyhowit
relatestomonitoringandreportingontestprogressandtheresultsandmanagingtheprocessof
decidingwhichbugstofix.
18.6 Resources
[PairWiseOnAllPairsTesting]Pairwise.OrgAwebsitededicatedtocatalogingallthetoolsavailablefor
allpairstesting.http://www.pairwise.org/tools.asp
[TabakaOnCollaboration]Tabaka,JeanCollaborationExplainedAddisonWesley.NJAddisonWesley.
NJ[Kaneretal]Kaner,C.etal.TestingComputerSoftware,2/e.Wiley,1999.

Chapter19 ManagingtheAcceptance
Process
Thepreviouschapterintroducedtheindividualtestlifecycleandthepracticestheassessorsusefor
identifyingtestconditions,designingthetests,executingthetestsandassessingtheoutcomes,and
maintainingthetests.Thischapterintroducesthemanagementpracticesweusewhileexecutingthe
testsandwhilemakingtheacceptancedecision.
Theprocessofacceptingsoftwareinvolvesmanyactivitiesthatgeneratethedataweuseasinputinto
theacceptancedecision.Eachoftheseactivitiestakestimeandconsumesvaluablehumanandother
resources.Awellmanagedacceptanceprocesswillusetheseresourceswiselywhileapoormanaged
processhasthepotentialtowastetheseresources,delaytheacceptancedecisionandevencompromise
thequalityofthedecisionmade.
Whileweareexecutingourtestplans,wewanttoknowtheanswerstotwoimportantandinterrelated
questions:
First,howistestingprogressing?Whenwillwehaveahighconfidenceassessmentofthe
productquality?Willitbeintimetomakeourreadinessoracceptancedecision?
Second,whatisourcurrentassessmentoftheproductquality?Isitgoodenoughtorelease
whethertocustomersortotheacceptancetesters?Whatactionsdoweneedtotaketomakeit
goodenough?
19.1 TestSchedulingPractices
Theoverallschedulethatdefineswhichtestswillberunandwhenshouldhavealreadybeendefinedas
partofthetestplan.Thatincludesthestrategicdecisionsaboutwhetheralltestingisdoneduringafinal
testphaseorincrementallythroughouttheproject.Inthissectionwefocusonthetechniquesweuse
forschedulingoftestexecution.Asareminder,undertestexecutionweincludebothdynamictesting
whentestcasesarerunthatinvolveexecutingsoftwareinthesystemundertestandstatictestingthat
isperformedintheformofreviewsorinspectionswithoutactuallyrunningcodeinthesystemunder
test.
19.1.1 PlanDrivenTestScheduling
Thetraditionalapproachtotestschedulinginvolvesdefiningaperiodoftime,sometimescalledatest
cycle,inwhichonecompleteroundoftestingcanbecompleted.Thetestplandefineshowmanytest
cyclesareplanned.Thedetailsofwhathappensineachtestcycleoftenemergeastheprojectunfolds.
Oneapproachistodefineadetailedprojectplanforthefirsttestcycle,onewhichdefinesthesetof
testingactivitiesthatwillbedoneonadaybydaybasis.ProjectmanagementtoolssuchasPertorGantt
chartsmaybeusedtodocumentthisdetailedtestexecutionplan.(SeePlanDrivenTestManagement.)

19.1.2 SessionBasedTestScheduling
Thealternativetodefiningadetailedprojectplanoftestactivitiesistoscheduleaseriesoftestsessions
andcreateabacklog(prioritizedlist)oftestsessionchartersthatareexecutedinthesetestsessions.
Eachsessionwouldbeofastandardduration,say90minutes,andmostchartersshouldbeexecutable
withinonesession.ThisSessionBasedTestManagementismosttypicallyusedformanaging
exploratorytestingbutcouldalsobeusedforexecutionofscriptedtesting.Wheneachtestsessionis
completedthetestermarksthetestcharterasoneofcompletedorneedsoneormoreadditional
sessions,orsuggestsadditionaltestchartersforfuturetestsession.Thisprovidesagoodindicationof
howtestingisprogressingasillustratedinfigureK(TestCharterBurndown)whichshowsthenumberof
testchartersdroppingovertimebutwiththeoccasionalupwardsspikeasnewtestchartersare
identified.

FigureK
TestCharterBurndown
Inthischart,thebarsrepresentingtheoriginalchartersshowthechartersthatweredefinedbeforetest
executionbegan.ThebarslabelledAddedChartersstackedontopofthemrepresentnewchartersthat
wereidentifiedwhiletestexecutionwasoccurring.TheTotalLeftlineindicateshowmanytestcharters
remaintobefullfiledwhiletheChartersCompletedindicatesthenumberofchartersalreadyfulfilled.
Wecangetagoodideaatanypointduringtestexecutionofwhentestingwillbecompletedby
projectingtheTotalLeftlineouttotherightuntilitintersectszerocharters.Thatistheearliestprobably
completiondate.

19.1.3 SelfOrganizedTestScheduling
Athirdapproachisoftenusedbyagileorselforganizingteams.Allthetestingtasksarepostedonawall
chart,variouslycalledabigvisiblechart[JeffriesOnBigVisibleCharts]oraninformationradiator
[CokburnOnAgileDevelopment],andteammemberssignuptodospecifictestingtasks.Astheyfinish
onetask,theymarkitdoneandpickanothertasktoperform.ThisiscalledSelfOrganizedTest
Scheduling.
19.1.4 EventTriggeredTestScheduling
Someformsofautomatedtestingcanbesetuptorunautomaticallyatcertaintimes(suchasevery
nightat2am)orwhencertaineventsoccur(suchaswhensomeonechecksinachangetopartofthe
codebase.)Theresultscanbepostedonawebsite,automaticallyemailed/IMdtoallteammembers,
orcommunicatedbyamulticoloredlightdisplayorlavalampsintheteamworkspaceorbyaniconin
everyonescomputerssystemtrayorsidegadget(foronesuchtoolseeTeamBuildMonitor
[LambOnTeamBuildMonitor]).ThispracticeisoftencalledContinuousIntegration
[MartinFowlerOnContinuousIntegration]orCIforshort.Teamsthatusecontinuousintegrationtypically
adoptastopthelinementalitytofailingtests.Thegoalistopromotecodehealthearly.Whenevera
testfailureisreportedbytheautomatedtestharness,fixingthefailedtestbecomesthetoppriorityfor
everyoneontheteam.Thisisthesoftwareequivalentofthestopthelinepracticeusedinlean
manufacturing.Oncethefailureisfixed,everyonecangobacktoworkingontheirrespectivetasks.This
focusonkeepingtheproductcodehealthyimprovesqualityandandreducesthedurationofthe
acceptancetestcycle.
Anotherapproachistopreventchangesthatbreakthebuildfrombeingcheckedin.TheGatedCheckin
featureoftheTeamFoundationServerdefersadeveloperscheckinuntilitcanbemergedand
validatedbyanautomatedbuild.Passingautomatedtestsorvalidatingresultsofthestaticanalysis
couldareexamplesofthevalidationpolicies.FormoreinformationonTeamFoundationBuild,see
[MSDNOnTeamFoundationBuild].
19.2 TestProgressReportingPractices
Whenwefirststartexecutingthetestswedontknowwhetherthequalityishighorlowbutwedo
knowthatwedonthaveahighdegreeofconfidenceyet.Astestingprogresses,weshouldbegettinga
betterideaofwhatthequalitylevelisandhowmuchlongeritwilltakeustogettotherequiredlevelof
confidence.Thisisverysimilartotheconeofuncertaintyconceptthatpredictsthecompletiondate
and/orcostofdevelopingthesoftware.FigureXillustratestheconeofuncertaintyforqualityfora
typicalproject.Theinitialguestimatewasanywherebetween10and100personmonths.Asthe
requirementswerebetterunderstood,therangereducedsomewhatbutasuddendiscoveryof
additionalscoperaisedtheestimatesonceagain.Aneffortwasmadetodescopetheprojecttorecover
theoriginaltimelines.Workcreepslowlyraisedboththeupperandlowerlimitandtheestimatesfinally
convergewhentheproductisacceptedasis.

140 Initial Scope NeededExtra

Guess Increases Test&FixCycle
120 Descoped
Detailed
Release1
Estimate Acceptance!
100
80
Latest(90%)
60
MostLikely(50%)
Earliest(10%)
40
20 Work
Creep
0

FigureX
ConeofUncertaintyforQuality
Wecan,ofcourse,assumethatwewillhaveareasonablyaccurateassessmentofqualitywhenwehave
finishedallourplannedtesting.Thisassumptionmayormaynotbecorrectbecauseitdependsonhow
effectivelyourplannedtestactivitieswillfindalltheimportantbugs.Thisalsoimpliesaclear
understandingofwhatisimportanttotheproductowner.Thisisverydifficulttoassessaheadoftime.
Wecertainlyneedtomonitorhowmuchtestingworkremainstobeexecutedinthecurrenttestcycle
andwhenweexpecttohaveitallcompleted.Thisappliestoeachtestcycleweexecute.
Sessionbasedtestingintroducesafeedbackmechanismthatexplicitlyallowsustoadjustthetestplans
aswelearnnewinformationbothabouttheproductandabouttheproductownerandtheproduct
ownersneeds.Aseachtestsessioniscompletedthetestersaddanynewlyidentifiedareasofconcern
tothebacklogoftestchartersyettobeexecuted.Developersmayalsosuggestadditionaltestcharters
toaddresstheriskassociatedwiththemodificationstheymaketothesoftwareasaresultofchange
requestsandbugfixesidentifiedbytests.Wemonitorthesizeofthebacklogoftestcharterstogeta
senseofwhetherwearemakingheadway.Astheconfidenceofthetestersanddevelopersimproves
theywillsuggestfewerandfewernewtestchartersandthereforethesizeofthecharterbacklogwould
dropfaster.
Predictingwhenthesoftwarewillbeofgoodenoughqualitytodeliverisdifficultbecausethatinvolves
predictinghowmanytestcycleswewillrequireandhowmuchtimetheproductdevelopmentteamwill
requirebetweenthetestcyclestofixthebugs.(SeeChapter16PlanningforAcceptanceforamore
detaileddiscussiononthistopic.)Thisrequiresmonitoringthebugbacklogandthearrivalrateofnew
bugstoassessandadjustthepredicteddeliverydate.

19.3 AssessingTestEffectiveness
Oneofthechallengesoftestingisassessinghoweffectiveourtestsreallyaresothatwecanknowhow
confidentweshouldbeinourassessmentofthequalitywehave.Thereareseveralwaystocalculate
theeffectivenessofthetestsincludingtheuseofcoveragemetricsandusingthefindrateof
intentionallyseededbugs.
19.3.1 CoverageMetrics
Wecanusemetricsliketestcoverageandcodecoveragetocalculatethetheoreticaleffectivenessof
ourtests.Thesemetricscantelluswhatpercentageoftherequirementshasbeentestedandwhat
percentageofthecodehasbeenexecutedbythetestsbutneitheroftheseisadirectmeasureofwhat
percentageofthebugswehavefound.Ofcourse,theprimaryissueisthatwehavenoideaofhow
manybugsreallyexistsoitisprettydifficulttosaywithanycertaintywhatpercentagewehavefound.
Foracautiousapproachtousingcodecoverage,seeBrianMaricksessay
[MarickOnCodeCoverageMisuse].
19.3.2 DefectSeeding
Onetechniquethecanprovideamoredirectmeasureisdefectseeding.Itinvolvesdeliberatelyplacinga
knownsetofdefectsinthesoftware.Asbugsarefoundduringtestingwecanestimatethepercentage
ofdefectsfoundbydividingthenumberofseededdefectsfoundbythetotalnumberofseededdefects
asshowninFigureY.
SeededDefectsFound
PercentDefectsFound
TotalSeededDefects
FigureY
PercentageofBugsFound
WecanprojectthenumberofdefectsyettobediscoveredusingtheformulainFigureZ.
FoundDefects SeededDefectsFound

TotalDefects TotalSeededDefects
Or,
TotalSeededDefects
TotalDefects FoundDefects
SeededDefectsFound
FigureZ
TotalBugsCalculation
ThesecalculationsaredescribedinmoredetailintheTestStatusReportingthumbnail.
19.4 BugManagementandConcernResolution
Akeyoutputoftestingandreviewsisalistofknowngapsbetweenwhattheproductownerexpectsof
theproductandhowitactuallybehaves.Whiletheprogressoftestingisusuallyreportedintermsof

whichtestshavebeenrunandwhichhavent,thegapbetweenexpectationsandrealityisusually
expressedasalistofknownbugsorissuesthatmayormaynothavetobeaddressedbeforethe
productwillbeaccepted.InChapter16PlanningforAcceptance,weintroducedtheConcern
ResolutionModelwhichdescribeshowconcerns,includingsoftwarebugs,changerequests,issuesand
documentationbugs,canbemanaged.Concernsthatfallintoanyofthesecategoriescouldbe
consideredgating(alsoknownasblockingorblockers),thatis,wouldpreventthesystemfrombeing
accepted.Oncewehavefinishedourfirstfullpassoftesting,presumablyattheendofourfirsttest
cycle,wecanthinkofthesetofgatingbugsasbeingtheoutstandingworklistfortheproduct
developmentteam.Ourgoalistodrivethelistofgatingbugsdowntozerosothatwecanconsider
acceptingandreleasingtheproduct.(Notethatjustbecauseitreacheszerodoesnotimplythatthere
arenogatingbugs,justnonewecurrentlyknowabout.)Theproductownercaninfluencethegating
countintwoways:theproductownercanclassifynewlyfoundbugsasgatingortheycanreclassify
existinggatingbugsasnongatingiftheydecidethatthebugcanbetoleratedbecausethereisalow
enoughlikelihood(reducedprobabilityasdescribedinChapter5RiskModel)itwillbeencounteredor
thereareacceptableworkaroundsintheeventthatitisencountered(reducedimpactpertherisk
model.)
19.4.1 BugTriage
Itiscommonpracticetoclassifytheseverityofbugsbasedontheirbusinessimpact.UsuallyaSeverity1
(orSev1asitiscommonlyabbreviated)bugisacompleteshowstopperwhileaSev5bugismerely
cosmeticandwontimpacttheabilityofuserstousetheproducteffectively.Manyproductowners
insistthatallSev1&2bugsbefixedbeforetheywillaccepttheproduct.Someproductownersconsider
Sev3bugscriticalenoughtoinsisttheyareresolvedbeforetheproductcanbeaccepted.Notethatthe
interpretationoftheseverityscaleismerelyavocabularyforcommunicationoftheimportanceofbugs
betweentheproductownerandtheproductdevelopmentteam;itisentirelyuptotheproductowner
tomakethedecisionwhetherthebugneedstobefixed.Theproductdevelopmentteammayinfluence
thatdecisionbypointingoutsimilaritiesordifferencewithotherbugsthathadthesameseverityrating
orbypointingoutthepotentialimpactaspartofanargumenttoincreaseordecreasetheseverity.They
mightalsopointoutpotentialworkaroundsorpartialfixesthattheyfeelmightjustifyreducingthe
severity.Butultimately,itistheproductownerdecisionsastowhethertheproductisacceptablewith
thebugstillpresent.
Theproductownerneedstobereasonableaboutwhatbugsshouldbeclassifiedasgating.Ifthereare
1,000bugsandtheteamcanfix20bugsperweek,itwilltaketheteam50weekstofixalltheexisting
bugsassumingthatnonewbugsarefoundandnoregressionbugsareintroducedbythebugfixes.Both
theseassumptionsarehighlyoptimistic.Therefore,thecustomerneedstoaskWhichofthesebugs
trulyneedtobefixedbeforeIcanaccepttheproduct?Thisispurelyabusinessdecisionbecauseevery
bughasabusinessimpact.Somemayhaveaninfinitesimallysmallbusinessimpactwhileothersmay
havealargebusinessimpact.Theproductownerneedstobeopinionatedaboutthisandtobeprepared
tolivewiththeconsequencesoftheirdecisionwhetherthatistodelayacceptance,deploymentand
usageoftheproductuntilaparticularbugisfixedortoputthesoftwareintousewiththeknownbug
present.Thereisnopointininsistingthatallbugsmustbefixedbeforeacceptancesimplytobeableto

saythattherearenoknownbugs.Doingsowilllikelydelayaccruinganybenefitsofthesystem
unnecessarily.Theprocessofdecidingtheseverityofeachbugandwhetherornotitshouldeverbe
fixedissometimescalledBugTriage.
19.4.2 BugBacklogAnalysis
Thislistofknownbugsisacollectionofusefulknowledgeaboutthestateoftheproduct.Mostbug
managementtoolsincludeasetofstandardreportsthatcanbeusedtobetterunderstandthebug
backlog.Theseinclude:
BugFixRateReportDescribestherateatwhichbugsarebeingfixed.
BugArrivalRateReportDescribestherateatwhichnewbugsarebeingreported.A
decreasingarrivalratemaybeanindicationthatthereturnoninvestmentoftestinghas
reachedthepointofdiminishingreturns.Oritcouldjustmeanthatlesstestingisbeingdoneor
thatthetestingisrepeatingitself.Longlivedproductsthatreleaseonaregularcycletypically
findthattheshapeofthecurveisfairlyconstantfromreleasetoreleaseandthiscanbeusedto
predictthereadytodeliverdatequiteaccurately.SeeFigureDBugArrivalRateandFigureE
CumulativeBugsFound.
BugDebtorBugBurnDownReportDescribestherateatwhichthenumberofbugsisbeing
reducedorisincreasing.Theburndownreportaggregatesthebugarrivalrateandthebug
fixingratetoallowustopredictwhenthenumberofgatingbugswillbelowenoughtoallow
acceptingtheproductandreleasingittousers.
BugAgingReportClassifiesbugsbyhowlongithastakentofixthemandhowlongithasbeen
sinceunresolvedbugswerefirstreported.Thelatterwillgiveanindicationofpotentiallevelof
customerdissatisfactioniftheaverageageofbugsislarge.
BugCorrelationReportClassifiesbugsbasedontheirrelationshipwithattributesofthe
systemundertest.Theproductowneristypicallymostinterestedinwhichfeatureshavethe
mostbugsbecausethishelpsthemunderstandthebusinessimpactofacceptingthesystem
withoutresolvingthem.Theproductdevelopmentteam,ontheotherhand,istypically
interestedinwhichcomponents,subsystems,ordevelopmentteamshavethemostbugs
associatedwiththembecausethishelpsthemunderstandwheretheirowninternalprocesses
needtobeimprovedmost.FigureZisanexampleofaBugsbyAreaorBugsbyFeatureTeam
report.

80
70 NumberofBugsFoundbyDate
60
50
40
30
20
Num
10
berof
0 Bug

FigureD
BugArrivalRate

CumulativeBugsFoundbyDate
1400
1200
1000
800
Cumulative
600
BugsFoundby
400 Date
200
0
01/03/2008 01/04/2008 01/05/2008 01/06/2008

FigureE
CumulativeBugsFound.

250
200 Usability
Installation
150 Documentation
Localization
100
Globalization
50 Logging/Tracing
Performance
0
Functionality
Security

FigureZ
BugCorrelationReport

FigureW
BugDebt/BurnDownReport

19.5 TestAssetManagementPractices
Theartefactsproducedwhileplanningandexecutingtheassessmentprocessareassets;thesetest
assetsneedtobemanaged.Ifrepeatabilityoftestingisconsideredimportant,suchaswhenthesame
testsneedtobeusedforregressiontestingsubsequentreleases,testscriptsandthecorrespondingtest
datasetsneedtobestoredinaversioncontrolledtestassetrepository.Thismaybeadocument
repositorysuchasSharePointoracoderepositorysuchastheTeamFoundationServerrepository,
SubversionorCVS.ThisisdescribedinmoredetailintheTestAssetManagementthumbnail.
Whentestassetsareexpectedtobelonglivedsuchaswhenthesametestswillbeusedtoregression
testseveralreleasesofaproduct,itisimportanttohaveastrategyfortheevolvingthetestassetsto
ensuremaintainability.TheTestEvolution,RefactoringandMaintenancethumbnaildescribeshowwe
cankeepourtestassetsfromdegradingovertimeastheproducttheyverifyevolves.
19.6 AcceptanceProcessImprovement
Theacceptanceprocessconsumesasignificantportionofaprojectsresources.Therefore,itisagood
placetolookwhentryingtoreducecostsandimprovetheeffectivenessofonesprocesses.Twogood
candidatesforprocessimprovementareimprovingtheeffectivenessofthetestpracticesand
streamliningtheacceptanceprocesstoreducetheelapsedtime.Thelatteristhesubjectofthenext
chapter.
19.6.1 ImprovingTestEffectiveness
Multireleaseprojectsandlonglivedproductswillgothroughtheacceptanceprocessmanytimesin
theirlifetime.Eachreleasecanbenefitfromlessonslearnedinthepreviousrelease,ifwecaretoapply
thelessons.Itisworthconductingoneormoreretrospectivesaftereachreleasetobetterunderstand
whichreadinessassessmentandacceptancetestingactivitieshadthemostimpactonproductquality
andwhichhadtheleast.Thehighlyeffectiveactivitiesshouldbecontinuedinfuturereleasesandthe
ineffectiveonesshouldbereplacedwithsomethingelse.Someteamsmakeitapointtotryatleastone
newpracticeeachreleasetoseeifitwilldetectbugsthatpreviouslyslippedthroughthereadiness
assessmentandacceptancetestingsafetynets.Itisalsoworthanalysingthelistofdefectfoundinthe
usagephasetoidentifyanyshortcomingsinthesafetynet.Bugcorrelationreportscancomeinhandyin
thisexercise.SeeFigureQBugsbyHowFoundReport.

NumberofBugsbyHowFound
350
300
250
200
150
100
50
NumberofBugs
0

FigureQ
BugsbyHowFoundReport.
19.7 Summary
Likeanyprocess,executionoftheacceptanceprocessneedstobemanaged.Thisincludesmonitoring
theexecutionoftheplannedtestingandensuringitresultsindatathatallowsforahighconfidence
acceptancedecision.Akeyaspectofmanagingacceptanceisthemanagingthebugbacklogtogetthe
bestpossiblequalityproductattheearliestpossibletime.Thesetwogoalsareatoddswitheachother
anddecidingwhichtakesprecedenceshouldbeabusinessdecision.Whenbuildingproductisan
ongoinggoaloftheproductownerorganization,continuousimprovementoftheacceptanceprocess
shouldalsobemanagedtofurtherreducetimetousageandimproveproductquality.
19.8 WhatsNext
Theacceptanceprocesstypicallytakesupasignificantportionofaprojectsresourcesandelapsedtime.
Thenextchapterlooksatwaysbothelapsedtimeandresourceusagecanbereduced.
19.9 Resources
[MartinFowlerOnContinuousIntegration]Fowler,M.ContinuousIntegration.
http://martinfowler.com/articles/continuousIntegration.html
[MSDNOnTeamFoundationBuild]http://msdn.microsoft.com/enus/library/bb668958.aspx
[CockburnOnAgileDevelopment]Cockburn,AlistairAgileSoftwareDevelopment:TheCooperative
GameAddisonWesleyProfessional

[JeffiresOnBigVisibleCharts]RonJeffries,BigVisibleCharts,
http://xprogramming.com/xpmag/bigvisiblecharts/
[LambOnTeamBuildMonitor]http://blogs.msdn.com/jimlamb/attachment/3467297.ashx
[MarickOnCodeCoverageMisuse]BrianMarick,HowtoMisuseCodeCoverage
http://www.exampler.com/testingcom/writings/coverage.pdf

Chapter20 FineTuningtheAcceptance
Process
ThepreviouschaptersofPartIIIintroducedthepracticesinvolvedinplanningandexecutingthe
acceptanceprocess.Byreadingthosechaptersthereadershouldbeabletogetabasic
understandingofwhatisinvolvedinacceptingsoftware.Thischapterdescribeshowwecanuse
theinformationweobtainwhileexecutingtheacceptanceprocessascluesforhowtoimprove
theproductdevelopmentprocessesofourorganizationtoreducedefectlevelsandtominimize
theelapsedtimeandresourcesconsumedwhilemakingtheacceptancedecision.
Theacceptanceprocessisanecessarybutpotentiallywastefulexercise.Weshouldstrivetokeepitas
simpleandvalueaddingaspossible.Thelongerittakes,themoreitcostsinwastedeffortanddelayed
deliveryofbusinessvalue.Wecantakestepstostreamlinetheacceptanceprocessbyreorganizinghow
wedoreadinessassessmentandacceptancetesting.Buttheacceptanceprocessisjustonepartofthe
productdevelopmentprocess.Theissueswefindwhileexecutingtheacceptanceprocessare
consequencesofthechoiceswehavemadeinhowwestructureourproducts,markets,andworkflows.
Wecanuserthesesymptomstomotivateabetterunderstandingoftheunderlyingproblemsinhowwe
work.
20.1 DebuggingtheAcceptanceProcess
Issuesencounteredduringtheacceptanceprocesscanbevaluablecluesaboutproblemsinhowour
organizationandworkflowsarestructured.Thesecluesprovidehintsabouthowweshouldrestructure
ourorganizationanditsprocessestoworkmoreefficiently.Thefollowingarealistofcommon
symptomsandpossiblesolutions.
20.1.1 OverlyLongDurationofAcceptanceProcess
Anacceptanceprocessthattakesalongperiodoftimehintsatissueswithhowtheorganizationis
structured.Itcanbecausedby:
Toomanygroupseachneedingtotheirownspecializedworkandtoomanyhandoffsbetween
them.Considerusingvaluestreammappingtoidentifywhichstepsaddrealvalueandwhich
couldbeeliminatedentirely(preferable)ordoneinparallel(secondchoice.)Therootcauseis
likelythewaytheorganizationisstructured;changingthestructuremaybeanecessarythough
highrisk/effortendeaverthoughonewithapotentiallyhugepayback.
Toomanytest&fixcyclesbeingneeded.Thisisdescribedbelow.
20.1.2 TooManyDefectsfoundduringAcceptanceProcess
Iftheacceptanceprocessfindsmanydefects,thisislikelyasignthatTheProductDevelopmentTeams():

Dontknowhowthefinishedproductshouldbehave.
ArenotheldaccountablefordeliveringafinishedproducttotheProductOwneracceptancetesting.
AlackofunderstandingofwhatdonelookslikeisasignthattheProductOwnerhasnotdonean
effectivejobofcommunicatingthenatureoftheproducttotheProductDevelopmentTeam.Thiscould
bebecausetheproductownerdoesntknoweither,oritcouldbeineffectivecommunicationsuchas
occurswhentheprimarymeansofcommunicationiswrittenrequirementspecifications.TheProduct
OwnercanlearnmorequicklywhattheyreallywantbydoingIncrementalAcceptanceTesting.This
givesthePDTmoretimetoaddressanychangesthaniftheacceptancetestingweretobedoneinafinal
AcceptanceTestPhase.Ineithercase,thejointunderstandingoftheacceptancecriteriacanbe
improvedbyincludingacceptancetestsaspartoftherequirementsprocess.Thesetestscanbe
providedbytheProductOwnerordevelopedjointlythroughcollaborationbetweentheProductOwner
andtheProductDevelopmentTeam.
Lackofaccountabilityoccursforseveralreasons.Oneiswhentheproductisdecomposedinto
componentsthatarefarremovedfromtheenduserfunctionality.TheProductDevelopmentTeam
oftendoesntunderstandhowtheircomponentsupportsthebusinessgoalsandthereforebuilds
functionalitythatmayactualbeatcrosspurposestoit.Thisoftenresultsinbugsfoundonlyafterthe
componentsareintegrated.ThisisoneofthedownfallsoftheComponentTeamapproachtoorganizing
thework.
Anotherreasonforlackofaccountabilityiswhenmanagementvaluesscheduleoverquality.Ofcourse
wesayqualityisimportantbutitsmanagementsactionsthatreallycount.Whenmanagement
pressuresdeveloperstomeetunrealistictimelineswearesayingscheduletrumpsquality.Aclear
definitionoftheMinimumQualityRequirement(MQR)iscrucial.WeneedtoencouragetheProduct
DevelopmentTeamtoimproveitsprocessestodeliverbetterqualitybymistakeproofingtheprocesses
asmuchaspossible.TestDrivenDevelomentandautomatedtestsarebothexamplesofhowtodothis.
Oncetheycanbuildsoftwarewithfewerdefects,theywillbeabletodeliverfasteraswell;theinverseis
nottrue.
20.1.3 TooManyTest&FixCyclesNeeded
WhentheproductrequirementshavebeenclearlycommunicatedandtheProductDevelopmentTeam
hasdoneaneffectivereadinessassessment,theacceptancetestingdoesnotfindmanydefects.Those
defectsthatarefoundshouldbefairlyminorandeasilyfixed.Butwhatiftherearealotofdefects?
Thentheproductwillneedtogobackthroughtheconstruction,readinessassessmentandacceptance
testingprocessanothertime.Whenthiscyclehastoberepeatedseveraltimesbeforetheproductis
goodenoughtoconsiderreleasing,itmaybeduetooneofseveralrootcauses:
Newbugsarebeingintroducedbymanyofthefixesfortheexistingbugs.Thiscouldbebecausethe
softwarehasbecomebrittleduetoageandexcessiveinternalcoupling(ormaybeitwasntdesigned
verywellevenwhenitwasnew.)Oritcouldbeduetorushingtheworkandalackofasafetynetto
catchthedefectsbeingintroduced.Thelattercanbeaddressedbyincorporatingautomatedunit
testingandpossiblytestdrivendevelopmenttoavoidinsertingthedefects.

TheproductDevelopmentTeamisnotdoingeffectivereadinessassessmentofeachreleasecandidate.
Thismaybebecausetheyarebeingrushedorbecausetheyarerelyingprimarilyonmanual
regressiontestingandcannothopetoretestalltheaffectedsoftware.Considerintroduceing
automatedregressiontestingatvariouslevels(unit,component,system)toreducethelengthof
timeittakestodoaneffectiveregressiontestcycle.
20.1.4 LotsofDebateBetweenBugsvs.ChangeRequests
Whenisabugadefectandwhenisitachangerequest?Ifthisdiscussionisoccurringalotduringthe
acceptanceprocessitmaybeanindicationthattheProductOwnerandtheproductDevelopmentTeam
arenotstrivingtosolvethesameproblem.Somepossiblerootcausesare:
FixedpricecontractsencouragetheproductDevelopmentTeamtoclassifyeverythingasaCRevenifit
islegimatelyabug.Solutionistoavoidcreatingdysfunctionalrelationshipsthatcomeoutoffixed
pricecontracts.ConsiderusingTargetPricecontracts[PoppendiecksOnLean]instead;these
motivatebothcustomerandsuppliertominimizethechangeswhilemaximizingvalue.Thereisno
distinctionbetweenabugandaCR;theonlydistinctionisbetweenchangesthatareworthmaking
andthosethatthecustomercanlivewithout.
VaugerequirementscausedbyalackofunderstandingoftheProductOwnerastowhattheywanted.
See
VaguerequirementsleadingtolackofclarityaboutwhatPOreallyaskedfor.Thegapbetweenwhatthe
POthoughttheyaskedforandwhattheproductDevelopmentTeaminterpretedtherequestcaused
thebugandthesubsequentdebate.ThesolutionistoimprovethecommunicationbetweenthePO
andtheproductDevelopmentTeam.Thisisbestachievedthroughcollaborationratherthansimply
writingmorecopiousrequirementdocumentation.Thecommunicationcanbesupportedby
detailedexampleswhichcanbeusedassampleacceptancetests.Considerincludingreadiness
assessorsandacceptancetestersintheproductdesigndiscussionsasthiswillusuallyunearth
interestingscenariousthattheProductOwnermayneedtoconsider.

DebuggingtheAcceptanceProcess
Issuesencounteredduringtheacceptanceprocesscanbevaluablecluesaboutproblemsinhowour
organizationandworkflowsarestructured.Thesecluesprovidehintsabouthowweshouldrestructure
ourorganizationanditsprocessestoworkmoreefficiently.Thefollowingflowchartsuggestpossible
solutionsbasedoncommonsymptoms:
Psuedocodeforaflowchart:
IfAcceptanceTakingTooLong
o IftoomanyTest&fixcycles
Iftoomanyregressionbugsintroduced

See0UseAutomatedTestExecutiontoReduceRegressionBugs
else
o Elseiftoolongpercycle
ConsiderstreamliningtheprocessusingValueStreamMapping
Else(nottakingtoolong)
o Iftoomannewbugsfound
Iffoundbyendusers
IfUsersfindsystemhardtouse
o See0UseIncrementalUsabilityTestingtoDiscoverDesign
DefectsEarlier
Ifbugsfoundinafewspecificareas
o See0IncreaseBreadthofAcceptanceTesting
Elseifbugsfoundeverywhere
o See0IncreaseDepthofAcceptanceTesting
Iffoundbyacceptancetesters
IfPDTisnotdoingthroughreadinessassessment
o IfPDTisorganizedaroundcomponents
See0UseFeatureTeamstoImproveAccountabilityof
ProductDevelopmentTeamforEndUserFunctionality
o ElseIfPDThasaccesstoacceptancetests
Runthemmorefrequently
See0UseAutomatedTestExecutiontoReduce
RegressionBugs
o Else
See0UseAcceptanceTestDrivenDevelpmenttoHelp
ProductDevelopmentTeamUnderstandRequirements
Better
isbeingrushedbymanagementorPOordeadlines
o See0FocusonQualitytoGetSpeedofDelivery
Elseifa

IffoundbyReadinessAssessors
Iffoundbyindependenttesters
o UseATTDtoimprovecommunicationofdone
SeealsoIffoundbyacceptancetesters
o ElseiftoomanyaccumulatedbugstofixtherebyrequiringBugTriage
Ifduetohighbugarrivalrates
seeIftoomanynewbugsfound
ElseifBugsarentbeingfixedduetolackoftime
EducateProductownerondecidingbetweenBugsvsNewFeatures
Ifduetofearofintroducingnewbugsduetofragilityofcodebase
Findwaystogetcodebaseunderautomatedtestandrefactor
o ElseifdisagreementsbetweenBugsandChangeRequests
IfFixedPriceContracts
See0UseTargetPriceContractstoAlignInterestsofProductOwner
and
Else
See0UseAcceptanceTestDrivenDevelpmenttoHelpProduct
DevelopmentTeamUnderstandRequirementsBetter
o Else
Isthereaproblem?
Thesolutionsreferencedinthisflowchartareelaboratedbelow.
PossibleRemedies
Basedontheresultsoftheanalysisofacceptanceprocessissuesencountered,oneormoreofthe
followingremediesmaybeuseful:
UseTargetPriceContractstoAlignInterestsofProductOwnerandProductDevelopment
Team
UseIncrementalUsabilityTestingtoDiscoverDesignDefectsEarlier
Incorporateusabilitytestingofearlyversionorprototype(includingpaperprototypes)

UseIncrementalAcceptanceTestingtoDiscoverDefectsEarlier
UseIncrementalAcceptanceTestingtoHelpProductOwnerDiscoverRequirementsEarlier
InvolveTestersDuringRequirementsDefiintiontoEnsureCompletenessofRequirements
UseAcceptanceTestDrivenDevelpmenttoHelpProductDevelopmentTeamUnderstand
RequirementsBetter
ProductOwnershouldprovideacceptanceteststoProductDevelopmentTeam
o CollaboratewithProductDevelopmenttodevelopthem
UseAutomatedTestExecutiontoReduceRegressionBugs
UseFeatureTeamstoImproveAccountabilityofProductDevelopmentTeamforEndUser
Functionality
IncreaseBreadthofAcceptanceTesting
Includemorevariedfunctionality/scopewithinthescopeoftesting.
IncreaseDepthofAcceptanceTesting
Usemore/differenttestdesign(e.g.Scenariobasedtesting)andexecutiontechniques(e.g.
ExploratoryTesting)togetbettertestcoverage.
FocusonQualitytoGetSpeedofDelivery
FocusonQuality;speedwillfollowduetolesstimespentfindingandfixingbugs.
StreamliningtheAcceptanceProcess
Theacceptanceprocessisanecessarybutpotentiallywastefulexercise.Weshouldstrivetokeepit
assimpleandvalueaddingaspossible.Thelongerittakes,themoreitcostsinwastedeffort
anddelayeddeliveryofbusinessvalue.itItcanbecomeaseriousimpedimenttobeing
responsivetoourusers(theproductownerscustomers.)Beforewecantakestepsto
streamlineitwemustfirstunderstandit.
20.1.5 UseValueStreamMappingtoUnderstandtheAcceptanceProcess
Wecanimproveourunderstandingofanexistingorproposedacceptanceprocessthroughan
exercisecalledValueStreamMapping.Thisisaformofbusinessprocessmodelingthatfocuses
ourattentionontheratioofelapsedtimeandtheamountofvalueadded.Figure1isavalue
streammapofahypotheticalacceptanceprocess.

Legend:
Value in days
Time in days
Severity 1 & 2 bugs found Avg 3 x Severity 1 & 2 bugs found Avg 0.5 x Valueadding
Bugs found Avg 4 x activities=
FixBugs
Insufficient Quality Data Insufficient Quality Data 60days
25% 25%
0 days
total=211days
Feature Feature Integration Integration
2 days Readiness Readiness Acceptance Acceptance Acceptance Acceptance
Assessment Decision Testing Decision Testing Decision
Construct
Software 2.5 day 5 days 5 days 28%efficiency
5 days 0.5 day 10 days 0.5 day 10 days 0.5 day
40* days 0.5 day 0.5 day 0.5 day
40 days
Testers50% Testers50% Testers50%
Inadequate design 25%
Insufficient information 30% Insufficient information 30%

Insufficient Quality Data 10%
Prepare Security Operations Prepare CMB

Operations
Security Queue Acceptance Acceptance CMB Queue Acceptance
Decision Assessment Decision
Document Decision Documents
1 day Avg 10 2.5 1 day Avg 10
10 days days 0.05 day days 0.5 day 10 days days 0.05 day
0.25 day 5 days 0.5 day 0.25 day
MonthlyReview MonthlyReview
Testers50%
(every20days) (every20days)

Figure1
AsIsAcceptanceProcess
Thisprocesstakesanaverageof211daystoexecuteandprovidesonly60daysofactualvalueresulting
inacycleefficiencyofonly28%.Someofthefactorsthatmakethisprocesstakesolongtoexecuteare:
Sixtydaysofsoftwaredevelopmentoutputissentthroughtheprocessinasinglebatch.Therefore,
thereisalargeinventoryofuntestedsoftwarewhichresultsinmanybugsbeingfoundduring
thereadinessassessmentandacceptancetestingactivities.Thefixingofthesebugsisdone
entirelyonthecriticalpathoftheproject.
Thereareseveralcyclesinprocesseachofwhicharetypicallyexecutedseveraltimes.Forexample,
ReadinessAssessmentsendsthecodeback,onaverage4timesforanadditional5dayselapsed
timewhichaddszerodaysofvaluebecauseitispurerework.Eachcycletakes7.5days(2.5days
fixingand5daysreadinessassessment)thereforethisfeedbackloopadds30daysofelapsed
time.
Sometaskstakelongerthannecessarybecausetheresourcesarenotdedicated.Forexample,the
readinessassessorshaveotherresponsibilitiesandittakesthem5daystodo2.5daysoftesting.
Customeracceptanceisdoneintwoseparatephasesandisfollowedbythreeotherformsof
acceptancedecisionmaking,twoofwhichcansendthesoftwarebackallthewaytobugfixing.
Thesoftwareneedstoberetestedeachtimethesoftwareissentbacktofixbugs.
Thesecurityteamisoverworkedresultinginanaveragewaittimeof10daysforthesecurityreview.
Thepreparationofthesecuritydocumentstakesanaverageof10daysasdevelopersdiscover
whatisrequiredbutonly1dayofthateffortaddsrealvalue.

TheChangeManagementBoard(CMB)meetsmonthlyresultinginanaveragewaitof10business
days.
Note:Thecalculationsassumethatnovalueisprovidedbythebugfixingpassesthroughthesame
activitiesasthisisunplannedrework.Ofcourse,thiswholecalculationisbasedontheassumptionthat
wecanaddvaluebyreviewingwork;anargumentcanbemadethatthewholeacceptanceprocessisa
formofwastecausedbythecustomersinabilitytoaccuratelydescribewhattheywantbuiltandthe
suppliersinabilitytofollowthespecificationadequately.
20.1.6 StreamlinetheAcceptanceProcess
Thisacceptanceprocesscanbestreamlinedbychanginghowtheworkpassesthroughthevarious
readinessandacceptanceactivities.Figure2illustratestheresultofhavingappliedanumber
transformationsontheprocess.ThesetransformationsareinspiredbyLeanThinkingwhichfocuseson
eliminatingwaste.SeethesidebarWasteinSoftwareDevelopmentforadescriptionoftheseven
commonformsofwaste.Whichformsofwastewetrytoeliminatefirstdependsonwhatisimportant
tous.Iftimetomarketisthekeyconsideration,wemaywanttotacklethequeuingdelaysfirstand
thenlookforwaystoreducetheamountofprocessing.Ifcost/resourceconstraintsareholdingusback,
wemaywanttolookforwaystoreducetheamountofprocessingfirst.
Legend:
Value in days PerIterationActivities
Time in days
Severity 1 & 2 bugs found Avg 1 x
Note:AllTesters
Insufficient Quality Data
100%Dedicated
4 Iterations +
25% Repeatedonceper
1 bug fix Feature
iterationplusonefinal
Feature
Acceptance Acceptance bugfixcycle
Testing Decision
Construct
Readiness
Software/
Fixbugs
10 days
Assessment
Readiness
Decision
1 day
1 day 0.1 day
0.1 day
2.5 days
0.5 day
10 days 2.5 days
0.5 day
25%
Valueadding
Integration
Acceptance
Integration
Acceptance
activities=
Testing Decision
1 day
58days
0.1 day
1 day
0.1 day Inadequate design
25%
total=110days
10%
Operations
Operations
Assessment
Acceptance
Decision
53%efficiency
1 day
1 day 0.1 day
0.1 day
Insufficient information Insufficient information 30%

< 10% CMB
Prepare 1on1 Security PrepareCMB
Security Security Security Queue Acceptance
Readiness Documents
Consultation Document Review Decision Decision
0.5 day 1 day 0.5 day 1 day WeeklyReview Avg 2.5
0.5 day 2 days 0.05 day 10 days days 0.05 day
0.5 day (every5days)
0.05 day 0.25 day
Consultations&
Interviewsprebooked
Figure2
StreamlinedAcceptanceProcess
Thisstreamlinedversionoftheprocessistheresultofthefollowingchanges:

Wehavebrokentheprojectinto4twoweek(10businessday)iterations.Eachiterationissentthrough
readinessassessment,featureacceptance,integrationacceptanceandoperationalacceptance.Bugs
foundinthefirstthreeroundsoftestingarefixedinthesubsequentiteration.Afterthefourth
iteration,thesoftwareisretestedandtheresultingbugsarefixedinasinglebugfixingiteration.
Contrastthiswiththe4roundsofbugfixasaresultofreadinessassessmentandthreeroundsdue
tofeatureacceptancetesting.Eachroundoftestingtakesroughly30%ofthetimeittookintheAs
Isprocessbecausethebacklogofuntestedsoftwareismuchsmaller.Thisisanexampleofreducing
wasteassociatedwithinventory.
Theacceptancetestsareprovidedtothedevelopmentteamalongwiththerequirements.Thisallows
thedevelopersandthereadinessassessmenttesterstoverifythefunctionalityisworkingcorrectly
beforehandingitofftotheacceptancetesterstherebyreducingthenumberofbugsfoundin
acceptancetesting.Thereadinessassessmentshownasoccurringaftertheiterationactually
happenscontinuously(assoonasthedevelopersaysthefeatureisready)thereforeanybugs
foundinreadinessassessmentarefixedbeforethedeveloperhasshiftedtheirfocusonthenext
fixture.Thisreducesthecostoffixingthefewbugsthatarefoundinreadinessassessmentbya
factorof4.Thisisanexampleofreducingthewasteassociatedwithdefects.
Theacceptancereviewforsecurityhasbeenchangedfromanacceptanceactivitytoareadinessactivity.
Thisallowsittobedoneinparallelwithdevelopment.Ithasalsobeenchangedfrombeingapush
modelwheretheteampreparesadocumenttobereviewedtoapullmodel.Thisstartswitha
consultationwiththesecurityspecialistwhohelpsthedevelopmentteamunderstandthe
appropriatesecurityrequirementsanddesignandwhatneedstobeinthesecuritydocument.This
providesinputintothesoftwareconstructionprocessresultinginasecuritycompliantdesignaswell
asmoreappropriatecontentinthesecuritydocument.Thisreducestheturnbackratefrom30%to
10%andthetotaleffortfrom10daysto3dayswithoutanyreductioninvalue.Theeffortto
producethedocumentisreducedandtheefforttoreadthedocumentisalsoreduced;definitelya
winwinsolution.Afinalsecurityreviewisheldpriortothefinaliterationtoreviewthefinished
documentaswellasanylatebreakingsecurityrelateddesignconsiderations.Thisisanexampleof
eliminatingwastebyreducingdelaysandbyreducingextraprocessing.
Thefeatureacceptancetesting,integrationacceptancetestingandoperationsacceptancetestingare
doneinparallelwithanunderstandingthatanyshowstopperbugsfoundinanyoftheparallel
testingactivitiescanresultinthesoftwarebeingrejected.Thisreducestheelapsedtimetothe
longestofthethreeinsteadofthesumofthethreeactivities.Thiseliminateswasteintheformof
delaysincurredwhileonegroupwaitsforanothergrouptofinishtheirwork.
Becauseonly25%ofthefunctionalityisbeingtestedanewineachiteration,theacceptancetestingcan
befinishedin1day.Thisisashortenoughperiodthatthetesterscanfocusontheoneprojectand
finishit1dayelapsedtime.Thisisntaformofwastereductionasmuchasitisanexampleof
improvingflowbymakingtheoutputoftheprocesslessburstyviasmallerbatchsizes.Achieving
flowisanotherofthekeyprinciplesofleandevelopment.

TheChangeManagementBoarddocumentationispreparedinparallelwiththefinalbugfixingiteration
andcanbesubmittedtotheCMBassoonasthethreeparallelacceptancedecisionsarepositive.To
furtherreducethedelay,theCMBnowmeetsweeklyfor1hourratherthanmonthlyforahalfday.
Thisreducestheaveragewaitfrom10businessdaysto2.5.Thisisanexampleofreducingwaste
associatedwithwaiting.
Thecollectedimpactofthesechangestotheacceptanceprocesshasreducetheaverageelapsedtime
to94daystoexecute110daysofvalueaddedprocessingthatdelivers58daysofvalueforacycle
efficiencyof53%.
Note:Thecalculationsassumethatvalueisprovidedbytheplanneddevelopmentiterationsbutno
valueisprovidedbythefinalbugfixingiteration.Thecycleefficiencywasbasedonthecumulative
durationincludingactivitiesdoneinparalleleventhoughtheydidnotaddanyelapsedtime.
Summary
Asignificantportionoftheelapsedtimebetweenthefinishofdevelopmentofsoftwareandwhenthe
softwarecanstartprovidingvaluetothecustomerisconsumedbytheacceptanceprocess.Theelapsed
timecanbereducedsignificantlybybuildingsoftwareincrementallyanddoingincrementalreadiness
assessmentandacceptancetestingaseachincrementofsoftwareisfinished.Differentkindsof
inspectionandtestingactivitiescanbedoneinparallelwithdevelopmentofthelaterincrements
reducingtheamountofworkthatneedstobedoneonthecriticalpathbetweencompletionof
developmentandfinalacceptance.Thetypesoftheissuesdiscoveredduringacceptanceprocesscanbe
usedtodiagnoseorganizational,culturalandprocessissuesintheorganizationsinvolved.Changingthe
processestoavoidtheissuesisusuallymorebeneficialthanimprovingtheefficiencyofaddressingthe
issuesoncefound.
WhatsNext?
VolumeIhasintroducedanumberoftoolsyoucanuseforreasoningabouthowyouacceptsoftware.It
hasdescribedwhentousealargenumberofacceptancerelatedplanning,requirements,inspection,
reviewandtestingpractices.Youmaywanttoresearchsomeofthesepracticesinmoredetail.Volume
2inthisseriesdescribesmanyofthesepracticesinmoredetailwhileVolume3providesexamplesof
theartefactsthatmighthavebeenproducedonafictionalproject.
Sidebar:FormsofWasteinSoftwareDevelopment
ThesevencommonwastesofmanufacturingcanberememberedusingtheacronymTIM
WOOD.Insoftware,thereisanequivalentofeachformwasteasexemplifiedbythelist
providedbyMary&TomPoppendieckintheirbookImplementingLeanSoftware

Development[PoppendiecksOnImplementingLean].Unfortunately,thesoftwarespecific
namesdontformaniceacronymsoweveincludedbothnameshere.Thesoftwarespecific
nameshavea*nexttothem.
T=ThewasteofTransport(Handoffs*)
Transportationiswastebecauseitdoesntaddanyvaluetotheendproductbutitincreases
thecostandtheelapsedtime.
Insoftware,transportcorrespondstohandoffsbetweenpartiesusuallyviadocuments.The
requirementsdocumenthandedbyspecifierstosoftwaredevelopersisoneexample,the
designdocumenthandedbyarchitectstodevelopersisanother.Thepreparationofthese
documentstakeslargeamountsofeffortoftenmuchmorethancommunicatingthesame
informationverbally.Handoffsusuallyresultinlossofinformationandthisistypicallyworse
whenthehandoffisasynchronous(e.g.documents)ratherthanfacetoface.Seethesidebar
UsingWholeTeamstoBuildProductsinChapter1forwaystoreducethenumberofhandoffs.
I=Inventory(PartiallyDoneWork*)
Inventoryisbadbecauseitcostsmoneytoproducetheinventoryandoftencostsmoneyto
storeormanagetheinventory.Inventoryalsomasksissuesbydelayingwhendefectiveparts
arediscovered.Justintimemanufacturingisallaboutreducinginventorytothelowestlevels
possible.
Insoftware,inventoryisanyartifactthathastakenefforttoproducebutwhichisnotyet
providingthecustomerwiththevalueexpectedtobeprovidedbyhavingthesoftwareinuse.
Somecommonformsofinventoryinclude:
UntestedsoftwareSoftwarethathasbeenwrittenbutnotyettested
Softwarewithalonglistofbugsuntriagedortriagedbutnotfixed
RequirementsdocumentsDocumentsthatprovidedetaileddescriptionsoffunctionality
thatwontbebuiltrightaway.
M=Movement(TaskSwitching*)
Movementisbadbecausewhileaworkerismovingtheycantbeproducing.Thisaddsno
valuetotheproductbutreducesproductivityoftheworkertherebyincreasingcost.
Insoftwaredevelopment,theequivalentofmovementistaskswitching.Thisiscausedby
askingpeopletoworkonseveralthingsatthesametime.Everytimetheyswitchbetweenone
taskandanothertask,timeiswastedwhiletheyreestablishtheirworkingcontext.
W=Waiting(Delays*orQueuing)
Wheneverworkstopswhilewaitingforsomethingtohappen,itisaformofwaste.
Commoncausesofwaitinginsoftwareincludewaitingforapprovals,waitingforclarificationof
requirementsandwaitingforslowcomputersortoolstofinishtheirprocessing.

O=Overprocessing(LostKnowledge*orProcessInefficiency)
Overprocessingiswastecausedbydoingunnecessarystepsordoingasteplongerthan
necessary.Thisaddsnoadditionalvaluebutitdoesaddcost.
Insoftware,overprocessingisanystepsintheproductionprocessthatarerequiredtoproduce
highqualitysoftware.Themostcommonexampleistheproductionofanydocumentation
thatnoonewilleverread.Anothercommonformishavingtorediscoverinformationthatwas
lostsomewherebeforeitcouldbeused.
O=Overproduction(ExtraFeatures*)
Overproductionisproducingtoomuch.Itislikeinventoryexceptthatitisfinishedproduct
whileinventoryisworkinprogress.
Insoftware,overproductionisthedevelopmentofunnecessaryfeatures.Ithasbeenreported
[REF]thatonaverage80%offeaturesdevelopedarerarelyorneverused.Thisisclearly
overproduction!
D=Defects
Defects,bugs,problemreports,usabilityissuesallrequireextraworktoanalyse,understand
andaddress.Thisextraworkispurewaste.Itisexactlythesameinsoftware.
Resources
[PoppendiecksOnLean]Poppendieck,M.LeanSoftwareDevelopmentAnAgileToolkit,AddisonWesley
Professional#ISBN10:0321150783ISBN13:9780321150783,
[PoppendiecksOnImplementingLean]Poppendieck,M.,Poppendieck,T.ImplementingLeanSoftware
DevelopmentFromConcepttoCash,AddisonWesley,.(Seerefinanotherchapter.)

AppendixAOrganizationStereotypes
andReaderPersonas
Thisappendixdescribestwostereotypicalorganizationsandthepeopleinvolvedinmakingorproviding
dataforthereadinessandacceptancedecisions.Thepeoplearedescribedaspersonas.
Notetoproduction
Reviewers:Thesetwostereotypesareincludedtoactasabusinesscontextforthepersonas.The
personasareincludedfortworeasons:
1. Toremindtheauthorsandreviewersofourprimarytargetaudience.
2. Togivereadersafiguretoempathizeorassociatethemselveswith.Thisshouldhelpthem
pickthepersonathatmostcloselyresemblestheirownjobresponsibilities.
Contents Primarypersonas Secondarypersonas
JosephKrawczak
LisaAndrews
Thinkingabout YvetteKirwan
VolumeI EricAndersen
Acceptance ChattyCathy
FredSpook
YvetteKirwan
Managing JosephKrawczak
VolumeII ChattyCathy
Acceptance LisaAndrews
FredSpook
EricAndersen
LisaAndrews
Functional JosephKrawczak
EricAndersen
VolumeIII Acceptance YvetteKirwan
Testing ChattyCathy
FredSpook
Operational LisaAndrews
ChattyCathy
VolumeIV Acceptance EricAndersen
FredSpook
Tetsing YvetteKirwan
OrganizationStereotypes
ITOrganizationofaBusiness
XYZCorpisalargetransportationcompany.Computersoftwarehasbeenintegraltothecompanys
businessforseveraldecades.Thecompanyhasseveralthousandapplications.Atthehighendare

systemsbuiltbytheITdepartmentrangingfromlargemainframesystemswritteninCobolandPL1to
WebbasedapplicationswritteninASP.NET.Atthelowendthereareenduserapplicationsbuiltin
VisualBasicandAccessandspreadsheetapplicationsbuiltbyendusersscatteredthroughoutthe
businessareas.
TheITdepartmentwillbuildthesoftwarebasedonthespecificationsprovidedbythebusinessand,
afterdoingitsownsystemtesting,expectsthebusinesstoconductuseracceptancetesting.Thereisno
separateQAdepartmentbuteachdevelopmentteamhasatleastonetestingspecialistwhohelpsthe
teamtesttheirownsoftware.TheITdepartmentsqualitygateprocessalsorequiressignoffsfromtheIT
ArchitecturegroupandtheOperationalsupportgroupbeforesoftwarecanbedeployedintothe
Acceptanceenvironment.Asaresultofrecentlyintroducedcorporategovernanceinitiatives(suchas
SOX),companypolicyrequiresapprovalbytheCorporateSecuritydepartmentbeforeanynew
applicationcanbedeployedintoproduction.
TheITArchitecturedepartmentincludesateamofdataarchitectswhoseprimaryresponsibilityisto
ownthecorporatedatamodel.Thecompanyviewsthedataintheapplicationsasacorporateassetand
thedataarchitectshavebeenworkinghardtomanagetheduplicationofdataacrossthem.TheIT
departmentalsomanagesthelifecyclesofthevariousITtechnologiesusedinthecompany.New
technologieshavetobeapprovedbytheITArchitecturegroupbeforetheycanbeusedandold
technologiesmaybemarkedforreplacement.TheCorporateSecuritydepartmenttakesanactive
interestinallnewapplicationsandmustapprovethesecuritymeasuresbeforeanynewapplicationcan
bedeployed.

AProductCompany
AdventureWorksisaproductcompanythatsellsbusinessproductivityimprovementproductsthat
includebothhardwareandsoftware.Someoftheproductsaresoldashardwarewithembedded
softwareandsomeassoftwarepackagesthatleverageexistinghardwareproducts.Thecompanyhasa
separateUxDandQAdepartments.TheUxDdepartmentisinvolvedinpartoftherequirements
definitionactivitiesassociatedwithdesigningtheproductaswellasensuringthattheproductshavea
consistentlook&fell.TheQAdepartmentteststhesoftwarebuiltbytheProductDevelopmentgroup
andadvisestheProductManageronwhetherthesoftwarecanbereleasedtothemarket.
ReaderPersonas
PersonasOftenFoundintheITOrganizationofaBusiness
ProductManagerorProductOwner
EricAndersenisaproductmanagerwhohasbeenaskedtoworkwithanagiledevelopmentteamasthe
ProductOwnerforoneofthesoftwareonlyproducts(anewsoftwarereleasethatworkswithexisting
hardwaresuppliedbythecompanyandthirdparties.)Becausethecompanysproductsareconsidered
hightechnology,Ericisfairlytechsavvy.Hehasagoodunderstandingofthecustomersrequirements
butdoesgetabitlostwhenthediscussiondescendstothelevelofnetworkingprotocols.
KeyquestionsEricneedstofindanswersto:
WhatkindsofinformationdoIneedfromtheQAdepartmenttodecidewhetherornotto
acceptthesoftware?Whatqualitycriteriamustbemet?HowdoIknowtheyhavebeen
met?

Whoisresponsibleforverifyingthesecriteriaaremet?Whoisresponsibleforensuring
thesecriteriaaremet?
HowdoestheUxDteamparticipateinacceptancetestingofthesoftware?
Howcanthereleasestrategy(multiplereleases,alphas,betas,etc.)helpme?
E.g.Improveusability,fitnessforpurpose
Getrevenueearlier
Viralmarketing
Whatistheprocessforaddressingdeficienciespriortorelease?
BusinessPersonintheMarketingDivision
LisaAndrewsisabusinesspersoninthemarketingdivisionofXYZCorp.Lisaisinherlatetwenties
(relativelyyoungforherpositioninthesomewhatconservativecompany)soshehasgoodcomputer
usageskillsbutisnotbyanystretchoftheimaginationconsideredtechnical.ThisisLisasfirstIT
project.Lisahasbeenaskedtobethebusinessleadoftheprojecttoreplaceseveralexistingsystems
usedbythestaffinthemarketing,salesandcontractsdepartmentswithasingleintegratedsystem
usingthelatesttechnologies.Inthisroleshewillhavethefinalsayonwhetherthesystemisacceptable
tobedeployed.Shewillalsoberesponsibleforensuringthatthebusinessbenefitsusedtojustifythe
projectareactuallyrealized.Shewillbeassistedonthisprojectbyseveralusersoftheexisting
applicationsaswellasabusinessanalystandatestprofessional.Shewillneedtooverseethisteamasit
definesthedetailedrequirementsforthenewsystem.
KeyquestionsLisaneedstoanswer:
Whatkindsoftestingwillmyteamneedtodotohelpmedecidewhetherornottoacceptthe
software?
Howcanthereleasestrategy(multiplereleases,alphas,betas,communitytechnologypreviews,
etc.)helpme?
E.g.Improveusability,fitnessforpurpose,userbuyin
Getcostsavingsearlier
WhenshouldIplanondoingAT?HowmuchtimeshouldIallowforAT?Howmucheffortwillbe
involved?HowmuchofstaffdoIneedtodoit?
Testlastorincremental?
AmIinvolvedinATordoIdelegateittosomebodyelse?Frommyorg?athirdpartylab?
WhoelsecanhelpmewithATifIdonthavetheskill/confidence
WhatkindsoftestingdoIexpecttheITdepartmenttodobeforeaskingmyteamtodoacceptance
testing?

WhatinformationdoIneedITtoprovidetomeaboutthekindsoftestingtheyhavedoneandhow
willIusethatinformationindecidingwhetherornottoacceptthesoftware(orevenstart
acceptancetesting?)
WhatinformationdoIneedtoprovidetoITpriortomedoingAT?
OperationsDepartmentManager
ChattyCathyisthemanageroftheoperationsdepartment.Sheisresponsiblefordeployingallserver
basedapplicationsandkeepingthemrunningaccordingtoeachapplicationsservicelevelagreement
(SLA).Shesbeenwiththecompany30yearshavingworkedherwayupfrombeinganadministrative
assistantinoneofthebusinessunits.Herformaltrainingwassecretarialschoolandsheslearned
whateversheneededtoknowonthejob.WhenshewasfirstmovedintoI.T.itwasasatesterfor
desktopapplicationsbutshequicklymovedtoupmanagethedesktopsupportteam.Whenthat
functionwasoutsourced,shewasaskedtotakeovertheoperationsgroup.Thatwastwoyearsagoand
shehasonlyrecentlybecomecomfortablewithherunderstandingofhowthegroupfunctions.Shehas
yettomakeanychangestothegroupsprocesses.
ITSecuritySpecialist
FredSpookistheITSecuritySpecialistintheCorporateSecuritydepartment.Hesaformerspy(IfItold
youwhichagencyIworkedforIwouldhavetokillyou,hesayswithawinkwheneverhesasked.)who
movedintotheprivatesectorwhenthefirstmajorsecuritybreecheswerepublicized.Hisbackground
hasmadehimratherparanoidandhedelightsinskeweringthedevelopmentteamswithhisarsenalof
nastysecurityscenarios.Asaresult,mostapplicationsfailtheirfirstsecurityreviewandrequire
significantarchitecturalrefactoringbeforepassing.Thebetterdevelopmentteamshavelearnedto
approachhimforanearlyofftherecordsecurityauditassoonastheirdesignhasstabilized.
PersonasOftenFoundinaProductCompany
TestManager
JosephKrawczakisthemanageroftheproductverificationgroupatAdventureWorks.Histeamof
professionaltestersverifiesthattheproductsbuiltbytheproductdevelopmentgroupworkasspecified
bytheproductmanagementteam.Hisbackgroundisintestingtelecomsystemswherehemanageda
30persontestingdepartment.Whenthetelecomindustrydownsizedhewasgivenasizablepackage
andwashiredbyABCCorptoheadupitsnewlycreatedIndependentTestdepartment.Hismandate
wastoimprovethequalityoftheproductsproducedbyAdventureWorks.Histeamofprofessional
testersisexpectedtoprovidehimwiththedatarequiredtodeterminewhethertheproductis
acceptable.Heprovidesthisdata,alongwithaaccept/rejectrecommendationtotheProductManager
whomakesthefinaldecision.Heisalwayspreparedtobackuphisrecommendationwithaclear

descriptionofthebusinessimpactofthebugsthathefeelsmustbefixedbeforehewouldbe
comfortableadvisingtheproductbeaccepted.
UserExperienceSpecialist
YvetteKirwanisaFineArtmajorwhogotajobasagraphicartistinamediacompany.When
conventionaladvertisingworktookadownturn,shewasmovedintotheproductdesigngroupwhere
shedidgraphicdesignforwebbasedsoftwareapplications.Shequicklyrealizedthatthegraphicswere
mainlyeyecandyandthattherealvaluewasprovidedbytheinteractiondesignsoshewentbackto
schoolparttimetogettrainedasaninteractiondesigner.Withthistrainingunderherbeltshequickly
becameakeyplayerinthedesignofsoftwareintensiveproductsatthecompany.Shewashiredby
AdventureWorks.specificallytoworkonthenewproduct.Herroleistodefinetheinteractiondesign
basedonmarketrequirements,usermodelsandfeedbackfromtheproductmanagerandthe
development.Thedesignsneedtobeimplementableinacostefficientmannerusingthetechnology
stackusedbytheproductdevelopmentteam.WhilesheistechnicallypartoftheUxDTeam,shesitsin
thedevelopmentareawiththedevelopmentteamasshehasfoundthistobeamoreeffectivewayto
transitiondesignknowledgetothedevelopersandtheirmanager.Shehasalsofoundittobeusefulto
enlistfriendsandpeopleoffthestreettodoinformalusabilitytestingofpaperprototypesaswellasthe
actualsoftware.Thisgivesherrealdatashecanusetohelpinfluencethetestdepartments
understandingtheusers.

AppendixBKeyPoints
ThechaptersinPartIIdescribedtheprocessleadinguptotheacceptancedecisionfromthe
perspectivesoftheBusinessLead,ProductManager,TestManager,DevelopmentManager,User
ExperienceSpecialist,SolutionsArchitect,EnterpriseArchitect,OperationsManager,andLegal.For
convenience,wesummarizedthekeypointshere.ThereareKeyPointscommontoallrolesandothers
specifictoeachindividualrole.
ThefollowingtablelistsKeyPointsthatarerolespecificandwhatrolestheyarespecificto.
<Thetablewillbeupdatedandavailableontestingguidance.codeplex.com>

Figure1
KeyPointsspecifictoindividualroles

KeyPointsApplicabletoAllRoles
Treatacceptanceasaprocessnotanevent:Appreciatethefactthatacceptancetestingwillnot
beasinglediscreteevent,butaprocesswithadistinctstartandend.
Definethescopeoftesting:Haveaclearunderstandingofwhatwillbetestedbythesupplier
andwhatyouareexpectedtotest.Typically,supplierisexpectedtodothorough(e.g.)testing
ofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,usabilityetc.)
beforehandingoffsystemforacceptancetesting.Doesthesupplierhavetheenvironmentsfor
integrationtestingwithoutinhousesystems?
Delegatedecisions:Ifyoudonthaveenoughtimeorunderstandingtotakepartintheproject,
assignaproxytorepresentyouandhaveaclearunderstandingofthekindsofdecisionsthey
canmakewith/withoutconsultingwithyou.
Delegatework/testing:Haveaclearunderstandingofyour(organizations)abilitiesw.r.t.
testingskills.Contractotherpartiestohelpyoutestifnecessary.Butdontunderestimatethe
effortinvolvedinmanagingthetestingoutsourcerastheywillneedtolearnyourbusiness
domainandrequirementsbeforetheycanbeeffective.
Understandthetestenvironments:Understandwhereyouwilltest.Beclearonhowyoullget
thenewreleasecandidatesdelivered
Viewtestsasrequirements:Acceptancecriteria/testsshouldbearticulatedandprepared
beforethesoftwareisbuiltandsharedwithvendorasawaytofleshoutthedetailsofthe
requirements.
Estimateacceptanceeffort&durationwhilerealizingitisdifficulttoestimate:Estimatingthe
effort&durationofacceptancetestingishard;timeboxingthefinalacceptancetestingcycleis
commonbutrequirespastexperiencetocomeupwithreasonableduration/effort.Ifindoubt,
guesshighandallowforseveralcyclesofacceptancetestingtogivethevendortimetofixbugs.
Useincrementalacceptancetestingtoreduceuncertaintyofestimating:Canavoidsomeof
theuncertaintyandmostoftheriskbydoingincrementalacceptancetesting.Bugsfound
earlieridentifyambiguityinthespecwhilethereisstilltimetoaddressitandcanbefixedlong
beforethefinalacceptancetestphase
Understandthedecisionmakingprocess:Makesureyouhaveaclearunderstandingofthe
decisionmakingprocess.OtherpotentialdecisionmakersincludeTestManagers,Security
Manager,UserExperience/UsabilityManager,Architect,LegalDepartment,etc.Ensureyou
haveagreementonwhethertheymakeadecisionorprovidedataorarecommendationtoyou.
Communicateinformationflowclearly:Ensureanyonewhoyouexpecttoprovideyouwith
dataonwhichtobaseyourdecisionunderstandswhatdatatheyareprovidingyouandwhen.
E.g.TheTestManagerprovidestestresultstoyouandyoumaketheacceptancedecision.
Understandtheoperationalrequirements:Makesureyouunderstandtheoperational
requirementsforyoursystem.Whetheryouaresellingasoftwarepackagetocustomersor

sellingSoftwareasaService(SaaS),beawarethatwhoeveroperatesthesoftwarewillhave
operationalrequirementsoverandabovethefunctionalrequirementsoftheendusers.
Clarifydecisionmakingvsdecisionsupporting:Makesureyouhaveaclearunderstandingof
whetheryouarethegatekeeper(acceptancedecisionmaker)orasupplierofinformationto
theProductManagerforhertomakethedecision(preferred)
Maximizelearning:Workwiththeproductownertoensurethateveryoneontheteamhas
accesstothedefinitionofsuccessintheformofacceptancetestsbeforethesoftwareisbuilt
KeyPointsforBusinesslead
3. Treatacceptanceasaprocessnotanevent:Appreciatethefactthatacceptancetestingwill
notbeasinglediscreteevent,butaprocesswithadistinctstartandend.
4. Recognizetheremaybemanydecisionmakers:Theprocessmayrequiredecisionsbymany
partieseachsupportedbydatacollectionactivities.Eachpartylikelyhasverydifferent
requirementsandinterests;youneedtobeawareofthemandcommunicatethemtothe
supplier(s)toensuretheyaddressthem.
5. Definethescopeoftesting:Haveaclearunderstandingofwhatwillbetestedbythe
supplierandwhatyouareexpectedtotest.Typically,supplierisexpectedtodothorough
(e.g.)testingofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,
usabilityetc.)beforehandingoffsystemforAT.Doesthesupplierhavetheenvironmentsfor
integrationtestingwithoutinhousesystems?
6. Delegatedecisions:Ifyoudonthaveenoughtimeorunderstandingtotakepartinthe
project,assignaproxytorepresentyouandhaveaclearunderstandingofthekindsof
decisionstheycanmakewith/withoutconsultingwithyou.
7. Delegatework/testing:Haveaclearunderstandingofyour(organizations)abilitiesw.r.t.
testingskills.Contractotherpartiestohelpyoutestifnecessary.Butdontunderestimate
theeffortinvolvedinmanagingthetestingoutsourcerastheywillneedtolearnyour
businessdomainandrequirementsbeforetheycanbeeffective.
8. Understandthetestenvironments:Understandwhereyouwilltest.Beclearonhowyoull
getthenewreleasecandidatesdelivered
9. Viewtestsasrequirements:Acceptancecriteria/testsshouldbearticulated/prepared
beforethesoftwareisbuiltandsharedwithvendorasawaytofleshoutthedetailsofthe
requirements.
10. Recognizethechoiceofprocessimpactseverything:TwokeystylesofAT:FinalTestPhase
andIncrementalAT.Theformerconcentratesmostacceptanceactivitiesattheendofthe
project.Incrementalacceptancetestingspreadsouttheworkandreducesriskbutrequires
allpartiestoworkindifferentways.

11. Estimateacceptanceeffort&durationwhilerealizingitisdifficulttoestimate:Estimating
theeffort&durationofacceptancetestingishard;timeboxingthefinalacceptancetesting
cycleiscommonbutrequirespastexperiencetocomeupwithreasonableduration/effort.If
indoubt,guesshighandallowforseveralcyclesofacceptancetestingtogivethevendor
timetofixbugs.
12. Useincrementalacceptancetestingtoreduceuncertaintyofestimating:Canavoidsomeof
theuncertaintyandmostoftheriskbydoingincrementalacceptancetesting.Bugsfound
earlieridentifyambiguityinthespecwhilethereisstilltimetoaddressitandcanbefixed
longbeforethefinalacceptancetestphase.
KeyPointsforProductManager
AllBusinessLeadsKeyPointsplus:
Understandthedecisionmakingprocess:Makesureyouhaveaclearunderstandingofthe
decisionmakingprocess.OtherpotentialdecisionmakersincludeTestManagers,Security
Manager,UserExperience/UsabilityManager,Architect,LegalDepartment,etc.Ensureyou
haveagreementonwhethertheymakeadecisionorprovidedataorarecommendationto
you.
Recognizethatcomplexproductsandorganizationshavecomplexdecisionmaking
processes:Thereareseveraldifferentwaystoorganizetheteamstodealwithverylarge
productsorproductsthataretheamalgamationofseveralproducts/productlines(e.g
SharePoint=Server+Office+IE).Thechoiceshouldconsidertheresultingdecisionmaking
model(e.g.componentteams,featureteams.)
Communicateinformationflowclearly:Ensureanyonewhoyouexpecttoprovideyouwith
dataonwhichtobaseyourdecisionunderstandswhatdatatheyareprovidingyouand
when.E.g.TheTestManagerprovidestestresultstoyouandyoumaketheacceptance
decision.
Ensureallsuppliersunderstandusersandusagemodel:Makesureyouunderstandthe
usersandcommunicatethistobothsupplierandtestorganizations.
Understandtheoperationalrequirements:Makesureyouunderstandtheoperational
requirementsforyoursystem.Whetheryouaresellingasoftwarepackagetocustomersor
sellingSoftwareasaService(SaaS),beawarethatwhoeveroperatesthesoftwarewill
haveoperationalrequirementsoverandabovethefunctionalrequirementsoftheend
users.

KeyPointsforTestManager
Determineyourtestmandate:Makesureyouunderstandwhetheryouaredoingreadiness
assessmentoracceptancetestingorboth.Foracceptancetestingbesurethetestershave
realistictestsfortheuserstheyproxy.Considerusingpersonas.Forreadinessassessmentit
isimportanttohaveagoodrelationshipwiththedevelopmentteamtohelpthem
understandthequalityoftheproductandwhatwillimproveit.
Clarifydecisionmakingvsdecisionsupporting:Makesureyouhaveaclearunderstanding
ofwhetheryouarethegatekeeper(acceptancedecisionmaker)orasupplierofinformation
totheProductManagerforhertomakethedecision(preferred)
Getagreementfromtheproductownerthatthetestcasesreflecttherequirements:Aim
towritetheacceptancetestcasesfirstandprovidethemtothedevelopmentteamsbefore
thesoftwareisbuilttobeusedasaidestoassistinunderstandinghowthesystemshould
behave.ThisisknownasAcceptanceTestDrivenDevelopment.
Planamultiprongedteststrategy:Planforamultiprongedteststrategythatincludes
incrementaltestingandbothscriptedandexploratorytests.
Collaboratewiththedevelopmentteam:Aimforanincrementaldeliveryoffunctionalityto
supportincrementalacceptancetesting.Workwiththedevelopmentteamtodesignfor
testabilitytofacilitateautomatedtestingandenablingthemtobeabletoruntestson
demand.
Dotestautomationwhenfeasibleandinthewayitsupports(notdictates!)meetingyour
testobjectives.Considerhavingadedicatedtestautomationengineer.
Agreeonadefinitionofwhatdoneis:Getthevariouspartiesinvolvedtoagreeona
definitionofwhatdoneisforsoftwarebeforeitgoesthrougheachofthequalitygatesofthe
gatingmodelandincludeachecklistofrequirements.
KeyPointsforDevelopmentManager
Performreadinessassessments:Allsoftwareshouldbeadequatelytestedbythe
developmentteambeforepresentingittothecustomer(s).
Agreeontheminimumcrediblerequirement(MCR)aheadoftime:Theproductowner,test
organizationandthesuppliershouldallagreeontheMCRaheadoftime,thatis,beforethe
softwareisbuilt.
Useacceptanceteststodefinesuccess:UseacceptancetestsbasedontheMQRtodefine
successbeforethesoftwareisbuiltandensurethateveryoneonthedevelopmentteamhas
accesstothemandrunsthetestsastheybuildthesoftware.
Runthetestsaspartofreadinessacceptance:Runalltheteststhattheproductownerwill
runandsomeyouvedefined.

Minimizeuntestedcode:Testasyougo,ideallyincrementaltesting.
Usetestingtopreventbugs:Testscanhelppreventbugsbeingintroducedduringthe
developmentcycle.Includefunctionaltesting,component/unittests,andparafunction
testingasearlyaspossible.
KeyPointsforUserExperienceSpecialist
Understandyourrole:Haveaclearunderstandingwiththeproductowner,product
managerorbusinessleadwhatyourroleshallbeinmakingtheacceptancedecision.
Maximizelearning:Workwiththeproductownertoensurethateveryoneontheteamhas
accesstothedefinitionofsuccessintheformofacceptancetestsbeforethesoftwareis
built.
Definetheusabilitycomponentoftheacceptancecriteriaasearlyaspossible:Workwith
thebusinessleadorproductmanagertodefinetheusabilitycomponentoftheacceptance
criteriaasearlyaspossible.
Addresstheusabilityrisksasearlyaspossible:Theareaswithhighestusabilityimpact
shouldbebuiltandtestedfirstsothatanyusabilityrelatedissuescanbeaddressedasearly
aspossibleinthedevelopmentcycle.
KeyPointsforOperationsManager
Understandroleinacceptancedecision:Youandeveryoneconcernedshouldbeclearon
yourroleinthemakingoftheacceptancedecision.Areyouadataprovidertothe
acceptancedecisionmakeroranacceptancedecisionmaker?
Identifywaystoreducethenumberoftest&fixcycles:Workwiththedevelopment
managertoidentifywaystoreducethenumberoftest&fixcyclesthroughearlydeliveryof
testablesoftwareandincrementalacceptancetesting.
Defineacceptancefromtheoperationalperspectiveasearlyaspossible:Ideallyyouhave
clearlydefinedacceptancecriteriabeforethesoftwareisbuilt.
KeyPointsforSolutionArchitect
Describethewholesolutiontoeveryone:Communicatethesolutioninitsentiretytothe
wholesupplierteamtoavoidtheMypartworksjustfine(evenifthesolutiondoesnt!)
mentality.
Ensureavailabilityofacceptancetests:Workwiththerequirementsanalystsandtestersto
ensurethattheacceptancetestsareavailabletothesupplierteambeforethesoftwareis
built.

Educateabouttradeoffs:Workwiththeproductownertohelpthemunderstandwhatkinds
offunctionalityiseasytobuildandwhatishard.Helpthemgetthemostbangfortheir
money.
Unbundlefunctionality:Workwiththeproductownertohelpthemidentifythelowervalue
functionalitythatcouldpotentiallybedeferredincasethereisaneedtotradeoff
functionalityforqualityordeliverydate.
DefineSLAmetrics
Engageenterprisearchitects:Engagetheenterprisewidearchitects(responsiblefor
infrastructure,standards,technologydecisions,longtermvision,security,dataarchitecture,
etc.)earlysotheycanhelpyouunderstandwhatrequirestheirapprovalbeforeitbecomes
partofthecriticalpath.
KeyPointsforEnterpriseArchitect
Publisharchitecturalacceptancecriteria:Publishgeneralacceptancecriteriasothatall
supplierteamsunderstandtherulesofthegameandwhatkindsofapprovalstheyneedto
getfromyou.
ArticulatecrosssystemintegrationrequirementsandSLAs:Identifyapplicableintegration
requirementsandservicelevelagreements.Makesurethesupplierteamisawareofthem
andthattheyareacceptancecriteria.
Ensurestandardscompliance:Verifythatproductscomplywithapplicablestandards.
Encouragereuse:Verifythatproductteamsareawareofandcapitalizeonopportunitiesto
reusecomponentsanddata.
Engagesupplierteamsearly:Engagesupplierteamsearlytoensurethatinspectionsprevent
defectsratherthanfindthem.
Delegatenoncriticaldecisions:Ifitisnotimportanttobeinvolvedinadecisionthenbow
outandlettheprojectteammakethedecision.
Keepapprovalprocesslightweight:Maketheprocessforseekingguidanceandapprovalas
lightweightaspossible.Emphasizeeffectivenessofcommunicationovercomprehensive
documentation.Avoidoffloadingwork(e.g.moredocumentation)totheprojectteamsto
saveyoueffortifthatresultsinincreasingthetotalwork.
Bepragmatic:Bepragmaticastowhatprogressindividualprojectorproductteamsare
expectedtomaketowardslongtermarchitecturalobjectives.Theyareresponsibletothe
productowner,nottoyou.

KeyPointsforLegal
Includeclearstipulationsofanycontractualimplications:Acceptancetypicallyisa
conditionforrenderingapaymentandaffectstheapplicationofanywarrantyprovisionsand
potentiallyanyremediesavailabletothecustomer.
Discourageonesidedcontracts:Thecontractshouldencouragebothpartiestogettoa
mutuallyagreeableacceptanceasquicklyaspossible.Contractsthatpenalizeoneparty
oftenleadtodysfunctionalbehaviorthatresultsinloseloseoutcomes.
Clearlydefinehowmuchtimeisallowedforacceptancetests:Acceptancetestingisalways
timeboxed.Thereisalimitedamountoftimeforacceptanceandreportingdeficiencies.
Includeacleardefinitionofacceptancecriteria:Besidestestingperformedatthelocationof
supplier,andfunctionalandoperationaltestsperformedwhenthesoftwareisdeployedto
thecustomer,softwaresystemoftenarerequiredtosuccessfullyrunwithcurrentdataina
liveoperatingenvironmentforaperiodoftime,withminimumbugs,beforethesystemis
formallyaccepted.
Knowtheexplicitandimplicitcontractualconditions:Acceptancecanbeviewedintermsof
explicitprovisionsspecifiedinthesoftwaredevelopmentcontractandthetermsimpliedinto
thecontractbylaw(whichalsodependsonthegeographicjurisdiction).


Acceptance Test Engineering Guide Vol I RC1 Full 102609 PDF

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Acceptance Test Engineering Guide Vol I RC1 Full 102609 PDF

Diunggah oleh

Hak Cipta:

Format Tersedia

AcceptanceTest

Classic Incremental Agile (Iteration) Agile (Kanban)

Number of releases 1 1 2 or more 2 or more

Number of iterations 1 26 4 or more 1

Maximum number of No maximum No maximum 1 iterations worth Less than the

Integration frequency Big Bang Quarterly Daily or hourly Daily or hourly

Requirement-to-test Months or years Months Days Days

Test timing Separate phase Separate phase Mostly incremental Mostly

Release criteria Scope-based Scope-based Time-boxed Time-boxed

Culture Hierarchical Hierarchical Collaborative Collaborative

Skills Highly Highly Generalists Generalists or

Determining progress Work Work completed Delivery of working Delivery of

Work remaining Estimate Estimate Estimated time for Estimated time

Analysis Business Tests Acceptance

Coding Units Tests Unit

[GouldLew wisOnDesignFForUsability]canbeusedttoverifythatyouare"buildingtherigh htsystem"very

Attribute Goal Importance Rationale

Reliability/availability 7x24x52 Critical Users require instant

Usability Easily discoverable High Most users will use

Low Medium High

answeris"Howmuchttimedoweh have?"Frequeently,thetimeavailableisnotlongeno oughtogatheer

140 Initial Scope NeededExtra

Testers50% Testers50% Testers50%

Inadequate design 25%

Insufficient information 30% Insufficient information 30%

Prepare Security Operations Prepare CMB

Insufficient information Insufficient information 30%

Contents Primarypersonas Secondarypersonas

Anda mungkin juga menyukai