E-ffective Testing for E-commerce

By Angelina Samaroo, Steve Allott and Brian Hambling, ImagoQA, UK

1. 2.

Introduction
What is e-commerce? For the purposes of this paper, e-commerce (also known as ebusiness) is defined as the software and business processes required to allow businesses to operate solel or primaril usin! di!ital data flows. "-commerce is often associated with web technolo! and is commonl transacted #ia web portals, but e-commerce is much more than the pro#ision of a web pa!e as the customer interface. $he creation of inte!rated business processes ("nterprise %esource &lannin!), the inte!ration of collections of disparate software applications, each desi!ned to facilitate a different aspect of the business ("nterprise 'pplication (nte!ration), the e)tension of software and business processes to embrace transactions with suppliers* s stems (+uppl ,hain -ana!ement), the need for increased securit for transactions o#er public networks, and the potential #olume demand at e-commerce sites all pro#ide new and unique challen!es to the e-commerce de#elopment communit . challen!es which will require no#el and inno#ator solutions and which will need thorou!h testin! before the are allowed to !o li#e. Wh is testin! important in the e-commerce en#ironment? $he first and primar reason is because e-commerce is, b its #er nature, business critical. (n the third quarter of 1//0, 1ell*s e-commerce site e)ceeded 213 million in dail sales4 the "5$rade site currentl e)ceeds 62, 333 transactions per da , !i#in! a cost of one-da failure of around 2033,3334 and the tra#el industr in "urope will be worth 22 billion b 2332, accordin! to 1atamonitor. $he immediac of the customer, with its implied promise of rapid deli#er at competiti#e prices, and the sheer accessibilit of the web, all combine to create potentiall massi#e demand on web sites and portals. $he second reason is that e-commerce is a massi#e and !rowin! market place but one which requires lar!e up-front in#estment to enter successfull . $here are alread 6.0 million web sites worldwide, 2.6 million of which ha#e been created this ear (1///). $he (nternational 1ata ,orporation ((1,) estimates that the e-commerce market will !row from o#er 26billion in 1//0 to 21trillion in 2337. $he a#era!e cost of de#elopment of an e-commerce site is 21 million, sa s the 8artner 8roup, and will increase b 269 annuall o#er the ne)t 2 ears. $he third reason is because the histor of e-commerce de#elopment has been littered with e)pensi#e failures, at least some of which could ha#e been a#oided b better testin!

before the site was opened to the !eneral public. ((n e-commerce terms, :the site* means the entire architecture from suppliers throu!h back-end s stems and front-end s stems to the customers4 it t picall includes (ntranet, (nternet and e)tranet applications as well as le!ac s stems and third part middleware). The Testing Challenges Business Issues ' successful e-commerce application is; <sable. &roblems with user interfaces lose clients. +ecure. &ri#ac , access control, authentication, inte!rit and non-repudiation are bi! issues. +caleable. +uccess will brin! increasin! demand. %eliable. Failure is unthinkable for a business critical s stem. -aintainable. =i!h rates of chan!e are fundamental to e-commerce. =i!hl a#ailable. 1owntime is too e)pensi#e to tolerate. $hese characteristics relate in part to the web technolo! that usuall underlies ecommerce applications, but the are also dependent on effecti#e inte!ration and effecti#e back-end applications. "-commerce inte!rates hi!h #alue, hi!h risk, hi!h performance business critical s stems, and it is these characteristics that must dominate the approach to testin! because it is these characteristics that determine the success of e-commerce at the business le#el. Technical Issues $he de#elopment process for e-commerce has unique characteristics and some associated risks. (t is !enerall reco!nised that a :web ear* is about 2 months lon!. (n other words, a credible update strate! would need to !enerate e-commerce site updates rou!hl monthl . For this reason, %apid 'pplication 1e#elopment (%'1) techniques predominate in the e-commerce en#ironment, and in some cases de#elopment is e#en done directl in a production en#ironment rather than in a separate de#elopment en#ironment. %'1 techniques are not new, and it is !enerall a!reed that the work best where functionalit is #isible to the user . so web site de#elopment would seem to be an ideal application area. <nfortunatel , thou!h, other aspects of e-commerce are at least as important as the front-end. $he end-to-end inte!ration of business processes and the consequent se#ere constraints placed on intermediate processes make them less than ideal application areas for %'1. $hese chan!es increase risk and create new challen!es for testers, because time pressures militate a!ainst spendin! a lon!er time testin! sites before the are released. 't the same time, the technical en#ironment of front-end s stems is chan!in! #er rapidl , so chan!e is imposed on e-commerce sites e#en when the site itself is not chan!in!. $his requires more re!ression testin! than would be e)pected in a con#entional application to ensure

that the site continues to function acceptabl after chan!es to browsers, search en!ines and portals. >ew issues ha#e also come to the fore for testers, notabl securit of transactions and the performance of web sites under hea# load conditions. (f we consider an e-commerce site as made up of a front end (the human-computer interface), a back end (the software applications underl in! the ke business processes) and some middleware (the inte!ratin! software to link all the rele#ant software applications), we can consider each component in isolation. Front End Systems Static Testing. $he front end of an e-commerce site is usuall a web site that needs testin! in its own ri!ht. $he site must be s ntacticall correct, which is a fairl strai!htforward issue, but it must also offer an acceptable le#el of ser#ice on one or more platforms, and ha#e portabilit between chosen platforms. (t should be tested a!ainst a #ariet of browsers, to ensure that ima!es seen across browsers are of the same qualit . <sabilit is a ke issue and testin! must adopt a user perspecti#e. For e)ample, the functionalit of buttons on a screen ma be acceptable in isolation, but can a user na#i!ate around the site easil and does information printed from the site look !ood on the pa!e when printed? (t is also important to !ain confidence in the securit of the site. -an of these tests can be automated b creatin! and runnin! a file of t pical user interactions . useful for re!ression testin! and to sa#e time in checkin! basic functionalit . Dynamic Testing. 'pplications attached to an e-commerce site, either b ,8( pro!rammin! or ser#er e)tensions, will need to be tested b creatin! scenarios that !enerate calls to these attached applications, for e)ample b requirin! database searches. $he ser#ices offered to customers must be s stematicall e)plored, includin! the turnaround time for each ser#ice and the o#erall ser#er response. $his, too, must be e)ercised across alternati#e platforms, browsers and network connections. "-commerce applications are essentiall transaction-oriented, based on ke business processes, and will require effecti#e interfacin! between intranet-based and e)tranet-based applications. Bac End Systems $he back end of e-commerce s stems will t picall include "%& and database applications. ?ack end testin!, therefore, is about business application testin! and does not pose an new or poorl understood problems from a business perspecti#e, but there are potential new technical problems, such as ser#er load balancin!. Fortunatel , clientser#er s stem testin! has tau!ht the testin! communit man #aluable lessons that can be applied in this situation. What is essential, howe#er, is to appl the ke front end testin! scenarios to the back end s stems. (n other words, the back end s stems should be dri#en b the same real transactions and data that will be used in front end testin!. $he back end ma well pro#e to be a bottleneck for user ser#ices, so performance under load and scalabilit are ke issues to be addressed. +ecurit is an issue in its own ri!ht, but also has potential to impact on performance.

!iddle"are and Integration (nte!ration is the ke to e-commerce. (n order to build an e-commerce application, one or more of the followin! components are usuall inte!rated; 1atabase +er#er +er#er-side application scripts@pro!rams 'pplication ser#er =$-A forms for user interface 'pplication scripts on the client &a ment ser#er +cripts@pro!rams to inte!rate with le!ac back-end s stems

$he process of de#elopin! an e-commerce site is si!nificantl different from de#elopin! a web site . commerce adds e)tra le#els of comple)it . Bne hi!hl comple) feature is that of inte!ration. (f an application is bein! built that uses a database ser#er, web ser#er and pa ment ser#er from different #endors, there is considerable effort in#ol#ed in networkin! these components, understandin! connecti#it -related issues and inte!ratin! them into a sin!le de#elopment (e)ecutable) en#ironment. (f le!ac code is in#ol#ed, this adds a new dimension to the problem, since time will need to be in#ested in understandin! the interfaces to the le!ac code, and the likel impact of an chan!es. (t is also crucial to keep in mind the steep learnin! cur#e associated with cuttin!-ed!e technolo!ies. Ceepin! pace with the latest #ersions of the de#elopment tools and products to be inte!rated, their compatibilit with the pre#ious #ersions, and in#esti!atin! all the new features for buildin! optimal solutions for performance can be a dauntin! task. 'lso, since e-commerce applications on the web are a relati#el new phenomenon, there are unlikel to be an metrics on similar proDects to help with proDect plannin! and de#elopment. $he maintenance tasks of installin! and up!radin! applications can also become #er in#ol#ed, since the demand e)pertise in; 1atabase administration. Web ser#er administration. &a ment ser#er administration. 'dministration of an other special tools that ha#e been inte!rated into the site.

$echnical support should also be borne in mind. ,orrectl functionin! back-end and front-end s stems offer no !uarantees of reliable o#erall functionalit or performance. "nd-to-end testin! of complete inte!rated architectures, usin! realistic transactions, is an essential component.
E

Ten Key Principles of Effective E-Commerce Testing

B#er the decades since (nformation $echnolo! (($) became a maDor factor in business life, problems and challen!es such as those now faced b the e-commerce communit ha#e been met and sol#ed. Ce testin! principles ha#e emer!ed and these can be successfull applied to the e-commerce situation. Principle 1. Testing is a risk management process. $he most important lesson we ha#e learned about software testin! is that it is one of the best mechanisms we ha#e for mana!in! the risk to businesses of unsuccessful ($ applications. "ffecti#e testin! adopts a strate! that is tailored to the t pe of application or ser#ice bein! tested, the business #alue of the application or ser#ice, and the risks that would accompan its failure. $he detailed plannin! of the testin! and the desi!n of the tests can then be conformed b the strate! into a business-focused acti#it that adds real business #alue and pro#ides some obDecti#e assessment of risk at each sta!e of the de#elopment process. &lans should include measures of risk and #alue and incorporate testin! and other qualit -related acti#ities that ensure de#elopment is properl focused on achie#in! ma)imum #alue with minimum risk. %eal proDects ma not achie#e e#er thin! that is planned, but the metrics will at least enable us to decide whether it would be wise to release an application for li#e use. Principle 2. Kno the value of the applications !eing tested. $o mana!e risk effecti#el , we must know the business #alue of success as well as the cost of failure. $he business communit must be in#ol#ed in settin! #alues on which the risk assessment can be based and committed to deli#erin! an a!reed le#el of qualit . Principle ". #et clear testing o!$ectives and criteria for successful completion %including test coverage measures&. When testin! an e-commerce site, it would be #er eas for the testin! to de!enerate into surfin!, due to the ease of searchin! related sites or another totall unrelated site. $his is wh the test pro!ramme must be properl planned, with test scripts !i#in! precise instructions and e)pected results. $here will also need to be some cross-referencin! back to the requirements and obDecti#es, so that some assessment can be made of how man of the requirements ha#e been tested at an !i#en time. ,riteria for successful completion are based on deli#erin! enou!h business #alue, testin! enou!h of the requirements to be confident of the most important beha#iour of the site, and minimisin! the risk of a si!nificant failure. $hese criteria . which should be a!reed with the business communit - !i#e us the critical e#idence that we need in decidin! readiness to make the site accessible to customers. Principle '. Create an effective test environment. (t would be #er e)pensi#e to create a completel representati#e test en#ironment for e-commerce, !i#en the #ariet of platforms and the use of the (nternet as a communications medium. ,ross-platform testin! is, naturall , an important part of testin! an multi-platform software application. (n the case of e-commerce, the term :cross-platform* must also e)tend to include :crossbrowser*. (n order to ensure that a site loads and functions properl from all supported

platforms, as much stress and load testin! as possible should be performed. 's an absolute minimum, se#eral people should be able to lo! into the site and access it concurrentl , from a mi)ture of the browsers and platforms supported. $he !oal of stress and load testin!, howe#er, is to subDect the site to representati#e usa!e le#els. (t would, therefore, be beneficial to use automated tools, such as +e!ue*s +ilk&erformer or -ercur (nteracti#e*s Aoad%unner, for performance@load testin!. Principle (. Test as early as possi!le in the development cycle. (t is alread well understood and accepted in the software en!ineerin! communit that the earlier faults are detected, the cheaper the cost of rectification. (n the case of an e-commerce site, a fault found after shippin! will ha#e been detected as a failure of the site b the marketplace, which is potentiall as lar!e as the number of (nternet users. $his has the added complication of loss of interest and possibl the loss of customer lo alt , as well as the immediate cost of fi)in! the fault. $he fact that e-commerce de#elopment is rapid and often based on chan!in! requirements makes earl testin! difficult, but testin! strate!ies ha#e been de#eloped b the %'1 communit , and these can be mobilised for support. &erhaps the most important idea in %'1 is the Doint de#elopment team, allowin! users to interact with the de#elopers and #alidate product beha#iour continuousl from the be!innin! of the de#elopment process. %'1 utilises product protot pes, de#eloped in a series of strictl controlled :timebo)es* . fi)ed periods of time durin! which the protot pe can be de#eloped and tested . to ensure that product de#elopment does not drift from its ori!inal obDecti#es. $his st le of web de#elopment makes testin! an inte!ral part of the de#elopment process and enhances risk mana!ement throu!hout the de#elopment c cle. Principle ). *ser +cceptance Testing %*+T&. $he client or ultimate owner of the ecommerce site should perform field testin! and acceptance testin!, with in#ol#ement from the pro#ider where needed, at the end of the de#elopment process. "#en if %'1 is used with its continuous user testin! approach, there are some attributes of an ecommerce site that will not be eas (or e#en possible, in some cases) to #alidate in this wa . +ome form of final testin! that can address issues such as performance and securit needs to be included as a final confirmation that the site will perform well with t pical user interactions. Where %'1 is not used, the scope of the pro#ider*s internal testin! co#era!e and user acceptance testin! co#era!e should be defined earl in the proDect de#elopment lifec cle (in the $est &lan) and re#isited as the proDect nears completion, to assure continued ali!nment of !oals and responsibilities. <'$, howe#er, should not be seen as a beta-testin! acti#it , dele!ated to users in the field before formal release. "commerce users are becomin! increasin!l intolerant of poor sites, and technical issues related to functionalit , performance or reliabilit ha#e been cited as primar reasons wh customers ha#e abandoned sites. "arl e)posure of users to sites with problems increases the probabilit that the will find the site unacceptable, e#en if de#elopers continue to impro#e the site durin! beta testin!. Principle ,. -egression testing. 'pplications that chan!e need re!ression testin! to confirm that chan!es did not ha#e unintended effects, so this must be a maDor feature of an e-commerce testin! strate! . Web-based applications that reference e)ternal links

need re!ular re!ression testin!, e#en if their functionalit does not chan!e, because the en#ironment is chan!in! continuousl . Where#er possible, re!ression testin! should be automated, in order to minimise the impact on the test schedule. Principle .. +utomate as much as possi!le. $his is a risk principle because test automation is frau!ht with difficulties. (t has been said that a fool with a tool is still a fool, and that the outcome of automatin! an unstable process is faster chaos, and both of these are true. >e#ertheless, the chances of !ettin! adequate testin! done in the ti!ht time scales for an e-commerce proDect and without automation are e)tremel slim. $he ke is to take testin! processes sufficientl seriousl that ou document them and control them so that automation becomes a feasible option . then ou select, purchase and install the tools. (t will not be quick or cheap . but it mi!ht Dust a#oid a #er e)pensi#e failure. Principle /. Capture test incidents and use them to manage risk at release time. ' test incident is an discrepanc between the e)pected and actual results of a test. Bnl some test incidents will relate to actual faults4 some will be caused b incorrect test scripts, misunderstandin!s or deliberate chan!es to s stem functionalit . 'll incidents found must be recorded #ia an incident mana!ement s stem ((-+), which can then be used to ascertain what faults are outstandin! in the s stem and what the risks of release mi!ht be. Butstandin! incidents can be one of the completion criteria that we appl , so the abilit to track and e#aluate the importance of incidents is crucial to the mana!ement of testin!. Principle 10. 1anage change properly to avoid undoing all the testing effort. $hin!s chan!e quickl and often in an e-commerce de#elopment and mana!ement of chan!e can be a bottleneck, but there is little point in testin! one #ersion of a software application and then shippin! a different #ersion4 not onl is the testin! effort wasted, but the risk is not reduced either. ,onfi!uration -ana!ement tools, such as &G,+ and ,lear,ase, can help to minimise the o#erheads of chan!e mana!ement, but the discipline is the most important thin!. Conclusions "-commerce is both familiar and no#el. +ome of the technolo! is relati#el no#el, and the application of that technolo! to a complete business is certainl no#el, but the problems of creatin! business processes to operate a business in a wholl new en#ironment o#ershadow all of that no#elt with some familiar and intractable problems. &arado)icall , it is in the more familiar areas of the technolo! that the most serious problems arise, because the emer!ence of e-commerce has placed new and challen!in! requirements on this relati#el old technolo! that was desi!ned for a quite different purpose. $estin! is crucial to e-commerce because e-commerce sites are both business critical and hi!hl #isible to their users4 an failure can be immediatel e)pensi#e in terms of lost re#enue and e#en more e)pensi#e in the lon!er term if disaffected users seek alternati#e sites. Het the time pressures in the e-commerce world militate a!ainst the thorou!h

testin! usuall associated with business criticalit , so a new approach is needed to enable testin! to be inte!rated into the de#elopment process and to ensure that testin! does not present a si!nificant time burden. $he #er familiarit of much of the technolo! means that tried and true mechanisms will either be suitable or can be modified to fit. %apid 'pplications 1e#elopment (%'1), in particular, su!!ests some promisin! approaches. Aike most new #entures, thou!h, ecommerce must find its own wa and establish its own methods. (n this paper we ha#e su!!ested some testin! principles that ha#e stood the test of time and intermin!led them with some lessons learned from similarl challen!in! de#elopment en#ironments to !i#e e-commerce testers a starin! point for their Dourne of disco#er .