LiveTech

Software QA and Testing Frequently-Asked-Questions

1) What is 'Software Quality Assurance'?
2) What is 'Software Testing'?
3) What are some recent major computer system failures cause !y
software !ugs?
") Why is it often har for management to get serious a!out #uality assurance?
$) Why oes software ha%e !ugs?
&) 'ow can new Software QA processes !e introuce in an e(isting organi)ation?
*) What is %erification? +aliation?
,) What is a 'wal-through'?
.) What's an 'inspection'?
1/) What -ins of testing shoul !e consiere?
11) What are $ common pro!lems in the software e%elopment process?
12) What are $ common solutions to software e%elopment pro!lems?
13) What is software '#uality'?
1") What is 'goo coe'?
1$) What is 'goo esign'?
1&) What is S01? 233? 2331? 1S4? Will it help?
1*) What is the 'software life cycle'?
1,) Will automate testing tools ma-e testing easier?
1.) What ma-es a goo Software Test engineer?
2/) What ma-es a goo Software QA engineer?
21) What ma-es a goo QA or Test manager?
22) What's the role of ocumentation in QA?
23) What's the !ig eal a!out 're#uirements'?
2") What steps are neee to e%elop an run software tests?
2$) What's a 'test plan'?
2&) What's a 'test case'?
2*) What shoul !e one after a !ug is foun?
2,) What is 'configuration management'?
2.) What if the software is so !uggy it can't really !e teste at all?
3/) 'ow can it !e -nown when to stop testing?
31) What if there isn't enough time for thorough testing?
32) What if the project isn't !ig enough to justify e(tensi%e testing?
33) What can !e one if re#uirements are changing continuously?
3") What if the application has functionality that wasn't in the re#uirements?
3$) 'ow can Software QA processes !e implemente without stifling proucti%ity?
3&) What if an organi)ation is growing so fast that fi(e QA processes are impossi!le?
3*) 'ow oes a client5ser%er en%ironment affect testing?
3,) 'ow can Worl Wie We! sites !e teste?
3.) 'ow is testing affecte !y o!ject6oriente esigns?
"/) What is 0(treme 7rogramming an what's it got to o with testing?
What is 'Software Quality Assurance'?
Software QA in%ol%es the entire software e%elopment 78420SS 6 monitoring an impro%ing the process9
ma-ing sure that any agree6upon stanars an proceures are followe9 an ensuring that pro!lems are
foun an ealt with: 1t is oriente to 'pre%ention':
What is 'Software Testing'?
Testing in%ol%es operation of a system or application uner controlle conitions an e%aluating the results
QA and Testing FAQs 7age 1 of 1*
LiveTech
;e:g:9 'if the user is in interface A of the application while using harware <9 an oes 29 then = shoul
happen'): The controlle conitions shoul inclue !oth normal an a!normal conitions: Testing shoul
intentionally attempt to ma-e things go wrong to etermine if things happen when they shouln't or things
on't happen when they shoul: 1t is oriente to 'etection':
4rgani)ations %ary consiera!ly in how they assign responsi!ility for QA an testing: Sometimes
they're the com!ine responsi!ility of one group or ini%iual: Also common are project teams
that inclue a mi( of testers an e%elopers who wor- closely together9 with o%erall QA processes
monitore !y project managers: 1t will epen on what !est fits an organi)ation's si)e an !usiness
structure:
Why is it often hard for anageent to get serious a!out quality assurance?
Sol%ing pro!lems is a high6%isi!ility process> pre%enting pro!lems is low6%isi!ility: This is illustrate !y an
ol para!le?
1n ancient 2hina there was a family of healers9 one of whom was -nown throughout the lan an employe
as a physician to a great lor: The physician was as-e which of his family was the most s-illful healer: 'e
replie9
@1 ten to the sic- an ying with rastic an ramatic treatments9 an on occasion someone is cure an
my name gets out among the lors:@
@3y eler !rother cures sic-ness when it just !egins to ta-e root9 an his s-ills are -nown among the local
peasants an neigh!ors:@
@3y elest !rother is a!le to sense the spirit of sic-ness an eraicate it !efore it ta-es form: 'is name is
un-nown outsie our home:@
Why does software have !ugs?
3iscommunication or no communication 6 as to specifics of what an application shoul or
shouln't o ;the application's re#uirements):
Software comple(ity 6 the comple(ity of current software applications can !e ifficult to
comprehen for anyone without e(perience in moern6ay software e%elopment: 3ulti6tiere
applications9 client6ser%er an istri!ute applications9 ata communications9 enormous relational
ata!ases9 an sheer si)e of applications ha%e all contri!ute to the e(ponential growth in
software5system comple(ity:
7rogramming errors 6 programmers9 li-e anyone else9 can ma-e mista-es:
changing re#uirements ;whether ocumente or unocumente) 6 the en6user may not unerstan
the effects of changes9 or may unerstan an re#uest them anyway 6 reesign9 rescheuling of
engineers9 effects on other projects9 wor- alreay complete that may ha%e to !e reone or thrown
out9 harware re#uirements that may !e affecte9 etc: 1f there are many minor changes or any
major changes9 -nown an un-nown epenencies among parts of the project are li-ely to interact
an cause pro!lems9 an the comple(ity of coorinating changes may result in errors: 0nthusiasm
of engineering staff may !e affecte: 1n some fast6changing !usiness en%ironments9 continuously
moifie re#uirements may !e a fact of life: 1n this case9 management must unerstan the
resulting ris-s9 an QA an test engineers must aapt an plan for continuous e(tensi%e testing to
-eep the ine%ita!le !ugs from running out of control 6 see 'What can !e one if re#uirements are
changing continuously?' in 7art 2 of the AAQ: Also see information a!out 'agile' approaches such
as B79 also in 7art 2 of the AAQ:
time pressures 6 scheuling of software projects is ifficult at !est9 often re#uiring a lot of
guesswor-: When ealines loom an the crunch comes9 mista-es will !e mae:
egos 6 people prefer to say things li-e?
'no pro!lem'
'piece of ca-e'
'1 can whip that out in a few hours'
'it shoul !e easy to upate that ol coe'
QA and Testing FAQs 7age 2 of 1*
LiveTech
instea of?
'that as a lot of comple(ity an we coul en up
ma-ing a lot of mista-es'
'we ha%e no iea if we can o that> we'll wing it'
'1 can't estimate how long it will ta-e9 until 1
ta-e a close loo- at it'
'we can't figure out what that ol spaghetti coe
i in the first place'
1f there are too many unrealistic 'no pro!lem's'9 the
result is !ugs:
poorly ocumente coe 6 it's tough to maintain an moify coe that is !aly written or poorly
ocumente> the result is !ugs: 1n many organi)ations management pro%ies no incenti%e for
programmers to ocument their coe or write clear9 unerstana!le9 maintaina!le coe: 1n fact9 it's
usually the opposite? they get points mostly for #uic-ly turning out coe9 an there's jo! security if
no!oy else can unerstan it ;'if it was har to write9 it shoul !e har to rea'):
software e%elopment tools 6 %isual tools9 class li!raries9 compilers9 scripting tools9 etc: often
introuce their own !ugs or are poorly ocumente9 resulting in ae !ugs:
"ow can new Software QA #rocesses !e introduced in an e$isting organi%ation?
A lot epens on the si)e of the organi)ation an the ris-s in%ol%e: Aor large organi)ations with
high6ris- ;in terms of li%es or property) projects9 serious management !uy6in is re#uire an a
formali)e QA process is necessary:
Where the ris- is lower9 management an organi)ational !uy6in an QA implementation may !e a
slower9 step6at6a6time process: QA processes shoul !e !alance with proucti%ity so as to -eep
!ureaucracy from getting out of han:
Aor small groups or projects9 a more a6hoc process may !e appropriate9 epening on the type of
customers an projects: A lot will epen on team leas or managers9 fee!ac- to e%elopers9 an
ensuring ae#uate communications among customers9 managers9 e%elopers9 an testers:
The most %alue for effort will often !e in ;a) re#uirements management processes9 with a goal of
clear9 complete9 testa!le re#uirement specifications em!oie in re#uirements or esign
ocumentation9 or in 'agile'6type en%ironments e(tensi%e continuous coorination with en6users9
;!) esign inspections an coe inspections9 an ;c) post6mortems5retrospecti%es:
What is verification? validation?
+erification typically in%ol%es re%iews an meetings to e%aluate ocuments9 plans9 coe9 re#uirements9 an
specifications: This can !e one with chec-lists9 issues lists9 wal-throughs9 an inspection meetings:
+aliation typically in%ol%es actual testing an ta-es place after %erifications are complete: The term '1+
C +' refers to 1nepenent +erification an +aliation:
What is a 'walkthrough'?
A 'wal-through' is an informal meeting for e%aluation or informational purposes: Dittle or no preparation is
usually re#uire:
What's an 'ins#ection'?
An inspection is more formali)e than a 'wal-through'9 typically with 36, people incluing a moerator9
reaer9 an a recorer to ta-e notes: The su!ject of the inspection is typically a ocument such as a
re#uirements spec or a test plan9 an the purpose is to fin pro!lems an see what's missing9 not to fi(
anything: Attenees shoul prepare for this type of meeting !y reaing thru the ocument> most pro!lems
will !e foun uring this preparation: The result of the inspection meeting shoul !e a written report:
Thorough preparation for inspections is ifficult9 painsta-ing wor-9 !ut is one of the most cost effecti%e
methos of ensuring #uality: 0mployees who are most s-ille at inspections are li-e the 'elest !rother' in
QA and Testing FAQs 7age 3 of 1*
LiveTech
the para!le in 'Why is it often har for management to get serious a!out #uality assurance?': Their s-ill may
ha%e low %isi!ility !ut they are e(tremely %alua!le to any software e%elopment organi)ation9 since !ug
pre%ention is far more cost6effecti%e than !ug etection:
What kinds of testing should !e considered?
<lac- !o( testing 6 not !ase on any -nowlege of internal esign or coe: Tests are !ase on
re#uirements an functionality:
White !o( testing 6 !ase on -nowlege of the internal logic of an application's coe: Tests are
!ase on co%erage of coe statements9 !ranches9 paths9 conitions:
unit testing 6 the most 'micro' scale of testing> to test particular functions or coe moules:
Typically one !y the programmer an not !y testers9 as it re#uires etaile -nowlege of the
internal program esign an coe: Eot always easily one unless the application has a well6
esigne architecture with tight coe> may re#uire e%eloping test ri%er moules or test
harnesses:
incremental integration testing 6 continuous testing of an application as new functionality is ae>
re#uires that %arious aspects of an application's functionality !e inepenent enough to wor-
separately !efore all parts of the program are complete9 or that test ri%ers !e e%elope as
neee> one !y programmers or !y testers:
integration testing 6 testing of com!ine parts of an application to etermine if they function
together correctly: The 'parts' can !e coe moules9 ini%iual applications9 client an ser%er
applications on a networ-9 etc: This type of testing is especially rele%ant to client5ser%er an
istri!ute systems:
functional testing 6 !lac-6!o( type testing geare to functional re#uirements of an application> this
type of testing shoul !e one !y testers: This oesn't mean that the programmers shouln't chec-
that their coe wor-s !efore releasing it ;which of course applies to any stage of testing:)
system testing 6 !lac-6!o( type testing that is !ase on o%erall re#uirements specifications> co%ers
all com!ine parts of a system:
en6to6en testing 6 similar to system testing> the 'macro' en of the test scale> in%ol%es testing of a
complete application en%ironment in a situation that mimics real6worl use9 such as interacting
with a ata!ase9 using networ- communications9 or interacting with other harware9 applications9
or systems if appropriate:
sanity testing or smo-e testing 6 typically an initial testing effort to etermine if a new software
%ersion is performing well enough to accept it for a major testing effort: Aor e(ample9 if the new
software is crashing systems e%ery $ minutes9 !ogging own systems to a crawl9 or corrupting
ata!ases9 the software may not !e in a 'sane' enough conition to warrant further testing in its
current state:
regression testing 6 re6testing after fi(es or moifications of the software or its en%ironment: 1t can
!e ifficult to etermine how much re6testing is neee9 especially near the en of the
e%elopment cycle: Automate testing tools can !e especially useful for this type of testing:
acceptance testing 6 final testing !ase on specifications of the en6user or customer9 or !ase on
use !y en6users5customers o%er some limite perio of time:
loa testing 6 testing an application uner hea%y loas9 such as testing of a we! site uner a range
of loas to etermine at what point the system's response time egraes or fails:
stress testing 6 term often use interchangea!ly with 'loa' an 'performance' testing: Also use to
escri!e such tests as system functional testing while uner unusually hea%y loas9 hea%y
repetition of certain actions or inputs9 input of large numerical %alues9 large comple( #ueries to a
ata!ase system9 etc:
performance testing 6 term often use interchangea!ly with 'stress' an 'loa' testing: 1eally
'performance' testing ;an any other 'type' of testing) is efine in re#uirements ocumentation or
QA or Test 7lans:
usa!ility testing 6 testing for 'user6frienliness': 2learly this is su!jecti%e9 an will epen on the
targete en6user or customer: Fser inter%iews9 sur%eys9 %ieo recoring of user sessions9 an
QA and Testing FAQs 7age " of 1*
LiveTech
other techni#ues can !e use: 7rogrammers an testers are usually not appropriate as usa!ility
testers:
install5uninstall testing 6 testing of full9 partial9 or upgrae install5uninstall processes:
reco%ery testing 6 testing how well a system reco%ers from crashes9 harware failures9 or other
catastrophic pro!lems:
failo%er testing 6 typically use interchangea!ly with 'reco%ery testing'
security testing 6 testing how well the system protects against unauthori)e internal or e(ternal
access9 willful amage9 etc> may re#uire sophisticate testing techni#ues:
compati!ility testing 6 testing how well software performs in a particular
harware5software5operating system5networ-5etc: en%ironment:
e(ploratory testing 6 often ta-en to mean a creati%e9 informal software test that is not !ase on
formal test plans or test cases> testers may !e learning the software as they test it:
a6hoc testing 6 similar to e(ploratory testing9 !ut often ta-en to mean that the testers ha%e
significant unerstaning of the software !efore testing it:
conte(t6ri%en testing 6 testing ri%en !y an unerstaning of the en%ironment9 culture9 an
intene use of software: Aor e(ample9 the testing approach for life6critical meical e#uipment
software woul !e completely ifferent than that for a low6cost computer game:
user acceptance testing 6 etermining if software is satisfactory to an en6user or customer:
comparison testing 6 comparing software wea-nesses an strengths to competing proucts:
alpha testing 6 testing of an application when e%elopment is nearing completion> minor esign
changes may still !e mae as a result of such testing: Typically one !y en6users or others9 not !y
programmers or testers:
!eta testing 6 testing when e%elopment an testing are essentially complete an final !ugs an
pro!lems nee to !e foun !efore final release: Typically one !y en6users or others9 not !y
programmers or testers:
mutation testing 6 a metho for etermining if a set of test ata or test cases is useful9 !y
eli!erately introucing %arious coe changes ;'!ugs') an retesting with the original test
ata5cases to etermine if the '!ugs' are etecte: 7roper implementation re#uires large
computational resources:
What are & coon #ro!les in the software develo#ent #rocess?
poor re#uirements 6 if re#uirements are unclear9 incomplete9 too general9 an not testa!le9 there
will !e pro!lems:
unrealistic scheule 6 if too much wor- is cramme in too little time9 pro!lems are ine%ita!le:
inae#uate testing 6 no one will -now whether or not the program is any goo until the customer
complains or systems crash:
featuritis 6 re#uests to pile on new features after e%elopment is unerway> e(tremely common:
miscommunication 6 if e%elopers on't -now what's neee or customer's ha%e erroneous
e(pectations9 pro!lems are guarantee:
What are & coon solutions to software develo#ent #ro!les?
soli re#uirements 6 clear9 complete9 etaile9 cohesi%e9 attaina!le9 testa!le re#uirements that are
agree to !y all players: Fse prototypes to help nail own re#uirements: 1n 'agile'6type
en%ironments9 continuous coorination with customers5en6users is necessary:
realistic scheules 6 allow ae#uate time for planning9 esign9 testing9 !ug fi(ing9 re6testing9
changes9 an ocumentation> personnel shoul !e a!le to complete the project without !urning
out:
ae#uate testing 6 start testing early on9 re6test after fi(es or changes9 plan for ae#uate time for
testing an !ug6fi(ing: '0arly' testing ieally inclues unit testing !y e%elopers an !uilt6in
testing an iagnostic capa!ilities:
QA and Testing FAQs 7age $ of 1*
LiveTech
stic- to initial re#uirements as much as possi!le 6 !e prepare to efen against e(cessi%e changes
an aitions once e%elopment has !egun9 an !e prepare to e(plain conse#uences: 1f changes
are necessary9 they shoul !e ae#uately reflecte in relate scheule changes: 1f possi!le9 wor-
closely with customers5en6users to manage e(pectations: This will pro%ie them a higher comfort
le%el with their re#uirements ecisions an minimi)e e(cessi%e changes later on:
communication 6 re#uire wal-throughs an inspections when appropriate> ma-e e(tensi%e use of
group communication tools 6 e6mail9 groupware9 networ-e !ug6trac-ing tools an change
management tools9 intranet capa!ilities9 etc:> insure that information5ocumentation is a%aila!le
an up6to6ate 6 prefera!ly electronic9 not paper> promote teamwor- an cooperation> use
prototypes if possi!le to clarify customers' e(pectations:
What is software 'quality'?
Quality software is reasona!ly !ug6free9 eli%ere on time an within !uget9 meets re#uirements an5or
e(pectations9 an is maintaina!le: 'owe%er9 #uality is o!%iously a su!jecti%e term: 1t will epen on who
the 'customer' is an their o%erall influence in the scheme of things: A wie6angle %iew of the 'customers' of
a software e%elopment project might inclue en6users9 customer acceptance testers9 customer contract
officers9 customer management9 the e%elopment organi)ation's
management5accountants5testers5salespeople9 future software maintenance engineers9 stoc-holers9
maga)ine columnists9 etc: 0ach type of 'customer' will ha%e their own slant on '#uality' 6 the accounting
epartment might efine #uality in terms of profits while an en6user might efine #uality as user6frienly
an !ug6free:
What is 'good code'?
'Goo coe' is coe that wor-s9 is !ug free9 an is reaa!le an maintaina!le: Some organi)ations ha%e
coing 'stanars' that all e%elopers are suppose to ahere to9 !ut e%eryone has ifferent ieas a!out
what's !est9 or what is too many or too few rules: There are also %arious theories an metrics9 such as
3c2a!e 2omple(ity metrics: 1t shoul !e -ept in min that e(cessi%e use of stanars an rules can stifle
proucti%ity an creati%ity: '7eer re%iews'9 '!uy chec-s' coe analysis tools9 etc: can !e use to chec- for
pro!lems an enforce stanars:
Aor 2 an 2HH coing9 here are some typical ieas to consier in setting rules5stanars> these may or may
not apply to a particular situation?
minimi)e or eliminate use of glo!al %aria!les:
use escripti%e function an metho names 6 use !oth upper an lower case9 a%oi a!!re%iations9
use as many characters as necessary to !e ae#uately escripti%e ;use of more than 2/ characters
is not out of line)> !e consistent in naming con%entions:
use escripti%e %aria!le names 6 use !oth upper an lower case9 a%oi a!!re%iations9 use as many
characters as necessary to !e ae#uately escripti%e ;use of more than 2/ characters is not out of
line)> !e consistent in naming con%entions:
function an metho si)es shoul !e minimi)e> less than 1// lines of coe is goo9 less than $/
lines is prefera!le:
function escriptions shoul !e clearly spelle out in comments preceing a function's coe:
organi)e coe for reaa!ility:
use white space generously 6 %ertically an hori)ontally
each line of coe shoul contain */ characters ma(:
one coe statement per line:
coing style shoul !e consistent throught a program ;eg9 use of !rac-ets9 inentations9 naming
con%entions9 etc:)
in aing comments9 err on the sie of too many rather than too few comments> a common rule of
thum! is that there shoul !e at least as many lines of comments ;incluing heaer !loc-s) as lines
of coe:
QA and Testing FAQs 7age & of 1*
LiveTech
no matter how small9 an application shoul inclue ocumentation of the o%erall program function
an flow ;e%en a few paragraphs is !etter than nothing)> or if possi!le a separate flow chart an
etaile program ocumentation:
ma-e e(tensi%e use of error hanling proceures an status an error logging:
for 2HH9 to minimi)e comple(ity an increase maintaina!ility9 a%oi too many le%els of
inheritance in class hierarchies ;relati%e to the si)e an comple(ity of the application): 3inimi)e
use of multiple inheritance9 an minimi)e use of operator o%erloaing ;note that the Ia%a
programming language eliminates multiple inheritance an operator o%erloaing:)
for 2HH9 -eep class methos small9 less than $/ lines of coe per metho is prefera!le:
for 2HH9 ma-e li!eral use of e(ception hanlers
What is 'good design'?
'=esign' coul refer to many things9 !ut often refers to 'functional esign' or 'internal esign': Goo internal
esign is inicate !y software coe whose o%erall structure is clear9 unerstana!le9 easily moifia!le9 an
maintaina!le> is ro!ust with sufficient error6hanling an status logging capa!ility> an wor-s correctly
when implemente: Goo functional esign is inicate !y an application whose functionality can !e trace
!ac- to customer an en6user re#uirements: ;See further iscussion of functional an internal esign in
'What's the !ig eal a!out re#uirements?' in AAQ J2:) Aor programs that ha%e a user interface9 it's often a
goo iea to assume that the en user will ha%e little computer -nowlege an may not rea a user manual
or e%en the on6line help> some common rules6of6thum! inclue?
the program shoul act in a way that least surprises the user
it shoul always !e e%ient to the user what can !e one ne(t an how to e(it
the program shouln't let the users o something stupi without warning them:
What is S'(? )**? )**(? (S+? ('''? A,S(? Will it hel#?
S01 K 'Software 0ngineering 1nstitute' at 2arnegie63ellon Fni%ersity> initiate !y the F:S:
=efense =epartment to help impro%e software e%elopment processes:
233 K '2apa!ility 3aturity 3oel'9 now calle the 2331 ;'2apa!ility 3aturity 3oel
1ntegration')9 e%elope !y the S01: 1t's a moel of $ le%els of process 'maturity' that etermine
effecti%eness in eli%ering #uality software: 1t is geare to large organi)ations such as large F:S:
=efense =epartment contractors: 'owe%er9 many of the QA processes in%ol%e are appropriate to
any organi)ation9 an if reasona!ly applie can !e helpful: 4rgani)ations can recei%e 2331
ratings !y unergoing assessments !y #ualifie auitors:
De%el 1 6 characteri)e !y chaos9 perioic panics9 an heroic
efforts re#uire !y ini%iuals to successfully
complete projects: Aew if any processes in place>
successes may not !e repeata!le:
De%el 2 6 software project trac-ing9 re#uirements management9
realistic planning9 an configuration management
processes are in place> successful practices can
!e repeate:
De%el 3 6 stanar software e%elopment an maintenance processes
are integrate throughout an organi)ation> a Software
0ngineering 7rocess Group is in place to o%ersee
software processes9 an training programs are use to
ensure unerstaning an compliance:
QA and Testing FAQs 7age * of 1*
LiveTech
De%el " 6 metrics are use to trac- proucti%ity9 processes9
an proucts: 7roject performance is preicta!le9
an #uality is consistently high:
De%el $ 6 the focus is on continuous process impro%ement: The
impact of new processes an technologies can !e
preicte an effecti%ely implemente when re#uire:
7erspecti%e on 233 ratings? =uring 1..*62//19 1/1, organi)ations
were assesse: 4f those9 2*L were rate at De%el 19 3.L at 29
23L at 39 &L at "9 an $L at $: ;Aor ratings uring the perio
1..26.&9 &2L were at De%el 19 23L at 29 13L at 39 2L at "9 an
/:"L at $:) The meian si)e of organi)ations was 1// software
engineering5maintenance personnel> 32L of organi)ations were
F:S: feeral contractors or agencies: Aor those rate at
De%el 19 the most pro!lematical -ey process area was in
Software Quality Assurance:
1S4 K '1nternational 4rganisation for Stanari)ation' 6 The 1S4 .//1?2/// stanar ;which
replaces the pre%ious stanar of 1..") concerns #uality systems that are assesse !y outsie
auitors9 an it applies to many -ins of prouction an manufacturing organi)ations9 not just
software: 1t co%ers ocumentation9 esign9 e%elopment9 prouction9 testing9 installation9
ser%icing9 an other processes: The full set of stanars consists of? ;a)Q.//162/// 6 Quality
3anagement Systems? 8e#uirements> ;!)Q.///62/// 6 Quality 3anagement Systems?
Aunamentals an +oca!ulary> ;c)Q.//"62/// 6 Quality 3anagement Systems? Guielines for
7erformance 1mpro%ements: To !e 1S4 .//1 certifie9 a thir6party auitor assesses an
organi)ation9 an certification is typically goo for a!out 3 years9 after which a complete
reassessment is re#uire: Eote that 1S4 certification oes not necessarily inicate #uality proucts
6 it inicates only that ocumente processes are followe: Also see http?55www:iso:ch5 for the
latest information: 1n the F:S: the stanars can !e purchase %ia the ASQ we! site at http?55e6
stanars:as#:org5
1000 K '1nstitute of 0lectrical an 0lectronics 0ngineers' 6 among other things9 creates stanars
such as '1000 Stanar for Software Test =ocumentation' ;10005AES1 Stanar ,2.)9 '1000
Stanar of Software Fnit Testing ;10005AES1 Stanar 1//,)9 '1000 Stanar for Software
Quality Assurance 7lans' ;10005AES1 Stanar *3/)9 an others:
AES1 K 'American Eational Stanars 1nstitute'9 the primary inustrial stanars !oy in the F:S:>
pu!lishes some software6relate stanars in conjunction with the 1000 an ASQ ;American
Society for Quality):
4ther software e%elopment51T management process assessment methos !esies 2331 an 1S4
./// inclue S71209 Trillium9 Tic-1T9 <ootstrap9 1T1D9 34A9 an 2o!iT:
See the '4ther 8esources' section for further information a%aila!le on the we!:
What is the 'software life cycle'?
The life cycle !egins when an application is first concei%e an ens when it is no longer in use: 1t inclues
aspects such as initial concept9 re#uirements analysis9 functional esign9 internal esign9 ocumentation
planning9 test planning9 coing9 ocument preparation9 integration9 testing9 maintenance9 upates9 retesting9
phase6out9 an other aspects:
Will autoated testing tools ake testing easier?
7ossi!ly: Aor small projects9 the time neee to learn an implement them may not !e worth it:
Aor larger projects9 or on6going long6term projects they can !e %alua!le:
QA and Testing FAQs 7age , of 1*
LiveTech
A common type of automate tool is the 'recor5play!ac-' type: Aor e(ample9 a tester coul clic-
through all com!inations of menu choices9 ialog !o( choices9 !uttons9 etc: in an application GF1
an ha%e them 'recore' an the results logge !y a tool: The 'recoring' is typically in the form
of te(t !ase on a scripting language that is interpreta!le !y the testing tool: 1f new !uttons are
ae9 or some unerlying coe in the application is change9 etc: the application might then !e
reteste !y just 'playing !ac-' the 'recore' actions9 an comparing the logging results to chec-
effects of the changes: The pro!lem with such tools is that if there are continual changes to the
system !eing teste9 the 'recorings' may ha%e to !e change so much that it !ecomes %ery time6
consuming to continuously upate the scripts: Aitionally9 interpretation an analysis of results
;screens9 ata9 logs9 etc:) can !e a ifficult tas-: Eote that there are recor5play!ac- tools for te(t6
!ase interfaces also9 an for all types of platforms:
Another common type of approach for automation of functional testing is 'ata6ri%en' or
'-eywor6ri%en' automate testing9 in which the test ri%ers are separate from the ata an5or
actions utili)e in testing ;an 'action' woul !e something li-e 'enter a %alue in a te(t !o('): Test
ri%ers can !e in the form of automate test tools or custom6written testing software: The ata an
actions can !e more easily maintaine 6 such as %ia a spreasheet 6 since they are separate from
the test ri%ers: The test ri%ers 'rea' the ata5action information to perform specifie tests: This
approach can ena!le more efficient control9 e%elopment9 ocumentation9 an maintenance of
automate tests5test cases:
4ther automate tools can inclue?
coe analy)ers 6 monitor coe comple(ity9 aherence to
stanars9 etc:

co%erage analy)ers 6 these tools chec- which parts of the

coe ha%e !een e(ercise !y a test9 an may
!e oriente to coe statement co%erage9
conition co%erage9 path co%erage9 etc:

memory analy)ers 6 such as !ouns6chec-ers an lea- etectors:

loa5performance test tools 6 for testing client5ser%er

an we! applications uner %arious loa
le%els:

we! test tools 6 to chec- that lin-s are %ali9 'T3D coe
usage is correct9 client6sie an
ser%er6sie programs wor-9 a we! site's
interactions are secure:

other tools 6 for test case management9 ocumentation
management9 !ug reporting9 an configuration management:
What akes a good Software Test engineer?
A goo test engineer has a 'test to !rea-' attitue9 an a!ility to ta-e the point of %iew of the customer9 a
strong esire for #uality9 an an attention to etail: Tact an iplomacy are useful in maintaining a
cooperati%e relationship with e%elopers9 an an a!ility to communicate with !oth technical ;e%elopers)
an non6technical ;customers9 management) people is useful: 7re%ious software e%elopment e(perience
can !e helpful as it pro%ies a eeper unerstaning of the software e%elopment process9 gi%es the tester
an appreciation for the e%elopers' point of %iew9 an reuce the learning cur%e in automate test tool
programming: Iugment s-ills are neee to assess high6ris- areas of an application on which to focus
testing efforts when time is limite:
QA and Testing FAQs 7age . of 1*
LiveTech
What akes a good Software QA engineer?
The same #ualities a goo tester has are useful for a QA engineer: Aitionally9 they must !e a!le to
unerstan the entire software e%elopment process an how it can fit into the !usiness approach an goals
of the organi)ation: 2ommunication s-ills an the a!ility to unerstan %arious sies of issues are
important: 1n organi)ations in the early stages of implementing QA processes9 patience an iplomacy are
especially neee: An a!ility to fin pro!lems as well as to see 'what's missing' is important for inspections
an re%iews:

What akes a good QA or Test anager?
A goo QA9 test9 or QA5Test;com!ine) manager shoul?
!e familiar with the software e%elopment process
!e a!le to maintain enthusiasm of their team an promote a positi%e atmosphere9 espite what is a
somewhat 'negati%e' process ;e:g:9 loo-ing for or pre%enting pro!lems)
!e a!le to promote teamwor- to increase proucti%ity
!e a!le to promote cooperation !etween software9 test9 an QA engineers
ha%e the iplomatic s-ills neee to promote impro%ements in QA processes
ha%e the a!ility to withstan pressures an say 'no' to other managers when #uality is insufficient
or QA processes are not !eing ahere to
ha%e people jugment s-ills for hiring an -eeping s-ille personnel
!e a!le to communicate with technical an non6technical people9 engineers9 managers9 an
customers:
!e a!le to run meetings an -eep them focuse
What's the role of docuentation in QA?
2ritical: ;Eote that ocumentation can !e electronic9 not necessarily paper9 may !e em!ee in coe
comments9 etc:) QA practices shoul !e ocumente such that they are repeata!le: Specifications9 esigns9
!usiness rules9 inspection reports9 configurations9 coe changes9 test plans9 test cases9 !ug reports9 user
manuals9 etc: shoul all !e ocumente in some form: There shoul ieally !e a system for easily fining
an o!taining information an etermining what ocumentation will ha%e a particular piece of information:
2hange management for ocumentation shoul !e use if possi!le:
What's the !ig deal a!out 'requireents'?
4ne of the most relia!le methos of ensuring pro!lems9 or failure9 in a large9 comple( software project is to
ha%e poorly ocumente re#uirements specifications: 8e#uirements are the etails escri!ing an
application's e(ternally6percei%e functionality an properties: 8e#uirements shoul !e clear9 complete9
reasona!ly etaile9 cohesi%e9 attaina!le9 an testa!le: A non6testa!le re#uirement woul !e9 for e(ample9
'user6frienly' ;too su!jecti%e): A testa!le re#uirement woul !e something li-e 'the user must enter their
pre%iously6assigne passwor to access the application': =etermining an organi)ing re#uirements etails
in a useful an efficient way can !e a ifficult effort> ifferent methos are a%aila!le epening on the
particular project: 3any !oo-s are a%aila!le that escri!e %arious approaches to this tas-: ;See the
<oo-store section's 'Software 8e#uirements 0ngineering' category for !oo-s on Software 8e#uirements:)
2are shoul !e ta-en to in%ol%e ADD of a project's significant 'customers' in the re#uirements process:
'2ustomers' coul !e in6house personnel or out9 an coul inclue en6users9 customer acceptance testers9
customer contract officers9 customer management9 future software maintenance engineers9 salespeople9 etc:
Anyone who coul later erail the project if their e(pectations aren't met shoul !e inclue if possi!le:
4rgani)ations %ary consiera!ly in their hanling of re#uirements specifications: 1eally9 the re#uirements
are spelle out in a ocument with statements such as 'The prouct shall:::::': '=esign' specifications shoul
not !e confuse with 're#uirements'> esign specifications shoul !e tracea!le !ac- to the re#uirements:
QA and Testing FAQs 7age 1/ of 1*
LiveTech
1n some organi)ations re#uirements may en up in high le%el project plans9 functional specification
ocuments9 in esign ocuments9 or in other ocuments at %arious le%els of etail: Eo matter what they are
calle9 some type of ocumentation with etaile re#uirements will !e neee !y testers in orer to
properly plan an e(ecute tests: Without such ocumentation9 there will !e no clear6cut way to etermine if
a software application is performing correctly:
'Agile' methos such as B7 use methos re#uiring close interaction an cooperation !etween programmers
an customers5en6users to iterati%ely e%elop re#uirements: The programmer uses 'Test first' e%elopment
to first create automate unit testing coe9 which essentially em!oies the re#uirements:
What ste#s are needed to develo# and run software tests?
The following are some of the steps to consier?
4!tain re#uirements9 functional esign9 an internal esign specifications an other necessary
ocuments
4!tain !uget an scheule re#uirements
=etermine project6relate personnel an their responsi!ilities9 reporting re#uirements9 re#uire
stanars an processes ;such as release processes9 change processes9 etc:)
=etermine project conte(t9 relati%e to the e(isting #uality culture of the organi)ation an !usiness9
an how it might impact testing scope9 approaches9 an methos:
1entify application's higher6ris- aspects9 set priorities9 an etermine scope an limitations of
tests
=etermine test approaches an methos 6 unit9 integration9 functional9 system9 loa9 usa!ility tests9
etc:
=etermine test en%ironment re#uirements ;harware9 software9 communications9 etc:)
=etermine testware re#uirements ;recor5play!ac- tools9 co%erage analy)ers9 test trac-ing9
pro!lem5!ug trac-ing9 etc:)
=etermine test input ata re#uirements
1entify tas-s9 those responsi!le for tas-s9 an la!or re#uirements
Set scheule estimates9 timelines9 milestones
=etermine input e#ui%alence classes9 !ounary %alue analyses9 error classes
7repare test plan ocument an ha%e neee re%iews5appro%als
Write test cases
'a%e neee re%iews5inspections5appro%als of test cases
7repare test en%ironment an testware9 o!tain neee user manuals5reference
ocuments5configuration guies5installation guies9 set up test trac-ing processes9 set up logging
an archi%ing processes9 set up or o!tain test input ata
4!tain an install software releases
7erform tests
0%aluate an report results
Trac- pro!lems5!ugs an fi(es
8etest as neee
3aintain an upate test plans9 test cases9 test en%ironment9 an testware through life cycle
What's a 'test #lan'?
A software project test plan is a ocument that escri!es the o!jecti%es9 scope9 approach9 an focus of a
software testing effort: The process of preparing a test plan is a useful way to thin- through the efforts
neee to %aliate the accepta!ility of a software prouct: The complete ocument will help people
outsie the test group unerstan the 'why' an 'how' of prouct %aliation: 1t shoul !e thorough enough to
!e useful !ut not so thorough that no one outsie the test group will rea it: The following are some of the
items that might !e inclue in a test plan9 epening on the particular project?
QA and Testing FAQs 7age 11 of 1*
LiveTech
Title
1entification of software incluing %ersion5release num!ers
8e%ision history of ocument incluing authors9 ates9 appro%als
Ta!le of 2ontents
7urpose of ocument9 intene auience
4!jecti%e of testing effort
Software prouct o%er%iew
8ele%ant relate ocument list9 such as re#uirements9 esign ocuments9 other test plans9 etc:
8ele%ant stanars or legal re#uirements
Tracea!ility re#uirements
8ele%ant naming con%entions an ientifier con%entions
4%erall software project organi)ation an personnel5contact6info5responsi!ilities
Test organi)ation an personnel5contact6info5responsi!ilities
Assumptions an epenencies
7roject ris- analysis
Testing priorities an focus
Scope an limitations of testing
Test outline 6 a ecomposition of the test approach !y test type9 feature9 functionality9 process9
system9 moule9 etc: as applica!le
4utline of ata input e#ui%alence classes9 !ounary %alue analysis9 error classes
Test en%ironment 6 harware9 operating systems9 other re#uire software9 ata configurations9
interfaces to other systems
Test en%ironment %aliity analysis 6 ifferences !etween the test an prouction systems an their
impact on test %aliity:
Test en%ironment setup an configuration issues
Software migration processes
Software 23 processes
Test ata setup re#uirements
=ata!ase setup re#uirements
4utline of system6logging5error6logging5other capa!ilities9 an tools such as screen capture
software9 that will !e use to help escri!e an report !ugs
=iscussion of any speciali)e software or harware tools that will !e use !y testers to help trac-
the cause or source of !ugs
Test automation 6 justification an o%er%iew
Test tools to !e use9 incluing %ersions9 patches9 etc:
Test script5test coe maintenance processes an %ersion control
7ro!lem trac-ing an resolution 6 tools an processes
7roject test metrics to !e use
8eporting re#uirements an testing eli%era!les
Software entrance an e(it criteria
1nitial sanity testing perio an criteria
Test suspension an restart criteria
7ersonnel allocation
7ersonnel pre6training nees
Test site5location
4utsie test organi)ations to !e utili)e an their purpose9 responsi!ilities9 eli%era!les9 contact
persons9 an coorination issues
8ele%ant proprietary9 classifie9 security9 an licensing issues:
4pen issues
Appeni( 6 glossary9 acronyms9 etc:
QA and Testing FAQs 7age 12 of 1*
LiveTech
What's a 'test case'?
A test case is a ocument that escri!es an input9 action9 or e%ent an an e(pecte response9 to
etermine if a feature of an application is wor-ing correctly: A test case shoul contain particulars
such as test case ientifier9 test case name9 o!jecti%e9 test conitions5setup9 input ata
re#uirements9 steps9 an e(pecte results:
Eote that the process of e%eloping test cases can help fin pro!lems in the re#uirements or esign
of an application9 since it re#uires completely thin-ing through the operation of the application:
Aor this reason9 it's useful to prepare test cases early in the e%elopment cycle if possi!le:
What should !e done after a !ug is found?
The !ug nees to !e communicate an assigne to e%elopers that can fi( it: After the pro!lem is
resol%e9 fi(es shoul !e re6teste9 an eterminations mae regaring re#uirements for regression testing
to chec- that fi(es in't create pro!lems elsewhere: 1f a pro!lem6trac-ing system is in place9 it shoul
encapsulate these processes: A %ariety of commercial pro!lem6trac-ing5management software tools are
a%aila!le ;see the 'Tools' section for we! resources with listings of such tools): The following are items to
consier in the trac-ing process?
2omplete information such that e%elopers can unerstan the !ug9 get an iea of it's se%erity9 an
reprouce it if necessary:
<ug ientifier ;num!er9 1=9 etc:)
2urrent !ug status ;e:g:9 '8elease for 8etest'9 'Eew'9 etc:)
The application name or ientifier an %ersion
The function9 moule9 feature9 o!ject9 screen9 etc: where the !ug occurre
0n%ironment specifics9 system9 platform9 rele%ant harware specifics
Test case name5num!er5ientifier
4ne6line !ug escription
Aull !ug escription
=escription of steps neee to reprouce the !ug if not co%ere !y a test case or if the e%eloper
oesn't ha%e easy access to the test case5test script5test tool
Eames an5or escriptions of file5ata5messages5etc: use in test
Aile e(cerpts5error messages5log file e(cerpts5screen shots5test tool logs that woul !e helpful in
fining the cause of the pro!lem
Se%erity estimate ;a $6le%el range such as 16$ or 'critical'6to6'low' is common)
Was the !ug reprouci!le?
Tester name
Test ate
<ug reporting ate
Eame of e%eloper5group5organi)ation the pro!lem is assigne to
=escription of pro!lem cause
=escription of fi(
2oe section5file5moule5class5metho that was fi(e
=ate of fi(
Application %ersion that contains the fi(
Tester responsi!le for retest
8etest ate
8etest results
QA and Testing FAQs 7age 13 of 1*
LiveTech
8egression testing re#uirements
Tester responsi!le for regression tests
8egression testing results
A reporting or trac-ing process shoul ena!le notification of appropriate personnel at %arious stages: Aor
instance9 testers nee to -now when retesting is neee9 e%elopers nee to -now when !ugs are foun an
how to get the neee information9 an reporting5summary capa!ilities are neee for managers:
What is 'configuration anageent'?
2onfiguration management co%ers the processes use to control9 coorinate9 an trac-? coe9 re#uirements9
ocumentation9 pro!lems9 change re#uests9 esigns9 tools5compilers5li!raries5patches9 changes mae to
them9 an who ma-es the changes:
What if the software is so !uggy it can't really !e tested at all?
The !est !et in this situation is for the testers to go through the process of reporting whate%er !ugs or
!loc-ing6type pro!lems initially show up9 with the focus !eing on critical !ugs: Since this type of pro!lem
can se%erely affect scheules9 an inicates eeper pro!lems in the software e%elopment process ;such as
insufficient unit testing or insufficient integration testing9 poor esign9 improper !uil or release
proceures9 etc:) managers shoul !e notifie9 an pro%ie with some ocumentation as e%ience of the
pro!lem:
"ow can it !e known when to sto# testing?
This can !e ifficult to etermine: 3any moern software applications are so comple(9 an run in such an
interepenent en%ironment9 that complete testing can ne%er !e one: 2ommon factors in eciing when to
stop are?
=ealines ;release ealines9 testing ealines9 etc:)
Test cases complete with certain percentage passe
Test !uget eplete
2o%erage of coe5functionality5re#uirements reaches a specifie point
<ug rate falls !elow a certain le%el
<eta or alpha testing perio ens
What if there isn't enough tie for thorough testing?
Fse ris- analysis to etermine where testing shoul !e focuse:
Since it's rarely possi!le to test e%ery possi!le aspect of an application9 e%ery possi!le com!ination of
e%ents9 e%ery epenency9 or e%erything that coul go wrong9 ris- analysis is appropriate to most software
e%elopment projects: This re#uires jugment s-ills9 common sense9 an e(perience: ;1f warrante9 formal
methos are also a%aila!le:) 2onsierations can inclue?
Which functionality is most important to the project's intene purpose?
Which functionality is most %isi!le to the user?
Which functionality has the largest safety impact?
Which functionality has the largest financial impact on users?
Which aspects of the application are most important to the customer?
Which aspects of the application can !e teste early in the e%elopment cycle?
Which parts of the coe are most comple(9 an thus most su!ject to errors?
Which parts of the application were e%elope in rush or panic moe?
Which aspects of similar5relate pre%ious projects cause pro!lems?
Which aspects of similar5relate pre%ious projects ha large maintenance e(penses?
Which parts of the re#uirements an esign are unclear or poorly thought out?
QA and Testing FAQs 7age 1" of 1*
LiveTech
What o the e%elopers thin- are the highest6ris- aspects of the application?
What -ins of pro!lems woul cause the worst pu!licity?
What -ins of pro!lems woul cause the most customer ser%ice complaints?
What -ins of tests coul easily co%er multiple functionalities?
Which tests will ha%e the !est high6ris-6co%erage to time6re#uire ratio?
What if the #ro-ect isn't !ig enough to -ustify e$tensive testing?
2onsier the impact of project errors9 not the si)e of the project: 'owe%er9 if e(tensi%e testing is still not
justifie9 ris- analysis is again neee an the same consierations as escri!e pre%iously in 'What if there
isn't enough time for thorough testing?' apply: The tester might then o a hoc testing9 or write up a limite
test plan !ase on the ris- analysis:
What can !e done if requireents are changing continuously?
A common pro!lem an a major heaache:
Wor- with the project's sta-eholers early on to unerstan how re#uirements might change so
that alternate test plans an strategies can !e wor-e out in a%ance9 if possi!le:
1t's helpful if the application's initial esign allows for some aapta!ility so that later changes o
not re#uire reoing the application from scratch:
1f the coe is well6commente an well6ocumente this ma-es changes easier for the e%elopers:
Fse rapi prototyping whene%er possi!le to help customers feel sure of their re#uirements an
minimi)e changes:
The project's initial scheule shoul allow for some e(tra time commensurate with the possi!ility
of changes:
Try to mo%e new re#uirements to a '7hase 2' %ersion of an application9 while using the original
re#uirements for the '7hase 1' %ersion:
Eegotiate to allow only easily6implemente new re#uirements into the project9 while mo%ing more
ifficult new re#uirements into future %ersions of the application:
<e sure that customers an management unerstan the scheuling impacts9 inherent ris-s9 an
costs of significant re#uirements changes: Then let management or the customers ;not the
e%elopers or testers) ecie if the changes are warrante 6 after all9 that's their jo!:
<alance the effort put into setting up automate testing with the e(pecte effort re#uire to re6o
them to eal with changes:
Try to esign some fle(i!ility into automate test scripts:
Aocus initial automate testing on application aspects that are most li-ely to remain unchange:
=e%ote appropriate effort to ris- analysis of changes to minimi)e regression testing nees:
=esign some fle(i!ility into test cases ;this is not easily one> the !est !et might !e to minimi)e
the etail in the test cases9 or set up only higher6le%el generic6type test plans)
Aocus less on etaile test plans an test cases an more on a hoc testing ;with an unerstaning
of the ae ris- that this entails):
What if the a##lication has functionality that wasn't in the requireents?
1t may ta-e serious effort to etermine if an application has significant une(pecte or hien functionality9
an it woul inicate eeper pro!lems in the software e%elopment process: 1f the functionality isn't
necessary to the purpose of the application9 it shoul !e remo%e9 as it may ha%e un-nown impacts or
epenencies that were not ta-en into account !y the esigner or the customer: 1f not remo%e9 esign
information will !e neee to etermine ae testing nees or regression testing nees: 3anagement
shoul !e mae aware of any significant ae ris-s as a result of the une(pecte functionality: 1f the
functionality only effects areas such as minor impro%ements in the user interface9 for e(ample9 it may not
!e a significant ris-:
QA and Testing FAQs 7age 1$ of 1*
LiveTech
"ow can Software QA #rocesses !e i#leented without stifling #roductivity?
<y implementing QA processes slowly o%er time9 using consensus to reach agreement on processes9 an
ajusting an e(perimenting as an organi)ation grows an matures9 proucti%ity will !e impro%e instea
of stifle: 7ro!lem pre%ention will lessen the nee for pro!lem etection9 panics an !urn6out will
ecrease9 an there will !e impro%e focus an less waste effort: At the same time9 attempts shoul !e
mae to -eep processes simple an efficient9 minimi)e paperwor-9 promote computer6!ase processes an
automate trac-ing an reporting9 minimi)e time re#uire in meetings9 an promote training as part of the
QA process: 'owe%er9 no one 6 especially talente technical types 6 li-es rules or !ureaucracy9 an in the
short run things may slow own a !it: A typical scenario woul !e that more ays of planning an
e%elopment will !e neee9 !ut less time will !e re#uire for late6night !ug6fi(ing an calming of irate
customers:
What if an organi%ation is growing so fast that fi$ed QA #rocesses are i#ossi!le?
This is a common pro!lem in the software inustry9 especially in new technology areas: There is no easy
solution in this situation9 other than?
'ire goo people
3anagement shoul 'ruthlessly prioriti)e' #uality issues an maintain focus on the customer
0%eryone in the organi)ation shoul !e clear on what '#uality' means to the customer
"ow does a client.server environent affect testing?
2lient5ser%er applications can !e #uite comple( ue to the multiple epenencies among clients9 ata
communications9 harware9 an ser%ers: Thus testing re#uirements can !e e(tensi%e: When time is limite
;as it usually is) the focus shoul !e on integration an system testing: Aitionally9
loa5stress5performance testing may !e useful in etermining client5ser%er application limitations an
capa!ilities: There are commercial tools to assist with such testing: ;See the 'Tools' section for we!
resources with listings that inclue these -ins of test tools:)
"ow can World Wide We! sites !e tested?
We! sites are essentially client5ser%er applications 6 with we! ser%ers an '!rowser' clients: 2onsieration
shoul !e gi%en to the interactions !etween html pages9 T27517 communications9 1nternet connections9
firewalls9 applications that run in we! pages ;such as applets9 Ia%aScript9 plug6in applications)9 an
applications that run on the ser%er sie ;such as cgi scripts9 ata!ase interfaces9 logging applications9
ynamic page generators9 asp9 etc:): Aitionally9 there are a wie %ariety of ser%ers an !rowsers9 %arious
%ersions of each9 small !ut sometimes significant ifferences !etween them9 %ariations in connection
spees9 rapily changing technologies9 an multiple stanars an protocols: The en result is that testing
for we! sites can !ecome a major ongoing effort: 4ther consierations might inclue?
What are the e(pecte loas on the ser%er ;e:g:9 num!er of hits per unit time?)9 an what -in of
performance is re#uire uner such loas ;such as we! ser%er response time9 ata!ase #uery
response times): What -ins of tools will !e neee for performance testing ;such as we! loa
testing tools9 other tools alreay in house that can !e aapte9 we! ro!ot ownloaing tools9 etc:)?
Who is the target auience? What -in of !rowsers will they !e using? What -in of connection
spees will they !y using? Are they intra6 organi)ation ;thus with li-ely high connection spees
an similar !rowsers) or 1nternet6wie ;thus with a wie %ariety of connection spees an !rowser
types)?
What -in of performance is e(pecte on the client sie ;e:g:9 how fast shoul pages appear9 how
fast shoul animations9 applets9 etc: loa an run)?
Will own time for ser%er an content maintenance5upgraes !e allowe? how much?
What -ins of security ;firewalls9 encryptions9 passwors9 etc:) will !e re#uire an what is it
e(pecte to o? 'ow can it !e teste?
'ow relia!le are the site's 1nternet connections re#uire to !e? An how oes that affect !ac-up
system or reunant connection re#uirements an testing?
QA and Testing FAQs 7age 1& of 1*
LiveTech
What processes will !e re#uire to manage upates to the we! site's content9 an what are the
re#uirements for maintaining9 trac-ing9 an controlling page content9 graphics9 lin-s9 etc:?
Which 'T3D specification will !e ahere to? 'ow strictly? What %ariations will !e allowe for
targete !rowsers?
Will there !e any stanars or re#uirements for page appearance an5or graphics throughout a site
or parts of a site??
'ow will internal an e(ternal lin-s !e %aliate an upate? how often?
2an testing !e one on the prouction system9 or will a separate test system !e re#uire? 'ow are
!rowser caching9 %ariations in !rowser option settings9 ial6up connection %aria!ilities9 an real6
worl internet 'traffic congestion' pro!lems to !e accounte for in testing?
'ow e(tensi%e or customi)e are the ser%er logging an reporting re#uirements> are they
consiere an integral part of the system an o they re#uire testing?
'ow are cgi programs9 applets9 ja%ascripts9 Acti%eB components9 etc: to !e maintaine9 trac-e9
controlle9 an teste?
Some sources of site security information inclue the Fsenet newsgroup 'comp:security:announce' an lin-s
concerning we! site security in the '4ther 8esources' section:
Some usa!ility guielines to consier 6 these are su!jecti%e an may or may not apply to a gi%en situation
;Eote? more information on usa!ility testing issues can !e foun in articles a!out we! site usa!ility in the
'4ther 8esources' section)?
7ages shoul !e 36$ screens ma( unless content is tightly focuse on a single topic: 1f larger9
pro%ie internal lin-s within the page:
The page layouts an esign elements shoul !e consistent throughout a site9 so that it's clear to
the user that they're still within a site:
7ages shoul !e as !rowser6inepenent as possi!le9 or pages shoul !e pro%ie or generate
!ase on the !rowser6type:
All pages shoul ha%e lin-s e(ternal to the page> there shoul !e no ea6en pages:
The page owner9 re%ision ate9 an a lin- to a contact person or organi)ation shoul !e inclue
on each page:
3any new we! site test tools ha%e appeare in the recent years an more than 2./ of them are liste in the
'We! Test Tools' section:
"ow is testing affected !y o!-ect-oriented designs?
Well6engineere o!ject6oriente esign can ma-e it easier to trace from coe to internal esign to
functional esign to re#uirements: While there will !e little affect on !lac- !o( testing ;where an
unerstaning of the internal esign of the application is unnecessary)9 white6!o( testing can !e oriente to
the application's o!jects: 1f the application was well6esigne this can simplify test esign:
What is '$tree /rograing and what's it got to do with testing?
0(treme 7rogramming ;B7) is a software e%elopment approach for small teams on ris-6prone projects
with unsta!le re#uirements: 1t was create !y Ment <ec- who escri!e the approach in his !oo- '0(treme
7rogramming 0(plaine' ;See the Software#atest:com <oo-s page:): Testing ;'e(treme testing') is a core
aspect of 0(treme 7rogramming: 7rogrammers are e(pecte to write unit an functional test coe first 6
!efore the application is e%elope: Test coe is uner source control along with the rest of the coe:
2ustomers are e(pecte to !e an integral part of the project team an to help e%elop scenarios for
acceptance5!lac- !o( testing: Acceptance tests are prefera!ly automate9 an are moifie an rerun for
each of the fre#uent e%elopment iterations: QA an test personnel are also re#uire to !e an integral part
of the project team: =etaile re#uirements ocumentation is not use9 an fre#uent re6scheuling9 re6
estimating9 an re6prioriti)ing is e(pecte: Aor more info on B7 an other 'agile' software e%elopment
approaches ;Scrum9 2rystal9 etc:) see resource listings in the Software#atest:com '4ther 8esources' section:
QA and Testing FAQs 7age 1* of 1*