Anda di halaman 1dari 4

Testing is famous for not quite going to plan, starting

Golden Rules too late and being squeezed by fixed end dates.

My Golden Rules are not an exhaustive list and I’m

sure all readers could provide others from their own

for experiences.

Here is my top 20 (not in order of importance) that

may help you navigate your Test Strategy.
Quality Testing by Andy Redwood

Golden Rule Number 1 If testing is an average 40% of project All the requirements get on the bus and the
Testing is a Risk Management Exercise. budget, why is training investment so doors close. Nothing gets on and nothing
low? gets off as the bus travel from A to B (ring-
One of the first principles I learned as a fenced scope). Requirements may change
tester – if there is no risk nobody will be Golden Rule Number 3 seats on the bus (change control). If any
worried. If there’s no worry, particularly The Testing Culture in an Organisation new requirements are identified, they get
from senior management or the Business, can Only Mature at the rate of its on the next bus coming along behind
why expend valuable resources searching Capability (good release management).
for defects that are not important or viable
to find. If you wish to improve the effectiveness of If this rule is broken (and it sometimes is
the testing, don’t try too much too soon. for very good reason), there will be rework
Ensure all the Requirements are risk- The scale of what can be achieved will associated with test preparation. The asso-
assessed to the best method that works for vary from one organisation to another and ciated cost, which can impact the original
you or your organisation. Prioritise all the will be dependent on many variables. Business Case, increases the later new
testing to target the most risk with the least requirements are added, or when major
test effort. When there are only minor risks The first objective must be to establish all changes in scope occur.
outstanding, an informed decision can be the perceived problems. Usually integra-
made on when to stop testing. tion and de-duplication within and across Golden Rule Number 5
different areas of responsibility will create Testing Estimates Should be Based on
Golden Rule Number 2 a balanced economy of scale. I have Business Risk.
Training in Testing Reduces Long- worked on a number of process improve-
Term Costs ment programmes, where the benefits It’s very tempting to give your best guess
have been measured in the multi-million when estimating the testing effort. If
It is not un-common for less than 1% of an pound mark for testing improvements you’re lucky, or you’ve been collecting
organisation’s training budget to be dedi- alone. testing measures and metrics, you may
cated to training in testing, although with have previous project information to guide
the introduction of ISEB Foundation Golden Rule Number 4 you.
Certificates this percentage is on the The Greater the Number of Changes to
increase. Requirements, the More Rework is If you adopt Golden Rule Number 1, you
Required. should be able to weight the risk with a
Formal training in Test Management, Test “time taken to test.” This is more difficult
Design, Static and Dynamic techniques, to How many times have you been in a situ- on the first pass but each proceeding proj-
name but a few of the many testing activi- ation where the requirements are not just ect will allow you to predict with ever
ties, will put systematic and structured changing but new ones are being created, increasing accuracy.
method into the preparation, design and even when you’re in test execution?
execution of tests. Try and influence business and senior A word of warning, this does not mean
managers to work to the “Bus to Bus that the same tasks always take the same
Stop” rule.

22 Journal of Software Testing Professionals September 2002

time. You will need to evaluate any new As part of the test design the test analyst This can be a difficult mindset for some
“hotspots.” However it is quite common to should continually scan back and forth people. There is a conflict with reporting a
hear - looking for requirements that have no con- high number of errors and making good
ditions and of equal importance, look for progress to plan. However, a high number
Manager: “How long does it take to test?” test conditions that do not map to a of found errors in test must remain an
Acceptance Testers: “We think about six requirement. important test objective, in order to ensure
weeks”. a high quality and robust system in
In this situation you may for example Production.
Manager: “How do you know?” come across missing business processes or
Acceptance Tester: “Because it does”. have an item that requires a business deci- Tests should be designed to challenge
sion. high-risk attributes at the earliest opportu-
Please don’t estimate by habit! nity.
Golden Rule Number 9
Golden Rule Number 6 Always Test the Test Environment. Golden Rule Number 11
Drafting a Test Strategy from a It is Better to Test the Errors out in
Facilitated Workshop is a Quick Win. “The log-ons were all wrong.” “The inter- Early Project Phases as Opposed to
face link was down”. “We were pointing at Having a Failure in Production.
Workshops firm up known testing require- the wrong files.” Just a few of the remarks
ments very early in the life cycle. you may experience on day one of the test- It has been suggested that 50% of all errors
ing. These are usually due to a lack of found in system testing originate before
They also allow unknown activities to sur- “pipe-cleaning” of the test environment coding.
face early, creating contingency within the prior to starting the real testing. Most of
programme. these tests (as with most testing activity) It can cost hundreds of times more to fix a
can be check-listed. problem in production than in the require-
My measures show that on average, of the ments phase.
last 100 test strategy workshops I have
facilitated an average thirty-six previously Golden Rule Number 10 A focus on static testing techniques would
unknown issues have been raised. Some Tests Should be Executed to Prove that be advantages, for example, walk-
of these would not otherwise have been the Item under Test Doesn’t Work, throughs, reviews, workshops and formal
spotted until system test execution. Rather than that it does Work. inspections will raise early incidents.

Golden Rule Number 7

Linking Development Documents to
Test Documents Improves Under-

Figure 1 is an old picture (thanks to Paul

Smith), but it demonstrates the relation-
ship between development and testing
documents. It is the documentation that
we as testers must test back to. In many
cases this relationship is implied and not

Golden Rule Number 8

All Test Conditions Should have a
Parent Requirement.

Traceability between requirements and

test conditions within a test repository (or
inventory) will ensure all requirements are
Figure 1.
tested and aid test progress reporting.

September 2002 Journal of Software Testing Professionals 23

Unfortunately in some organisations it is embark on a path of continuous process Test packs run as regression tests:
still radical for testing to be perceived as improvement with reduced costs.
an across the life-cycle group of activities. • provide the status of changes since the
In general, the more senior the manage- Golden Rule Number 14 previous release
ment level, the harder it gets to persuade All Incidents Should be Traceable to a • can be constructed at all test levels to
people that testing can begin as soon as the Test Condition and a Requirement. enable maximum flexibility for testing
requirements are defined. large or small projects
This is fundamental if you are to know • form a baseline from which to com-
Golden Rule Number 12 that: mence preparation of new tests.
Each Test Phase Should Attempt to Test
Something Different with as Little (a) you have mitigated all the risks Golden Rule Number 16
Duplication as Possible. (b) you know you can stop testing. If you can’t Measure, you will not
Know if what you Achieve is Good or
Duplication costs. Some duplication is This activity has been made easier over the Bad.
unavoidable. years by test management tools.
Measures lead to metrics lead to trends.
Testing is more efficient and effective if It is also possible to evaluate data on types Metrics form management information
the testing undertaken is defined for each of incidents raised against requirements and allow informed decisions. Measures
test phase, with minimal duplication, with- (or test conditions). make perceptions factual.
in the test strategy. Details for each test
phase will appear in phase specific test Where you have raised few incidents Golden Rule Number 17
plan documents, each linked to correspon- against certain requirements or conditions, Measures Point to Root Causes. Fix the
ding development deliverables. you may conclude that these are likely to Root Cause and You Improve the
be stable parts of the system. In my expe- Quality.
I think it needs to be questioned if a par- rience areas of least testing, point at the
ticular test phase is actually required. The high-risk areas in the production system to Fixing the root cause in processes and pro-
basis for this decision could be risk and monitor in the first few weeks post imple- cedures so that they never recur, will
viability. In my travels in and out of organ- mentation. reduce costs and improve quality of deliv-
isations I see test phases executed only ery.
because they appear in the life cycle or the Golden Rule Number 15
programme management method as an Test Assets and Project Histories Must Care is required, as it is easy to mistake
activity. The objective for undertaking the be Considered for Re-use. analysis for the root cause for a witch-
phase, in some cases, has already been hunt.
proven in previous phases. Project histories give factual information
and documentation of the previous The root cause could be anywhere and is
Golden Rule Number 13 release. Having access to all documenta- likely not to be in the phase the incident
An Incident is not a Error or a Fault, tion sets, test logs, measures and metrics was identified. The important thing is to
but is Anything that was not Expected. from the previous project will: fix it and learn from it, not apportion any
Incidents can occur in any programme • reduce analysis, design and build costs
phase. Incidents need not be testing spe- • allow better understanding of the exist- Golden Rule Number 18
cific. An incident that results from human ing functionality Tools Can Only Automate Existing
error or a component fault will be logged • allow more accurate project estimates Process. Tools Require Support.
as a problem. • enable earlier test preparation
Existing process needs to be in a state that
It is important to evaluate all incidents, Testware is worth an average 40% of allows it to be automated. This can be
categorise them under strategically every project budget; why not re-use this done as the programme evolves, but in my
defined incident types and allow sufficient on iterative projects to reduce the test experience automation should run as a
time to conduct root cause analysis. In this preparation time? parallel project to the main testing activity
way you will gather meaningful measures, on a project.
establish trends, calculate metrics and

24 Journal of Software Testing Professionals September 2002

Build the tests, run them manually and Golden Rule Number 20 CALL FOR SUBMISSIONS
then follow up behind with a small team Sign off Criteria Should Never be a
who will automate the testing for the Surprise to Management JSTP is looking for articles reflecting real
life experiences with any of the following
future. Less risk, more control. areas:
When criteria are agreed in advance via • Testing (test process, test management,
My old friend Terry Stevens use to have a the testing strategy or lower level test testing techniques, performance test
ing, security testing, test automation, etc.)
catch phrase, “the cost of the tool is not the plans and reviewed at set points in the life • Requirements
cost of the tool.” This is still true. Tools are cycle, sign off into production is the norm, • Bug tracking and reporting
licensed software and require some • Incident management
not the exception. • Configuration management
expertise to obtain full benefit. • Release management
Nothing escapes into production with • Risk assessment
• Test process measurement
Those of you thinking of embarking on high-risk caveats or with insufficient test- and improvement
automation projects can now make use of ing.
the many user groups, limited (sometimes All submissions for publication in JSTP
must adhere to the following guidelines:
free) on-site vendor support or help from Build yourself a list of all interfacing man-
the many consultancies with experience in agers that need to always be informed as to 1. Author’s Bio. A 25 word or less author’s
this field. biography shall accompany each submis-
the status and progress of testing across all sion.
projects. Recommend via the staged 2. All graphics must be in high resolution
Golden Rule Number 19 update reports if sign off is ‘ok to go’ or format and submitted in a separate graphic
If You Don’t Ask a Third Party file such as a TIF.
what the deviation is from the standard 3. Original Content. Unless otherwise
Supplier the Question, You May Never approach detailed in the test strategy. agreed, all article submissions shall contain
be Told the Answer. Managers will be able to make proactive original content not previously published.
Exceptions may apply only if written per-
decisions as the project progresses rather mission is granted by the holder of any pre-
I’ll try not to get on a soapbox about this than have to wait until it has finished to vious copyright.
even though it is one of the most frustrat- react. 4. Grant of Copyright. Unless otherwise
ing things that a Test Manager may have to agreed, the copyright for articles submitted
shall be granted to JSTP for publication and
deal with. About the Author reprints.
5. Format. Unless otherwise agreed, sub-
Andy Redwood is a Principal Consultant missions shall be provided in MS Wordtm
Here are some of the most common ques- with Cresta Group, working with clients to format with page numbers (x of y) in the
tions you might wish to ask. I’d be happy help them improve the quality, efficiency footer.
to supply a more detailed set (email 6. Size. The length shall not exceed 15 pages
and effectiveness of their testing of MS Wordtm single-spaced, including all resources. Andy is a frequent public charts and graphical illustrations.
speaker at European Conferences and is 7. Multiple authors. Submission of multi-
• How did you test this package before author papers is taken as an indication that
an active member within the British all authors agree to its publication.
you handed it over to us? Computer Society and ISEB (Information 8. Style. The text of the paper must be in
clear, concise and grammatically correct
Systems Examinations Board). English. Abbreviations and acronyms must
• Do you have any test documentation I be clearly defined.
could look at? 9. Attribution. Any quotes or excerpts from
any other person or publication shall be duly
attributed either in the text or in a footnote.
• What errors do you know to be out- 10. Introduction. The first paragraph(s) of
standing that are in this release? the article should provide a concise summa-
ry of the article’s topic, including the bene-
fit(s) to the reader.
• Does this release include changes for 11. Sub-Headings. Where appropriate, sub-
any other client and if so, how does it headings should be added to indicated main
affect us? ideas and to assist the reader in locating spe-
cific information.
12. Summary. The final paragraph(s) of the
• What’s the turnaround time for fixing article should provide a concise summary of
the article.
our priority 1 problems? 13. References. References in the text
should include the authors’ names together
with the year.
14. Bibliography. In the bibliography, list
references in alphabetical order with the
titles of journals cited in full.

September 2002 Journal of Software Testing Professionals 25