1.
INTRODUCTION
Effective testing comes from a well-structured approach and a well-defined testing methodology
so the game product is highly satisfying to our Publisher and the game player. In contrast, poor
testing results in a buggy game or software that gives rise to a long stream of repeated testing and
project delays.
The purpose of this document is to explain the testing methodology for game development. This
document is intended for the person performing the testing. The topics and the level of details
provide an interesting and useful set of information for the Game Tester or Software Tester. There
are a number of Quality Assurance and testing initiatives that are covered in this document,
including a testing strategy, testing processes and techniques, test coverage criteria, Testing
Requirements, a Test Plan definition, testing documentation, and Full Cycle Testing.
There are few, if any fixed rules for this testing methodology; however, there are many
suggestions, ideas, and guidelines for improving the quality and effectiveness of testing for our
game projects. Hopefully, testers will be able to learn, understand, plan and carry out effective and
efficient testing in a structured manner. If you have any questions about the information, please
contact your Software Quality Director.
This document must be used in conjunction with the other companion QA documents
1. Game Testing Primer,
2. Testing Design for a game project, and
3. Defect Reporting Guideline.
2.
TESTING STRATEGY
2.1
In a simplistic view, testing is to identify bugs found in the software, so the problem can be
removed. There are different forms of tests and testing that can be categorized as Black-Box
testing and Clear-Box testing (Clear Box testing is also known as White-Box testing in the
software industry). Their testing objective and overall processes are indifference (e.g., test
planning, test design, testing execution, regression testing and bug reporting), but their focus of
attention puts emphasis on different aspects of the game:
Black Box focuses on the functional or the playability aspects of the game. For
examples, testing the user interface (e.g., the selection menus and the use of buttons), the
look and feel (e.g., the graphics and animation), and the actual gameplay.
Clear Box is focus on the architecture and integration aspects of the game software.
For examples, the use of a database, pipelines, the interaction/integration of game
components like the rendering engine, the AI engine, sound, and so on.
For Black Box testing, the tester must know how to play the game (e.g., use of the game pad,
know the rules and the game flow). For Clear Box testing, the tester must understand what
coding is. The Software Tester uses a run-time debugging environment, feeds the code or chunks
of code with input (i.e., data, setting variables, etc.) and analyzes the test result.
2.2
TESTING PHILOSOPHY
Testing is NOT a single persons job, nor solely the responsibility of the Game Tester and the
Software Tester for a game project. Every team member working in a game project must have
quality in mind, and each person is responsible for the accuracy and completeness of the work
that he/she produces.
This testing methodology is NOT the only process and it should NOT be used in isolation. The
reader must be aware that this testing methodology is considered as an integral part to the Game
Pre-production and Production processes.
In reality, no one can test a program COMPLETELY, i.e., testing every single part of the game
using different and all available characters, so triggering different path of the logic and all the
possible variations of input, interfaces, AI, and then output. Our testing strategy is to develop
excellent, full-coverage, and effective testing (i.e., 80/20 rule).
3.
3.1
A requirement is an objective that must be met. The tester must understand most of the game
requirements and translate them into functional terms (e.g., what needs to be tested, what things
are testable and what not, what are the targets and measures, etc.), leaving the design and
implementation details to the developers.
As part of the testing requirement development process, the testers read the system
documentation, gather all the visual and written information, analyze what and how the game
components can be tested. It is the responsibility of testers to read all the relevant documentation
so they can share (to understand and appreciate) the mission of the project. Please refer to the
Full Cycle Testing section about how to conduct a quality review.
You are required to develop a Testing Requirements document for each game by outlining what
and how the game and game components will be tested. The document includes:
a list of features,
the specifics of the internal design and external designs of the game. This may require
a description of the possible implementations if it makes the testing requirements
easier to understand (e.g., certain theme of the game, the characters, the animation,
the AI, cinematic or camera view, and so on). For example, to test the multidirectional fight action for the Chan PS2 game, you must make reference to the use of
Ring of Engagement, describe how the opponents engage into the fight scene, and
what you expect the single/combination fighting actions.
a testing structure detailing if and how Game Testing and/or Software Testing is
applied (i.e., in a spreadsheet format for the items identified above),
the testing criteria (e.g., ideas for testing), and
the completion criteria (e.g., what are the expected results, what does something is
done or something is working mean to you in game testing?)
After the testing requirements are identified, the Technical Director for the game and the Software
Quality Director will review the document to confirm scope and priority. Following the test
requirements, the testers work on their own test design and develop a Test Plan and Test Cases.
Any testing dependency requirements must be identified and communicated to the game team so
the game code is test-enabled, i.e., what kind of cheat codes, or test automation-enabled code
are required?
The testing documentation is expected to be developed in the early stage of the project, i.e., draft
testing documentation is produced when we have the first playable build. It is important to note
that the testing requirements will not cover every single detail of the game, but it must cover
testing all the contractually required elements (e.g. specific features and the major game
functionality). You can obtain this information from the game Technical Director.
3.2
When a game project is pressed for time, we must recognize the existence of a threshold point
where sufficient timeline must be provided for the testers to perform:
This threshold point varies from game to game, the tester is expected to communicate the
bare bone testing requirement with the Producer and/or Project Manager.
4.
The Game Testing Primer document describes a few aspects and focuses of game testing. This
section outlines a systematic testing approach, and explains how you break down a game feature
into its smallest testable components:
4.1
Systematic game testing means examining every part of a game. These parts include:
4.2
Must examine the entire screen and not just a small part of it.
Familiar with the game rules and test the gameplay against these rules.
Test for clipping (two or more polygons or polygon objects overlapping each other or
canceling each other out).
Examine the overlap (where a semi-transparent object shown on top of another solid colour
object and/or the background), and check if the overlap is appropriate in terms of size,
placement, the purpose and the information that is provided).
Test for incorrect and inappropriate collision (the condition when two objects should be
colliding). Imagine when two cars collide, they should get bumped by each other with about
the same effect (e.g. damage).
In terms of physics, when a collision involves two objects with the same mass, they should
generate the same amount of momentum in the opposite direction. In contrast, when a plane
hits the ground, should only the plane show the effect of collision because the mass of a plane
is trivial compared to the mass of the earth.
Move the character through all the available objects and all the levels and closely examine the
effect (it could be collision, an event trigger, or an object trigger).
Test for grayed out or grey out (when an option or icon cannot be selected)
Test loading/saving from a game save device (e.g. hard drive or a memory card) and ensure
the correct messages are displayed on the screen.
Ensure a game load or data load message and an appropriate loading indication (e.g.
loading meter) displayed on the Loading Screen.
Ensure the loading time is acceptable (no one likes to wait for more than 20 seconds).
Look for micro-pause where slight pauses occur in the game but do not actually cause a full
freeze or crash. When it happens, it will temporarily impede the gameplay.
Test for the multi-player mode (the game runs on one machine, not on-line gaming) where
two or more players engage in one game.
Test the game for memory leak or memory overload by leaving the game on/running for a few
days.
Test end of bound (also known as end of the world or edge of the world). The game should
not allow the character to move out of bound, and/or the game becomes erratic (e.g. game
crash/freeze, the texture becomes deteriorated, the character gets trapped in certain area of
the map).
Test for platform compatibility. Platform testing requires some extra effort for the PC game
and on-line gaming:
For a PC game, test for all the supported operating systems like Microsoft Windows
95/97/98/2000 and NT (this includes Install, Uninstall, and actual gameplay). Also, test
for different sound/graphic cards and peripherals that are available in the market (discuss
with your Technical Director for the make/model of the peripherals).
This is a list of common testing techniques used by a Game Tester. This is not meant to be a
complete list, an experienced Game Tester will tell you that you should use your mind this is
where a little creativity, a little imagination, and a little bit of common sense become very useful.
4.3
Before I let you go, I must warn you about Crack bugs and Placeholder, they are often mistaken
as real bugs:
Crack bugs (i.e. a bug which in fact is NOT a bug, but someones crazy imagination).
Placeholder (i.e. something dummy-up on the screen, it is used to fill a spot until the
correct video, artwork, sound or object is implemented and incorporated into the game).
5.
There are two types of test cases; one is testing the functionality of the product; commonly
known in the software industry as Positive Testing. The other one is testing the durability and
the level of tolerance resistance of the game; known as Negative Testing (it is also known as
Stress Testing or Load Testing). Negative test cases are created with breaking the game in
mind, so we will test for certain odd situations and determine if the game knows how to respond,
and reacts correctly. Examples of this odd situation are,
Load a game without a MEMORY card, or pulling out the MEMORY card during
game loading.
Running a game for more than 48 hours in order to test for memory allocation and
management. When the game component does not handle the use of memory
properly, some of the memory may become unusable (the memory does not get
recycled). When it happens, a game will become non-playable (e.g. freeze) after a
long-playing time.
For the playability testing, say, a scenario is developed for the Head-to-Head mode
in Snowboarding, we have Player-1 boarding very fast, and finishing much earlier than
Player-2, which is boarding extra slow. We expect the game should be able handle
this extreme situation (e.g. no freeze, the scoring continues to work the way it should
be) till Player-2 is finished.
Define Smoke Testing to test a new CD burn. The name of Smoke Testing comes from the
engineering lab in testing new car engines. Before a new engine is put out for a road test, the
engineers would simply start the engine and see if it works. If smoke comes out from the engine,
they would know immediately that some parts do not work properly. Just like creating a new
build, we simply run the game, if the game keeps on crashing, we would know some parts are not
right. The Smoke Test for a new CD burn needs to be defined. The test plan is a simple list of
what main game features and options need to be tested so you are able to confirm a new build is
successful. Just like testing a new car engine in the lab, you would start it and let it run for a few
cycles.
6.
6.1
The Game Testing Primer document outlines the role of software testing. The processes that are
commonly used for Software Testing are:
A) Code Inspection: review the source code and focus on logic structure, conformity
with standards, the use of comments, and the completeness of program documentation
and/or definition.
B) Incremental Focus Testing: test chunk or pieces of code by feeding values/input to an
isolated module, and analyze the intermediate result from the module. Usually a run-time
facility, a debugging tool, a dummy front-end, which allows invocation of objects by
their underlying method(s).
C) Data Analysis: examine the data in the database; or analyze which part of the system
uses or modifies any item of data. By tracing data items through the game play, the tester
can validate if data usage, interpretation and manipulation by the modules or objects are
appropriate.
D) Path and Flow testing: verify if the correct sequence of objects or components is
executed for an end-to-end path.
E) Algorithm-Specific testing: test a specific game scenario or feature by setting (or
tweaking) data variables, data values to the code, and execute it in the run-time
environment. Different from the Incremental Focus Testing, the testing is to verify a suite
of codes that make up a specific game feature.
F) Artificial Intelligence Analysis: generate the run statistic of the programmable moves
and plays of the AI components. The result is to validate if all the pre-programmed moves
(e.g. side grip on the snowboard) and plays (e.g. Jackie Chans combination punch/kick in
a multi-directional action) are used. Verify if the AI logic is appropriate and reasonable
(e.g. in a basketball game, it would not be reasonable to have the scoring 90% by dunk;
and 10% by jump shot).
6.2
The following provides a brief overview of specific tasks and techniques that a Software Tester
should learn or know while working on a game project.
1) File structures
A new software tester should get familiar with Radicals file structure. There are several good
documents pertaining to this from the global game and foundation technologies technical
documents. The learning objective is basically to learn the overall game component architecture,
and understand how it all flows together and to familiar yourself with the file dependencies of the
game. When checking a game module, it is important to understand what parts its inner workings
7.
7.1
TESTING COVERAGE
It will be way too much for a Software Tester to handle if someone has to do 100% of above.
Unlike Game Testing, I believe some of these tasks are discretionary. You must learn and decide
how much testing is enough. As a rule of thumb, the Software Tester should take a best-effort
based approach and time-box his or her work and involvement for a game project. Once you
come up with a test design and some high-level test plan, discuss with the game Technical
Director to confirm the scope and priority.
7.2
We all know good analysis of a bug simplifies the debugging process. However, there is a balance
to strike between the amount of investigation and analysis done by a Software Tester and the
amount done by a Developer. Here are some factors to consider:
7.3
Isolate the problem so it can be assigned to a single person in the bug report.
Describe the problem at the appropriate level of details so the developer can work on
it without wasting too much time to re-investigate and explore the problem.
Are we in the crunch time is the developer on the critical path?
Are you a more skilled debugger than the developer who is assigned for this bug?
How we can get more productivity and better use of time?
The Game Tester requires a fully functional Test Environment for testing. This test environment is
comparable to a debugging environment that is used by the game developer or the Software
Tester. Also, the test environment must support testing the graphic components of the game (e.g.
providing some of the artwork run-time information; not just showing the artwork itself).
The Software Tester is expected to provide support and help out the Game Tester in using a new
Test Environment platform.
8.
Game development goes through a cycle of stages. The game software is defined, evaluated,
designed, built, tested, fixed, Alpha, Beta and then Final. The game development cycle involves
many tasks by multiple functional teams, and in multiple stages. Most of the stages are often
described sequentially but the underlying tasks are often proceeding in parallel, especially during
game production. Testing and fixing can happen at any stages in the project cycle. However, the
cost of fixing an error increases exponentially as project progresses.
The objective of Full Cycle Testing is to identify and fix the error (aka defect or bug) as early as
possible so a quality product will be delivered at the end. In a simple terminology, the techniques
used for Full Cycle Testing are 1) validation for accuracy and correctness, and 2) verification
for completeness.
8.1
8.2
Here are the testing criteria for the requirement phase and the design phase:
Requirement Document:
Are these the right requirements for the new game?
Are they complete?
Are they achievable?
Are they realistic?
Are they reasonable?
Are they compatible to other competitor game?
Are they testable?
9.
Effective testing relies on a complete Test Plan package. The document should be written in a
way that promotes reusability (reuse a large portion of the previous test plan documents for a
new game) and consistency (to minimize learning curve; the testing documentation produced for
the game projects all have a similar format).
The hierarchical listing provided below is to outline a structure and the testing document
framework that the testing will be handled for each game project.
The Game Testing Test Plan is a positive testing document that explains WHAT and HOW to
test for the playability of the game (e.g. options and features).
QA has developed a guideline to explain the SONY requirements, section by section. We also
reviewed the SONY Bug Report from our previous game submissions and develop a our lesson
learned checklist so we wont make the same mistake twice. As far as timing goes, the review
for the manufacture standards and technical requirements is often conducted after Alpha.
In the Test Design document, you will learn the details (e.g. definition, guideline, templates and
ideas) of preparing the test plan documents.
10.
These are some of the good ideas from the experienced QA testers:
Make sure you are testing on the most up-to-dated version of the game. With at least one
version possibly created on the same day, it is easy to find yourself testing the wrong version.
Make sure that every time you get a new game build, you are getting every part of a new
game (a new game has more than one file!) and it may not work against a previous saved
game from a Game Save/Load device like MEMORY card.
Keep track of your hours so you have time to look for new bugs and regression test the
fixed bugs that you need to verify for closure.
A good understanding of the Tracker database is a MUST. You must understand the Tracker
template, the definition and attributes of each field. An optional field does not mean the field
is not required; the field is not required only if it is not applicable to the bug.
We must regression test the previous bugs. Just because something got fixed in the previous
versions is not a guarantee that it still works in the current version. This is our mandate to retest the previously closed bugs with the severity of A-Crash, B-Non functioning Feature, and
C- Major Functional every week and for each major milestone.
Regression testing can also be frustrating, make sure you stay focused and dont think of it as
a trivial task.
It is NOT only tester who can enter bugs to the defect database. However, you should
review the bug report if that is entered by a non-tester person (why? It is because you are
responsible for regression testing, and the testers are responsible for all the bugs in the defect
database).
Keep in mind that you will find bugs, however your productivity is not measured by the
number of bugs that you find. There is no fixed rule for a game tester as the number of bugs
that he or she must report in a day.
11.
EXTERNAL TESTING
Pre-submission: The game is sent to the manufacturer (e.g. SONY, Microsoft) so they will
briefly look at it and provide us with some feedback. Pre-submission gets us some idea about
the readiness of the game. This is sort of an informal review, if we dont receive any problem,
it doesnt necessarily mean the game is bug free. We often have a couple of pre-submissions
for a brand new game (e.g. our first action game of Jackie Chan Stuntmaster), and we may
skip the pre-submission for the second generation of a previous game title.
Submission: The game is formally reviewed by the manufacturer, usually a team of people is
assigned to test the game. The result in the form of a manufacturer bug report is sent to us
through the publisher. The submission process takes about a week. A game must meet
certain standards and requirements (e.g. no crash bugs, no more than certain number of noncritical bugs identified by the manufacturer) before it is considered commercial ready by
the manufacturer, and no game can be published without manufacturers approval.
There are certain rules that must be followed for the submission procedure, e.g. the use of special
CD media, the number of copies, and the game packaging, etc. Please refer to the manufacturer
specification for details. The Test Lead must work closely with the Producer, the Publisher and the
testing group of the manufacturer for the submission process.