April 2011
The information technology industry increasingly realizes the im- be developed in short development cycles (“sprints”), whereby at
portance of conducting, in a careful and efficient manner, verifi- the end of each sprint a new version of the product was released
cation and validation activities, which includes software testing. to the client. In our project, each sprint lasted 10 work days, and
Here the testing individuals, along with the rest of the team, work the items to be developed were chosen by the team during sprint
to assure that the developed software meets all clients’ needs planning meetings.
and is of a high quality standard. To achieve this goal, an effec-
tive test plan is indispensable. From the beginning of the project, During the sprint planning meeting, all team members had to pri-
a test engineer should be present, because this allows us to plan oritize all BLIs, in order to help decide which tasks had to be de-
ahead and to find and fix defects as soon as possible in the de- veloped during the following sprint. From a test engineer’s point
velopment life cycle. After all, as mentioned in testing literature, of view, the approach was to always try to anticipate the features
the sooner defects are found, the lower will be the costs to fix that appeared to be critical for system behavior and for meeting
them and the higher is the probability that their correction won’t the client’s needs. Prioritization could be assigned due to com-
cause new bugs (Glenford J. Myers, “The art of software testing”). plexity or importance; the intention was to prevent bugs relating
to these features from being found late in the project life cycle.
This article, describes how testing activities were performed in
a mobile game development project using SCRUM as the man- Testing strategy
agement process. It will describe in detail the testing strategies Planning – First sprint
used, along with the best practices identified and the learned Testing activities started before the end of the project’s first
lessons. The main goal of the article is to assist other test en- sprint with the arrival of a test engineer. As a first task, a com-
gineers who are starting in game development projects, so that plete analysis of the Basic Game Design Specification (BGDS)
they can easily and rapidly adapt to the differences compared was made. This document summarizes all basic game features.
to standard software development projects. This will also allow After evaluating the BGDS and all client’s requests and needs,
them to contribute to the creation of new and even more effec- a simple test strategy was defined which involved documenting
tive testing techniques. manual test cases as soon as the sprint started using a simple
priority scheme based on the complexity of the selected story and
The project the implementation order. At this initial stage we also identified
All the techniques and learned lessons described in this article and solved all test environment needs, like available hardware,
were experienced during a project developed at C.E.S.A.R. (Re- SIM cards, bug tracking software, etc. Finally, a set of general
cife’s Center of Advanced Studies and Systems) from December test cases made available by the client were also evaluated prior
2007 to March 2008. to starting test execution.
Since the project included, amongst others, elements like short Test case design
duration, a small team, frequent client involvement, and con- One of the initially defined constraints to the project was that all
stant requirement changes, it was decided to apply SCRUM and documented BLIs should be covered by test cases, and the BGDS
an agile development methodology to help manage all activities were considered as the test oracle. Based on project characteris-
during the project. In accordance with SCRUM, all tasks to be de- tics like short duration, scarce BLI documentation, and frequent
veloped were listed as backlog items (BLI). These were elected to changes, we decided to design test cases in a more general way
26 www.agilerecord.com
with a focus on testing the game’s basic functionalities for each At the end of each sprint an intermediate version of the game
BLI. This resulted in a set of test cases similar to those used for was released to the client, who could analyze the delivery and
“sanity” tests. This way we expected to maximize the time spent provide feedback to our team. This usually involved aspects like
on test execution and to avoid spending excessive time on docu- game play, game design and defects. Through an analysis of this
mentation. feedback we could figure out which areas were more relevant to
our client, adjust our test strategy accordingly, and then focus on
Test execution the missed defects in the next sprint.
The test specification consisted of a spreadsheet with a set of
test cases sent by the customer and a group of specific scenar- Exploratory tests
ios designed specifically for the game by the test engineers., A Exploratory tests, which were chosen as our main test execu-
MANTIS bug tracker (http://www.mantisbt.org/) was used as the tion strategy due to the project profile, began early in the initial
defect management tool. sprints. In a traditional approach, informal test charters were pre-
pared focusing on a specific area or BLI to be tested. During the
During the execution phase, the following activities were planned course of the project, as the Game Design Specification became
to be executed in each sprint: The main focus was the execution more mature, we started using two new approaches for running
of the exploratory tests as features were released throughout the the exploratory tests, which will be described below.
sprint, along with the execution of the client set of test cases and
the game’s specific group of test cases. The use of exploratory Test case based
testing is generally encouraged for projects with characteristics In this approach the actual test cases were used as the focus
similar to ours, and when executed by experienced test engi- area for each exploratory test, whereby the steps of the test case
neers, such”exploratory tests can be much more efficient than were run in an unusual way. The tester is encouraged to diverge
the tests performed following scripted test cases” (James Bach). as much as possible from the specified test steps, and to try and
think of alternative paths that could be taken instead of the one
During the course of each sprint, an important task performed suggested by the test case. The main idea is to use the existing
by the test engineer was to effectively monitor the progress of test cases just as a reference in order to cover all the features
all developers’ activities on the current sprint. This was done in of the application and to leave it to the test engineer to evalu-
order to define the best time to request the release of intermedi- ate relevance of the features to the system. By doing this, we
ate versions of the game for component testing and also to avoid encourage the test engineer to think creatively to find new ways,
defects or change requests being raised for features that had not or ways not previously foreseen, to break the software. This ap-
been fully released by the development team. This monitoring of proach achieved very good results and certainly increased the
activities was made easier by the SCRUM methodology where, total number of relevant defects found.
during the daily meetings, we could easily follow project activities
through the burndown graph. If this approach is to achieve a high degree of success, it is very
important for the test engineer to know all existing features of
As previously mentioned, the decision to prioritize the explora- the system, the market and the client’s expectations. He needs
tory tests was made due to the project’s main characteristics, to fully concentrate on his work in order to notice details that may
such as a lack of a extensive documentation at the beginning of have escaped before. It is also important that the engineer can
the project, and frequent changes of client’s needs and require- work in a comfortable environment during test execution without
ments. being constantly interrupted, thus allowing an efficient analysis
of the existing test cases and unexplored possibilities.
Formal tests
The complete set of test cases consisted of the client’s standard GDS scanning
test cases together with game- specific test cases which came to A different approach, which we used in the later sprints, was
a total of around 15O tests. However, not all specific tests were based on the execution of the exploratory tests through a com-
executed in the initial sprints, since most of the features were plete scan of the GDS. This approach couldn’t be applied fully in
not yet developed. A complete test execution cycle would take the initial sprints, because only the BGDS was available, which
an average of three days. During the rest of the sprint other test- didn’t contain enough information to allow a more detailed ex-
ing activities like test design and maintenance were performed ecution.
along with exploratory testing and change request validation.
For this technique to be applied successfully, the test engineer
The client’s set of test cases mainly focused on assuring that needs to have already read and understood the document,, and
the company’s standards were being followed by our team. This there should be no open questions. The main idea of this ap-
concerned features like key mapping, performance, interaction proach is to make sure that every description included in the
and user interface. Taking this into consideration, it was manda- GDS is correctly coded in the game. By simultaneously analyz-
tory to run these tests for each sprint in order to assure that the ing the document and exploring the game, it becomes visible if
developed game suited all clients’ demands and, above all, that any important scenario isn’t well described in the text. With this
they wouldn’t interfere with the mobile phone’s basic features. approach we can combine the benefits of static testing of docu-
www.agilerecord.com 27
mentation with the advantages of exploratory testing of the ap- Art
plication. Any gaps between game and documentation can easily All change requests of the “art” category are related to features
be found and the game designer can clarify the type of any bugs like image rendering, lightening, and any other aspects related to
found. the elements produced by the art team (although some of them
may also be performed by the development team).
Registered defects
During the course of the project, 122 change requests (CRs) This type of defect varies widely from huge perceptible failures,
were registered, which were simply classified for this article into which can be easily noticed, to specific scenarios that are caused
four categories, according to the type of defect, as follows: func- only by a defined sequence of actions. This kind of defect can be
tional, user interface, art, and sound effects. Later on in this found not only by focusing on this aspect during testing, but also
article is described how each of these areas presented unique while running any other type of test. All that is needed is a highly
important characteristics. In addition to these descriptions, two alert tester who pays attention to details.
graphs will indicate the amount of CRs per type and the severity
of the registered defects. It is highly important for the test engineer, especially if he is not
experienced with this kind of defect, to interact with the art team
The severity of each bug was classified as follows: (i) minor (for to clarify the questions related to possible defects, and in doing
defects that do not block the game’s correct behavior, e.g., the so beginning to understand the features, their limits and their
phone doesn’t vibrate when a new level is unlocked); (ii) normal solutions.
(for defects that affect important elements of the game, but
do not block the game’s execution, e.g., a special effect lasts Sound effects
shorter than the value specified in the GDS); (iii) major (for de- We also assigned defects to a sound effect category, because
fects that directly affect gameplay or user satisfaction, that have it turned out to be a key area where initially we did not expect
a direct impact on the game’s level design, and that prohibit the to find a relevant amount of bugs. However, testing showed that
player from proceeding, e.g., scenarios where the game freezes); this assumption was wrong. Several defects were found which
(iv) critical (for defects similar to major defects, but with an even demanded considerable work from the development team to get
higher impact on the game’s correct operation, e.g., level is not them fixed. One aspect observed, the concurrent events execu-
unlocked after completing tasks or phone doesn’t receive calls tion, caused complications in the game’s general behavior. This
while game is started). concerns scenarios such as executing a sound while another one
is already playing, user-initiated pauses of the game, disabling
Functional and enabling the sound. In addition, severe performance prob-
The CRs from this group are related to inconsistencies regarding lems could result from some of the game’s sound events.
the game’s designed rules and logic. This type of defect, even
those with lower severity, have a direct impact on the game’s Throughout the course of the project, this kind of the defect,
success, because they usually get in the way of a smooth un- which initially was underestimated, gained higher priority and at-
derstanding of the game’s objectives. They may even block the tention. We found that in this area we had a higher probability of
player from overcoming the challenges presented, turning the causing other defects while fixing one.
game into an impossible mission.
At first, test execution for this aspect was impacted by the quality
Scenarios like score limits, lousy player and excellent player of the available hardware, which presented a bad sound quality.
approach, and other possible features that involve testing the Later on, with the arrival of new hardware, tests could be easily
game’s features and limits, were tests that usually detected this executed and showed better results. Therefore it is important for
kind of defect. These scenarios were not always well described in the testing team to ensure the availability of the correct hardware
the GDS, and developers didn’t take them into consideration or at the beginning of the project.
unit test them properly.
User interface
Interface defects were connected to failures during display of
texts, opening of pop-up windows, screen limits, etc. These is-
sues could be easily noticed by any player, and would definitely
give the idea of a poorly developed game, without care for details.
28 www.agilerecord.com
To assist in this task, one real benefit is to add a recorded video
of the issue or, if that is not possible, at least a screenshot. If the
issue can’t be reproduced on the simulator, use a regular camera
to capture it. Just remember that this is not a rule. It’s up to you
to evaluate whether a CR should be improved by adding some
extra resource (after all, not all issues have a visual feedback).
www.agilerecord.com 29
If correctly used, cheat codes can increase the team’s perform- This approach can also be applied to different type of scenarios,
ance and even help anticipating bugs. such as performance, sound effects, rendering and other fea-
tures which can be compared with similar games.
Team communication
It is also important for the testing team to define specific time CONCLUSION
intervals throughout the project where reports from test execu- No matter what stage we are at in the development life cycle
tion cycles will be sent for the rest of the team. This helps us to or the experience level of the developing team, there are many
make our work visible and understandable for the entire team. causes for software bugs. Most of them are not related to the
Although SCRUM already makes tasks communication easier code itself, but to other problems, such as incomplete or unclear
among team members by applying the daily meetings and burn- requirements, hardware issues and integration. Therefore, con-
down graphs, we still need a more formal type of communication. sidering the practices and lessons learned from this project, we
A recommended moment for these reports is at the end of each are convinced that software quality is a high priority for modern
sprint. software products, like mobile games, and that achieving this
shall be a common goal for the entire team with a clear division
Activities followed up using the burndown graph of responsibilities. Making sure that there are no conflicts be-
As test engineers we commonly need to assure that we are us- tween developers, testers and other team members regarding
ing the latest available versions for testing. During the course quality, everyone must work together to deliver a high quality
of a sprint, the time for requesting new partial versions can be product. Only by placing priority on quality we will be able to de-
determined through the daily meetings and the burndown graph, liver products that fully meet our clients’ needs and expectations.
where we can be aware of the stage of each BLI and set up with
the developers the number of features available for testing. The
test engineer needs to keep constantly in touch with the team
leader to assure that these intermediate versions get released
for component and exploratory testing.
30 www.agilerecord.com