Testing – Aztecsoft
Recommended
Approach and
Execution Model
Vishal Talreja
Aztecsoft itest
itest.aztecsoft.com Page 1 of 17
MULTIMEDIA INTERFACE TESTING
Customer Inc. is the industry's leading wireless semiconductor company focused on radio frequency (RF)
and entire cellular system solutions for mobile communications applications. Customer is working on
building a framework for Multimedia applications for the mobile industry. This framework is a layer
below the API layer that talks to the MMI developed by the Phone manufacturer/Multimedia application
providers.
The purpose of this document is to provide an approach to Customer Inc. to help test their new
platform/framework for multimedia on mobile devices. The approach is based along three lines of
action:
Audio and Video Testing Approach
Identification of the types of bugs common in embedded systems software
Discussion of the various methods used to find bugs
Some of the approach paths detailed in the sections below have been executed on desktops in a
windows environment while some pertain to mobile and handheld devices. The baseline philosophy of
testing remains agnostic of the platform under test.
itest.aztecsoft.com Page 2 of 17
MULTIMEDIA AUDIO TESTING
It is extremely difficult to define a term as vague as audio quality; everybody has a different view of
the subject. Standardizing external and integrated audio around established technology specifications
is very important for mobile audio to succeed. The quality of audio in a mobile device is extremely
important for a rich multimedia experience on a handheld device.
Internal audio can affect playback on stereo Bluetooth headphones, Bluetooth headsets, wired
headphones, wired headsets and externally amplified players and hence control of audio settings via
software is very important from an end-user perspective.
We believe that the multimedia platform (audio portion only) will be used in (but not limited to) the
following environments:
Playback of audio files (MP3, AAC, WAV, WMA, Ogg-Vobis, MIDI)
Playback of polyphonic ringtones including iMelody, MIDI, MP3, WAV
Playback of game audio
Speech Recognition – useful in voice-based dialing
Conversation Recording on device
Voice Memo Recording
itest.aztecsoft.com Page 3 of 17
The above approach includes the following:
As the above settings are altered by software, corresponding audio parameters need to be measured to
ascertain that the change in audio settings accurately changes the audio output and its characteristics.
Audio parameters that need to be measured include:
The above tests can be performed using the Customer platform API and audio spectrum analyzer
software such as RightMark Audio Analyzer test suite.
itest.aztecsoft.com Page 4 of 17
Audio Device State Testing
Audio device states need to be measured via API during the course of audio testing. These are:
With the proliferation of audio streams, podcasts, and digital-music capable mobile devices, there is a
growing adoption of Digital Rights Management. DRM tags are included in audio content. If the audio
device driver does not support DRM, then it is difficult to playback audio content that is intended for
trusted devices. Aztecsoft will test for DRM support using the Customer platform APIs.
Drivers with DRM support: Devices with DRM-enabled drivers are considered "trusted." The DRM
system will play all encrypted and non-encrypted content on these devices.
Drivers without DRM support: Drivers without DRM signatures can render unencrypted content.
These drivers can also play DRM content that does not require a "trusted audio device."
However, if the usage rules in the content require that it play only on trusted devices, the DRM
system will not work.
When requested, the audio driver must disable the ability to capture the stream
currently being played back. That is, whenever Content Copy Protection rights are
asserted through the DRM interface:
The driver does not save the unprotected digital content in any non-volatile storage.
This may be EEPROM, hard disk, memory card, memory stick, or any other similar
storage medium.
The driver disables the capture multiplexer on an output D/A converter or otherwise
prevents loop-back of digital content.
When requested, the audio driver must disable the digital audio output on the device.
That is, whenever the content asserts Digital Output Disable rights through the DRM
interface, the driver must disable all digital audio outputs that could transmit content
over a standard interface through a standard interconnection scheme.
The audio driver must rely only on other components that also contain DRM signatures.
The driver must never facilitate the transfer of audio data to any component that does
not have a DRM signature.
itest.aztecsoft.com Page 5 of 17
For instance, if an audio file is transmitted from the mobile device to a Windows-based
computer, the driver must utilize the Windows DRM kernel APIs to make the Windows
DRM system aware of the movement of digital content.
The audio device and driver must not include user-selectable options to defeat or subvert
the DRM components. Specifically, the driver must not provide user control panels, or
other methods of disabling the DRM functions.
Multimedia video quality is easier to assess as compared to audio, which is very subjective.
We believe that the multimedia platform (video portion only) will be used in (but not limited to) the
following environments:
Playback of video files (MPEG-2, MPEG-4, 3GPP)
Playback of static image files including WBMP, BMP, JPG, GIF, PNG formats
Playback of video audio
Video quality can be determined by measuring many parameters such as resolution, refresh rate,
support of video file formats, video memory etc.
itest.aztecsoft.com Page 6 of 17
Determining frame rate is an automated process that requires code to be written to divide the number
of frames rendered with the time it took to render them. One can choose to measure the FPS over as
many frames as one wants. If it is measured over just the last frame one can get very accurate FPS but
it will vary too fast for most applications' need. If, instead, it is measured over the last 10 frames one
can get a more stable number.
A common approach to testing video streaming is Mean Opinion Scores (MOS) based that evaluates
overall user satisfaction by relating output to human visual parameters such as „blur‟ and „blockiness‟
and thus help pinpoint errors in video display. This approach applies to both video content streaming
and video telephony.
Qvoice has good testing tools that can test this functionality very effectively. Genista, a company
based in Japan, provides software solutions to measure the perceptual quality of electronic media
including desktop, video, mobile, media streams, digital audio and images.
The above audio-video testing will be carried out under various modes and states that include:
itest.aztecsoft.com Page 7 of 17
Test Mode State
Battery Full
Battery Battery Low
Battery Critical State
Vibration ON
Vibration
Vibration OFF
Mute
Un-mute
Pause
Resume
Play
Playback Functions (Audio
Stop
and Video)
Forward
Rewind
Volume UP
Volume DOWN
Seek
Stereo
Audio Modes
Mono
itest.aztecsoft.com Page 8 of 17
TESTING IN THE EMBEDDED SYSTEMS ENVIRONMENT
When devising a plan to remove bugs from software, it helps to know what you're trying to find.
Software can fail in many ways, and mistakes are introduced into the code from many different
sources. Some bugs have greater repercussions than others, and almost all of them have consequences
determined by the type of application and the domain in which it operates.
What follows is a partial catalog of errors found in embedded systems software. Without understanding
how embedded and mobile software can err, it's difficult to find the potential errors.
There are many types of software defects that are common to all classes of computer software:
In addition to all these, real-time embedded systems have potential problems in many additional areas:
Exceeding the capability of the microprocessor to perform needed operations in time
Spurious resets caused by the watchdog timer
Power mode anomalies
Incorrect interfacing to the hardware peripherals in the system
Exceeding the capacity of limited resources like the stack, heap, others
Missing event response deadlines
The list below is only partial. The relative frequency and severity is listed for each type of bug.
itest.aztecsoft.com Page 9 of 17
Error Type Error Description Frequency Severity
codes can cause problems. functions with
many return codes.
The result of an operation should be
checked to ensure an overflow
/underflow did not occur, before using
the result in any meaningful way. Failure
to check for overflow/underflow can
Common where
result in data-sensitive problems that
arithmetic
can be difficult to track down. If an
Math operations are
overflow condition is detected, it must High
overflow/underflow performed using
be handled in some appropriate way
integer or fixed-
(often by limiting the data to the largest
point math.
number that can be represented in the
data type). These checks are
unnecessary only when the input data is
well known and it's impossible for the
operation to ever overflow or underflow.
Incorrect decision logic grows common in
complicated functions and deeply nested
Logic/math/processin decisions. Boolean operations and
Common High
g error mathematical calculations can also be
easily misunderstood in complicated
algorithms.
If a section of code can be interrupted
before it completes its execution, and
can be called again before the first
execution has completed, the code must
be designed to be reentrant. This
typically requires that all variables Rare since most
referenced by the reentrant routine exist embedded systems
Reentrance Problem Critical
on a stack, not in static memory. In code is not
addition, any hardware resources used reentrant
must be manipulated carefully. If not,
data corruption or unexpected hardware
operation can result when the
interrupted (first) execution of the
routine finally completes.
The intended sequence of operations can
be corrupted by incorrectly designed for
and while loops, if then else structures,
switch case statements, goto jumps, and
Non-functional
Incorrect control flow so on. This causes problems such as Common
to critical
missing execution paths, unreachable
code, incorrect control logic, erroneous
terminal conditions, unintended default
conditions, and so on.
Data errors
Pointer errors are often more common
when certain types of structures are used Common in
in the code. Doubly linked lists make languages that
Pointer Error High or Critical
heavy use of pointers, so it's easy to support pointers,
point to the wrong node or link to a NULL such as C.
pointer.
itest.aztecsoft.com Page 10 of 17
Error Type Error Description Frequency Severity
Index registers (or similar types of
registers in other architectures) are
commonly used. High-level language
programs often make heavy use of
arrays. Many times, strings are stored as
Indexing Error Common High or Critical
arrays of characters. Individual elements
within an array are identified with an
array index. Accessing the wrong
element within an array is another
example of an indexing problem.
Sometimes improper initialization can be
obvious, as when reading a variable that
Improper variable has never been written. Other times it's
Not very common Low
initialization more obscure, such as reading a filtered
value before the proper number of
samples have been processed.
To get the expected results, the correct
data must be processed. The same name
can be applied to different data items
that exist at different scopes. For
example, an automatic variable can
coexist with a static variable of the same
Variable scope error Not very common Low to High
name in that file. Different objects
instantiated from the same class refer to
their members with the same name.
When pointers are used to reference
these objects, it becomes even easier to
make a mistake.
Converting a data value from one
representation to another is a common
operation, and often a source of bugs.
Data sometimes needs to be converted
from a high-resolution type used in
calculations to a low-resolution type
used in display and/or storage.
Incorrect
Conversion between unsigned and signed
conversion/type- Common Low to Critical
types, and string and numeric types is
casting/scaling
common. When using fixed-point math,
conversion between data types of
different scales is frequent. Typecasts
are useful to get data into whatever
representation is needed, but they also
circumvent compiler type checking,
increasing the risk of making a mistake.
Many real-time embedded systems need
to share data among separate threads of
execution. For example, suppose an
operation that uses a number of
Data Synchronization
different data inputs is performed. This Not very common Low to High
Error
operation assumes these data are
synchronous in order to perform its
processing. If the data values are
updated asynchronously, the processing
itest.aztecsoft.com Page 11 of 17
Error Type Error Description Frequency Severity
may be using some "new" data items with
some "old" data items, and compute a
wrong result. This is especially true if a
control flag is used to interpret the data
in some way.
itest.aztecsoft.com Page 12 of 17
Error Type Error Description Frequency Severity
appear when the emulator is used.
Reported bugs could also be a result of
improper use of the instrumentation.
Structural Testing
Sometimes called "glass-box testing," this activity uses an unobstructed view of how the code does its
work. This type of test is usually performed on a single unit of the software at a time. Test procedures
are written to exercise all the important elements of the code under test. This may include exercising
all the paths in the module. It often involves many executions of the same code, with different values
of data. Boundary conditions are typically exercised. This test is also used to determine the consistency
of a component's implementation with its design.
Cost
- Time High It takes time to write the procedures, and examine what is
critical to test. Once the test procedures are written, they
can be reused to retest the same module later when minor
modifications are performed.
- Money Varies Depends on how the testing is done. Simulators are less
expensive than In-Circuit Emulators (ICEs).
Effectiveness High If a module passes a thorough white box test, the level of
confidence is high that it won't cause problems later.
Functional Testing
In functional test, the program or system is treated as a black box. This implies the tester has no
knowledge of what's in the box, only its inputs and outputs. The system is exercised by varying the
inputs and observing the outputs. This type of test is usually performed on the entire software system.
In complex applications, the software is broken down into components or subsystems, which are tested
individually. All the individual parts are then brought together (usually one at a time) and the
integrated system is tested.
Test procedures are usually written to describe the expected behavior in response to a given
environment and input stimulation. In the absence of formal test procedures, some engineers simply go
through the system or software's actual behavior to its specified behavior without regard to the
software implementation.
In this phase, Aztecsoft plans to perform a system-level inspection of the software, checking specific
items:
itest.aztecsoft.com Page 13 of 17
Insure all peripheral registers are understood and accessed correctly.
Insure that the watchdog timer is enabled. (Many times, engineers keep it disabled while
the code is being developed to ease debugging.)
Insure all digital inputs are de-bounced appropriately.
Insure that the turn-on delay for analog to digital converters (and digital to analog
converters) is handled appropriately.
Carefully examine power-up and power-down behavior, and the entrance and exit from
any low power modes.
Insure an appropriate interrupt vector is defined for every interrupt, not just those that are
expected.
Verification Testing
Different organizations mean different things when they use the term verification. Some use
"verification" and "validation" interchangeably; others make a clear distinction between the terms. The
IEEE defines validation as "the process of evaluating a system or component during or at the end of the
development process to determine whether it satisfies specified requirements."
Verification, on the other hand, is defined as "The process of evaluating a system or component to
determine whether the products of a given development phase satisfy the conditions imposed at the
start of that phase. It is a formal proof of program correctness."
Aztecsoft views verification testing as a detailed analysis of software, independent of its function, to
determine if common errors are present. It is generally performed on the code as a whole, not on
individual units. It's very white box-like, in the sense that verification examines the structural integrity
of the code, and not functionality. Examples of typical verification checks include: stack depth
analysis, proper watchdog timer usage, power-up/power-down behavior, singular use of each variable,
and proper interrupt suppression.
itest.aztecsoft.com Page 14 of 17
Cost Severity Details
This is the only way to develop confidence in some areas
of the code. It's difficult to generate tests that will
produce certain conditions (for example, worst-case stack
Effectiveness High depth, maximum interrupt latency). However, by
analyzing the code, the engineer can determine what the
worst-case condition is and show that the system has been
designed to handle it.
Stress/Performance Testing
Stress tests are designed to load the system to the maximum specified load and then beyond, to see
where it breaks. "Load" could be number of users, number of multimedia messages per unit time,
number of multimedia formats that the chip can handle at the same time, size of multimedia play-lists,
picture messages, frequency of a periodic interrupt, number of dynamically allocated tasks, and so on.
Knowing at what point the system fails tells us how much overhead (factor of safety) we have.
Performance testing is conducted to measure the system's performance. This may be important to
verify that a requirement is met, to ensure the design is adequate, or to determine resources available
to add more features. (Examples: processor throughput, interrupt latency, worst case time to complete
a given task, playback quality and so on.)
Acceptance Testing
Acceptance tests represent the customer‟s interests. The acceptance tests give the customer
confidence that the application has the required features and that they behave correctly. In theory,
when all acceptance tests pass the project is considered complete.
They capture user requirements in a directly verifiable way, and they measure how well the
system meets expressed business and functional requirements.
They expose problems that unit tests miss and verify if the system executes the way it was
intended.
They provide a ready-made definition of how “ready” the system is i.e. satisfies all the
requirements in a usable way.
Aztecsoft provides Acceptance Testing Certification services to customers. This service has four distinct
phases:
Plan - This is a phase where we understand and analyze the customer‟s functional and user
requirements
Design - In this phase, we design an Acceptance Test Plan on behalf of the customer
itest.aztecsoft.com Page 15 of 17
Implement - This phase involves running the Acceptance Test on behalf of the customer along
the guidelines defined in the plan
Assess- In this phase we assess the results of the Acceptance Test and certify the
project/product
In addition to the above, Aztecsoft‟s engineers will also perform the following critical analyses on the
software:
Stack Depth – Determine the worst-case stack depth and insure adequate stack has been
allocated.
Watchdog Usage – Using the timing data obtained in the unit tests, find all places where the
watchdog timer is reset and insure that all correctly operating modes of the system will always
reset the watchdog timer before it expires.
Shared Data – Locate all data that is accessed by interrupt service routines and other areas of
the code. If a multi-tasking design is used, also locate all data that is shared by tasks operating
at different priority levels. Verify that adequate protection measures are used on the data to
insure it never becomes corrupted.
Deadlock – Build an allocation graph of all the data shared among different tasks that could end
up causing a deadlock. Insure that deadlock is prevented by the design of the system and the
order in which shared data items are locked.
Utilization – Using the timing information obtained in the unit tests, determine the worst-case
execution times of all tasks in the system. Verify that the microprocessor can complete all
tasks by their deadlines, at their maximum rate of occurrence, under worst-case situations.
Schedulability – Using the worst-case task execution times obtained, determine if all the tasks
can be scheduled and meet their deadlines under all situations.
Timing – Insure all other timing requirements of every task are met under worst-case situations.
For example, release-time jitters, end-to-end completion, etc.
itest.aztecsoft.com Page 16 of 17
EXECUTION MODEL
Aztecsoft recommends an execution model that will comprise of a team of 4 key individual roles to
help test this platform effectively.
itest.aztecsoft.com Page 17 of 17