Testing Tools
By Elfriede Dustin
Agenda
Who is IDT
State of Software Testing
What is Automated Test and Re-test (ATRT)
Approach to ATRT
Advantage of open source
ATRT and open source Our IDE and solution
ATRT Challenges
2
http://www.idtus.com
Who is IDT?
IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re-Test (ATRT) and Quality Assurance (QA)
solutions
Deliver turn key automated test suite
Automated test strategy and identification of highest payoff areas
to apply automation for your project
Selection of best automation tools for your project
Using your existing test cases or develop test cases for you
Test results documented in requirements traceability matrix
http://www.idtus.com
Who is IDT?
IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re-Test (ATRT) and Quality Assurance (QA)
solutions
http://www.idtus.com
?
5
http://www.idtus.com
Spiral / Release 4
Test
Days
Spiral / Release 2
Test
Days
Spiral / Release 1
Test
Days
Test
Days
ua
l
Reduction in
Test Days
Spiral / Release 3
Increased
Test
Coverage
M
an
Automated
What is ATRT?
Our definition of ATRT is:
http://www.idtus.com
Build
Assemble
Round-Trip Engineering
Development Tools
Use
cases
Components
Test
Test
http://www.idtus.com
Components
http://www.idtus.com
Development
Management
Components
Execution
AutomatedSoftware
Testing Testing
Tools
Automated
http://www.idtus.com
10
Visual Modeling
Infrastructure
Development Tools
Components
Middleware
Automated Testing Tools
Software Configuration Management
Defect Tracking
11
12
http://www.idtus.com
Type of Tool
Tool Description
Business Modeling
Configuration
Management
Defect Tracking
Technical Review
Management
Requirements
Definition
Phase
Documentation
Generators
Requirements
Management
Requirements
Verifiers
Use Case
Generators
http://www.idtus.com
13
Type of Tool
Database Design
Structure Charts,
Flowcharts, and
Sequence Diagrams
Test Procedure
Generators
Programming
Phase
Metrics Tools
Syntax Checkers/
Debuggers
http://www.idtus.com
Tool Description
14
http://www.idtus.com
Type of Tool
Tool Description
Simulation
Test Management
Tests management
Network Testing
GUI Testing
(Capture/Playback)
Load/Performance Testing
Security Testing
15
Deployment Approach
Adapted ATLM
Populate Test Manager
with Test Cases and
Requirements
Test Results:
Pass/Fail by Test Case
Pass/Fail Summary
Test Time Required
Identify or Develop
& Pilot
Test Manager
Test Scripts
Data Comparators
Performance Test Tools
GUI Record/Playback
Storing/Retrieving Results
Collect/Create:
Requirements
Test Cases
Test Procedures
Expected Results
Interface Specifications
System/Component
Configuration Description
16
http://www.idtus.com
17
http://www.idtus.com
18
http://www.idtus.com
http://www.idtus.com
19
http://www.idtus.com
20
http://www.idtus.com
http://www.idtus.com
http://www.scl.com/products/ics/motif/testing/replay/
www.ics.com
QF-Test
http://www.qfs.de/en/qftestJUI/index.html
Froglogic
http://www.froglogic.com/pg?id=Google&category=squishjava
Redstone Software
Eggplant
KD Executor
http://www.kdab.net/?page=products&sub=kdexecutor
Jameleon
http://sourceforge.net/projects/jameleon/
ShUnit2
http://ldtp.freedesktop.org/wiki/Home
Cantana
http://www.ipl.com/pdf/p0003.uk.pdf
crontab
Part of Unix
crontab, cruisecontrol, www.adminschoice.com/docs/crontab.htm
Dogtail
http://people.redhat.com/zcerza/dogtail/about.html
http://www.cyberciti.biz/tips/linux-automated-gui-testing-with-python-baseddogtail-tool.html
Expect
http://expect.nist.gov/
23
http://www.idtus.com
http://www.idtus.com
What is STAX?
STAX is an automation system
With STAX you can automate and monitor your entire test environment
System/product setup
Testcase distribution
Testcase execution
Testcase results collection and analysis
STAX consists of
A programming language designed for automation
Programs written in this language are called jobs
An execution engine (similar to an interpreter for other programming/scripting
languages) which manages the execution and runtime behavior of the job
A GUI application, called the STAX Monitor, which provides a dynamically
updated view of your executing jobs
A tool (STAXDoc) that is used to generate documentation for your STAX jobs
25
http://www.idtus.com
http://www.idtus.com
27
CRON Calls into STAF services at a specified time interval External (Java)
EMAIL Allows you to send email messages External (Java)
EVENT Provides a publish/subscribe notification system External (Java)
EVENTMANAGER Allows you to call STAF services when a specified Event
occurs External (Java)
HTTP Allows you to make HTTP requests which can be grouped together in a
session External (Java)
LOG Provides a full-featured logging facility External (C++)
MONITOR Allows a testcase to publish its current running execution status for
others to read External (C++)
RESOURCE POOL Allows you to manage exclusive access to pools of elements,
e.g. VM UserIDs or Software Licenses External (C++)
STAX Provides an XML-based execution engine External (Java)
ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives
External (C++)
28
http://www.idtus.com
XML
The STAX programming language is based on XML. This provides built-in
structure to your jobs, as well as providing a set of existing tools for use in
constructing your jobs, such as XML (aware) editors and XSLT.
Python
The STAX programming language builds on Python to provide a rich and
accessible data model
STAX's Python integration also allows you to access the wealth of existing
Python libraries
Java
The STAX programming language allows you to access existing Java
classes/libraries, providing another source for reuse
http://www.idtus.com
29
Automation Tasks
Automation Startup
System Setup
Testcase Execution
Execution
Monitoring
Synchronization
Resource Management
Testcase Cleanup
Results Notification
Automation Completion
30
http://www.idtus.com
FS, Process
Process, Monitor, Log, Variable,
Queue, ResPool
Log, Process
Process
Email, HTTP
S
T
A
X
J
o
b
<function>
<parallel>
<sequence>
<process>
<stafcmd>
<testcase>
<timer>
<block>
<loop>
<message>
<log>
<import>
<job>
<iterate>
<paralleliterate>
<hold>
<release>
<terminate>
31
http://www.idtus.com
32
Display Application
Windows
Based
Simulation of Application
Processing Messages
Eggplant
R/P
VNC Robot
R/P
Simulation of Application
Processing Messages
ATRT
Test Manager
RTM
Bugzilla
http://www.idtus.com
STAF/STAX
Linux Based
33
34
http://www.idtus.com
Additional Requirement
Pristine test environment required automated
testing tool cannot be installed on ApplicationUnder-Test PC
Eggplant allows for remote capture/playback via
VNCserver and KVM switch
VNCRobot
35
http://www.idtus.com
36
http://www.idtus.com
Effective Automated
Software Testing
Current State:
Developing software in order to test software
Automate the test support development: automated IDL
to code generation; test data generation (using FireEye
from NIST); compare utilities; log file analyzers;
37
http://www.idtus.com
38
http://www.idtus.com
Changing requirements
GUI interfaces
Code access/availability
Defects uncovered earlier in the software
development lifecycle are cheaper to fix
http://www.idtus.com
Proposed solutions
Focus on Reusable Components
Test Data / Expected Results
Interfaces
Middleware
40
http://www.idtus.com
GUI Testing
Jemmy - java library to test UI
http://jemmy.netbeans.org/
http://abbot.sourceforge.net/doc/overview.shtml
Perl modules
run via VNCServer?
41
http://www.idtus.com
Questions
?
Email edustin@idtus.com with any follow-up questions
42
http://www.idtus.com
IDT
We are hiring.
Please come see me now if interested or send
email edustin@idtus.com with any follow-up
43
http://www.idtus.com